Up until now, we've learned about tabular methods. Methods that store table containing separately learn values, for each possible state. In real-world problems, these tables will become intractably large. Imagine a robot that sees the world through a camera. We clearly cannot store a table entry for every possible image. Luckily, this is not the only possibility. Today we'll talk about more general ways to approximate value function. By the end of this video, you'll be able to understand how we can use parameterized functions to approximate values, explain linear function approximation, recognize that the tabular case is a special case of linear value function approximation, and understand that there are many ways to parameterize an approximate value function. In previous courses, we spent a lot of time thinking about how to estimate value functions. So far, our value approximations have always had the same form. For each state, we store separate value in a table. We look at values, and modify them in the table as learning progresses. But this is not the only way to approximate a value function. In principle, we can use any function that takes a state, and produces a real number. For example in a grid like this, our value function approximation could take the X and Y position, and add them together to produce a value estimate. This was just to get your intuitions going. We wouldn't want to use this as a value function, because we can't modify the approximation. So we have no way to do learning. This is where the idea of a parameterized function comes in. We incorporate a set of real valued weights, which we can adjust to change the function. For example in this grid, instead of using a fixed sum of X and Y, we could use a function of the form w-1 times X, plus w-2 times Y. The weights w-1 and w-2 parameterize our function. They allow us to change the output the function generates. To indicate that this function approximates the true value function, we use the notation v hat. W is a vector containing all the weights that parameterize the approximation. We do not have to store a whole table of values. We only have to store two weights to represent our value function approximation. You can imagine how this compact representation is useful for large state spaces. The tabular representation modifying the value estimates for one state, leaves the others unchanged. This is no longer the case with a parameterized function. Watch what happens to the value estimates of each state as we change the weight w-1. If we change either w-1 or w-2, we will change the value estimate for every state. Now, our learning outcomes will modify the weights, instead of the individual state values. The example we just discussed is a special case called linear value function approximation. In this case, the value of each state is represented by a linear function of the weights. This simply means that the value of each state, is computed as the sum of the weights multiplied by some fixed attributes of the state called features. We can express this compactly. We write that the approximate value is given by the inner product of this feature vector, and the weight vector. We will use bold X of S to denote the feature vector. Our choice of feature impacts the kinds of value functions we can represent. For example, consider the approximation we've discussed so far. With it, we can only represent value functions which changed linearly as a function of the features X and Y. Consider the case where the true value function is given by the numbers on this grid. We cannot represent this as a linear function of X and Y. To get these outer values correct, w-1 and w-2 both need to be zero. However that would mean these inner values are not correct because at least one of w-1 and w-2, must be non-zero for the sum that equal five. But we don't have to use X and Y as features. Linear function approximation relies on having good features. Here X and Y are not good features for this problem. But as you'll see later, there are many powerful methods to construct features. Linear function approximation is actually very general. In fact, even a tabular representation is a special case of linear function approximation. To build a linear value function to represent this tabular value function, we need to define the features. Let's choose our features to be indicator functions for particular states. For state S_i, feature i is one and the remaining features are zero. We have 16 features, one for each state. Let's compute the approximate values for a state with these features. Since all the features are zero except for one of them, the inner product is equal to the single weight associated with that state. Since every state's value is represented by a separate weight, this is equivalent to a tabular value function. These parameterized values are general, and we can consider lots of different types of functions. Neural networks are an example of a non-linear function of state. The output of the network is our approximate value for a given state. The state is passed to the network as the input. All the connections in the network correspond to real valued weights. When data is passed through a connection, the weight is multiplied by its input. This process transforms the input state through a sequence of layers to finally produce the value estimate. We will take a deep dive into neural networks later. That's it for this video. You should now be comfortable with the idea of parameterized functions, and how they might be used to approximate value functions. In the coming weeks, we will discuss how to update these approximate value functions while the agent interacts with the world.