In this lab, you will define a RNN2_model to find the next value of a time series. In this lab, we get to implement our first recurrent neural network. Will do it on the same data set we've been working with, which represents the height of a sprinkler over time. You should already have your data generated from the previous lab. To confirm, we'll run the bash command head which prints out the first five sequences from our training and evaluation data sets. You should see an output that looks like this. If you don't see any data output, please go back and run the cells from the top of this notebook to generate the data. Now, since we're implementing the RNN model, make sure to change your dash dash model parameter and the G Cloud ML Engine local trends cell to equal RNN. Once I do that, if I run this cell to execute a local training job, you'll see I get an error. That's because I haven't actually implemented the RNN model yet. To do that, I'll need to edit the model that PY file located in the sine model directory. There's a hyperlink to that directory and the markdown. So, I'll click on that and then open the model.py. If I scroll down to the function rnn_model, I'll see where I need to implement my code. Note that our first line of code has already been given to us. It takes our time series feature, reshapes it and stores it as variable x. Now, why is this reshape necessary? Well, our features tensor is passed from our input function as a rank two tensor. Specifically, it has the shape, batch size by number of inputs. Number of inputs is also referred to as sequence length or unroll length. However, RNNs require a rank three tensor. The third dimension represents how many features we have for each input. In this case, it's just one, the height of a sprinkler, which is why we can easily represent this as a 2D tensor to begin with. The third dimension represents how many features we have for each input. In this case it's just one, the height of the sprinkler, which is why we could easily represent this as a 2D tensor to begin with. But you can imagine, if we had additional time series measurements like wind speed, then we would need that third dimension to be something greater than one to represent the total number of features input at each time step. Now, to build our RNN model, we need three things. We need to define our RNN cell type, we need to instantiate our dynamic RNN and we need to extract or final state to feed it into a DNN. I'll paste in some place holders for that code now, starting with define our RNN cell type. To do that, I'll type cell equals tf.nn.rnn_cell.GRUCell and pass in a cell size. Cell size is a hyperparameter that defines the size of your hidden state. You tune it the same way you would tune the number of units in a given dense layer. Too few and your network may not have enough capacity to capture the patterns. Too many and your network will be unnecessarily slow and may struggle to converge. Usually, the capacity of our rnn cell needs to scale with the number of inputs. So here, we'll express it as a function of that. Cell size is going to equal the number of inputs integer division by three. Next, we'll instantiate our dynamic RNN. The dynamic RNN operation returns a tuple. The first object in the tuple is called outputs and it contains the activations for every single timestamp. We'll see why that may be useful later on. For now, we just care about the activation for the last time step. This is what the second object in the tuple called state, contains. We take our state and pass it as the input to a DNN to generate our predictions. Since we've already implemented our DNN model code, we can just copy that here and make sure to change the input to be the state tensor. Now we'll save my changes and switch back to the notebook. Now let's run our local training job again to make sure our code compiles. Awesome. We can now see it's training. We'll save running on the cloud for the last lab. That way we can compare the results of all model types at once. But before we conclude this lab, there's one more thing I'd like to do. Let's go back to our model.py and add print statements for all of our tensors. Here I use the convention, function name, followed by tensor name, followed by tensor value. Remember, that since TensorFlow uses delayed execution, this won't print the actual values of the tensor but it will print the dimensions and types of our tensors. I highly recommend this practice especially when you're first learning a new model type. It both helps with debugging and also gives you intuition into what your model is actually doing. Now that we have our print statements, I'll save my model.py and re-run our local training to analyze the shapes. Now, I can see the shapes of my tensors as they flow through the RNN model. I encourage you to pause the video now and see if you can explain to yourself what each of these dimensions represents. Then, we'll go over them together. Okay, let's start at the top. X represents our input features tensor and it has shape question mark by 49 by one. The question mark represents our batch size and the question mark means it's a placeholder dimension. That is to say, TensorFlow doesn't yet know what the batch size will be because this session hasn't been run yet, but there will be some value there. The 49 represents our number of inputs, sometimes called sequence length or unroll length. It's 49 because we generated sequences of size 50 but we saved the last value for the label. Finally, the one is our input dimension, which is to say, at each of the 49 time steps, we only input one feature which in this case is height. Next, we have our state tensor. This represents the final hidden state in our unrolled RNN. It has shape, question mark by 16. Once again, the question mark is a placeholder for batch size and 16 is what we defined our cell size to be. We then take this size 16 vector representing the output of the rnn and feed it through a RNN. First condensing it to 10 units, then to three and ultimately to one prediction. You'll notice the first dimension for all these shapes is still a question mark reminding us that all these operations are happening on a batch of records and not just one. You'll also notice that these same print statements repeat two more times. One is for the evaluation loop and one is for the prediction loop. You can ignore these values as they will be the same. All right, that's it for now. In future labs, we won't walk through all the tensor dimensions but I encourage you to get into this habit and continue the practice even if you don't see it in a solution video.
