We're going to look at our success in, out of sample predictions comparing with the betting odds and with Nate Silver's FiveThirtyEight website. We're now going to work with the games played up until the end of 2019. We're only going to work with those games to which we have data played in 2020, and that's games in rows 198-288. We want to take a subset of these rows, notice that when you want to take a subset, you start with the first row that you want to count, and you actually end one row after the one you want to finish with. So 198 colon 288, means all the rows from row 198 up to but not including row 288. So if we run that, we now have this. You can see there are, if we describe here you can see that there are 90 games that were played up until the start of the shutdown in March. Let's see how many games we got right, and we can see, we have a 47 percent success rate here, which is slightly below the 52.7 success rate we had with our model for the out-of-sample predictions, sorry, for the within-sample predictions that we were looking at in the previous video. But that's not really the relevant measure of success, the relevant measure of success is to compare us with the bookmakers, and what was the bookmaker's success rate? Well, if we work out when the bookmakers were correct, which we do here when was B365 true, and then take the average of that value. You can see the bookmakers were right 48.9 percent of the time. In fact, the difference between our model's performance in this out-of-sample period compared to the bookmakers is quite close again. In fact, it's worth bearing in mind that not only is this quite impressive, that a very simple model like this can generate reasonably accurate predictions compared to the bookmakers, it's even more impressive if you stop to think for a minute that the bookmakers were making predictions between January the 1st and March 14th for games that were played. For example, think of a game played at the beginning of March. They were no restricted to information up until December 31st, 2019 as we were, we were only using the data up to that point. They were able to use data in January and February. For example, suppose a player got injured in January or February, that would affect your prediction. We weren't able with our data to take account to that effect and the bookmakers were, and yet our model comes out pretty close. Again, that should also make you realize it was really unlikely that you would significantly outperform the bookmakers ever because of their ability to use all up-to-date information. But it does show that our very simple model does remarkably well. So if we want to work out the Brier score, we need to generate the outcome variables that we did before. The value of one for each possible outcome, if that is the outcome or zero if that's not the outcome. Now, we do need to generate the probabilities for the bet365 data because the brier score is defined in terms of probabilities, so we need to take the inverse of the decimal odds and we need to normalize for the over around. So divide by the sum of the probabilities for the three events in order to standardize the probabilities and we do that now. So if we can now calculate the Brier score for our model. So our model has a Brier score of 0.62. Now, what was the brier score for the book is the brier score was 0.587. Remember, lower brier score are better, zero is the perfect brier score. Again, the bookmakers are slightly better than our model, but not that much better. We can also do a crosstab for our data. If you look at this, the results are very very similar for the bookmakers and for our model. For example, if you look at the top row of the crosstab for Bet365 of the away wins in the data, Bet365 got it right 11 times and wrong 11 times. They were right 50 percent of the time. Our model got it right nine times was wrong 13 times, somewhat smaller number. There were 24 draws in the model. Bet365 never predicted a draw, we never predicted a draw. That's a wash. Then when it comes to home games, Bet365 predicted 33 of the home games correctly and eleven incorrectly. They got three-quarters of the results correct and our model did exactly the same. The difference is a couple of games. Out of 90 predictions our model got it wrong two times more often than the Bet365 betting odds. Overall, not bad. What about FiveThirtyEight? FiveThirtyEight puts out these predictions and claims that the panthers might find these useful in terms of thinking about gambling. They're not advising people to bet on their results. But the idea is that people might find these useful guide if they're thinking about betting odds and thinking about probabilities. How do they compare with our model? One thing to say is their model is far more complicated than our model. They actually give a description and there's a link here in the notebook to the description. It's a very detailed and quite involved description of a model which was much more complicated to estimate than ours. In fact, they don't even provide full details. You couldn't replicate their model directly because they don't provide the data and they don't explain what the parameters are. They don't give you regression models and so forth. It's a slightly opaque, although much better than a lot of similar prediction models that you see out there. We want to compare FiveThirtyEight performance to the bookmakers and our model. We need to do some of the same things that we've done for the bookmakers thoughts in our model, we need to have a statement of what the likely outcome would be. Based on the probability, what's the most likely outcome away win, a home win, or draw? Then we need to identify the success rate. We need to say, when was the prediction of FiveThirtyEight the most likely outcome the same as the actual outcome, the [inaudible]. You can see that FiveThirtyEight true gives you a value of zero or one depending on the success. We can then work out what the mean success rate is for the FiveThirtyEight model. FiveThirtyEight model has a success rate of 48.9 percent. The bookmaker's success rate is 48.9 % and our success rate of 46.7. FiveThirtyEight generated forecasts that were exactly the same as the bookmaker's forecasts and slightly better than ours. What about the Brier score? The Brier score for the FiveThirtyEight model is 0.5935926. That Brier score is not as good as the bookmakers. Slightly worse, but again slightly better than our Brier score. In that sense, the FiveThirtyEight model is a lot more complicated than our model and produces slightly better results, but does not produce results that are better than the bookmakers. Again, if we do the cross tabs, again, you see the FiveThirtyEight, like our model and like the bookmaker's never predicts a draw. If you compare that with the other two models, you can see what is the strength of the five. How does the FiveThirtyEight model compare? Well, when it comes to away wins, FiveThirtyEight is as bad as our model in terms of away wins. Getting only nine right and 13 wrong. That's where it's falling behind the bookmakers. The bookmakers are getting 11 away wins correct and 11 away wins wrong. Where FiveThirtyEight does well, they don't predict any draws. No one predicts any draws, so they get all of those wrong. Of the home wins, they get 35 correct and only nine wrong. Which is better both than the bookmakers and than our model. You can see then why the ordering here is bookmakers best. Sorry, bookmakers bet 365 best, FiveThirtyEight, second best, and our model third best. But bearing in mind again that they're all very close to each other in terms of the overall outcome. Again, as we observed in the previous session, one of the reasons why the bookmakers model is so close to the success rate of our model is that they're generating pretty much the same prediction throughout. We'll show that again; 85.6 % of our forecasts were the same as the bookmakers. What's that percentage for the FiveThirtyEight model and the bookmakers? Well, the FiveThirtyEight is the same as the bookmaker in 88.9 % of occasions. Again, all of these models are generating pretty much the same prediction, and therefore they produce very similar test results. Here in this notebook, we've seen how to generate out-of-sample forecasts. When you're taking this course, it will be the case that the premiere league will have finished its season, I think. Therefore, you can actually go on and see how successful the model was in predicting all of the gains right to the end of the season. That would be a nice little exercise for you to try. But also we've shown you the principle here of how you can use a simple model in just to generate forecasts of games that have not yet been played and showed how in this particular case, actually these forecasts are really surprisingly accurate, at least by the standards of bookmakers. With bookmakers really being the gold standard for prediction.
