You can't test everything, how many test should you have at a given layer in the test stack? Or as it's often referred to, test pyramid. Well, there are three really key differences between these different types of tests. First, is the amount of time they take to run that escalates a lot actually in a non-linear way and particularly as we get up to the system level here. These tests take a long time to run and the reason why speed is so important is that we want a situation where a developer has spent time and energy to invest in these tests and while they're sitting and working on changing, extending, adding to the code they can push a button, push to their CI or CD system that's going to run these tests for them and in a few minutes get a result back that says either, "Hey here's specifically what's broken, and what it's saying is not right", or "Everything's fine, let's keep rolling. " And if that takes longer and they need to get up and get coffee or makes sense to move on to something else then it's just that much harder for them to go back and find what they were doing and fix it, these tests become less useful for them. On a related item, we have this idea of isolation. So we would never want to catch something here at the system level that we can get here and we would never want to catch something here that we could catch down here because the quality of the information, the isolation of the output, is much better as we go down in the stack mean we much rather know specifically what little individual piece of code has malfunction than hey just generally something's gone wrong with the user's logging in or whatever. So, the isolation goes down as we go up the test stack. And then finally, cost goes up a lot as we go up the stack. The system tests in particular are fairly expensive and difficult to maintain because there is just so much going on at this system level. And so, we look at all of this particularly at the system level, we probably want, I mean a lot of cases five to 20 system tests might be the right amount for kind of a mid-size application and we'd just be trying to test really the core things that the user is going to do. So, for example if it's an e-commerce site just you're really basic search, select and purchase stuff or whatever is the equivalent for your system. You may hear from a developer, "Yeah I left that job because I was drowning in Selenium tests, and what they mean by that is Selenium is a tool that helps you run these system tests. And what they mean by that is we were just constantly getting all of these tests that said that something was wrong but really it was just that an element on the page moved, or the wording of something changed, or whatever that, stuff that they really didn't actually need to do anything about at all and that's not a situation that anybody wants to be in. There you spend a lot of money, you wasted a lot of developer time and you don't want that. There are a few specific anti-patterns that you want to watch out for and act on right away. One is what's lovingly called the ice cream cone pattern. And this is where basically we just have a lot of system tests, this happens when for whatever reason the development team isn't really investing in their own test coverage and we have a system test group that's doing this black box or phenomenological testing where they just look at how do we manually log into the system and do stuff to it and see what happens. And this is where you'll get that comment, "Hey I was drowning in Selenium tests perhaps." The other reason that this can happen is that if you haven't seen a Selenium tests or tests like that run before where there's browser automation it looked super cool if you've never dealt with debugging code or maintaining software you might think well why would we test any other way this looks great, let's practice like we play and it looks like a user is really using the system. And the reality is it's not good to over-invest in those tests and now you know why. The other pattern a little less acute probably in most cases is called the hourglass pattern. And that's just where we have fewer integration tests or no integration tests than we otherwise might want it. And this can happen for example, if we have developers that are finding the rewards of having good unit test coverage and doing that and a system test team that's got good coverage on the system level but maybe this is like an interdisciplinary area that's sort of ground ball, and nobody's really sure who should do it or whether they should do it or not. You'll know that this is the case if you're finding that either you're catching things up here that could have been caught down here which testers and engineers can figure out together when doing a retrospective or things are going to production that when you do a retrospective on those they could've been caught with an integration test. Again these are when data's passing between two functions, two functions are interacting and this isn't a place you want to under invest in because you don't want to have these be caught here and you certainly don't want them going out to production. The last thing that I'll mention is that if you use feature flags, and this is a way to have features be turned off when they go to production, you can sometimes get away with abbreviating some of these upper level tests and teams will find that they can get away with that because they're able to for instance just turn these features on for their internal users or very tiny set of customers that they carefully monitor and make sure everything's okay. So, that is one pattern the teams have found allows them to push up through the stack a little bit quicker and accelerate their path to a more continuous pipeline. Lastly, I would hesitate to prescribe any specific number, but just to give you a very general sense of proportions there is a post from the Google test team that I included about test proportions and their idea is, well roughly it's a 70, 20, 10 mix in terms of number of tests. Now, whether that's the right number for you it's better to figure out adaptively as you look at things that happen in production and your retrospectives on your iterations. But those are some ideas about how to figure out where to invest in your testing and how many tests you should have.
