In computer science we have two kinds of scaling. In horizontal scaling, where we add more machines to a cluster to serve many clients at the same time. With vertical scaling we \u201cbeef up\u201d the machines we have, adding more powerful CPU, more RAM, or faster hard drives. In Software Engineering, scaling can also mean coordinating a large project, across teams of teams and larger groups. All that raises the question \u2013 what does \u201cscaling testing\u201d even mean?<\/p>\n
Today I\u2019ll address scaling test automation. To do that, I\u2019ll start at the word itself \u2013 very quickly \u2013 then move on to the important bit, what that means to you, the reader, right now.<\/p>\n
If you look up the word scaling in the dictionary, you\u2019ll find a reference to, well \u2026 a fish. Let\u2019s throw away that definition for now. In computer science, when we use the term \u201cscaling,\u201d the image that comes to mind might be \u201cscaling a cliff,\u201d that is, getting to the highest point. The logo of the\u00a0Scaled Agile, Inc<\/a>\u00a0is a mountain, while\u00a0Leading Agile<\/a>\u00a0uses a mountain as the entire metaphor for their transformation services, with basecamps, a compass, field notes, and so on.<\/p>\n The\u00a0root word<\/em>\u00a0here is scale. That is the thing you use to find out how much something weighs. Or, perhaps the scope of things. \u201cWe need to consider this issue on a global scale,\u201d the President might say, because to do \u201cthe right thing\u201d in a border city could lead to complications. If you\u2019ve used scales, you know they have limits. When you add too much weight, the variables get too large, things break. This is another definition of scaling \u2013 when paint cracks off in small pieces.<\/p>\n To get\u00a0test automation<\/a>\u00a0to scale, then, means to do a lot more of it. A lot more\u2026without things breaking down<\/em>, which is exactly what happens on a large project, or program of projects if you\u2019re not paying attention.<\/p>\n Ironically, people often turn to automation because they believe software testing doesn\u2019t scale. The logic here is pretty simple: Every new sprint, the programmers add more features, but the testing team does not add more time to test. In the waterfall era, this was less of an issue, as bigger projects just meant a longer testing effort. When Scrum made two week sprints the standard, teams turned to test automation to reduce the testing time. Yet if you run tests on a laptop and the time to execute those tests grows by 15 minutes every two weeks, after two years, that could be a\u00a0thirteen-hour delay<\/em>. That amounts to 1.5 business days. I call this\u00a0automation delay<\/a>, and that is a major source of failure of test tooling projects. It fits one definition of scaling, as the team is trying to cover more features or more software. Let\u2019s look at some other ones.<\/p>\n \u2022 More coverage (Deeper testing).<\/em><\/strong>\u00a0It\u2019s easy enough to create a sort of \u201csmoke test\u201d that runs the bare minimum of scenarios. This shows it is possible to use the software as intended, and it is not broken in some horrendous way for the majority of users. Smoke tests cover something between build verification and the core scenarios. Of course, the software could still be horrendously broken for some minority of users, or modestly broken for all users. So management attempts to scale by adding deeper testing.<\/p>\n \u2022 Find problems faster<\/em><\/strong>. When I started at Socialtext, we ran our Selenium-based test suite on all supported browsers toward the end of each sprint. More than twelve years later, those tests all run before a human starts exploratory testing the feature. Other companies I work with run the tests immediately after the code is committed and before it is merged into the master branch. This isolates defects when they are introduced, to the exact change that created them.<\/p>\n \u2022 Cross team \/ More applications<\/em><\/strong>. This takes the approach of a successful automation effort for one team, or project, and applies it to many teams and projects. There may be re-use of components or code libraries involved, especially for a large website, like an eCommerce site, with multiple teams. Notice that the first two goals conflict with each other.\u00a0The more testing you have, the longer it takes to run.\u00a0<\/em>Yet one of the goals of test automation, in general, is earlier feedback. This creates automation delay. In more metaphorical terms, these are the cracks in the wall when you have too many layers of paint; the paint\u00a0scales off<\/em>.<\/p>\n The first step is to figure out exactly what problem your company is trying to solve. Do you want to have multiple teams, find problems faster, or have deeper coverage? How deep is deep enough?<\/p>\n In my experience, these are tough questions. Companies like to defer them. Even changing the conversation from testing \u201cdone or not\u201d or \u201cscale of one to ten\u201d can be incredibly frustrating. As a consultant, one common reply I hear is \u201cI want it all! That\u2019s why we hired you; if I wanted just one thing, my team could do that. I want to have my cake and eat it too.\u201d<\/p>\n Yes, I have literally heard executives use those words. Which is fine, as far as it goes.<\/p>\n For today, step one is defining the particular challenge you want to overcome. What problems do you want to solve first? One way to do figure this out is with a two-step survey. First, ask people the challenges to scaling \u2014 these are the ways the wall is cracking. Then make a list, give them a hundred points, and ask how many they would invest in each. That frames the problem of where to focus investment in solving a problem where it is not possible to do everything at once. If there are ten items, and the solution is ten points each, then expect slow, incremental improvements, and make a plan to address that. It is possible that the word \u201cscale\u201d is a red herring, a term thrown around as in \u201cthat won\u2019t scale\u201d simply to stop the business.<\/p>\nScaling Test Automation<\/h2>\n
Solving the Scaling Problem<\/h2>\n