{"id":3072,"date":"2021-07-20T13:39:25","date_gmt":"2021-07-20T13:39:25","guid":{"rendered":"https:\/\/elementordzyns.com\/subject7\/?p=3072"},"modified":"2024-12-26T15:36:42","modified_gmt":"2024-12-26T15:36:42","slug":"should-you-focus-on-unit-versus-end-to-end-tests","status":"publish","type":"post","link":"https:\/\/elementordzyns.com\/subject7\/should-you-focus-on-unit-versus-end-to-end-tests\/","title":{"rendered":"Should you Focus on Unit versus End-to-End Tests?"},"content":{"rendered":"

\"Post
\nThe best part of a presentation might just be the Q&A. \u00a0After all, that is the part where the material meets the actual life experiences of the audience. \u00a0About a month ago, we hosted a panel discussion titled \u201cRevisiting the Test Automation Pyramid \u2013 20 years later.\u201d\u00a0 The panel attracted a lot of great questions, so we\u2019re delving deeper into some of the concepts explored by the panel.\u00a0 As a side note, if you missed the webinar, you can\u00a0<\/em>Find the Replay Here<\/em><\/a>.<\/em><\/p>\n

Question:<\/strong>\u00a0\u201cLet\u2019s talk about unit testing versus functional and E2E testing \u2014 how much of each type of testing should I do?\u201d<\/p>\n

This seems like the perfect follow-up question for a presentation on the test automation pyramid. How steep are the sides? What is the appropriate number of unit tests, API tests, and end-to-end tests?<\/p>\n

We can provide guidance for figuring out how to shift testing, what to add more of, and what to do less of. To get there, I\u2019ll start at the very beginning, and go very fast.<\/p>\n

THE MEASUREMENT PROBLEM<\/h2>\n

In order to answer how much, how many, or what ratios, we need a way to count and compare types of tests. A small unit test might take a minute or two to write and 10 milliseconds to run; a small end-to-end test might take a half-hour to create and a minute to run. In the time it takes a tester to create one end-to-end test, a programmer might create several unit tests.<\/p>\n

Does that satisfy the pyramid\u2019s requirements of a broad base? Or are we just comparing apples to oranges?<\/p>\n

Talking to the people who kicked around the original ideas, I\u2019ve come to the conclusion that the intent was one of focus: The team should focus on the unit tests more than the higher-level tests. In other words, they should build a solid foundation first.<\/p>\n

These ideas are conceptual advice, but they are not very practical. To figure out the ratio, we would need to measure the number of minutes people spend on activities at various levels, then compare high-functioning teams to low-functioning, and then eliminate other independent variables.<\/p>\n

However, the reality is that no one does this sort of measurement at industry scale. In his landmark paper\u00a0Software Engineering Metrics: What Do They Measure and How Do We Know?<\/a>, Cem Kaner suggests we have so little agreement on what words mean \u2013 is the buildmaintainer a developer? A tester? What about user experience specialists? Kaner goes on to argue that comparisons like figuring out a developer-to-tester ratio make little sense. Even with a survey of every company, the metrics would reflect an average of all the people who responded to the survey. Companies with an external API as a main product would certainly have a different ideal ratio from those that do electronic data interchange (EDI) as a business. Likewise, software as a service (SaaS) companies, like GitHub or Atlassian (who makes Jira), are going to have a different approach from insurance companies and banks that do data processing.<\/p>\n

With that in mind, the question becomes less about the \u201cright\u201d industry-standard number of each type of test, or even the right number for your company. Instead, the question is, \u201cshould our team be doing more or less of each type of testing?\u201d<\/p>\n

CONSIDERING CONTEXT<\/h2>\n

Unit tests tend to find low-level, isolated problems, and also provide incredibly fast feedback to the programmer. Integration and API tests find problems gluing together components, often due to misunderstanding the protocol \u2014 how the information is transferred. End-to-end tests exercise the entire system, making sure the complete flow works for the customer.<\/p>\n

The next question is: What kind of problems is your team experiencing right now?<\/p>\n

It is tempting to get the data you need from a Scrum retrospective. But I find that people in Scrum retrospectives tend to focus on a single incident or something that has happened only a few times. What we want is an objective assessment that looks at what has happened over time, not what people experienced in the past two weeks.<\/p>\n

Instead of a retrospective, I suggest looking at bugs found recently. That includes bugs found in testing and bugs that are found later by customers after they escape to production. This evaluation includes quantitative data, but also qualitative data from actually reading the bug reports, particularly for impact.<\/p>\n

When I\u2019ve done this, I\u2019ve gone back somewhere between a month and six months and pulled the bugs that qualify into a spreadsheet, with a column for root cause and another for where the defect should have been found. Most recently I\u2019ve just looked at the last hundred bugs. If your team does a \u201czero-known-bug\u201d policy and doesn\u2019t write them up but just fixes them, you might need to write down what you find for two to four weeks.<\/p>\n

One team I worked with recently had a great number of user interface bugs that were based on regressions \u2014 that is, a change in one piece of the code had unintended consequences in a different area of the code that seemed to be unrelated. It was a mobile application, and the screens used shared code libraries. The changes were visual and mostly concerned the look and feel of elements like text boxes and combo boxes.<\/p>\n

Another team had mobile application problems that showed up mostly when testing with real devices, such as a sticky scroll on an iPhone, or changes in screen design that did not render properly when the screen resolution changed for various devices. This was an e commerce application for a luxury brand, and as it turned out, iOS represented an extremely large percentage of actual dollars spent on checkout.<\/p>\n

None of these bugs would have been caught by unit tests. In these cases, the focus clearly needed to shift to more user interface and end-to-end tests.<\/p>\n

Another data point comes from looking at how often testers are waiting for a fix, how long that wait is, and if the problem could reasonably have been found by end-to-end tests. Waiting is one of the\u00a0seven wastes<\/a>\u00a0of the Toyota Production System. In software, we tend to \u201cpaper over\u201d waiting by having people work on other tasks, but the wait (and delay) is still there. Gathering data for this problem can be as simple as going to the daily standup and getting a list of stories (and bugs) the testers are waiting for, then working backwards to see if end-to-end tests would have found those defects or if unit tests could have found them.<\/p>\n

Bugs that escape to production can be a little trickier. In some cases, the team needs additional end-to-end tests. In others, the problem was a lack of creativity and freedom within the testers \u2014 \u201cNobody thought to test that,\u201d or in some cases, \u201cWe thought to test that, but setup was hard, and the programmer said it could never happen.\u201d<\/p>\n

The classic question \u201cWhy didn\u2019t QA find that bug?<\/a>\u201d is really unproductive here. Instead, the questions are \u201cCan this bug reoccur?\u201d and, if so, \u201cHow should we shift the test effort to find the defect?\u201d<\/p>\n

For one team I worked with, the bugs that escaped to production were relatively unique: a configuration flag was left pointing to test.companyname.com when an API was called, or a database indexer ran for too long and caused all requests from the database to block. These were big problems, but relatively rare, best addressed by policy and procedure. Certainly, a test running on test.companynname.com could never find that problem. In other cases, the problems in production were infrastructure-based and could have been found by making the change in a test environment, then running the end-to-end tests against that environment.<\/p>\n

A final question to consider is how many of the important defects that are discovered are through exercising the entire system (often regressions), and how many are limited to new feature development. A great number of new feature bugs would imply the need for better exploratory testing, and perhaps, more unit tests. You would need to dig into the details.<\/p>\n

End-to-end and GUI tests provide value when the workflow breaks. If testers spend a great deal of time dead in the water waiting for bug fixes, it\u2019s likely time for more\u00a0end-to-end testing<\/a>. On the other hand, I\u2019ve worked with teams where the first build simply did not work at all, and the simplest combination of inputs rendered an error. In that case, it might be best to focus on unit tests, or at least programmer-testing.<\/p>\n

FROM THEORY TO ACTION<\/h2>\n

Here\u2019s a place to start: Is the code quality on the first build delivered to test \u201cgood enough?\u201d If not, then is the problem one of insufficient regression testing? Or do the issues stem from new features?<\/p>\n

If the problem is insufficient regression testing, analyze whether the issues could be found by end-to-end testing. Test the login, search, creating a new page, creating a profile \u2014 errors along these lines that block the flow of work might be findable by end-to-end testing. Other problems, like a sticky scroll, might require more engineering work to build better code. If the problem is new feature development, examine whether the programmers should do more unit tests or better check their own work before passing it off.<\/p>\n

These questions don\u2019t provide a percentage or ratio, but they can help set expectations for change.<\/p>\n

On a full delivery team of 10 or more developers, analysts and testers, the next step is likely to take someone half-time to work on the area that needs emphasis. Often programmers feel like they need permission to do more unit tests (or write end to end tests themselves), while testers feel like they need permission to beef up the regression-check automation. Insisting on obtaining this permission will slow down the pace of delivery.<\/p>\n

Still, if the heavier concentration on unit-testing was buggy, and those bugs delayed release, a heavier emphasis on end-to-end testing may result in less back-and-forth and better quality in less total time. Think of this as reducing the big, ugly delays by investing in a little more end-toend testing. You may even end up going faster. That is a win.<\/p>\n

Get winning.<\/p>\n","protected":false},"excerpt":{"rendered":"

The best part of a presentation might just be the Q&A. \u00a0After all, that is the part where the material meets the actual life experiences of the audience. \u00a0About a month ago, we hosted a panel discussion titled \u201cRevisiting the Test Automation Pyramid \u2013 20 years later.\u201d\u00a0 The panel attracted a lot of great questions, […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"inline_featured_image":false,"footnotes":""},"categories":[1],"tags":[],"class_list":["post-3072","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"acf":[],"_links":{"self":[{"href":"https:\/\/elementordzyns.com\/subject7\/wp-json\/wp\/v2\/posts\/3072","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/elementordzyns.com\/subject7\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/elementordzyns.com\/subject7\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/elementordzyns.com\/subject7\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/elementordzyns.com\/subject7\/wp-json\/wp\/v2\/comments?post=3072"}],"version-history":[{"count":11,"href":"https:\/\/elementordzyns.com\/subject7\/wp-json\/wp\/v2\/posts\/3072\/revisions"}],"predecessor-version":[{"id":3212,"href":"https:\/\/elementordzyns.com\/subject7\/wp-json\/wp\/v2\/posts\/3072\/revisions\/3212"}],"wp:attachment":[{"href":"https:\/\/elementordzyns.com\/subject7\/wp-json\/wp\/v2\/media?parent=3072"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/elementordzyns.com\/subject7\/wp-json\/wp\/v2\/categories?post=3072"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/elementordzyns.com\/subject7\/wp-json\/wp\/v2\/tags?post=3072"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}