It’s really, really difficult to design automated tests that are useful at Web scale. The Web is littered with blog posts, Quora posts and StackOverflow questions about how to deal with the consequences of test systems that were built without the benefit of rigorous analysis.
What sort of analytics and algorithms are useful in designing tests?
To write effective automated tests, engineers have to switch their minds over to thinking about the whole system rather than a single unit of functionality. This is one reason why expert testers are so useful: they don’t have to context switch from “developer mode” in order to get work done.
The expert tester designs her tests by implicitly referencing a state diagram of a whole system. But, if you draw the state diagram of a system like, say, login and authentication — that’s a very complex set of paths.
So the expert tester also decomposes the state diagram of each feature into multiple paths. A state diagram is then implicitly constructed for each of those paths. She then prioritizes the paths and assesses the feasibility of actually implementing a computer program that can traverse those that are most important.
It’s never the case that all of the highest priority paths can actually be tested given our current technology. Figuring that out up front is essential, otherwise it’s very easy to blow all the available development time on a test that looks superficially easy but can’t actually be implemented.
How is test design done? And how specifically is it different from “customer-facing” software architecture?
Note that I keep saying “implicitly constructs a state diagram.” That’s because the expert tester is a domain expert who’s been building her skills for years. Plus, she’s gotten very good at state and path analysis. She doesn’t need to draw diagrams and perform explicit path reduction. She can do that stuff in her head. She doesn’t need explicit mathematical models any more than a front end engineer needs to see an explicit graph of the DOM.
That’s a longwinded way of saying: test design can look easy — when it’s being done by highly trained professionals. But everyone has to spend years climbing a very painful learning curve in order to get good at test design. In that respect test design is just like any other programming skill.
So it’s important to approach large-scale automated testing projects knowing that the work involved is not intuitive or obvious to engineers who design client-facing features for a living! There is a lot of specialized analysis that needs to happen before the first line of code is written. The expert tester does this analysis implicity, and can get the job done very fast. But for engineers who don’t think about testing 24-7, it won’t hurt to grab a conference room and chart out some finite state diagrams on the whiteboard!
Remember: you cannot write a useful test unless you can represent the scenario as a finite state machine. If you can’t do that then the set of paths you have chosen is too complex and/or non-deterministic; and you need to re-apply the path reduction heuristic described above.