Testing strategy · first published on Medium, 6 November 2023
The True Purpose of Tests in Software Development
The role of tests often carries misconceptions that can hinder the effectiveness of your development process. It's time to set the record straight and shed light on the true purpose of tests. Understanding this purpose can revolutionize your workflow and contribute to the delivery of higher-quality software.
Ensuring an Accurate State for Informed Decisions
As developers, we often hear that the purpose of tests is to find bugs. While bug detection is a vital aspect, it's only the tip of the iceberg.
The ultimate purpose of tests is to provide an accurate state for your application, enabling you to make informed decisions on whether it's ready for production. In other words, tests determine if your software is truly worthy of reaching your customers.
Now, let's debunk some common misconceptions that have clouded the understanding of this crucial purpose:
Misconception 1: Tests are only for finding bugs
Tests do indeed uncover bugs, but they do much more. They serve as a form of documentation, outlining how your application is expected to behave. This documentation becomes invaluable when new team members join or when revisiting code after months. Tests not only spot issues but also communicate your software's intended functionality.
Misconception 2: Tests guarantee a bug-free application
While tests significantly reduce the likelihood of bugs slipping through, they can't promise a bug-free application. Software is complex, and interactions between different components can lead to unforeseen issues. Tests serve as a safety net, but they don't eliminate the need for vigilance, monitoring and further testing, especially in real-world scenarios.
Misconception 3: More tests always mean better quality
Quantity doesn't equate to quality when it comes to tests. It's crucial to focus on the relevance and effectiveness of your tests rather than sheer numbers. A handful of well-designed tests that cover critical scenarios are often more valuable than an abundance of redundant tests. Prioritize quality over quantity.
Misconception 4: Tests are a one-time effort
Tests are not a one-and-done task. As your software evolves, so should your tests. Refactoring, adding new features, and addressing bugs will require adjustments to your test suite. Maintaining and updating your tests is an ongoing process that ensures they remain valuable throughout your application's lifecycle.
Additionally, over time, you'll expand your test suite to cover more scenarios and validations as your application evolves. While this expansion is essential for comprehensive coverage, it may also lead to an increase in test run time. To maintain efficiency, it's crucial to invest continuous effort in optimizing your tests. This can be achieved through improvements in your code, such as using more efficient queries, enhancing test execution strategies (e.g., parallelization, domain-specific execution), or making informed choices about which scenarios to prioritize.
Misconception 5: Testing is solely for the eyes of code writers
In many development practices, it's common for developers and testers to keep their test cases and designs within their own realm, excluding other vital stakeholders like product managers and application security managers. However, the insights and perspectives of these individuals can be incredibly valuable. Simply knowing that your test cases will be presented to this diverse group can stimulate fresh thinking, leading to the creation of new scenarios and validations that can significantly enhance our product's quality and align it more closely with user requirements and security standards.
Misconception 6: Automated tests replace manual testing entirely
Automation and manual testing are not mutually exclusive. They complement each other. Automated tests are excellent for repetitive and regression testing, while manual testing offers a human touch to exploratory and usability testing. The two approaches work together to provide a holistic view of your software's quality.
Purpose of Tests: Context Reigns Supreme
The journey from code creation to production readiness is far from black and white. It's a path where context reigns supreme, where the 'right' approach depends on your application's unique needs and objectives.
As we'll see in the following examples, understanding that the ultimate purpose of tests is to provide an accurate state for your application is the key to crafting effective test cases, interpreting results, and answering the pivotal question — 'Is this application worthy of our clients?'
A note, this is just one example out of many aspects we need to consider in our tests, but it would provide you with a clear understanding of truly grasping the purpose of our tests.
Over-the-Air (OTA) updates and mobile UI End-to-End (E2E) tests. These examples highlight how understanding the purpose of tests can lead to different decision-making approaches, aligning your testing strategy with the specific goals of your project.
A glitch in the OTA
In the OTA update scenario, products often accept a small ratio of failures due to the nature of OTA updates. Retrying an update or addressing the issue in the next update cycle is a common practice, ensuring that customer devices remain functional. Such scenarios can be an OS version update in an Android or iOS device, or a firmware update of a Wi-Fi router.
Consider a scenario involving Over-the-Air (OTA) updates. Some software products, like mobile apps and IoT devices, rely on OTA updates to deliver new features and security patches. In these cases, understanding the purpose of tests can directly impact decision-making.
Imagine your test results reveal a minor glitch in an OTA update process. It's tempting to view this as a showstopper and delay the release. However, when you grasp the true purpose of tests, you can take a more nuanced approach. Accepting a small ratio of issues is an acknowledged and often necessary practice. This acceptance stems from the understanding that several factors are beyond our control in the vast and diverse landscape of connected devices.
In such cases a failure (packet loss) is decided and then used as a benchmark for our tests. To be clear, we don't just accept failures and move on to a retry, but we understand that in this realm a retry is part of nature, so we analyze failures and decide whether these retries are part of the acceptable range (e.g. for bluetooth headphones we'll be much less forgiving).
User Experience is a blocker
However, in mobile UI E2E testing, retries can be a double-edged sword. These tests aim to simulate user interactions and verify the user experience.
Retrying tests in this context can provide false information on the actual application state and the potential user experience. If a test fails, it may indicate a real usability issue that users would encounter. Therefore, understanding the purpose of these tests means recognizing that retries can mask critical problems and misrepresent the user experience.
In the world of UI testing, every retry isn't just another test attempt; it's a reflection of the user's journey. A test retry here can mirror a real-life experience for an end user. And when that experience is marred by repeated retries, it can spell disaster for user satisfaction and even result in customer attrition.
Consider this: when a user interacts with a website or app, they expect a seamless experience. Every click, swipe, or button press should result in an immediate and accurate response. Retries in UI testing can directly impact this ideal.
A retry in a UI test may indicate an initial failure — be it a slow response, an unresponsive button, or an unexpected error message. If this same scenario were to occur in a real user's hands, it would lead to frustration, potentially a loss of trust, and a higher likelihood of abandonment. In today's competitive digital landscape, users have little patience for technical glitches and are quick to switch to alternatives that offer a smoother, hassle-free experience.
In conclusion
Understanding the true purpose of tests in software development is a game-changer. It transforms testing from a mere bug-hunting activity into a strategic process that ensures your application is worthy of your customers. Embrace the multifaceted role of tests, and you'll navigate the software development journey with greater confidence and success.
So, next time you write a test, remember that you're not just seeking out bugs; you're crafting a blueprint for your software's excellence. Tailor your testing approach to your specific context, whether it's OTA updates or UI E2E tests, and make decisions that align with the ultimate goal of delivering high-quality software.
And to help you prepare for your next test strategy, here are some questions that would assist you in your task:
1. What is the primary purpose of this feature/application? Understanding the core functionality and goals of your software is crucial for effective testing.
2. Who are the end users, and what are their expectations? Identifying the user personas and their expectations helps tailor testing to match user needs.
3. What are the critical features and functionalities that must work flawlessly? Prioritizing key features guides the focus of testing efforts.
4. Are there specific industry or regulatory standards that must be adhered to? Ensuring compliance with relevant standards is essential in fields like healthcare, finance, and more.
5. What are the potential risks associated with my application? Identifying potential risks, such as security vulnerabilities or performance bottlenecks, informs the choice of testing types.
6. What are the different environments my application will run in (e.g., browsers, devices, operating systems)? Recognizing the diverse environments your application will encounter informs compatibility and cross-platform testing.
7. What is the expected user load, and what are the performance requirements? Understanding the expected user load and performance expectations guides load and stress testing.
8. Is my application subject to frequent updates or changes? Frequent updates may require a more robust regression testing strategy.
9. What is the budget and timeline for testing activities? Budget and time constraints impact the depth and breadth of testing that can be conducted.
10. What is the level of collaboration between developers, testers, and other stakeholders? Assessing the collaboration dynamics informs the testing process and responsibilities.
11. Do we have access to real user data for testing, or should we use simulated data? Data privacy considerations and the availability of real data impact test data generation.
12. What tools and technologies are available for testing? Evaluating available testing tools and technologies can influence the testing approach.
13. Is the application designed for scalability or redundancy? Design considerations impact testing for high availability and fault tolerance.
14. How critical is the user experience in my application? Understanding the importance of the user experience guides usability and UI testing.
15. What level of security testing is necessary to protect user data and the application? Security considerations dictate the depth of security testing required.
16. Is there a need for automated testing, and if so, which areas should be automated? Identifying opportunities for automation streamlines testing processes.
17. What errors can potentially occur? Are all handled? What is the end user impact for each error? Brainstorming potential errors and error handling significantly improves the overall approach in terms of design, dev and testing.
These questions provide a starting point for assessing the unique characteristics of your application and determining the most effective testing approach based on your specific context and goals.
Want a guardrailed assistant like this on your site, or AI automation with the same rigour?