Common sense coding: Test automation isn’t really that automated

May 29, 2023

Common sense coding: Test automation isn’t really that automated

May 29, 2023

two people reviewing a test automation on a computer screen

A time and money saving venture, software test automation for quality assurance must be done properly to pay off.

Garret brings more than 15 years of experience providing software development leadership, application architecture expertise, and reviewing and implementing software delivery lifecycle best practices for his clients. From analysis and architecture, to implementation and delivery, to quality and support, Garret is an end-to-end specialist.

Are you looking to reduce testing costs and improve your efficiency throughout the testing process? Automated testing brings people and technology together to simplify and improve application testing but while it may seem straight forward, it isn’t completely automated, at least not in the beginning.

With manual testing taking up as much as one-third of an IT department’s budget, automated testing’s biggest draw is that it can help free up manual testing hours. But it also goes a long way toward improving quality assurance and driving innovation forward.  

Consider your adoption capabilities and future goals for testing. Is automation in this space right for you and your business?

The Right Methodology for Innovation has 6 steps: plan, build, deliver, run, manage, and learn

The development project lifecycle includes many stages, all of which build on each other to ensure activities are well-organized. The most successful organizations use this methodology and improve processes of each stage, creating a culture of continuous advancement and innovation.

This article focuses on the “building” stage.

What is test automation?

Test automation is the process of using tools to automatically execute pre-defined tests to review and verify software products, such as web applications, and their results.

Using scripted sequences, this method examines the software, reports outcomes, and can compare results with previous tests.

Some main benefits of automating testing include decreasing manual testing and associated costs, increasing accuracy, coverage and reusability as well as improving reporting, bug detection, and efficiency.

What uses can test automation have?

Test automation can be used to perform a wide variety of tasks, including:

  • Unit Testing – Isolating a single unit of an application from the rest of the software to test and validate its behaviour.
  • Integration Testing – Testing how units are integrated and working as a group to verify how they communicate and behave together.
  • Smoke Testing – Quickly verifying whether a new build is stable and meets a base level of functionality to allow for more thorough testing activity or greenlighting a return to normal operations.
  • Regression Testing – Thorough validation that a change did not break existing features of functionality of a system.

Limitations to consider

  • Lacks the ability to do exploratory testing by responding to unanticipated results or performing unexpected actions.
  • It takes considerable time and effort to build. It can often take a comparable amount of time to build the tests as it does to build the code and it will take longer to build a test than it does to run the test manually.
  • The test creator can make mistakes and as a result, the test itself may have bugs.
  • Tests must be maintained and updated when the code it is testing is modified. This can be inefficient and take time if the code being tested is new or being frequently changed.
  • Creating tests for code that doesn’t already exist is challenging and can require significant changes to a team’s process to do test driven development (TDD) or behaviour driven development (BDD) efficiently.

Example of challenges you may run into with test automation

To examine some challenges you may run into when implementing test automation, and how to get ahead of them, let’s look at an example of a team that was using new technology to create a new version of one of their systems to replace an old version that was being discontinued.

As part of the new project, the team wanted to implement as many modern best practices as they could to increase efficiency and quality. This included incorporating automated testing.

In an effort to leverage automation wherever possible, the team chose to allocate their quality assurance (QA) staff to test automation development at the expense of manual verification.

This is where they ran into some problems. Without a strong skillset in TDD/BDD, exclusive reliance on post-development validation test automation was not effective or efficient for quick turnaround on volatile features in active development.

Development would take place over a six-to-eight-week cycle prior to a two-week user acceptance testing (UAT) period and test automation only had access to the completed features for script development at the start of UAT.

Because test automation was inappropriately used as the primary QA, this produced a situation where test automation didn’t have sufficient time to complete scripting work. It also created a conflict between test automation and UAT needs because there was a single QA environment.

Features were then deployed to production where the only QA was performed by the developer who created it and a cursory time-constrained UAT tester. This left test automation spending the next development cycle catching up to develop tests for the previous cycle.

Ultimately, the solution quality did not improve, any savings gained by automating testing were lost from a lack of timely QA on new features and there were no qualified QA staff performing manual testing, increasing the overall risk of introducing new issues with higher severity and impact.

These challenges were caused by a fundamental misunderstanding of where test automation would be most effective, which process changes needed to be implemented to realize the benefits of test automation, and how to reallocate resources to maintain application quality.

Best practices and learnings to optimize opportunities of test automation

First and foremost, test automation is not a replacement for manual QA. Rather, it’s a method meant to improve long-term stability of an application and minimize the risks and costs associated with regression testing.

In addition to developing a detailed plan based on your needs and team’s abilities, consider these best practices when entering the world of test automation:

  • Focus manual QA staff on testing new and volatile functionality, and reallocate staff required for regression testing, and testing existing and stable functionality, to creating test automation.
    • Manual staff can also be test automation developers, to help focus manual testing during UAT phase, and test automation development during development phases.
  • Allocate a separate QA environment dedicated to test automation activity and implement an efficient process for backup/restore of data assets.
  • Prioritize mature features or features that are out of scope for the next development phase with defined, passing test cases for automation during the next development phase.
  • Allocate manual QA staff to work with business while planning and prioritizing work for future development to define acceptance criteria, create test scripts and test cases.
    • If your organization is adopting TDD/BDD practices, QA staff can also assist with developing TDD/BDD unit test scripts during planning phases.

It’s important to ask yourself how much time and money is currently being spent on manual testing and what level of payoff, and investment, it would take to move your team to automated testing.

While this isn’t to say test automation is a clear solution to all of your current testing challenges, there are solutions that can bring down cost, save time, and improve the quality of the product.

Get in touch

To learn more about how you can get support throughout the cyber insurance process, contact our team of experts today.