Chasing the Perfect Balance

Frontend test automation faces a fundamental trade-off: it can be very fast but often at the expense of stability, or it can be more stable but slower. Teams are frequently divided on which aspect to prioritize. Some advocate for speed, others for stability, and many desire both — yet achieving both simultaneously often seems unattainable.

When automated test cases fail, quality assurance or test automation teams must investigate each individual error and sometimes rerun the tests, which wastes valuable time and resources. Network errors, slow loading times, or even various spinners complicate test stability. Unfortunately, to prevent unnecessary error messages, test speed must suffer somewhat.

The definition of "stable" is open to interpretation. Does stability mean 100% successful test runs every day? Or is a 95% success rate sufficient? True stability means tests only fail when a genuine issue exists. However, ensuring 100% stability requires that both the system under test and its environment are also perpetually stable — an almost impossible standard.

"Fast" is another relative term. The duration of a test depends on the scope: is it a simple check on the landing page, or does it involve multiple steps after login? Not every test needs to run in under 20 seconds to be considered fast. If the average test runs in about 30 seconds, that is already highly efficient, especially when parallelization can further accelerate overall execution.

Examples:

There are many suggestions for speeding up tests: running them in parallel, mocking certain components, using API requests for setup, or navigating directly to URLs instead of simulating user clicks. While these approaches can increase speed, they may not accurately reflect real user behavior and could mask potential issues.

Stability can be improved by introducing retries, adding pauses between steps, or switching testing frameworks. However, these measures can slow down execution or require significant effort to implement in large, established test suites.

The optimal approach is to make tests as fast as possible while maintaining the highest achievable stability. This requires continuous optimization tailored to the system and environment under test. Sometimes, this means accepting that tests won't be exceptionally fast.

However, tests cannot be excessively slow. For example, if you have 500 test cases, each taking 60 seconds and unable to run in parallel, the suite will take over 8 hours to complete. In such scenarios, running all tests after every deployment or automation update becomes impractical.

While the ‘Lean Principle’ says that you should reduce waste, maximize customer value, and promote continuous improvement, it is also very important to run these most important automated test cases as stable and as fast as possible.

The goal is to strike a balance: achieve moderate execution times with maximum stability. In my experience, it is preferable for test execution to take a bit longer if it means the automation is exceptionally stable. With effective parallelization, execution time becomes less critical, but as the number of test cases grows, longer total durations are inevitable.

For each project, identify the optimal path to highly stable test cases and enhance execution speed through parallelization or earlier automated runs. It is far better to accept longer execution times with near-perfect stability than to settle for rapid tests plagued by daily failures.

Our experts are happy to support you!