Following the performance load test process improves the accuracy and confidence of results together with objectivity of lessons that can be learned when conducting performance load testing on your websites and applications.
Performance load testing (load testing) is an important and integral component of the development and maintenance of a website or web application (website).
Its objective is to ensure that the key performance-based business criteria of the website and its environment platform can be achieved and benchmarks for future comparison can be established.
Although each website and application are different, load testing is a repeatable process and once implemented it can be used to produce repeatable results.
Figure 1 provides a step by step high level overview of the performance load test process. Although load testing is a highly technical activity, formalizing it as a defined project with gated milestones can deliver significant advantages.
Invariably preparation for a load test occurs to late in the overall development project cycle and this can lead to many important aspects of the load test from being missed, rushed or overlooked. However, with proper preparation, this does not have to be the case.
Scoping
As the first task, the objective of scoping is to provide the foundation on which the load test project will be delivered. A thorough understanding of the scope and objectives of the load test is vital to ensure that the key performance criteria are identified to enable the expected performance-based business outcomes to be proven.
Ideally, the key performance criteria should be defined in the business analysis and requirements definition process and documented as formal non-functional requirements (NFRs). This is rarely the case and so it becomes the task of the load testing team to develop and agree the key performance criteria.
When a website is being developed, as new, as a partial or complete replacement, or it is part of a platform migration project, performance criteria could be set as being ‘better than the old platform’. Although this approach can help to set the target criteria, it does assume that the key performance criteria of the existing platform is known and has been established in a previous benchmark load test.
Historical benchmarks are useful in developing key performance criteria for a load test. The objectives should, as a minimum, include the following:
- Measuring the performance of the system to be tested at:
- Normal, business-as-usual levels of load
- Current, peak levels of load
- Expected future, higher levels of load
- Measure the end user response time of the system being tested under normal, peak and expected future levels of load
- Measure the number of successful business activities that the system being tested can support and identify the specific types of errors encountered
- Understand the capacity and throughput capabilities and limits of the system being tested, and establish the basis for future capacity planning decisions
- Identify bottlenecks and understand how performance degrades with increasing numbers of users or new applications or features
- Provide realistic levels of load that enable the measurement of system resources
- Support performance tuning activities through the provision of repeatable tests
- Determine if an application is ready for deployment to production
- Formalize the success criteria to determine if the website meets the technical and business objectives of the load test.
Defining success criteria will require technical decisions to be made based on analysis of the test results. For example, it is probably not acceptable if the peak levels of load can be sustained but end user response time has quadrupled over that observed at normal load.
Project Definition
With an understanding of the key performance criteria objectives the scope of the project, the resources necessary and project timelines can be defined.
Environment Scope
To achieve this a comprehensive definition of the system to be tested must be made. This includes the environment it will process on, its network connectivity (CDNs, Load balancing etc), any related applications, third party dependencies and the all business-critical and architectural features that are to be tested should also be identified.
With this identified, a set of user journeys should be outlined that matches the way end user’s interact with the website. It is then possible to identify the ‘mix’ of these user journeys. The mix, expressed in percentage terms, identifies how many end users will be processing for each user journey concurrently. Figure 2 shows an example of this for three user journeys and how the percentage of concurrent users is different for each user journey.
Load Test Types
Achieving testing objectives will depend on the type of load tests conducted. There are many different types of tests each of them are designed to test the website in different ways and give different set of results. These include:
- Normal – designed to apply normal load to the website to ensure that expected normal operations can be achieved.
- Peak – designed to apply peak loading to the website to prove that the website delivers the expected levels of throughout and end user service.
- Stress tests – designed to place the website under extreme levels of load for a short period of time to determine how the website responds to a sudden major surge in traffic.
- Spike testing – designed to show how the website responds to regular spikes in traffic, such as after a series of TV adverts, to ensure it recovers and that the end user experience is not impacted.
- Endurance – designed to ensure that the website operates without impediment, such as memory leak or database issues, under a consistently high level of load.
Invariably a selection of these test types will be necessary to provide sufficient and accurate detail so that the load test objectives can be proven.
Load Testing Tools
There are a high number of load testing tools available commercially and although it is possible to create your own load test tools, invariably it is more cost effective to select one of the industry proven tools that are available. The benefits of using such a tool includes:
- Provision of scalable load testing capabilities
- Enabling a selection of load testing types
- Providing scripting capabilities
- Access to comprehensive reporting capabilities
- Identifying performance bottlenecks
- Minimizing risk
- Reduced costs
- Repeatable solution
To use load testing tools programming knowledge and skills will be necessary. Modern testing tools invariably use script-based languages which are relatively easy to learn and apply.
Load Test Plan
With the detail gathered the Load Test Plan (LTP) can be constructed. This is the formal planning document generated for the project and as a project planning document, it should have gated milestones set out within it to ensure the project occurs as planned.
The LTP is a comprehensive document and should, as a minimum, include the following items:
- Business and technical objectives – formalising the scoping
- Testing environment – ensuring that all components are identified
- Roles and responsibilities – so that all tasks and activities are assigned.
- Load test components – detailing user journeys, scripts, test data and any technical considerations.
- Testing approach – confirming test types, different processing scenarios, user concurrency, mix etc.
- Pre-test checklist – list any pre-test actions necessary, such as load test data or conduct a database backup.
- Schedule of tests – identifying the schedule for each test with any success / fail criteria listed and how many test windows are required.
- Expected deliverables – confirming the outcomes and documentation expected from the load testing.
- Communication – ensuring communication and escalation details are known and for all project members.
- Training – identifying training requirements for any project member.
- Risk register – listing all identified risks and mitigations
With the LTP completed the success of the load test is enhanced and an understanding of how to proceed in the event the load tests do not complete as expected.
Script Generation
Scripts are necessary in load testing as each test of a user journey is conducted by a ‘virtual end user’ client that must mimic the actions of a real visitor. Consequently, the actions of the real user must be programmed into the test so that the virtual user knows what actions to take.
This is achieved through scripting and the complexity of the scripts will depend on what activities need to be reproduced in the user journeys.
How scripts are created should be considered when selecting a load testing tool as script development can vary considerably in technical complexity.
There are several methods for creating scripts, each with their own advantages.
Build the Script from the HTTP Stream
This is a highly technical process and implies a strong understanding of HTTP and also programming capabilities to control the stream. This approach provides the greatest flexibility and control over the actions the virtual end user can take but due to the technical complexity and skills required, it may also be costly.
Use Script Generation Tools
This approach automates the majority of the script building process as the load test tool will provide a facility to collect the necessary HTTP stream automatically while you to perform the user journey manually. In the majority of cases this approach will generate the entire script that is capable of performing as a virtual end user. However, where a special case needs to be accommodated, facilities will generally exist to enable manual script programming to be incorporated within the script.
The majority of load test tools are covered by these two approaches, but if you prefer not to build scripts at all, then there are some load testing companies that will provide a managed service that negates any requirement for technical programming at all.
Script Verification
With the scripts written, they will need to be verified against the target website. This important step is necessary to ensure that when you start the scheduled load testing, confidence in the scripts is assured.
A common mistake is to assume that a single run of a script that completes successfully is fully verified, but it is recommended that verification runs between 5 and 10 scripts concurrently, especially on a new website, to prove the capability of concurrency of execution.
Although scripts may be proven weeks ahead of load testing a final verification step should be conducted immediately prior to the load test to ensure correct operation against the system to be tested. Often final website updates can negate previously working scripts, especially when CSS selectors or HTML changes have been applied and the changes have not been updated in the scripts.
Environment setup
As websites operate in complex technical environments, specific attention must be given to the demands the load testing will place upon it.
This is specifically important if the test is to be conducted in the live production environment as testing may impact on critical production data. As part of the planning process, test data necessary for the load test should be identified and developed and it should be verified as part of environmental setup.
Performance testing must be run against a stable and consistent codebase and as such requires the implementation of a code and architecture freeze prior to final script verification until after the load testing has been completed. This should ensure that the load tests progress smoothly and eliminates the need for rework due to unexpected changes.
Load testing is about ensuring the capability of your own infrastructure services so calls to third party software should be removed from user journeys in script generation. However, some third-party software will be necessary for the normal operation of the website, so in these cases you will need to communicate with the third party about the load test and the impact it may have on the provision of their services. This should include key infrastructure services, such as CDN providers who, if unaware may consider your website is under a DDoS attack and act accordingly.
All aspects of the environment should be prepared as referenced in the LTP and scheduled testing should not begin until all aspects of the environment preparation are complete.
Test Execution
With everything in place, load testing can begin. A considerable amount of work and cost will have gone into getting to this point and so it is important to follow the agreed load test plan.
This requires starting and ending the load test window as planned as other business activities may require the resources the load test utilizes, especially if the live production environment is hosting the tests.
Each test is processed sequentially and the next test should not proceed until the success or fail criteria of the preceding test has been confirmed. This is to ensure that resources are not wasted. For example, if you are planning a test for peak load and the test of normal load is unsuccessful, then it may not be prudent to proceed until the cause of any failures are understood and rectified.
Irrespective of the success criteria, sufficient time for the environment to reset itself should be given between tests. 15 minutes should be sufficient for this unless there are other activities, such as database recovery, that must be completed as a test pre-requisite.
It is also important to ensure there is clear and concise communication between all members of the project, as listed in the LTP, during the test window. This ensure that activities occur quickly and efficiently and the use of the time and resources available in the load testing window are maximized.
Results Analysis and Reporting
Some of these activities will occur during the test window to enable the success criteria to be determined. However, it is after the testing window has finished that real analysis can begin.
Load testing tools generate a considerable amount of data and present it in topical reports enabling the metrics necessary to compare against the key performance criteria to be obtained.
It is often necessary to drill down deeper to find opportunities for performance improvement, especially when key performance criteria are not met.
The reporting requirements defined in the LTP should be completed and approved within the appropriate project milestone. However, where performance opportunities have been identified, these should also be included in an addendum if a pre-defined section does not exist in the report.
As the load test will have an executive sponsor, it is important to ensure that the report has a concise executive summary section that identifies the successes and failures, if any, together with any recommended next steps.
Project Complete
Based on the results, it is possible that several steps of the LTP are iterative as identified problems are resolved and retested. Once all project milestones have been passed, the project can be considered completed.
Constructed correctly, the load test plan can be used as a repeatable process for load testing. It should ensure that all changes necessary in each iteration are accounted for in the scoping and planning and it is important to incorporate lessons learned from previous load tests.
The performance load test process enables organizations to encompass a highly technical set of activities within the bounds of a repeatable project. This can establish a high degree of confidence in the website’s capability of delivery of services under different load conditions and can also provide the basis for continual improvement of its performance and the end user experience it delivers.