Leveraging GitHub Actions for Apache JMeter Load Testing
Load testing is crucial for ensuring the performance and reliability of web applications. Apache JMeter, an open-source tool, is widely used for load testing. GitHub Actions provides automation within the software development lifecycle. Combining these enables seamless automation of load testing processes within GitHub workflows.
Introduction to Apache JMeter
Apache JMeter is a Java-based open-source tool designed for load testing and performance measurement of various services, focusing primarily on web applications. It allows users to simulate different types of loads, including heavy loads, concurrent loads, and distributed loads, to assess the performance under various scenarios.
Key features of Apache JMeter include:
- Ability to create test plans with a graphical user interface (GUI).
- Support for various protocols such as HTTP, HTTPS, FTP, JDBC, SOAP, and more.
- Comprehensive reporting and analysis capabilities.
- Extensibility through plugins to enhance functionality.
GitHub Actions Overview
GitHub Actions automates workflows within GitHub repositories.
Key features of GitHub Actions include:
- YAML-based syntax for defining workflows.
- Integration with GitHub repositories and events.
- Extensive marketplace for reusable Actions.
- Support for continuous integration (CI) and continuous deployment (CD) pipelines.
Integrating JMeter Load Testing with GitHub Actions
To integrate Apache JMeter load testing into your GitHub workflow, you can leverage GitHub Actions to automate the execution of JMeter test plans. Here’s a step-by-step guide to get you started:
Step 1: Installation
Apache JMeter installation is straightforward. However, some considerations should be taken into account. Ensure to provide the correct Java/JMeter path in your environment variables. Additionally, allocate sufficient memory to execute your test cases.
There are two use cases for Apache JMeter installation:
- GitHub Hosted Agents: If you’re not dealing with single sign-on APIs, you can install Apache JMeter on GitHub agents instantaneously.
- Single Sign-On (SSO) Method: If you’re testing APIs that use a single sign-on method, you need to install Apache JMeter in the regular way.
Lets look further how you can install them individually.
Github Hosted Agents.
If your dealing with Non Single Sign on API’s then you can use below script to install Apache JMeter on your github agents on the fly. Some of the components which are must for Apache JMeter installation.
- Java
- valid python libraries (use requirements.txt)
When you are dealing with GitHub hosted agent, some of the additional components which are:
Make sure you install valid Apache JMeter binaries currently in my case I have used JMeter version 5.5 but you can use latest 5.6 version as well. Once you install Apache JMeter there are some additional components you need to install such as command runner, plugins manager.
Additional plugins using the Plugins Manager to extend the capabilities of Apache JMeter. In the provided example, the PerfMon plugin is installed, which allows monitoring of system resources during the performance test. Depending on your specific testing requirements, you may need to install additional plugins for advanced functionalities.
In the provided example, plugins such as jpgc-synthesis
, jpgc-graphs-basic
, jpgc-graphs-additional
, jpgc-cmd
, jpgc-graphs-dist
, jpgc-standard
, and jpgc-casutg
are installed. These plugins offer various features such as advanced graphing capabilities, command-line integration, and custom test components, enhancing the effectiveness of performance testing scenarios.
Along with that Install additional plugins such as the PerfMon plugin using the Plugins Manager Command Line Tool. The PerfMon plugin allows monitoring of system resources during performance tests, providing valuable insights into the performance characteristics and resource utilization of the target system under load.
name: Performance Testing
on:
push:
branches:
- feature/prashanth-perf-test
jobs:
performance_test:
runs-on: ubuntu-latest
env:
JMETER_THREADS: 10
JMETER_DURATION: 60
JMETER_PATH: 'Performance/filename.jmx'
WORKING_DIRECTORY: Performance
steps:
- uses: actions/checkout@v3
- name: Set up Python 3.11
uses: actions/setup-python@v4
with:
python-version: 3.11
- name: Setup-Jmeter
run: |
python3 -m pip install --upgrade pip
pip3 install -r requirements.txt -q
sudo apt-get update
sudo apt install curl -y
sudo apt install -y default-jdk
sudo curl -O https://archive.apache.org/dist/jmeter/binaries/apache-jmeter-5.5.tgz
sudo tar -xvf apache-jmeter-5.5.tgz
cd $GITHUB_WORKSPACE/apache-jmeter-5.5/lib && sudo curl -O https://repo1.maven.org/maven2/kg/apc/cmdrunner/2.2.1/cmdrunner-2.2.1.jar
cd $GITHUB_WORKSPACE/apache-jmeter-5.5/lib/ext && sudo curl -O https://repo1.maven.org/maven2/kg/apc/jmeter-plugins-manager/1.6/jmeter-plugins-manager-1.6.jar
cd $GITHUB_WORKSPACE/apache-jmeter-5.5/lib && sudo java -jar cmdrunner-2.2.1.jar --tool org.jmeterplugins.repository.PluginManagerCMD install jpgc-synthesis,jpgc-graphs-basic,jpgc-graphs-additional,jpgc-cmd,jpgc-graphs-dist,jpgc-standard,jpgc-casutg
- name: Install PerfMon Plugin
run: |
cd $GITHUB_WORKSPACE/apache-jmeter-5.5/lib
sudo java -jar cmdrunner-2.2.1.jar --tool org.jmeterplugins.repository.PluginManagerCMD install jpgc-perfmon
For performance test you can use parameters such as the number of threads (JMETER_THREADS
), duration of the test (JMETER_DURATION
), and the path to the JMeter test plan (JMETER_PATH
). This allows for easy customization and adaptation of the testing parameters based on specific requirements.
Apache JMeter Installation on Self hosted agent
Now lets look into next scenario of Installing Apache JMeter, make sure you download all required binaries and install it in Azure VM. Currently in my case I have used Windows 2022 server. Couple of items to remember make sure once you install Apache JMeter you add below variables in your environment variables.
- Java path
- JMeter path.
Next in order to run Apache JMeter make sure you allocate enough memory or else your GitHub workflow will fail.
Here is the YAML file if you are using self hosted agent.
name: Performance Testing
on:
push:
branches:
- feature/prashanth-perf-test
jobs:
performance_test:
runs-on: [self-hosted, Windows, DemoMachine]
env:
JMETER_THREADS: 10
JMETER_DURATION: 60
JMETER_HOME: 'C:\apache-jmeter-5.6.3'
WORKING_DIRECTORY: 'Performance'
PYTHON_HOME: 'C:\actions-runner\_work\_tool\Python\3.9.13\x64'
steps:
- name: Checkout code
uses: actions/checkout@v3
- name: Clear JTL file
run: |
$env:JMETER_RESULT="$GITHUB_WORKSPACE\${{ env.WORKING_DIRECTORY }}\result.jtl"
Clear-Content -Path $env:JMETER_RESULT
- name: Run Performance Tests
run: |
$env:JMETER_SCRIPT="$GITHUB_WORKSPACE\${{ env.WORKING_DIRECTORY }}\filename.jmx"
$env:JMETER_RESULT="$GITHUB_WORKSPACE\${{ env.WORKING_DIRECTORY }}\result.jtl"
& "$env:JMETER_HOME\bin\jmeter.bat" -Jthreads=${{ env.JMETER_THREADS }} -Jloadtime=${{ env.JMETER_DURATION }} -n -t "$env:JMETER_SCRIPT" -l "$env:JMETER_RESULT" -e -o "$env:GITHUB_WORKSPACE\${{ env.WORKING_DIRECTORY }}\html-reports"
- name: Upload JMeter Test Results
uses: actions/upload-artifact@v3
if: always()
with:
name: perf_test_results
path: |
${{ env.JMETER_RESULT }}
${{ github.workspace }}\${{ env.WORKING_DIRECTORY }}\html-reports
Some of the steps which I am using it in my workflow are
I am clearing old results file as i want to see new results everytime so I have added a step to clear old content and it should be replaced with new one.
As you want to start Apache JMeter inside VM so in “Run Performance Tests” step I have added my .jmx file path which needs to be launched and then result.jtl file path which needs to be generated everytime.
Along with that I am passing additional variables such as JMeter threads, Duration.
Step2: Create JMeter Test Plan
Begin by creating your JMeter test plan using the JMeter GUI or by writing the test plan in XML format. Your test plan should include details such as target URLs, thread groups, samplers, listeners, and assertions based on your load testing requirements.
Lets look into individual steps.
- First step in your workflow is to add user variables if you are using any specific username or password you can add them in “User Parameters”.
2. Next step you need to start adding the actual workflow and backend calls, the best way to capture these calls is via Inspect option and then the same can be added in your JMeter workflow. In our case I am using OIDC authentication and capturing “code_challenge”, “client_id”, “state” etc.
Next step is to add your Authorization step
and along with this I am extracting CertID
Step 3: Store Test Plan in Repository
Store your JMeter test plan within your GitHub repository. This ensures that the test plan is version-controlled and accessible to your GitHub Actions workflows. Currently I have saved my .jmx file in my github repo and added this path in “WORKING_DIRECTORY”.
WORKING_DIRECTORY: ‘Performance’
Step 4: Define GitHub Actions Workflow
Create a YAML file named workflow.yml
in the .github/workflows
directory of your repository. This file defines the workflow for executing the load tests using JMeter. Here's an example workflow:
Here is the full extracted YAML file with Self Hosted agent and Github Agent.
Github Agent
name: Performance Testing
on:
push:
branches:
- feature/prashanth-perf-test
jobs:
performance_test:
runs-on: ubuntu-latest
env:
JMETER_THREADS: 10
JMETER_DURATION: 60
JMETER_PATH: 'Performance/filename.jmx'
WORKING_DIRECTORY: Performance
steps:
- uses: actions/checkout@v3
- name: Set up Python 3.9
uses: actions/setup-python@v4
with:
python-version: 3.9
- name: Setup-Jmeter
run: |
python3 -m pip install --upgrade pip
pip3 install -r requirements.txt -q
sudo apt-get update
sudo apt install curl -y
sudo apt install -y default-jdk
sudo curl -O https://archive.apache.org/dist/jmeter/binaries/apache-jmeter-5.5.tgz
sudo tar -xvf apache-jmeter-5.5.tgz
cd $GITHUB_WORKSPACE/apache-jmeter-5.5/lib && sudo curl -O https://repo1.maven.org/maven2/kg/apc/cmdrunner/2.2.1/cmdrunner-2.2.1.jar
cd $GITHUB_WORKSPACE/apache-jmeter-5.5/lib/ext && sudo curl -O https://repo1.maven.org/maven2/kg/apc/jmeter-plugins-manager/1.6/jmeter-plugins-manager-1.6.jar
cd $GITHUB_WORKSPACE/apache-jmeter-5.5/lib && sudo java -jar cmdrunner-2.2.1.jar --tool org.jmeterplugins.repository.PluginManagerCMD install jpgc-synthesis,jpgc-graphs-basic,jpgc-graphs-additional,jpgc-cmd,jpgc-graphs-dist,jpgc-standard,jpgc-casutg
- name: Install PerfMon Plugin
run: |
cd $GITHUB_WORKSPACE/apache-jmeter-5.5/lib
sudo java -jar cmdrunner-2.2.1.jar --tool org.jmeterplugins.repository.PluginManagerCMD install jpgc-perfmon
- name: Run Performance Tests
run: |
$GITHUB_WORKSPACE/apache-jmeter-5.5/bin/./jmeter.sh -Jthreads=$JMETER_THREADS -Jloadtime=$JMETER_DURATION -n -t ${{ env.WORKING_DIRECTORY }}/filename.jmx -l ${{ env.WORKING_DIRECTORY }}/result.jtl
sudo java -jar $GITHUB_WORKSPACE/apache-jmeter-5.5/lib/cmdrunner-2.2.1.jar --tool Reporter --generate-csv aggregate_report.csv --input-jtl ${{ env.WORKING_DIRECTORY }}/result.jtl --plugin-type AggregateReport
sudo $GITHUB_WORKSPACE/apache-jmeter-5.5/bin/./jmeter.sh -g ${{ env.WORKING_DIRECTORY }}/result.jtl -o ${{ env.WORKING_DIRECTORY }}/graph_results
- name: Upload JMeter Test Results
uses: actions/upload-artifact@v3
with:
name: perf_test_results
path: |
result.jtl
aggregate_report.csv
Performance/graph_results
- name: Deploy JMeter HTML Report
uses: peaceiris/actions-gh-pages@v3
with:
github_token: ${{ secrets.GITHUB_TOKEN }}
publish_dir: ${{ env.WORKING_DIRECTORY }}/graph_results
Self Hosted agent
name: Performance Testing
on:
push:
branches:
- feature/prashanth-perf-test
jobs:
performance_test:
runs-on: [self-hosted, Windows, WellTracker]
env:
JMETER_THREADS: 10
JMETER_DURATION: 60
JMETER_HOME: 'C:\apache-jmeter-5.6.3'
WORKING_DIRECTORY: 'Performance'
PYTHON_HOME: 'C:\actions-runner\_work\_tool\Python\3.9.13\x64'
steps:
- name: Checkout code
uses: actions/checkout@v3
- name: Replace Password values
run: |
$file = "${{ env.WORKING_DIRECTORY }}/filename.jmx"
$content = Get-Content $file
$content | ForEach-Object { $_ -replace "passwordwhichneedstobechanged", "${{ secrets.SSO_PASSWORD }}" } | Set-Content $file
- name: Print GitHub Workspace
run: |
echo "GitHub Workspace: ${{ github.workspace }}"
- name: Print Working Directory
run: |
echo "Working Directory: ${{ env.WORKING_DIRECTORY }}"
- name: Clear JTL file
run: |
$env:JMETER_RESULT="$GITHUB_WORKSPACE\${{ env.WORKING_DIRECTORY }}\result.jtl"
Clear-Content -Path $env:JMETER_RESULT
- name: Run Performance Tests
run: |
$env:JMETER_SCRIPT="$GITHUB_WORKSPACE\${{ env.WORKING_DIRECTORY }}\filename.jmx"
$env:JMETER_RESULT="$GITHUB_WORKSPACE\${{ env.WORKING_DIRECTORY }}\result.jtl"
& "$env:JMETER_HOME\bin\jmeter.bat" -Jthreads=${{ env.JMETER_THREADS }} -Jloadtime=${{ env.JMETER_DURATION }} -n -t "$env:JMETER_SCRIPT" -l "$env:JMETER_RESULT" -e -o "$env:GITHUB_WORKSPACE\${{ env.WORKING_DIRECTORY }}\html-reports"
- name: Upload JMeter Test Results
uses: actions/upload-artifact@v3
if: always()
with:
name: perf_test_results
path: |
${{ env.JMETER_RESULT }}
${{ github.workspace }}\${{ env.WORKING_DIRECTORY }}\html-reports
I am using replace step as you might be using some username or password in your .jmx file, so it is better to get them from Github secrets. Currently I am using “Replace Password Values
" using PowerShell as its a Windows VM.
If you are using Linux Agent then you can use sed command.
- name: Replace Password values
run: |
sed -i "s|passwordwhichneedstobechanged|${{ secrets.CYPRESS_SSO_PASSWORD }}|" welltracker1.jmx
working-directory: ${{ env.WORKING_DIRECTORY }}
Step 5: Execute Load Tests
Commit and push your changes to the main branch of your GitHub repository. This triggers the GitHub Actions workflow defined in the workflow.yml
file. The workflow downloads JMeter, executes the specified test plan, and saves the results.
When you execute your workflow it will generate 2
- Results.jtl
- Index.html
If you are testing it locally then you can view everything using “View Results Tree”.
Step 6: View Test Results
Once the workflow execution is complete, you can view the test results within the GitHub Actions interface or by downloading the artifacts generated during the workflow run. Analyze the results to identify performance bottlenecks and optimize your application accordingly.
Once you extract zip file you will see graph_results, here is the quick snippet.
Apart from that report will show some additional details such as:
- Statistics
- Errors
- Top 5 Errors by sampler
Conclusion
Overall never thought setting up and configuring JMeter is so easy.
By integrating Apache JMeter load testing with GitHub Actions, teams can automate their load testing processes, ensuring consistent performance evaluation throughout the software development lifecycle. This combination facilitates faster feedback loops, improves code quality, and enhances the overall reliability and scalability of web applications. Start leveraging GitHub Actions for your load testing needs today to streamline your development workflows and deliver high-quality software with confidence.