DevOps is taking over all the IT industry. Neoload is one of the Automated testing environment that helps to test software in an. Hi there I am unable to execute a user path in 7.0.2 version and the log info from the Agent file shows below. 2019/11/22 12:32:04 INFO - neoload.agent.Agent: Starting.
Latest versionReleased:
A command-line native utility for launching and observing NeoLoad performance tests
Project description
Overview
This command-line interface helps you launch and observe performance tests on the Neotys Web Platform. Since NeoLoad is very flexible to many deployment models (SaaS, self-hosted, cloud or local containers, etc.), configuration and test execution parameters depend on your licensing and infrastructure provisioning options.
Property | Value |
---|---|
Maturity | Stable |
Author | Neotys |
License | BSD 2-Clause 'Simplified' |
NeoLoad Licensing | License FREE edition, or Enterprise edition, or Professional |
Supported versions | Tested with NeoLoad Web from version 2.3.2 |
Download Binaries | See the latest release on pypi |
TL;DR . What
The goal of this guide is to demonstrate how you can:
- create API load tests using code (YAML)
- run them from any environment
- visualize test results in web dashboards
TL;DR . How
NOTE: For Windows command line, replace the ' multi-line separators above with '^'
Contents
- Setup a test
- Upload a Neoload project
- [Excluding files from the project upload] (#excluding-files-from-the-project-upload)
- Upload a Neoload project
- Run a test
- Continuous Testing Examples
Prerequisites
The CLI requires Python3
- Download and install python3 for Windows 10 from Python.org
- Make sure you check the option 'Add Python to the environment variables' option
- Install pip:
python -m pip install -U pip
- Download and install python3 for Mac OS X from Python.org - Python3 on Mac OS X
Optional: Install Docker for hosting the test infra on your machine (this feature does not work with Docker for Windows).
Installation
NOTE: if you receive SSL download errors when running the above command, you may also need to install sources using the following command:
Login to Neoload Web
NeoLoad CLI defaults to using the NeoLoad Web APIs for most operations. That's why you need to login.
The CLI will connect by default to Neoload Web SaaS to lease license.
For self-hosted enterprise license, you must specify the Neoload Web Api url with --url.
The CLI stores data locally like api url, token, the workspace ID and the test ID you are working on. The commands can be chained !
Setup a test
Optionally Choose a workspace to work with
Since Neoload Web 2.5 (August 2020), assets are scoped to workspaces.The CLI allows you to choose your workspace at login or with the 'use' sub-command, otherwise the 'Default Workspace' is used.
/! The zones are shared between workspaces.
Setup resources in Neoload Web
Run a test requires an infrastructure that is defined in Neoload Web Zones section (see documentation how to manage zones)You must at least have either a dynamic or a static zone with one controller and one load generator. At First, you could add resources to the 'Default zone' since the CLI use it by default.
Define a test settings
Test settings are how to run a test, a sort of template. Tests are stored in Neoload Web.
You must define :
- Which scenario of the Neoload project to use
You can optionally define :
- The test-settings description
- The controller and load generator's zone to use (defaultzone is set by default)
- How many load generators to use for the zone (1 LG on the defaultzone is set by default)
- Advanced users who already have several zones with available resources in it, specify all zones and the number of LGs with --controller-zone-id and lg-zone-ids
Neoload 7 0 25
To work with a specific test already created and be able to chain commands
Upload a Neoload project
See basic projects examples on github tests/neoload_projects folderTo upload a NeoLoad project zip file or a standalone as code file into a test-settings
To Validate the syntax and schema of the as-code project yaml files
Excluding files from the project upload
If you are uploading a project directory that contains non NeoLoad as-code YAML files (such as .gitlab-ci.yml) you will need to create a .nlignore file (exactly the same as .gitignore) that excludes these files from the project upload process so that NeoLoad Web does not parse them and fail them as if they should be the NeoLoad DSL.
Please see Gitlab and Azure pipeline examples for more detail.
Run a test
This command runs a test, it produces blocking, unbuffered output about test execution process, including readout of current data points.At the end, displays the summary and the SLA passed & failed.
- detach option kick off a test and returns immediately. Logs are displayed in Neoload Web (follow the url).
- as-code option specify as-code yaml files to use for the test. They should already be uploaded with the project.
- Test result name and description can be customized to include CI specific details (e.g. CI job, build number.).
- Reservations can be used with either the reservationID or a reservation duration and a number of Virtual users.
If you are running in interactive console mode, the NeoLoad CLI will automatically open the system default browser to your live test results.
When hitting Ctrl+C, the CLI will try to stop the test gracefully
Stop a running test
View results
Metadata on a test can be modified after the test is complete, such as name, description, and status.
To work with a specific test result and be able to chain commands
Detailed logs and results are available on Neoload Web. To get the url of the current result :
View zones
Display in a human readable way the list of all static and dynamic zones registered on Neoload Web, and the resources attached (controllers and load generators).
Create local docker infrastructure to run a test [EXPERIMENTAL]
WARNING: Docker features are not officially supported by Neotys as they rely heavily on your own Docker setup and environment. This command is only for local/dev test scenarios to simplify infrastructure requirements.
NOTE: this functionality is not in the 1.0.0 version (May 2020 on Pypi), but is scheduled for inclusion by June 16th 2020.If you want to obtain this version before that time, please intall the version 1.1.0 release candidate from pypi:
If you want to use the latest commit of this feature, please pull this Git repo, checkout the topic-docker-commandbranch and install locally. You may need to uninstall your existing version of this CLI first:
In certain environments, such as on a local dev workstation or in a Docker-in-Docker CI build node, it is usefulto 'bring your own infrastructure'. In other words, when you don't already have a controller and load generatorsavailable in a zone, you can spin some up using Docker before the test starts. An example of an all-on-one approach:
Neoload 7 0 2 Sezonas
Delicious panel retouch 4 0 download free. What the 'docker prepare' CLI command does is to look at the test-settings for what zones and how many resourceswe need, then create local Docker containers and attach them to NeoLoad Web accordingly in preparation for 'run'.
NOTE: Docker CLI must be installed on the system using these commands. This will usethe Docker daemon, however it is configured. In a Docker-in-Docker context, this is inferred.For local workstations, it is sufficient to install Docker Desktop or Docker for Mac.
NOTE: If the 'prepare' or 'attach' actions are used before the 'run' command, the test will use or reusethe Docker configuration for infrastructure. This requires that all zones in test-settings be static zones.
NOTE: The 'forget' action undos the above note, in cases where static zones were in use by test-settingsat first, but then were changed to use dynamic zones where Docker attaches make no sense.
NOTE: When using the 'detach' or 'forget' actions and containers are running, they will be stopped.There will be a prompt/check if stdin is attached to this process (typically not the case in CI)
Pre-connecting Docker in Preparation for Consecutive Test Runs
You may also want to spin up Docker containers and keep them around for multiple test runs using the sameinfrastructure, such that:
Continuous Testing Examples
The main goal of the NeoLoad-CLI is to standardize the semantics of how load tests are executed across development, non-prod, and production environments.While the above instructions could be run from a contributor workstation, they can easily be translated to various continuous build and deployment orchestration environments, as exampled:
- Sorry AWS CodeBuild, haven't seen any F100 clients using the pform
- CircleCI, TBD when @punkdata gets back to @paulsbruce :)
NB: When chaining commands, the return code of the whole command is the return code of the last command. That's why you should not chain the two commands 'run' and 'test-results junitsla'.
NOTE: When combining NeoLoad projects and YAML-based pipeline declarations, please see [Excluding files from the project upload] (#excluding-files-from-the-project-upload) to ensure that unecessary artifacts aren't included in the project upload process.
Support for fast-fail based on SLAs
Not all tests succeed. Sometimes environments are down. Sometimes 3rd parties are surprisingly slow. You don'twant to wait for your build pipelines to conduct the whole test duration if it's possible to identify theseissues early. Applying proper SLAs to your tests allows you to monitor for errors and latency during the test.
Consider the following SLA:
If you want to fail the pipeline if either of these thresholds are exceeded over a certain percent of their times,you must:
- run the test in 'detached' mode to allow for non-blocking execution of a test
- use the fastfail command to monitor for early signals to stop the test if SLAs are violated
- finally wait for the test results
To run the test in detached mode:
Then immediately afterwards, use the fastfail command:
In the above example, '25' represents the percent of times where the SLA was violated, such as 'on a particularrequest with an SLA applied, 10 out of 50 times it was executed, the SLA failed'.
Finally, because the test was executed in non-blocking mode, you should wait for the final test result.
Packaging the CLI with Build Agents
Many of the above CI examples include a step to explicitly install the NeoLoad CLI as part of thebuild steps. However, if you want the CLI baked into some build agent directly so that itis ready for use during a job, here's a Docker example:
For Docker builds See the test harness Alpine-based Dockerfile
IDE Integrations
Since most of what we do in an IDE is create/edit code, we're mostly interested in how to:
- make it easy to write API tests in YAML (automatic syntax validation)
- validate that tests do not contain unanticipated errors even at small scale
- dry-run small (smoke) load tests locally so that code check-ins will work in CI/pipeline tests
Since the latter two cases are already covered by command-line semantics, our primary focusis to accelerate test authoring by providing NeoLoad as-code DSL (Domain-specific Language) validationand in some cases editor auto-complete.
Status of IDE / editor integrations
IDE / Editor | Syntax checks | Auto-complete | Setup steps |
---|---|---|---|
Visual Studio Code | [x] | [x] | see instructions |
PyCharm | [x] | [x] | Mark 'neoload' directory as 'Sources Root' |
Contributing
Feel free to fork this repo, make changes, test locally, and create a pull request. As part of your testing, you should run the built-in test suite with the following command:
NOTE: for testing from Mac, please change the PYTHONPATH separators below to colons (:) instead of semicolons (;).
Additionally, any contributions to the DSL validation functionality, such as on the JSON schema or the validate command, should execute the following tests locally before pushing to this repo:
This command executes a number of NEGATIVE tests to prove that changes to the JSON schema or validation process produce failures when their input is malformed in very specific ways (common mistakes).
Release historyRelease notifications | RSS feed
1.1.4
1.1.4rc8 pre-release
1.1.4rc7 pre-release
1.1.4rc6 pre-release
1.1.4rc5 pre-release
1.1.4rc4 pre-release
1.1.4rc3 pre-release
1.1.4rc2 pre-release
1.1.4rc1 pre-release
1.1.3
1.1.2
1.1.1rc1 pre-release
1.1.0
1.1.0rc2 pre-release
1.1.0rc1 pre-release
1.0.1
1.0.0
1.0.0rc3 pre-release
1.0.0rc2 pre-release
0.4.6
0.4.5
0.4.4
0.4.3
0.4.2
0.4.1
Neoload 7 0 2 0
0.4
0.3.10
0.3.9
0.3.8
0.3.7
0.3.6
0.3.5
0.3.4
0.3.3
0.3.2
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Filename, size | File type | Python version | Upload date | Hashes |
---|---|---|---|---|
Filename, size neoload-1.1.4.tar.gz (454.5 kB) | File type Source | Python version None | Upload date | Hashes |
Hashes for neoload-1.1.4.tar.gz
Algorithm | Hash digest |
---|---|
SHA256 | c85f7021f15bdf0bd11af983bb704cca4d67d3a28ff66d0fdb16d1740cd2818b |
MD5 | 079b131fbdc84b063cef3c51baed772d |
BLAKE2-256 | fc7ec3683bb55fd27c39255ad7f7cf3412b316d2dfb1e764c06463ef153468bf |
This plugin allows you to monitor load tests performed by NeoLoad from Jenkins. NeoLoad test results are combined with the other integration jobs in the Jenkins dashboard. Performance regression issues are raised quickly to make Jenkins integration projects more relevant. For regular jobs or pipeline jobs.
Older versions of this plugin may not be safe to use. Please review the following warnings before using an older version:
After every build of a job configured with the NeoLoad plugin, Jenkins can display:- the HTML summary of the NeoLoad test result per build available with the Performance Result command
- the JUnit details of every passed or failed test available with the Test Result command---also used in the test result trend graph
See https://www.neotys.com/documents/doc/neoload/latest/en/html/#5769.htm for detailed documentation.
Prerequisites
The NeoLoad plugin, from version 2.1.0 requires:
- Jenkins 1.609.1 or later,
- Java 7 or later.
Collaboration Configuration
The plugin allows to configure a collaboration server like Neotys Team Server (NTS) with the Manage Jenkins > Configure System command.
Build Step
The plugin adds a new build step: Execute a NeoLoad Scenario.
The following steps are required:
- Type the path of the NeoLoad executable.
- Select Local Project or Shared Project and configure it.
- Type the name of the NeoLoad scenario in the Scenario Name field.
- Select Default Report File Names or Custom Report File Names and configure it.
- Select Existing License or Shared License and configure it.
- Select Display Trend Graph: Average Response Time (all pages) to include the Avg. Resp. Time trend graph in Jenkins.
- Select Display Trend Graph: Error Rate % to include the Error Rate trend graph in Jenkins.
- (Recommended) Add as many user-defined graphs as wanted with several curves on each graph.
Post-Build Action
The plugin needs the Archive the artifacts post-build action. The regeneration of trends could be triggered. Please archive the artifact before Refresh trends.
Example Trend Graphs
Pipelines Steps
The 'neoloadRun' step in the Jenkins Snippet Generator makes it possible to run a NeoLoad scenario from Jenkins. It also archives the reports and refreshes the graphs.
- Warning: To use the Snippet Generator, the Jenkins project including the job to configure must be compliant with Pipeline as code. For more information, see Pipeline as code.
Once the Jenkins project is selected, the Snippet Generator is accessible with a click on the Pipeline Syntax link.
This plugin provides two steps:
neoloadRun: to run NeoLoad scenario, archive report and refresh the trends.
- neoloadRefreshTrends: to refresh or change the trends only.
Refresh graph
FAQ
Why don't I see any trend graphs?
In order to see trend graphs, please verify:
- The Archive the artifacts post-build action has been added.
- Either Default Report File Names or Custom Report File Names is selected and an xml report is defined.
- At least two executions were run.
- Date and time is synchronized between the Jenkins machine and the build machine.
Known Issues
1. The NeoLoad report file (via artifacts and the Performance Result link) displays a blank page. This affects versions released before NeoLoad 5.2.
Workaround
Use the Jenkins Script Console to disable the sandboxing security by executing the following script. The Script Console is under Jenkins -> Manage Jenkins -> Script Console.
Clear the cache afterwards (hold shift and reload the page).
See https://wiki.jenkins-ci.org/display/JENKINS/Configuring+Content+Security+Policy for more information.
2. The NeoLoad Graphs aren't displayed in the main page of my job.
Workaround
Make sure you used a 'Freestyle project' for your job. If you use (for example) the Maven Plugin for your job, create a 'Freestyle project' then add Maven configuration build step.
Changelog
Version 2.2.6 (released October 03, 2019)
FIXED: Passwords storage has been changed for security reasons.
Version 2.2.5 (released May 20, 2019)
IMPROVED: Allow usage of YAML or JSON local project.
Setup resources in Neoload Web
Run a test requires an infrastructure that is defined in Neoload Web Zones section (see documentation how to manage zones)You must at least have either a dynamic or a static zone with one controller and one load generator. At First, you could add resources to the 'Default zone' since the CLI use it by default.
Define a test settings
Test settings are how to run a test, a sort of template. Tests are stored in Neoload Web.
You must define :
- Which scenario of the Neoload project to use
You can optionally define :
- The test-settings description
- The controller and load generator's zone to use (defaultzone is set by default)
- How many load generators to use for the zone (1 LG on the defaultzone is set by default)
- Advanced users who already have several zones with available resources in it, specify all zones and the number of LGs with --controller-zone-id and lg-zone-ids
Neoload 7 0 25
To work with a specific test already created and be able to chain commands
Upload a Neoload project
See basic projects examples on github tests/neoload_projects folderTo upload a NeoLoad project zip file or a standalone as code file into a test-settings
To Validate the syntax and schema of the as-code project yaml files
Excluding files from the project upload
If you are uploading a project directory that contains non NeoLoad as-code YAML files (such as .gitlab-ci.yml) you will need to create a .nlignore file (exactly the same as .gitignore) that excludes these files from the project upload process so that NeoLoad Web does not parse them and fail them as if they should be the NeoLoad DSL.
Please see Gitlab and Azure pipeline examples for more detail.
Run a test
This command runs a test, it produces blocking, unbuffered output about test execution process, including readout of current data points.At the end, displays the summary and the SLA passed & failed.
- detach option kick off a test and returns immediately. Logs are displayed in Neoload Web (follow the url).
- as-code option specify as-code yaml files to use for the test. They should already be uploaded with the project.
- Test result name and description can be customized to include CI specific details (e.g. CI job, build number.).
- Reservations can be used with either the reservationID or a reservation duration and a number of Virtual users.
If you are running in interactive console mode, the NeoLoad CLI will automatically open the system default browser to your live test results.
When hitting Ctrl+C, the CLI will try to stop the test gracefully
Stop a running test
View results
Metadata on a test can be modified after the test is complete, such as name, description, and status.
To work with a specific test result and be able to chain commands
Detailed logs and results are available on Neoload Web. To get the url of the current result :
View zones
Display in a human readable way the list of all static and dynamic zones registered on Neoload Web, and the resources attached (controllers and load generators).
Create local docker infrastructure to run a test [EXPERIMENTAL]
WARNING: Docker features are not officially supported by Neotys as they rely heavily on your own Docker setup and environment. This command is only for local/dev test scenarios to simplify infrastructure requirements.
NOTE: this functionality is not in the 1.0.0 version (May 2020 on Pypi), but is scheduled for inclusion by June 16th 2020.If you want to obtain this version before that time, please intall the version 1.1.0 release candidate from pypi:
If you want to use the latest commit of this feature, please pull this Git repo, checkout the topic-docker-commandbranch and install locally. You may need to uninstall your existing version of this CLI first:
In certain environments, such as on a local dev workstation or in a Docker-in-Docker CI build node, it is usefulto 'bring your own infrastructure'. In other words, when you don't already have a controller and load generatorsavailable in a zone, you can spin some up using Docker before the test starts. An example of an all-on-one approach:
Neoload 7 0 2 Sezonas
Delicious panel retouch 4 0 download free. What the 'docker prepare' CLI command does is to look at the test-settings for what zones and how many resourceswe need, then create local Docker containers and attach them to NeoLoad Web accordingly in preparation for 'run'.
NOTE: Docker CLI must be installed on the system using these commands. This will usethe Docker daemon, however it is configured. In a Docker-in-Docker context, this is inferred.For local workstations, it is sufficient to install Docker Desktop or Docker for Mac.
NOTE: If the 'prepare' or 'attach' actions are used before the 'run' command, the test will use or reusethe Docker configuration for infrastructure. This requires that all zones in test-settings be static zones.
NOTE: The 'forget' action undos the above note, in cases where static zones were in use by test-settingsat first, but then were changed to use dynamic zones where Docker attaches make no sense.
NOTE: When using the 'detach' or 'forget' actions and containers are running, they will be stopped.There will be a prompt/check if stdin is attached to this process (typically not the case in CI)
Pre-connecting Docker in Preparation for Consecutive Test Runs
You may also want to spin up Docker containers and keep them around for multiple test runs using the sameinfrastructure, such that:
Continuous Testing Examples
The main goal of the NeoLoad-CLI is to standardize the semantics of how load tests are executed across development, non-prod, and production environments.While the above instructions could be run from a contributor workstation, they can easily be translated to various continuous build and deployment orchestration environments, as exampled:
- Sorry AWS CodeBuild, haven't seen any F100 clients using the pform
- CircleCI, TBD when @punkdata gets back to @paulsbruce :)
NB: When chaining commands, the return code of the whole command is the return code of the last command. That's why you should not chain the two commands 'run' and 'test-results junitsla'.
NOTE: When combining NeoLoad projects and YAML-based pipeline declarations, please see [Excluding files from the project upload] (#excluding-files-from-the-project-upload) to ensure that unecessary artifacts aren't included in the project upload process.
Support for fast-fail based on SLAs
Not all tests succeed. Sometimes environments are down. Sometimes 3rd parties are surprisingly slow. You don'twant to wait for your build pipelines to conduct the whole test duration if it's possible to identify theseissues early. Applying proper SLAs to your tests allows you to monitor for errors and latency during the test.
Consider the following SLA:
If you want to fail the pipeline if either of these thresholds are exceeded over a certain percent of their times,you must:
- run the test in 'detached' mode to allow for non-blocking execution of a test
- use the fastfail command to monitor for early signals to stop the test if SLAs are violated
- finally wait for the test results
To run the test in detached mode:
Then immediately afterwards, use the fastfail command:
In the above example, '25' represents the percent of times where the SLA was violated, such as 'on a particularrequest with an SLA applied, 10 out of 50 times it was executed, the SLA failed'.
Finally, because the test was executed in non-blocking mode, you should wait for the final test result.
Packaging the CLI with Build Agents
Many of the above CI examples include a step to explicitly install the NeoLoad CLI as part of thebuild steps. However, if you want the CLI baked into some build agent directly so that itis ready for use during a job, here's a Docker example:
For Docker builds See the test harness Alpine-based Dockerfile
IDE Integrations
Since most of what we do in an IDE is create/edit code, we're mostly interested in how to:
- make it easy to write API tests in YAML (automatic syntax validation)
- validate that tests do not contain unanticipated errors even at small scale
- dry-run small (smoke) load tests locally so that code check-ins will work in CI/pipeline tests
Since the latter two cases are already covered by command-line semantics, our primary focusis to accelerate test authoring by providing NeoLoad as-code DSL (Domain-specific Language) validationand in some cases editor auto-complete.
Status of IDE / editor integrations
IDE / Editor | Syntax checks | Auto-complete | Setup steps |
---|---|---|---|
Visual Studio Code | [x] | [x] | see instructions |
PyCharm | [x] | [x] | Mark 'neoload' directory as 'Sources Root' |
Contributing
Feel free to fork this repo, make changes, test locally, and create a pull request. As part of your testing, you should run the built-in test suite with the following command:
NOTE: for testing from Mac, please change the PYTHONPATH separators below to colons (:) instead of semicolons (;).
Additionally, any contributions to the DSL validation functionality, such as on the JSON schema or the validate command, should execute the following tests locally before pushing to this repo:
This command executes a number of NEGATIVE tests to prove that changes to the JSON schema or validation process produce failures when their input is malformed in very specific ways (common mistakes).
Release historyRelease notifications | RSS feed
1.1.4
1.1.4rc8 pre-release
1.1.4rc7 pre-release
1.1.4rc6 pre-release
1.1.4rc5 pre-release
1.1.4rc4 pre-release
1.1.4rc3 pre-release
1.1.4rc2 pre-release
1.1.4rc1 pre-release
1.1.3
1.1.2
1.1.1rc1 pre-release
1.1.0
1.1.0rc2 pre-release
1.1.0rc1 pre-release
1.0.1
1.0.0
1.0.0rc3 pre-release
1.0.0rc2 pre-release
0.4.6
0.4.5
0.4.4
0.4.3
0.4.2
0.4.1
Neoload 7 0 2 0
0.4
0.3.10
0.3.9
0.3.8
0.3.7
0.3.6
0.3.5
0.3.4
0.3.3
0.3.2
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Filename, size | File type | Python version | Upload date | Hashes |
---|---|---|---|---|
Filename, size neoload-1.1.4.tar.gz (454.5 kB) | File type Source | Python version None | Upload date | Hashes |
Hashes for neoload-1.1.4.tar.gz
Algorithm | Hash digest |
---|---|
SHA256 | c85f7021f15bdf0bd11af983bb704cca4d67d3a28ff66d0fdb16d1740cd2818b |
MD5 | 079b131fbdc84b063cef3c51baed772d |
BLAKE2-256 | fc7ec3683bb55fd27c39255ad7f7cf3412b316d2dfb1e764c06463ef153468bf |
This plugin allows you to monitor load tests performed by NeoLoad from Jenkins. NeoLoad test results are combined with the other integration jobs in the Jenkins dashboard. Performance regression issues are raised quickly to make Jenkins integration projects more relevant. For regular jobs or pipeline jobs.
Older versions of this plugin may not be safe to use. Please review the following warnings before using an older version:
After every build of a job configured with the NeoLoad plugin, Jenkins can display:- the HTML summary of the NeoLoad test result per build available with the Performance Result command
- the JUnit details of every passed or failed test available with the Test Result command---also used in the test result trend graph
See https://www.neotys.com/documents/doc/neoload/latest/en/html/#5769.htm for detailed documentation.
Prerequisites
The NeoLoad plugin, from version 2.1.0 requires:
- Jenkins 1.609.1 or later,
- Java 7 or later.
Collaboration Configuration
The plugin allows to configure a collaboration server like Neotys Team Server (NTS) with the Manage Jenkins > Configure System command.
Build Step
The plugin adds a new build step: Execute a NeoLoad Scenario.
The following steps are required:
- Type the path of the NeoLoad executable.
- Select Local Project or Shared Project and configure it.
- Type the name of the NeoLoad scenario in the Scenario Name field.
- Select Default Report File Names or Custom Report File Names and configure it.
- Select Existing License or Shared License and configure it.
- Select Display Trend Graph: Average Response Time (all pages) to include the Avg. Resp. Time trend graph in Jenkins.
- Select Display Trend Graph: Error Rate % to include the Error Rate trend graph in Jenkins.
- (Recommended) Add as many user-defined graphs as wanted with several curves on each graph.
Post-Build Action
The plugin needs the Archive the artifacts post-build action. The regeneration of trends could be triggered. Please archive the artifact before Refresh trends.
Example Trend Graphs
Pipelines Steps
The 'neoloadRun' step in the Jenkins Snippet Generator makes it possible to run a NeoLoad scenario from Jenkins. It also archives the reports and refreshes the graphs.
- Warning: To use the Snippet Generator, the Jenkins project including the job to configure must be compliant with Pipeline as code. For more information, see Pipeline as code.
Once the Jenkins project is selected, the Snippet Generator is accessible with a click on the Pipeline Syntax link.
This plugin provides two steps:
neoloadRun: to run NeoLoad scenario, archive report and refresh the trends.
- neoloadRefreshTrends: to refresh or change the trends only.
Refresh graph
FAQ
Why don't I see any trend graphs?
In order to see trend graphs, please verify:
- The Archive the artifacts post-build action has been added.
- Either Default Report File Names or Custom Report File Names is selected and an xml report is defined.
- At least two executions were run.
- Date and time is synchronized between the Jenkins machine and the build machine.
Known Issues
1. The NeoLoad report file (via artifacts and the Performance Result link) displays a blank page. This affects versions released before NeoLoad 5.2.
Workaround
Use the Jenkins Script Console to disable the sandboxing security by executing the following script. The Script Console is under Jenkins -> Manage Jenkins -> Script Console.
Clear the cache afterwards (hold shift and reload the page).
See https://wiki.jenkins-ci.org/display/JENKINS/Configuring+Content+Security+Policy for more information.
2. The NeoLoad Graphs aren't displayed in the main page of my job.
Workaround
Make sure you used a 'Freestyle project' for your job. If you use (for example) the Maven Plugin for your job, create a 'Freestyle project' then add Maven configuration build step.
Changelog
Version 2.2.6 (released October 03, 2019)
FIXED: Passwords storage has been changed for security reasons.
Version 2.2.5 (released May 20, 2019)
IMPROVED: Allow usage of YAML or JSON local project.
Version 2.2.4 (released July 26, 2018)
FIXED: Graph with ' character in name was not available.
Version 2.2.3 (released July 25, 2018)
IMPROVED: The log on NeoLoad error. Jackpot knights casino.
Version 2.2.2 (released July 12, 2018)
- FIXED: Confusion of launching method when Master and Slave are running in different operating system.
Version 2.2.1 (released July 10, 2018)
FIXED: Process is interrupted on Windows Slave.
Version 2.2.0 (released June 26, 2018)
ADD: Pipeline as code support Pastebot 2 1.
- ADD: SAP license capability
Version 2.1.1 (released May 23, 2018)
- FIXED: Runtime Exception 'Issue executing password scrambler' was thrown when NeoLoad path executable could not be found. (#11841)
Version 2.1.0 (released February 1, 2018)
- IMPROVED: The trends are now generated at end of Neoload Job with new post-build Action
Version 2.0.2 (released March 17, 2017)
- FIXED: Randomly, the password scrambler could not be executed, so usage of shared project of shared license could fail. (#10743)
- FIXED: Avoid scanning result files if project does not have NeoLoad configuration.
Version 2.0.1 (released September 1, 2016)
Neoload 7 0 20
- FIXED: Job could not be executed if Jenkins master and slave were on different OS. (#10588)
- FIXED: Job configuration warned about missing executable and nlp file on master even though it could be executed on slave. (#10635)
- FIXED: Graphs trend did not work when job was executed on Windows. (#10629)
- IMPROVED: To install the plugin, Java version 8 was required. Now Java version 7 and higher are supported. (#10610)
Version 2.0.0 (released June 20, 2016)
- FIXED: When creating a new Neoload job there is no choice selected by default for Report File Details. (#10272)
- FIXED: PDF report option is not kept in Project details after a save or apply. (#10253)
- FIXED: Exception: Fail to convert sharedProject. (#10278)
- FIXED: When using NTS server, job failed due to NTS password 'PASSWORD' . Input length must be multiple of 16 when decrypting. (#10248)
- FIXED: Label have to be reviewed for Neoload servers. (#10246)
- FIXED: Project details field ordering need to be review for better understanding. (#10244)
- FIXED: Help mention VU ( User path) when field label mention User Path (VU). (#10243)
- FIXED: Even Customize Report . settings are not set they are filled in executed command. (#10236)
- FIXED: Jenkins 2.0: Deleting a server from the main config page makes the job fail. (#10260)
- FIXED: Job can't be run if there is no Test description: Invalid argument ', Expected value of type 'STRING'. (#10235)
- FIXED: Use new NL logo for link to Performance result. (#10249)
Version 1.0.2 (released February 9, 2016)
- FIXED: HTML report files sometimes didn't display. (#9257)
Version 1.0.1 (released February 6, 2014)
- FIXED: Performance results links were lost when a job was renamed. (#5308)
- FIXED: Performance results linked to a report artifact that was not related to the current build after renaming a job. (#5309)
- FIXED: Displayed graph values did not match displayed html report values. (#5813)
Version 1.0 (released November 5, 2013)
- Initial release.