One of the most important features of openQA is that it can be used to test several combinations of actions and configurations. For every one of those combinations, the system creates a virtual machine, performs certain steps and returns an overall result. Every one of those executions is called a 'job'. Every job is labeled with a numeric identifier and has several associated 'settings' that will drive its behavior.

A job goes through several states. Here is (an incomplete list) of these states:

Jobs in the final states 'cancelled' and 'done' have typically gone through a whole sequence of steps (called 'testmodules') each one with its own result. But in addition to those partial results, a finished job also provides an overall result from the following list.

Sometimes, the reason of a failure is not an error in the tested operating system itself, but an outdated test or a problem in the execution of the job for some external reason. In those situations, it makes sense to re-run a given job from the beginning once the problem is fixed or the tests have been updated. This is done by means of 'cloning'. Every job can be superseded by a clone which is scheduled to run with exactly the same settings as the original job. If the original job is still not in 'done' state, it’s cancelled immediately. From that point in time, the clone becomes the current version and the original job is considered outdated (and can be filtered in the listing) but its information and results (if any) are kept for future reference.

One of the main mechanisms for openQA to know the state of the virtual machine is checking the presence of some elements in the machine’s 'screen'. This is performed using fuzzy image matching between the screen and the so called 'needles'. A needle specifies both the elements to search for and a list of tags used to decide which needles should be used at any moment.

A needle consists of a full screenshot in PNG format and a json file with the same name (e.g. foo.png and foo.json) containing the associated data, like which areas inside the full screenshot are relevant or the mentioned list of tags.

Some actions in openQA require special privileges. openQA provides authentication through openID. By default, openQA is configured to use the openSUSE openID provider, but it can very easily be configured to use any other valid provider. Every time a new user logs into an instance, a new user profile is created. That profile only contains the openID identity and two flags used for access control:

Many of the operations in an openQA instance are not performed through the web interface but using the REST-like API. The most obvious examples are the workers and the scripts that fetch new versions of the operating system and schedule the corresponding tests. Those clients must be authorized by an operator using an API key with an associated shared secret.

For that purpose, users with the operator flag have access in the web interface to a page that allows them to manage as many API keys as they may need. For every key, a secret is automatically generated. The user can then configure the workers or any other client application to use whatever pair of API key and secret owned by him. Any client to the REST-like API using one of those API keys will be considered to be acting on behalf of the associated user. So the API key not only has to be correct and valid (not expired), it also has to belong to a user with operator rights.

For more insights about authentication, authorization and the technical details of the openQA security model, refer to the detailed blog post about the subject by the openQA development team.

A job can belong to a job group. Those job groups are displayed on the index page when there are recent test results in these job groups and in the Job Groups menu on the navigation bar. From there the job group overview pages can be accessed. Besides the test results the job group overview pages provide a description about the job group and allow commenting.

Job groups have properties. These properties are mostly cleanup related. The configuration can be done in the operators menu for job groups.

It is also possible to put job groups into categories. The nested groups will then inherit properties from the category. The categories are meant to combine job groups with common builds so test results for the same build can be shown together on the index page.

The following cleanup settings can be done on job-group-level:

NOTE Deletion of job results includes deletion of logs and will cause the job to be completely removed from the database.

NOTE Checkout the Cleanup section for more details and the Build tagging section for how to mark a job as important.

NOTE New groups use the limits configured in the [default_group_limits] section of /etc/openqa/openqa.ini. Jobs outside of any group use the limits configured in the [no_group_limits] section of /etc/openqa/openqa.ini.

NOTE Archiving of important jobs can be enabled. Checkout the related settings within the [archiving] section of the config file for details.

You can point CASEDIR and NEEDLES_DIR to Git repositories. openQA will checkout those repositories automatically and no manual setup is needed.

Otherwise you will need to clone tests and needles manually. For this, clone a subdirectory under /var/lib/openqa/tests for each test distribution you need, e.g. /var/lib/openqa/tests/opensuse for openSUSE tests.

The repositories will be kept up-to-date if git_auto_update is enabled in openqa.ini. The updating is triggered when new tests are scheduled. For a periodic update (to avoid getting too far behind) you can enable the systemd unit openqa-enqueue-git-auto-update.timer.

You can get openSUSE tests and needles from GitHub. To make it easier, you can just run /usr/share/openqa/script/fetchneedles. It will download tests and needles to the correct location with the correct permissions.

Fedora’s tests are also in git. To use them, you may do:

To get everything configured to actually run the tests, there are plenty of options to set in the admin interface. If you plan to test openSUSE Factory, using tests mentioned in the previous section, the easiest way to get started is the following command:

This will load some default settings that were used at some point of time in openSUSE production openQA. Therefore those should work reasonably well with openSUSE tests and needles. This script uses /usr/share/openqa/script/openqa-load-templates, consider reading its help page (--help) for documentation on possible extra arguments.

For Fedora, similarly, you can call:

Some Fedora tests require special hard disk images to be present in /var/lib/openqa/share/factory/hdd/fixed. The createhdds.py script in the createhdds repository can be used to create these. See the documentation in that repo for more information.

To start testing a new ISO put it in /var/lib/openqa/share/factory/iso and call the following commands:

If your openQA is not running on port 80 on 'localhost', you can add option --host=http://otherhost:9526 to specify a different port or host.

For Fedora, a sample run might be:

More details on triggering tests can also be found in the Users Guide.

The easiest and quickest way to spawn a single instance of openQA with a single command line using the podman container engine is the following:

Once the startup has finished, the web UI is accessible via http://localhost:1080 or https://localhost:1443.

As an alternative also separate containers are provided for both the web UI and worker.

For example the web UI container can be pulled and started using the podman container engine:

The worker container can be pulled and started with:

To supply a custom openQA config file, use the -v parameter. This also works for the database config file. Note that if a custom database config file is specified, no database is launched within the container. By default, the web UI container uses the self-signed certificate that comes with Mojolicious. To supply a different certificate, use the -v parameter. Example for running openQA with a custom config and certificate:

The same works for the workers container where you most likely want to to supply workers.ini and client.conf:

This examples assume the working directory is an openQA checkout. To avoid doing a checkout, you can also grab the config files from the webui/conf and worker/conf directory listings on GitHub.

To learn more about how to run workers checkout the Run openQA workers section.

For creating a first test job, checkout the Triggering tests section. Note that the commands mentioned there can also be invoked within a container, e.g.:

Checkout the containerized setup section for more details.

Take a look at openSUSE’s registry for all available container images.

On openSUSE Leap and openSUSE Tumbleweed to setup openQA on your machine simply download and execute the openqa-bootstrap script as root - it will do the rest for you:

The script is also available from an openSUSE package to install from:

openQA-bootstrap supports to immediately clone an existing job simply by supplying openqa-clone-job parameters directly for a quickstart:

The above command will bootstrap an openQA installation and immediately afterwards start a local test job clone from a test job from a remote instance with optional, overridden parameters. More information about openqa-clone-job can be found in Cloning existing jobs - openqa-clone-job.

You can also run openqa-bootstrap repeatedly. For example when you stop a container and the openQA daemons and database are stopped, calling openqa-bootstrap start will start necessary daemons again.

You can try out openqa-bootstrap in a container environment like GitHub Codespaces.

On GitHub openQA, click on the "Code" button and select "Codespaces". Just click on the plus sign to create a new Codespace. Or use this link as a quickstart to resume existing instances or create new ones.

It will run openqa-bootstrap in the background. If the codespace environment is ready, open a new VSCode terminal and type

The Web UI instance can be opened as soon as you get a popup that there is a webserver available on port 80.

You can now use openqa-clone-job to run jobs in this instance.

After stopping and resuming a codespace instance, run

to start the openQA daemons again.

Be sure to delete codespace instances if you don’t use them anymore, as even stopped instances will consume storage of your monthly limit.

You can also setup a systemd-nspawn container with openQA with the following commands. and you need to have no application listening on port 80 yet because the container will share the host system’s network stack.

The easiest way to install openQA is from distribution packages.

You can find the development version of openQA in OBS in the openQA:devel repository.

To add the development repository to your system, you can use these commands.

To make everything work correctly on openSUSE when using Apache, you need to enable the 'headers', 'proxy', 'proxy_http', 'proxy_wstunnel' and 'rewrite' modules using the command 'a2enmod'. This is not necessary on Fedora.

For a basic setup, you can copy openqa.conf.template to openqa.conf and modify the ServerName setting if required. This will direct all HTTP traffic to openQA.

For a basic setup, you can copy openqa.conf.template to openqa.conf and modify the server_name setting if required. This will direct all HTTP traffic to openQA.

Note that the default config in openqa.conf.template is using the keyword default_server in the listen statement. This will only change the behaviour when accessing the server via its IP address. This means that the default vhost for localhost in nginx.conf will take precedence when accessing the server via localhost. You might want to disable it.

If you use the openqa-upstreams.inc which is included with the upstream sources and openQA packages, you may want to customize the size of the shared memory segment according to the formula: page_size * 8

For openQA you need to set httpsonly = 0 as described in the TLS/SSL section below, if you do not setup NGINX for SSL.

By default openQA expects to be run with HTTPS. The openqa-ssl.conf.template Apache config file is available as a base for creating the Apache config; you can copy it to openqa-ssl.conf and uncomment any lines you like, then ensure a key and certificate are installed to the appropriate location (depending on distribution and whether you uncommented the lines for key and cert location in the config file). On openSUSE, you should also add SSL to the APACHE_SERVER_FLAGS so it looks like this in /etc/sysconfig/apache2:

If you don’t have a TLS/SSL certificate for your host you must turn HTTPS off. You can do that in /etc/openqa/openqa.ini:

openQA uses PostgreSQL as database. By default, a database with name openqa and geekotest user as owner is used. An automatic setup of a freshly installed PostgreSQL instance can be done using this script. The database connection can be configured in /etc/openqa/database.ini (normally the [production] section is relevant). More info about the dsn value format can be found in the DBD::Pg documentation.

openQA supports three different authentication methods: OpenID (default), OAuth2 and Fake (for development).

Use the auth section in /etc/openqa/openqa.ini to configure the method:

Independently of method used, the first user that logs in (if there is no admin yet) will automatically get administrator rights!

Note that only one authentication method and only one OpenID/OAuth2 provider can be configured at a time. When changing the method/provider no users/permissions are lost. However, a new and distinct user (with default permissions) will be created when logging in via a different method/provider because there is no automatic mapping of identities across different methods/providers.

For authentication to work correctly the clocks on workers and the web UI need to be in sync. The best way to achieve that is to install a service that implements the time-sync target. Otherwise a "timestamp mismatch" may be reported when clocks are too far apart.

The automated cleanup is enabled and configured by default. Cleanup tasks are scheduled via systemd timer units and run via openqa-gru.service. The configuration is done in /etc/openqa/openqa.ini and various places within the web UI. If you want to tweak the cleanup to your needs, have a look at the Cleanup of assets, results and other data section.

Editing needles from web can optionally commit new or changed needles automatically to git. To do so, you need to enable git support by setting

in /etc/openqa/openqa.ini. Once you do so and restart the web interface, openQA will automatically commit new needles to the git repository.

You may want to add some description to automatic commits coming from the web UI. You can do so by setting your configuration in the repository (/var/lib/os-autoinst/needles/.git/config) to some reasonable defaults such as:

To enable automatic pushing of the repo as well, you need to add the following to your openqa.ini:

Depending on your setup, you might need to generate and propagate ssh keys for user 'geekotest' to be able to push.

It might also be useful to rebase first. To enable that, add the remote to get the latest updates from and the branch to rebase against to your openqa.ini:

If rebasing, it may be useful to perform a hard reset of the local repository to ensure that the rebase will not fail. To enable that, add the following to your openqa.ini (along with the previous snippet):

Automatic cleanup of old results (see GRU jobs) can sometimes render important tests useless. For example bug report with link to openQA job which no longer exists. Job can be manually marked as important to prevent quick cleanup or referer can be set so when job is accessed from particular web page (for example bugzilla), this job is automatically labeled as linked and treated as important.

List of recognized referrers is space separated list configured in /etc/openqa/openqa.ini:

Default behavior for all workers is to use the 'Qemu' backend and connect to 'http://localhost'. If you want to change some of those options, you can do so in /etc/openqa/workers.ini. For example to point the workers to the FQDN of your host (needed if test cases need to access files of the host) use the following setting:

Once you got workers running they should show up in the admin section of openQA in the workers section as 'idle'. When you get so far, you have your own instance of openQA up and running and all that is left is to set up some tests.

The following information is partially openSUSE specific. The openQA-worker package provides further systemd units:

There are some additional requirements to get remote worker running. First is to ensure shared storage between openQA web UI and workers. Directory /var/lib/openqa/share contains all required data and should be shared with read-write access across all nodes present in openQA cluster. This step is intentionally left on system administrator to choose proper shared storage for her specific needs.

Example of NFS configuration: NFS server is where openQA web UI is running. Content of /etc/exports

NFS clients are where openQA workers are running. Run following command:

You can configure openQA to send events (new comments, tests finished, …) to an AMQP message bus. The messages consist of a topic and a body. The body contains json encoded info about the event. See amqp_infra.md for more info about the server and the message topic format. There you will find instructions how to configure the AMQP server as well.

To let openQA send messages to an AMQP message bus, first make sure that the perl-Mojo-RabbitMQ-Client RPM is installed. Then you will need to configure amqp in /etc/openqa/openqa.ini:

For a TLS connection use amqps:// and port 5671.

When there are multiple openQA web interfaces (openQA instances) available a worker can be configured to register and accept jobs from all of them.

Requirements:

In the /etc/openqa/workers.ini enter space-separated instance hosts and optionally configure where the shared storage is mounted. Example:

Configuring SHARE_DIRECTORY is not a hard requirement. Workers will try following directories prior registering with openQA instance:

Once a worker registers to an openQA instance, scheduled jobs (of matching worker class) can be assigned to it. Dependencies between jobs will be considered for ordering the job assignment. It is possible to mix local openQA instance with remote instances or use only remote instances.

If your network is slow or you experience long time to load needles you might want to consider enabling caching on your remote workers. To enable caching, CACHEDIRECTORY must be set in workers.ini. There are also further settings one can optionally configure. Example:

The specified CACHEDIRECTORY must exist and must be writable by the cache service (which usually runs as _openqa-worker user). If you install openQA through the repositories, said directory will be created for you.

The shown configuration causes workers to download the assets from the web UI and use them locally. The TESTPOOLSERVER setting causes also tests and needles to be downloaded via rsync from the specified location. You can find further examples in the comments in /etc/openqa/workers.ini.

It is suggested to have the cache and pool directories on the same filesystem to ensure assets used by tests are available as long as needed. This is achieved by using hard links, resorting to symlinks in other cases with the risk of assets being deleted from the cache before tests relying on these assets end.

The caching is provided by two additional services which need to be started on the worker host:

The rsync server daemon needs to be configured and started on the web UI host.

Example /etc/rsyncd.conf:

Specific job settings might refer to files within the test distribution. You can configure openQA to display links to these files within the job settings tab. To enable particular settings to be presented as a link within the settings tab one can setup the relevant keys in /etc/openqa/openqa.ini.

The files referenced by the configured keys should be located either under the root of CASEDIR or the data folder within CASEDIR.

If a job is done, especially if no label could be found for carry-over, often more steps are needed for the review of the test result or providing the information to either external systems or users. As there can be very custom requirements openQA offers a point for optional configuration to let the instance administrators define specific actions.

By setting custom hooks it is possible to call external scripts defined in either environment variables or config settings.

If an environment variable corresponding to the job result is found following the name pattern OPENQA_JOB_DONE_HOOK_$RESULT, any executable specified in the variable as absolute path or executable name in $PATH is called with the job ID as first and only parameter. For example for a job with result "failed", the corresponding environment variable would be OPENQA_JOB_DONE_HOOK_FAILED. As alternative to an environment variable a corresponding config variable in the section [hooks] in lower-case without the OPENQA_ prefix can be used in the format job_done_hook_$result. The corresponding environment value has precedence. The exit code of the externally called script is not evaluated and will have no effect.

It is also possible to specify one general hook script via job_done_hook and enable that one for specific results via e.g. job_done_hook_enable_failed = 1.

The job setting _TRIGGER_JOB_DONE_HOOK=0 allows to disable the hook script execution for a particular job. It is also possible to specify _TRIGGER_JOB_DONE_HOOK=1 to execute the general hook script configured via job_done_hook regardless of the result.

The execution time of the script is by default limited to five minutes. If the script does not terminate after receiving SIGTERM for 30 seconds it is terminated forcefully via SIGKILL. One can change that by setting the environment variables OPENQA_JOB_DONE_HOOK_TIMEOUT and OPENQA_JOB_DONE_HOOK_KILL_TIMEOUT to the desired timeouts. The format from the timeout command is used (see timeout --help).

For example there is already an approach called "auto-review" https://github.com/os-autoinst/scripts/#auto-review---automatically-detect-known-issues-in-openqa-jobs-label-openqa-jobs-with-ticket-references-and-optionally-retrigger which offers helpful, external scripts. Config settings for openqa.opensuse.org enabling the auto-review scripts could look like:

or for a host openqa.example.com:

The environment variable should be set in a systemd service override for the GRU service. A corresponding systemd override file /etc/systemd/system/openqa-gru.service.d/override.conf could look like this:

When using apparmor the called hook scripts must be covered by according apparmor rules, for example for the above in /etc/apparmor.d/usr.share.openqa.script.openqa:

Additions should be added to /etc/apparmor.d/local/usr.share.openqa.script.openqa after which the apparmor service needs to be restarted for changes to take effect. Note that in case of symlinks the target must be specified, and the link itself is irrelevant. So for example Can’t exec "/bin/sh" can occur if /bin/sh is a link to a path that’s not allowed.

Apparmor denials and stderr output of the hook scripts are visible in the system logs of the openQA GRU service, except for messages in "complain" mode which end up in audit.log. General status and stdout output is visible in the GRU minion job dashboard on the route /minion/jobs?offset=0&task=finalize_job_results of the openQA instance.

By default, when a worker reports an incomplete job due to a cache service related problem, the job is automatically cloned. It is possible to extend the regex to cover other types of incompletes as well by adjusting auto_clone_regex in the global section of the config file. It is also possible to assign 0 to prevent the automatic cloning.

Note that jobs marked as incomplete by the stale job detection are not affected by this configuration and cloned in any case.

Some of these events are very common and may clutter audit database. For this reason job_grab and job_done events are on the blocklist by default.

These statistics help to identify the most time-consuming queries.

For WireGuard using wg-quick is recommended.

To generate a private (first line) and a public (second line) key for each peer use this command:

Create a config in /etc/wireguard/openqa.conf on the webui host:

Create a config in /etc/wireguard/openqa.conf on the worker1 host (and analog on other worker hosts):

On all peers run now:

Then update /etc/openqa/workers.ini on the workers like this:

Same for /etc/openqa/client.conf

Check the log files in /var/lib/openqa/testresults

www.opensuse.org’s OpenID provider may have trouble with IPv6. openQA shows a message like this:

To avoid that switch off IPv6 or add a special route that prevents the system from trying to use IPv6 with www.opensuse.org:

If openQA is very slow and e.g. the test setup times out because the asset caching downloads take too long it makes sense to cross-check the networking performance. This can be done via iperf3.

Launch the server via iperf3 -s on one host (e.g. the openQA web UI host). Then run a test on another host (e.g. an openQA worker host) like this:

Use -4/-6 to check IPv4 vs. IPv6 performance. Use -R to check in the other direction. Both can make a huge difference.

More examples: https://fasterdata.es.net/performance-testing/network-troubleshooting-tools/iperf

When testing an operating system, especially when doing continuous testing, there is always a certain combination of jobs, each one with its own settings, that needs to be run for every revision. Those combinations can be different for different 'flavors' of the same revision, like running a different set of jobs for each architecture or for the Full and the Lite versions. This combinational problem can go one step further if openQA is being used for different kinds of tests, like running some simple pre-integration tests for some snapshots combined with more comprehensive post-integration tests for release candidates.

This section describes how an instance of openQA can be configured using the options in the admin area to automatically create all the required jobs for each revision of your operating system that needs to be tested. If you are starting from scratch, you should probably go through the following order:

Machines, mediums, test suites and job templates can all set various configuration variables. The so called job templates within the job groups define how the test suites, mediums and machines should be combined in various ways to produce individual 'jobs'. All the variables from the test suite, medium, machine and job template are combined and made available to the actual test code run by the 'job', along with variables specified as part of the job creation request. Certain variables also influence openQA’s and/or os-autoinst’s own behavior in terms of how it configures the environment for the job. Variables that influence os-autoinst’s behavior are documented in the file doc/backend_vars.asciidoc in the os-autoinst repository.

In openQA we can parameterize a test to describe for what product it will run and for what kind of machines it will be executed. For example, a test suite kde can be run for any product that has the KDE software stack installed, like openSUSE-DVD-x86_64 and openSUSE-NET-i586, and can be tested in different x86-64 and i586 machines like 64bit, 64bit_USBBoot, 32bit. In this example we could have the following test scenarios considering that the “x86_64” flavor is not compatible with the 32bit machine:

For every test scenario we need to configure a different instance of the test backend, for example os-autoinst, with a different set of parameters.

You need to have at least one machine set up to be able to run any tests. Those machines represent virtual machine types that you want to test. To make tests actually happen, you have to have an 'openQA worker' connected that can fulfill those specifications.

A medium type (product) in openQA is a simple description without any concrete meaning. It basically consists of a name and a set of variables that define or characterize this product in os-autoinst.

Some example variables used by openSUSE are:

A test suite consists of a name and a set of test variables that are used inside this particular test together with an optional description. The test variables can be used to parameterize the actual test code and influence the behaviour according to the settings.

Some sample variables used by openSUSE are:

Some of the variables usually set in test suites that influence openQA and/or os-autoinst’s own behavior are:

The job groups are the place where the actual test scenarios are defined by the selection of the medium type, the test suite and machine together with a priority value.

The priority value is used in the scheduler to choose the next job. If multiple jobs are scheduled and their requirements for running them are fulfilled the ones with a lower priority value are triggered. The id is the second sorting key: Of two jobs with equal requirements and same priority value the one with lower id is triggered first.

Job groups themselves can be created over the web UI as well as the REST API. Job groups can optionally be nested into categories. The display order of job groups and categories can be configured by drag-and-drop in the web UI.

The scenario definitions within the job groups can be created and configured by different means:

Any job setting can refer to another variable using this syntax: %NAME%. When the test job is created, the string will be substituted with the value of the specified variable at that time.

The variable expansion applies to job settings defined in test suites, machines, products and job templates. It also applies to job settings specified when creating a single set of jobs and to variables specified in the worker config.

Consider this example where a variable is defined within a test suite:

It may expanded to this job setting:

It is possible to define the same variable in multiple places that would all be used for a single job - for instance, you may have a variable defined in both a test suite and a product that appear in the same job template. The precedence order for variables is as follows (from lowest to highest):

That is, variable values set as part of the API request that triggers the jobs will 'win' over values set at any of the other locations. In the special case of the BACKEND variable, if there is a MACHINE specified, the BACKEND value for this machine defined in openQA has highest precedence.

If you need to override this precedence - for example, you want the value set in one particular test suite to take precedence over a setting of the same value from the API request - you can add a leading + to the variable name. For instance, if you set +VARIABLE = foo in a test suite, and passed VARIABLE=bar in the API request, the test suite setting would 'win' and the value would be foo.

If the same variable is set with a + prefix in multiple places, the same precedence order described above will apply to those settings.

Note that the WORKER_CLASS variable is not overridden in the way described above. Instead multiple occurrences are combined.

Test suites can be described using API commands or the admin table for any operator using the web UI.

If a description is defined, the name of the test suite on the tests overview page shows up as a link. Clicking the link will show the description in a popup. The same syntax as for comments can be used, that is Markdown with custom extensions such as shortened links to ticket systems.

The overview page is configurable by the filter box. Also, some additional query parameters can be provided which can be considered advanced or experimental. For example specifying no build will resolve the latest build which matches the other parameters specified. Specifying no group will show all jobs from all matching job groups. Also specifying multiple groups works, see the following example.

Specifying multiple groups with no build will yield the result for the latest build of each group. This can be useful to have a static URL for bookmarking.

Based on comments in the individual job results for each build a certificate icon is shown on the group overview page as well as the index page to indicate that every failure has been reviewed, e.g. a bug reference or a test issue reason is stated:

“progress.opensuse.org” is used to track test issues, bugzilla for product issues, at least for SUSE/openSUSE. openQA bugrefs distinguish this and show corresponding icons

At the top of the test results overview page is a form which allows filtering tests by result, architecture and TODO-status. "TODO" means that tests still require review.

There is also a similar form at the bottom of the index page which allows filtering builds by group and customizing the limits. Also the 'All tests' table allows filtering by the TODO-status.

When hovering over the branch icon after the test name children of the job will be highlighted blue and parents red. So far this only works for jobs displayed on the same page of the table.

On a tests result page there is a tab for “Next & previous results” showing the result of test runs in the same scenario. This shows next and previous builds as well as test runs in the same build. This way you can easily check and compare results from before including any comments, labels, bug references (see next section). This helps to answer questions like “Is this a new issue”, “Is it reproducible”, “has it been seen in before”, “how does the history look like”.

Querying the database for former test runs of the same scenario is a rather costly operation which we do not want to do for multiple test results at once but only for each individual test result (1:1 relation). This is why this is done in each individual test result and not for a complete build.

Related issue: #10212

Screenshot of the feature:

Find the always latest job in a scenario with the link after the scenario name in the tab “Next & previous results” Screenshot:

Should always refer to most recent job for the specified scenario.

Examples:

This allows e.g. to show only failed builds. Could be included like in http://lists.opensuse.org/opensuse-factory/2016-02/msg00018.html for “known defects”.

Example: Add query parameters like …&result=failed&arch=x86_64 to show only failed for the single architecture selected.

The query parameter `limit_builds' allows to show more than the default 10 builds on demand. Just like we have for configuring previous results, the current commit adds web UI selections to reload the same page with higher number of builds on demand. For this, the limit of days is increased to show more builds but still limited by the selected number.

Example screenshot:

By using advanced query parameters in the URLs you can configure the search for builds. Higher numbers would yield more complex database queries but can be selected for special investigation use cases with the advanced query parameters, e.g. if one wants to get an overview of a longer history. This applies to both the index dashboard and group overview page.

Example to show up to three week old builds instead of the default two weeks with up to 20 builds instead of up to 10 being the default for the group overview page:

Using a new query parameter `only_tagged=[0|1]' the list can be filtered, e.g. show only tagged (important) builds.

Example screenshot:

Related issue: #11052

For each job result including the latest job result page, there is a corresponding route to get an SVG status badge that can eg. be used to build a status dashboard or for showing the status within a GitHub comment.

There is an optional parameter 'show_build=1' that will prefix the status with the build number.

Many test failures within the same scenario might be due to the same reason. To avoid human reviewers having to add the same bug references again and again, bug references are carried over from previous failures in the same scenario if a job fails. The same behaviour can be achieved by adding flag:carryover to a comment. This idea is inspired by the Claim plugin for Jenkins.

This is possible by adding the keyword pinned-description anywhere in a comment on the group overview page. Then the comment will be shown at the top of the group overview page. However, it only works as operator or admin.

A dark mode theme can be enabled via "Appearance" settings for all logged in users. It can either be forced with the "dark mode" setting, or left to browser detection. Switching automatically between light and dark mode is natively supported by most modern browsers and can also be controlled manually via flags:

For more information, see developer.mozilla.org/CSS/@media/prefers-color-scheme

The developer mode allows to:

It can be accessed via the "Live View" tab of a running test. Only registered users can take control over tests. Basic instructions and buttons providing further information about the different options are already contained on the web page itself. So I am not repeating that information here and rather explain the overall workflow.

In case the developer mode in not working on your instance, try to follow the steps for debugging the developer mode under 'Pitfalls'.

Scenarios are defined as part of a job group. The Edit job group button exposes the editor.

Now there are two scenarios for x86_64, one by giving just the name of the test suite and another which has a machine, priority and settings. Both are allowed. However since at least one scenario relies on defaults those need to be specified once in their own section:

The defaults section is only required whenever a scenario is not completely defined in-place. When it is used, the available parameters are identical to those for a single scenario. For instance the example could be amended to use settings and run every test suite for that architecture on several machines by default.

Defaults are always overwritten by explicit parameters on scenarios. Further more, all settings can be specified in YAML. Using this together with custom job template names, variants of a scenario can even be specified when they would normally be considered duplicated:

Even more flexibility can be achieved by using aliases in YAML, or in other words re-using a scenario by reference, such as to run the same scenarios in two different mediums. & is used to define an anchor, while * is the alias referencing the anchor:

Also YAML Merge Keys are supported. This way you can reuse previously defined anchors and add other values to it. Values in the merged alias will be overridden.

You can even merge more than one alias.

The job templates are written in YAML 1.2. In YAML, strings usually do not have to be quoted, except if it is a special value that would be loaded as a Boolean, NULL or Number. The following table shows all special values (See the documentation for the default YAML 1.2 Core Schema for more information).

Because we are using the Merge Keys feature, also the unquoted string << is special. If you need the literal string << (for example as a value in the job settings), you have to quote it.

The following example lists all jobs within the job group with the ID 1 and the setting BUILD=20210707 on openqa.opensuse.org:

The tool jq is used for pretty-printing in this example but it is also useful for additional filtering (see js’s tutorial).

However, openQA’s API provides many more filters on its own. These can be used by adding additional query parameters, e.g.:

Tests can be triggered over multiple ways, using openqa-clone-job, jobs post, isos post as well as retriggering existing jobs or whole media over the web UI.

Job groups can be queried via the experimental REST API:

The GET request will get the YAML for one or multiple groups while a POST request conversely updates the YAML for a particular group.

Two scripts using these routes can be used to import and export YAML templates:

The following job settings are specifying that an asset is required by a job:

Where you see e.g. ISO_n, that means ISO_1, ISO_2 etc. will all be treated as assets.

The values of the above parameters are expected to be the name of a file - or, in the case of REPO_n, a directory - that exists under the path /var/lib/openqa/share/factory on the openQA web-UI. That path has subdirectories for each of the asset types, and the file or directory must be in the correct subdirectory, so e.g. the file for an asset HDD_1 must be under /var/lib/openqa/share/factory/hdd. You may create a subdirectory called fixed for any asset type and place assets there (e.g. in /var/lib/openqa/share/factory/hdd/fixed for hdd-type assets): this exempts them from the automatic cleanup described in the section about asset cleanup. Non-fixed assets are always subject to the cleanup.

UEFI_PFLASH_VARS is a special case: whether it is treated as an asset depends on the value. If the value looks like an absolute path (starts with /), it will not be treated as an asset (and so the value should be an absolute path for a file which exists on the relevant worker system(s)). Otherwise, it is treated as an hdd-type asset. This allows tests to use a stock base image (like the ones provided by edk2) for a simple case, but also allows a job to upload its image on completion - including any changes made to the UEFI variables during the execution of the job - for use by a child job which needs to inherit those changes.

You can also use special suffixes to the basic parameter forms to access some special handling for assets.

Jobs can upload assets to the web-UI so other jobs can used them as HDD_n and UEFI_PFLASH_VARS assets as described in the previous section.

To declare an asset to be uploaded, you can use the job settings PUBLISH_HDD_n and PUBLISH_PFLASH_VARS. For instance, if you specify PUBLISH_HDD_1=updated.qcow2, the HDD_1 disk image as it exists at the end of the test will be uploaded back to the web-UI and stored under the name updated.qcow2. Any other job can then specify HDD_1=updated.qcow2 to use this published image as its HDD_1.

To find out whether an asset should be removed, openQA determines by which groups the asset is used. If at least one job within a certain job group is using an asset, the asset is considered to be used by that job group. If that job group is within a parent job group, the asset is considered part of that parent job group.

So an asset can belong to multiple job groups or parent job groups. The assets table which is accessible via the admin menu shows these groups for each asset and also the latest job.

While an asset might belong to multiple groups it is only accounted to the group with the highest asset limit which has still enough room to hold that asset. That basically mean that an asset is never counted twice.

If the size limit for assets of a group is exceeded, openQA will remove assets which belong to that group:

Assets which do not belong to any group are removed after a configurable duration unless the files are still being updated. Keep in mind that this behavior is also enabled on local instances and affects all cloned jobs (unless cloned into a job group).

If an asset is just a symlink then only the symlink is cleaned up (but not the file or directory it points to).

'Fixed' assets - those placed in the fixed subdirectory of the relevant asset directory - are counted against the group size limit, but are never cleaned up. This is intended for things like base disk images which must always be available for a test to work. Note that relative symlinks in the regular assets directory that point into the fixed subdirectory are also preserved.

To configure the maximum size for the assets of a group, open 'Job groups' in the operators menu and select a group. The size limit for assets can be configured under 'Edit job group properties'. It also shows the size of assets which belong to that group and not to any other group.

The default size limit for job groups can be adjusted in the default_group_limits section of the openQA config file.

Assets not belonging to jobs within a group are deleted automatically after a certain number of days. That duration can be adjusted by setting untracked_assets_storage_duration in the misc_limits section of the openQA config to the desired number of days.

In less trivial cases where a common limit is not enough or certain assets need more fine-grained control, patterns based on the filename can be used. The patterns are interpreted as Perl regular expressions and if a pattern matches the basename of an asset the specified duration in days will be used. In simple cases the pattern is just a match on a word.

Consider the following examples to specify custom limits that would match assets with the names testrepo-latest and openSUSE-12.3-x86_64.iso.

Note that modifications to the file will count against the limit, so if an asset was updated within the timespan it will not be removed.

Cleanup can be triggered in different ways. One option is to use minion_task_triggers and specify tasks via on_job_done. Another way to do that is to use the systemd timers openqa-enqueue-*-cleanup to periodically run tasks. Both can be used separately or in combination.

The relevant Minion tasks are:

These are no-ops if a task is already running so they can safely be enqueued repeatedly. Note that the tasks can still take considerable time computing what to delete, from seconds to minutes. The tasks can be enabled in the corresponding config file section.

By default the cleanup is enabled with systemd timers if available. To completely disable cleanup make sure that no minion cleanup tasks are enabled over the config file and prevent individual or all cleanup systemd timers, for example for the asset cleanup:

An openQA test needs to implement at least the run subroutine containing the actual test code and the test needs to be loaded in the distribution’s main.pm.

Here is an example in Perl:

And here is an example in Python:

There are more optional subroutines that can be defined to extend the behavior of a test. A test must comply with the interface defined below. Please note that the subroutine marked with *1 are optional.

The Python integration that OpenQA offers through Inline::Python also allows the test modules to import other Perl modules with the usage of the perl virtual package provided by Inline::Python.

Because of the way Inline::Python binds Perl functions to Python it is not possible to use keywords arguments from Python to Perl functions. They must be passed as positional arguments, for example "key", "value".

See the following snippet of Perl code

versus the equivalent python code:

Additionally, Python tests do not support run_args. An error will be present when a Python test detects the presence of run_args on schedule.

This is because of the way Inline::Python handles argument passing between Perl <→ Python, references to complex Perl objects do not reach Python properly and they can’t be used.

The following examples are short complete test modules written in Perl implementing the interface described above.

The following examples are short complete test modules written in Python implementing the interface described above.

By default, tests are aborted after two hours by the worker. To change this limit, set the test variable MAX_JOB_TIME to the desired number of seconds.

The download of assets, synchronization of tests and other setup tasks do not count into MAX_JOB_TIME. However, the setup time is limited by default to one hour. This can be changed by setting MAX_SETUP_TIME.

To save disk space, increasing MAX_JOB_TIME beyond the default will automatically disable the video by adding NOVIDEO=1 to the test settings. This can be prevented by adding NOVIDEO=0 explicitly.

The variable TIMEOUT_SCALE allows to scale MAX_JOB_TIME and timeouts within the backend, for example the test API. This is supposed to be set within the worker settings on slow worker hosts. It has no influence on the video setting.

Soft and hard failures can be triggered on demand by regular expressions when they match the serial output which is done after the test is executed. In case it does not make sense to continue the test run even if the current test module does not have the fatal flag, use fatal as serial failure type, so all subsequent test modules will not be executed if such failure was detected.

To use this functionality the test developer needs to define the patterns to look for in the serial output either in the main.pm or in the test itself. Any pattern change done in a test it will be reflected in the next tests.

The patterns defined in main.pm will be valid for all the tests.

To simplify tests results review, if job fails with the same message, which is defined for the pattern, as previous job, automatic comment carryover will work even if test suites have failed due to different test modules.

There are two sets of packages that can be included in test logs: 1. Packages installed on the worker itself - stored as worker_packages.txt. 2. Packages installed on SUT - stored as sut_packages.txt.

For both sets, if present, openQA will include the difference to the last good job in the "Investigation" tab of a failed job.

To enable logging of worker package versions, set PACKAGES_CMD in workers.ini. The command should print installed packages with their version to stdout. For RPM-based systems it can be for example rpm -qa.

To enable logging of SUT package versions, make the test create the file sut_packages.txt in the current worker directory. If upload_logs() is used, the resulting file needs to be copied/moved.

By default, any worker can get any job with the matching architecture.

This behavior can be changed by setting the job variable WORKER_CLASS taking a comma-separated list of worker class values. The values are combined from multiple places where defined. Typically machines and test suite configurations set the worker class. Jobs with this variable set are assigned only to workers, which have all corresponding worker class values in their configuration (and-combination).

For example, the following configuration ensures, that jobs with WORKER_CLASS=desktop can be assigned only to worker instances 1 and 2. Jobs with WORKER_CLASS=desktop,foo can only be assigned to worker instance 2 which has both the values desktop and foo:

Worker class values can also be set to additionally qualify workers or worker instances for informational purposes, for example region and location tags based on company conventions:

By default the openQA workers run the "isotovideo" application from PATH on the worker host, that is in most cases isotovideo. A custom worker engine command can be set with the test variable ISOTOVIDEO. For example to run isotovideo from a custom container image one could use the test variable setting ISOTOVIDEO=podman run --pull=always --rm -it registry.example.org/my/container/isotovideo /usr/bin/isotovideo -d

You might encounter flaky openQA tests that fail sporadically. The best way to address flaky test code is of course to fix the test code itself. For example, if certain steps rely on external components over network, retries within the test modules should be applied.

However, there can still be cases where you might want openQA to automatically retrigger jobs. This can be achieved by setting the test variable RETRY in the format <retries>[:<description>] to an integer value with the maximum number of retries and an optional description string separated by a colon. For example triggering an openQA job with the variable RETRY=2:bug#42 will retrigger an openQA test on failure up to 2 totalling to up to 3 jobs. Note that the retry jobs are scheduled immediately and will be executed as soon as possible depending on available worker slots. Many factors can change in retries impacting the reproducibility, e.g. the used worker host and instance, any network related content, etc. By default openQA tests do not retry. The optional, additional description string is used only for reference and has no functional impact.

See Automatic cloning of incomplete jobs for an additional solution intended for administrators handling known issues causing incomplete jobs.

Custom hook scripts on "job done" based on result can be used to apply more elaborate issue detection and retriggering of tests.

There are different dependency types, most importantly chained and parallel dependencies.

A dependency is always between two jobs where one of the jobs is the parent and one the child. The concept of parent and child jobs is orthogonal to the concept of types.

A job can have multiple dependencies. So in conclusion, a job can have multiple children and multiple parents at the same time and each child/parent-relation can be of an arbitrary type.

Additionally, dependencies can be machine-specific (see Inter-machine dependencies section).

Scenarios requiring more than one system under test (SUT), like High Availability testing, are covered as multi-machine tests (MM tests) in this section.

openQA approaches multi-machine testing by assigning parallel dependencies between individual jobs (which are explained in the previous section). For MM tests specifically, also take note of the following remarks:

In short, writing multi-machine tests adds a few more layers of complexity:

openQA provides a locking API. To use it in your test files import the lockapi package (use lockapi;). It provides the following functions: mutex_create, mutex_lock, mutex_unlock, mutex_wait

Each of these functions takes the name of the mutex lock as first parameter. The name must not contain the "-" character. Mutex locks are associated with the caller’s job.

mutex_lock tries to lock the mutex for the caller’s job. The mutex_lock call blocks if the mutex does not exist or has been locked by a different job.

mutex_unlock tries to unlock the mutex. If the mutex is locked by a different job, mutex_unlock call blocks until the lock becomes available. If the mutex does not exist the call returns immediately without doing anything.

mutex_wait is a combination of mutex_lock and mutex_unlock. It displays more information about mutex state (time spent waiting, location of the lock). Use it if you need to wait for a specific action from single place (e.g. that Apache is running on the master node).

mutex_create creates a new mutex which is initially unlocked. If the mutex already exists the call returns immediately without doing anything.

Mutexes are addressed by their name. Each cluster of parallel jobs (defined via PARALLEL_WITH dependencies) has its own namespace. That means concurrently running jobs in different parallel job clusters use distinct mutexes (even if the same names are used).

The mmapi package provides wait_for_children which the parent can use to wait for the children to complete.