AZOV Tests

Overview

AZOV tests (or Shallow tests) – simple tests with the only guaranteed purpose of ensuring the interface does not crash (in many cases it is additionally checked that the interface doesn’t return an error code) being called with some particular correct parameters and in the correct environment. This is close to “existence” or “smoke” or “sanity” tests (but beware - these terms are interpreted differently by different experts). We use a new technology and tools (AZOV Framework) for automatic shallow tests generation based on some description of the interfaces, their parameters, interlinks and default values in the LSB database. The core idea here is to augment the database to contain enough information about the interfaces and their dependencies that would allow automatically building correct call chains representing typical scenarios of interface usage.

AZOV Framework

The Azov framework is designed for automated development of shallow-quality tests that check run-time presence of interfaces and absence of critical errors in simple use cases. Azov helps to quickly create numerous API tests for large systems using structured syntactic information about target interfaces from the main LSB Database augmented by additional information assigned by test developers.

The Azov test development framework consists of the following components:

1. LSB Database Extension contains 11 new tables (linked with the existing ones) to keep additional information about initializing correct parameters and execution environment for target interfaces and about constraints on return values.
2. Web UI for test developers to specify the additional information (i.e. visually work with data in the DB extension).
3. Code Generator that automatically generates tests from the extended database. It can generate tests in both plain C (for standalone debugging of failures) and T2C (for reusing good test execution and reporting facilities) formats.

The basic idea is to provide the code generator with enough information to generate correct invocation of the target interface in a typical use case and check basic constraints on the return value. The issue is that, for many interfaces, correct invocation requires preparing correct environment and correct parameter values. And this in turn may require calling other interfaces. Azov database extension keeps information about various initialization means for various environments and semantic parameter types (specialized types). Such information can be reused for multiple interfaces each time the same semantic types are used, which greatly speeds up the development. Correct values for simple types are created by generator completely automatically. When an interface has initialization means assigned for all of its parameters (and if necessary for initial environment), the generator can automatically compose a test for this interface.

Key Features:

1. Test development for numerous interfaces requires relatively small effort per test and thus is cheap enough. The efficiency is best revealed for interface sets with the number of interfaces starting from hundreds.
2. Fully automatic generation of tests based on information in the extended LSB Database.
3. Existing information about interfaces is reused from the LSB Database.
4. Additional information about interfaces is handled via handy visual user interface.
5. T2C Framework features can be reused for debugging and executing shallow tests.

Development of generator of the tests (that is a part of AZOV framework) leaded to creation a separate tool - API Sanity Autotest.

Results

Test suites for Qt 3, Qt 4 and xml 2 libraries are developed with the help of the Azov technology. They are included into the LSB 4.0 certification test suite.

Azov-Qt3:

Azov-Qt4:

Azov-xml2:

Downloads

The latest Azov-Qt3 and Azov-Qt4 release packages can be downloaded from this page.

The latest Azov-xml2 release packages can be downloaded from this page.

The tests from the suite can be executed either by Distribution Checker or manually.