What is Tested by OLVER?

Fig. 1. The structure of a Linux system.

From Applications To Hardware

Every computing system is built to run one or more applications. However, the code of applications alone is not enough to get necessary services from hardware. Most applications use library calls to communicate with the operating system and get common utility services. LSB Core 3.1 standard defines the following base and utility libraries: libc, libcrypt, libdl, libm, libpthread, librt, libutil, libpam, libz, libncurses. In the modern Linux systems, the interfaces of these libraries are implemented by glibc, Linux-PAM, zlib and ncurses upstream components, which actually provide even more interfaces than the 1532 functions required by LSB Core.

The functions of the system libraries can be classified as follows depending on the kernel usage degree.

  • The function is implemented entirely in the library. The Linux kernel is not used (e.g. strcpy, tsearch).
  • The library implements just a trivial wrapper for using corresponding kernel call (e.g. read, write).
  • The function implementation contains both kernel calls (possibly many different) and essential part in the library itself (e.g. pthread_create, pthread_cancel).

The Linux kernel contains a lot of exported entry points, but most of them are for internal use by subsystems and modules of the kernel itself. The external kernel interface contains about 250 functions (version 2.6). GLIBC 2.3.5 uses 137 of them in its implementation.

Variety of Linux Configurations

It may seem that testing of the kernel and the libraries by its original developers is enough to ensure high quality of Linux systems. But it is not true. Even at the library API level there are many dimensions that make almost every particular Linux system unique from the QA (quality assurance) point of view.

As shown in the figure 1, the behavior of library API is defined by the library itself, by the Linux kernel and by hardware. In its turn, the libraries and the kernel are defined by version (including specific patches) and by build configuration. The thing is that even the same library or kernel version can behave differently depending on the parameters that were used during building from sources. Hardware architecture can potentially affect behavior semantics as well. This means that every specific Linux configuration needs separate testing to ensure that everything works well together.

By Linux configuration we [Linux Verification Center] mean particular combination of the Linux kernel version, versions of the system libraries, parameters used for building the kernel and the libraries, and finally specific hardware architecture that runs all these staff.

System Under Test of the OLVER Test Suite

OLVER project of the Linux Verification Center aims at developing a test suite for checking behavior of the system libraries defined in LSB Core 3.1 standard (total 1532 functions). This is a "black box" testing based on requirements of the LSB Core and related standards such as POSIX.

The main testing focus is on the base and utility libraries itself. Meanwhile, the whole Linux configuration is tested actually, which means checking the joint work of the libraries, the kernel and hardware. The tests actually simulate an application that calls library APIs in different combinations and with various parameters. But the tests differ from applications in that the tests "know" how the interfaces should work (which is defined by specifications) and thus they can check the correctness of return values, resulting system states, reaction sequences, etc.

Meanwhile, the testing technology, tools and framework used in the OLVER project allow advanced testing of arbitrary programming interfaces (including asynchronous interactions and protocols). This allows using our results and technologies as a good basis for testing Linux interfaces beyond or different from LSB standard, for example for testing custom Linux systems with possibly pared-down, modified or extended kernel and/or libraries, as well as systems on top of special hardware. The Center provides services in this area.