README File for the OLVER Test Suite

This readme file describes how to build and execute the tests and how to view test results.

If you have just installed binary package of the test suite you may skip "Building the Test Suite" section of this readme.

If you just unpacked tarball with sources, you should run install.sh script with root priviledges in the sources directory first, and then you may build and run OLVER.

Install/Uninstall Notes

During installation OLVER creates a special user account. It creates user group 'olver' and user 'tester' with no user password. All tests should be run under this account.
During uninstall OLVER does not remove 'tester' user account. If you want to do it, do it manually.

Software and Hardware Requirements
Currently all seven archtectures IA32, IA64, x86_64, s390, s390x, PPC64, PPC32 are supported.

We've checked our test suite on the following Linux distributions and
architectures:
1. SuSE Linux Enterprise Server 10.0 on IA32, IA64, PPC32, PPC64, s390, s390x hardware architectures
2. RedHat Enterprise Linux 4 Update 2 on IA32 and x86_64 architectures
3. Debian GNU/Linux 4.0 'etch' on IA32, IA64, PPC32 architectures
4. Ubuntu 6.06 LTS on IA32 architecture
5. Mandriva Linux 2006 on IA32 architecture
6. Fedora release 8 (Werewolf) on IA32 architecture
For building and running the test suite additional software requirements should
be met listed in "Building the Test Suite" and "Running the Test Suite" sections.
Structure of the Directory Tree

Test Suite Root Directory is /opt/olver
Files
    install.sh
        Appears in the test suite directory only if you unpacked tarball
        with sources. This script should be run first and it prepares
        environment for building and running OLVER.
    build_tests.sh
        Build script.
    agent_perm.sh
        Sets root permissions for the agent.
    run_tests.sh
        Run script.
    ask_root.sh
        Performs su or sudo command to set up agent permissions during build.
    build_prerequisites.sh
        Checks system components needed to build OLVER.
    prerequisites.sh
        Checks system components needed to run OLVER.
    testplan
        List of tests to be run by run_tests.sh.
    create_report.sh
        Report generator script.
    olver_report.pl
        Generates OLVER summary report.
    tjreport
        Generates reports in TET format.
    clean_tests.sh
        Clean script.
    README
        This readme file.
    README_RUS
        Russian translation of readme file.
    LICENSE
        Apache License Version 2.0, January 2004
    times.ref
        This file contains scenario execution times from the last run.
        OLVER initially contains a starting version of this file for
        Athlon XP 2700+ 1GB RAM. times.ref is used to show estimated
        run times for scenarios during execution.
    etc/olver.conf
    etc/olver_ia64.conf
    ...
    etc/olver_s390.conf
        Contains global parameters for the test suite for the given architecture.
    etc/Makefile_agent_ppc64
    etc/Makefile_agent_s390
    etc/Makefile_agent_s390x
    etc/Makefile_model_ppc
        Architecture specific makefiles.
    etc/stochastic.conf
        Contains information about scenarios that sometimes behave unstable.
    src/agent/build_config.sh
        Builds agent even if some LSB functions are missing on the
        target machine.
Directories
    bin
        Compiled binaries are put in this directory:
        bin/olver         - test system executable
        bin/agent         - test agent executable
        bin/olverterm     - terminal emulator for ncurses testing
    src/agent
        Contains the source code for the test agent.
    src/model
        Contains the source code for the test system.
    src/olverterm
        Comtains the source code for OLVER terminal emulator.
    src/testdata
        Contains the source for test data generation.
    tools/CTesK
        Contains binaries for the test generator.
    tools/TraceTools
        Contains binaries for the test report generator.
    tools/reportgen
        Contains the report generator for coverage of requirements.
    tools/tet-report
        Contains the generator to create test execution journal in
        TET format.
    tools/BugDB
        Contains information about bugs in Linux distributions
        that were found using OLVER. It is used to provide the user
        with more detailed failure descriptions if these bug are
        dicovered on user's system.
    doc
        Contains OLVER documentation.

Test Results Root Directory is /var/opt/olver
Results of the test runs will be stored into this folder.

    /var/opt/olver/<DATE>-<TIME>
        One of the folders with test results, it is automatically
        created after the test run.
        Folder name format is "%Y-%m-%d_%H-%M-%S".
    /var/opt/olver/<DATE>-<TIME>/*.utt
        XML files with raw results of the scenario runs.
        File name format is "<scenario name>.utt".
        E.g. heap_scenario.utt
    /var/opt/olver/<DATE>-<TIME>/report
        HTML report for the test run. It contains cumulative results
        for all scenarios being executed.
    /var/opt/olver/<DATE>-<TIME>/res.xml
        Raw information about coverage of the requirements.
    /var/opt/olver/<DATE>-<TIME>/result.htm
        HTML report presenting coverage of the requirements.
        The report uses the following color schema:
            - covered requirements are displayed in green
            - those covered requirements that are simply checks of the
              internal model of the test system (the case of the closed
              state), are displayed in yellow
            - requirements to the calling application are displayed
              in grey
            - not covered requirements are displayed in black
            - failed requirements are displayed in red
    /var/opt/olver/<DATE>-<TIME>/tjreport.log
        Test execution journal in TET format.
    /var/opt/olver/<DATE>-<TIME>/nice_tet.log
        Test execution summary report in TET format.
    /var/opt/olver/<DATE>-<TIME>/run.log
        Test execution log.
    /var/opt/olver/<DATE>-<TIME>/summary.htm
        OLVER summary report.
Building the Test Suite

Prerequisites

To build OLVER one need to have the following components installed in the
system:

1. LSB 3.1 Software Development Kit (SDK). You can find it on
       http://www.freestandards.org/en/Downloads
2. GNU C/C++ Compiler 3.3.5 or later
3. Sun or IBM Java Runtime Environment version 5 or later. Note
   that 'java' command should be on your path.
4. autoconf
5. automake
6. libtool

In order to build the test suite run under user 'tester' script:

    ./build_tests.sh

It takes approximately 25 minutes to complete the build on Athlon XP 2700+.

Note: during the build you'll be prompted to enter the root password.
This is needed to set appropriate privileges for the agent in order
to test ncurses and process functionality. If you have already got root
privileges password prompt will be answered automatically.
Making Test Plan

To decide which scenarios must not be included in the test run, please
open testplan and comment out lines with scenario names.

#str_scenario (will not be executed)
integer_scenario (will be executed)
Running the Test Suite

Prerequisites

To run OLVER and obtain execution reports one need to have
the following components installed in the system:

1. LSB runtime loader: /lib/ld-lsb.so.3. If it is not found, you should
   install LSB 3.1 Software Development Kit (SDK). You can find it on
       http://www.freestandards.org/en/Downloads
2. Sun or IBM Java Runtime Environment version 5 or later and
   'java' command need to be on your path.
3. Perl language (e.g. perl-5.8.8) to generate summary report.
4. XML::Parser Perl module (e.g. perl-XML-Parser-2.34 (rpm),
                                 libxml-parser-perl-2.34 (deb)).
5. xsltproc (e.g. libxslt-1.1.15) to generate result.htm with
   covered requirements.

In order to execute the test suite run under user 'tester' script:

    ./run_tests.sh

During run of the test suite you will be informed about scenario execution
timeout. This is a maximum time single test from the test plan can be
executed. After this period of time the test is considered as halted and
it is killed. This parameter in seconds can be adjusted in run_tests.sh script
on line 17:

    TEST_TIMEOUT= ...
Viewing Test Results

Test Execution Summary Reports:
    1. In OLVER format: /var/opt/olver/<DATE>-<TIME>/summary.htm
    2. In TET format: /var/opt/olver/<DATE>-<TIME>/nice_tet.log

Test Execution Report:
    1. Go to /var/opt/olver/<DATE>-<TIME>/report/ directory.
    2. Open index.html using any web browser you like.

Requirements Coverage Report:
    1. Go to /var/opt/olver/<DATE>-<TIME>/
    2. Open result.htm using any web browser you like.

For detailed information about test reports see
http://linuxtesting.org/downloads/olver-reports.pdf
Cleaning the Build

Run script: ./clean_tests.sh
Troubleshooting

1. SuSE. Fail to build the agent. Add symbolic links:
    a) libz
        su
        cd /lib
        ln -s libz.so.1 libz.so
    b) libpam
        su
        cd /lib
        ln -s libpam.so.0 libpam.so
    c) libncurses
        su
        cd /lib
        ln -s libncurses.so.5 libncurses.so

2. SuSE. Can not build olverterm because can not find
   tic (terminfo compiler). Install ncurses-devel package.

3. Mandriva. Fail to build the agent. Add symbolic links:
    a) libpam
        su
        cd /lib
        ln -s libpam.so.0 libpam.so

4. If you experience problems with building model of the test suite
   make sure that your java environment is Sun or IBM Java Runtime Environment
    version 5 or later. Sun Java is preffered.

5. On some systems problem may occure with agent build - clog10() function is
   incorrectly defined as
        extern double complex clog10(void);
   in LSB headers on these systems. This is LSB 3.1 Software Development Kit (SDK)
   problem. Correct in /opt/lsb/include/complex.h the above definition to
        extern double complex clog10(double complex)

Main

About Us

Activities

Results

FAQ

Links

README File for the OLVER Test Suite

Contents