path: root/doc/TESTS

Contents
========

1. Running the tests
  1.1 Standard test procedure
  1.2 Summary report
  1.3 Prerequisites
  1.4 Repeatability
  1.5 Execution time
  1.6 Testing package in designated directory
  1.7 Post-install check
  1.8 Skipping test items
  1.9 Valgrind

2. Troubleshooting
  2.1 Missing programs
  2.2 Broken programs
  2.3 Color name file
  2.4 Multiple versions
  2.5 System dependent issues

3. Reporting test failures

------------------------------------------------------------------------------

1. Running the tests
====================

1.1 Standard test procedure
===========================

The recommended method of running the tests is after packaging:

   make
   make package
   make check

To capture the output do:

   make check 2>&1 | less

Or:

   make check > check.log  2>&1



1.2 Summary report
==================

Like most other test routines, these produce much output.  A summary will
appear at the end:

  Test summary:
  ==================
  SUCCESS 83
  FAILURE 1
  TOTAL TESTABLE 84
  ==================
  All tests done.
  Sat, 08 Jun 2013 09:30:33 +0000
  make: *** [check] Error 1



1.3 Prerequisites
=================

The tests require the Bash command shell.  The script Execute-Tests has some
expressions unique to bash.  Quite old versions work, at least back to bash
v. 2.05b.

The tests also use the following utilities:

  - awk 
  - cat
  - cmp
  - egrep
  - head
  - mktemp
  - sed
  - seq
  - tee



1.4 Repeatability
=================

The current version of the test package produces identical results if you run
"make check" repeatedly.  The tests contain no random elements; some
Netpbm programs use randomness internally, but the tests seed their random
number generators with fixed values so that they too have repeatable results.



1.5 Execution time
==================

Currently "make check" takes no more time to execute than "make package",
and much less than "make".



1.6 Testing package in designated directory
===========================================

If you specify the package directory when you do "make package",
you must do the same with "make check":

   make
   make package pkgdir=/tmp/package
   make check pkgdir=/tmp/package



1.7 Post-install check
======================

You can run the tests after installation as well as from the package
directory.  Just set $PBM_TESTPREFIX to the bin directory.  For example, if
the Netpbm executables are in /usr/local/bin/ do

   PBM_TESTPREFIX=/usr/local/bin make check

If you want to test the executables in the default search path do

   PBM_TESTPREFIX="" make check

If the color dictionary file rgb.txt is in a non-standard location, you must
set RGBDEF.  If you don't, the tests will report several false positives.

   PBM_TESTPREFIX=/usr/local/bin RGBDEF=/etc/colors/rgb.txt make check



1.8 Skipping test items
=======================

The file test/Test-Order is a list of tests which are run.  If you want to
skip any test, remove the line or comment it out with a "#".



1.9 Valgrind
============

You can run the whole test under Valgrind.  This is an advanced feature
intended for programmers who work on Netpbm code.

To turn on Valgrind, uncomment this line in test/Execute:

# export PBM_TESTPREFIX="valgrind --log-fd=4 "${PBM_TESTPREFIX}

   make check 4> valgrind.log

Note that Valgrind output is directed to file descriptor 4.

Valgrind version 3.5.0 or newer is recommended.  Older versions do
not report the command that is being executed.

You can add any options for Valgrind should by editing the above mentioned
line.  To run "valgrind --track-origins=yes", you must make two changes in
config.mk:

  - Add -g to CFLAGS.
  - Turn stripping off.

Valgrind significantly increases execution time.  You should consider paring
down the items in Test-Order.  You probably don't need to run
"all-in-place.test", which has restrictions in the way errors are reported
when PBM_TESTPREFIX is not a simple directory path.  Another test
"pamslice-roundtrip.test" is time-consuming, you may want to skip this one
too.



2. Troubleshooting
==================

2.1 Missing programs
====================

The first test run, "all-in-place.test" detects missing programs.

If you work around a build glitch with "make --keep-going" you
will get a few errors here.

A wholesale failure with "all-in-place.test" indicates a systemic
problem, such as a misconfigured dynamic library or a directory-wide
permission issue.  This kind of failure is known to happen when N is
set too high with "make -jN" (overambitious parallel make.) 

The current test routines assume a typical build configuration - they are not
aware of the actual configuration you chose.  If a choice you make during
configure causes "make" to skip compilation of certain programs, the test
routines won't know and will report failures.

For details read 'Netpbm Library Prerequisites':
http://netpbm.sourceforge.net/prereq.html .



2.2 Broken programs
===================

Broken programs will invariably lead to failures.  Certain programs
(for instance, image generators 'pbmmake' and 'pgmmake') are used in numerous
test scripts.  Problems in them will lead to multiple failures.

To aid you in this situation each test routine lists the necessary programs
near the top.

Each test routine comes in two parts, a ".test" file which is a
shell script and a ".ok" file which denotes its proper output.
When a test fails, a ".out" file will be produced in the
/tmp/netpbm-test/ directory.  By comparing ".ok" and ".out" you
can tell exactly what went wrong.  Often one does not need to
go this far; the error messages tell enough.



2.3 Color dictionary file
=========================

If you get the following error message, it indicates a problem with
the color dictionary file rgb.txt.

  ppmmake: can't open color names dictionary file from the path '/usr/share/
  netpbm/rgb.txt:/usr/lib/X11/rgb.txt:/usr/share/X11/rgb.txt:/usr/X11R6/lib/
  X11/rgb.txt' and Environment variable RGBDEF not set.  Set RGBDEF to the
  pathname of your rgb.txt file or don't use color names.

This is highly unlikely to occur with "make check" right after packaging,
but may appear after installation.

To check manually after installation, execute the following and see
whether the proper output or the error message appears:

   ppmmake red 1 1 -plain

Proper output:

   P3
   1 1
   255
   255 0 0

The simple remedy is properly setting the environment value RGBDEF to
the location of rgb.txt.

If you want to hardcode the path, modify RGB_DB_PATH in pm_config.h
and run "make" again.  Note that running "configure" will cause
pm_config.h to be overwritten; changes by editing will be lost.



2.4 Multiple versions
=====================

If multiple versions of Netpbm executables are installed on your
system, you should do a post-installation check to ensure that
the newly built version is in place and in working condition.

The test routines can test binaries other than the intended
target, for example pre-compiled binaries distributed in .rpm
or .deb format.  If PBM_TESTPREFIX is explicitly set to a directory
that contains programs from such a source, you should expect multiple
failures due to missing programs, missing features, etc.  If
PBM_TEXTPREFIX is set to "" (null), the outcome will depend heavily
on what version has precedence in PATH.

Netpbm distributed with Debian or Ubuntu is called "Netpbm-Free" and
is based on a fork which broke off in 2002.  There are many differences.
Many tests will fail.  However, the test framework itself is valid for
these distributions.



2.5 System dependent issues
===========================

The tests have worked on x86 and x86_64 GNU/Linux systems, with several
versions of GCC and Clang.  Reports from users of other systems including Mac
OS, Sun SPARC and BSD and compilers other than GCC are highly welcome.

Floating point math precision seems to be an issue.  Some discrepancies
have been observed between x86 32 bit and 64 bit; the tests are written to
work around them as much as possible.  The use of the "--fast-math"
flag by default may also be a factor.

The current test framework checks whether the random number generator
is the one from glibc and skips certain tests if a different one is
detected.


3. Reporting test failures
==========================

When reporting problems with the tests, please give both
the output of "make check" and the contents of the "netpbm-test"
directory.