[patch] Characterization scripts

[Jules Bergmann]

This patch adds a characterization script (char.pl) and parameter data 
(char.db).  It automates the running of multiple benchmarks with various 
configurations.

The basic flow for char.pl is:

First, you create a parameter entry for the benchmark in the char.db 
file (or in another .db file if you're doing something special).  For 
example, if you want to characterize vector multiply (the 'vmul' 
benchmark) and are interested in single precision scalar (case 1) and 
complex (case 2) performance, you would create an entry like so:

	# comment ... vmul entry
	set: vmul
	  pgm:   vmul
	  cases: 1 2
	
The .db file format is a bit of a hack.  In particular, indentation 
matters.  'set:' must be at the beginning of a line, the parameters 
('pgm:', 'cases:', etc) must be indented.
	

Then, you use the 'char.pl' script to run the benchmarks.  It assumes 
you are in the build directory (i.e. where you ran configure).  To run 
the vmul benchmark cases, you would type:

	> $srcdir/scripts/char.pl			\
		-db $srcdir/scripts/char.db		\
		vmul

This will
  - Build the benchmark executable ('benchmarks/vmul' in this case) if it
    doesn't already exist (hacker beware: it doesn't check dependencies.
    If the executable is there, but hopelessly out of data ... oops!)
  - Run the benchmark for each of the cases, putting the output in the
    file:

	$pgm-$case-$np.dat

    For the vmul, this will create 'vmul-1-1.dat' and 'vmul-2-1'.dat.

    If the .dat file is already there, the case will be skipped.
    (Again, dependencies are not checked).

The benchmark is run in the '-data' mode, which collects enough info to 
reconstruct the ops/s, iob/s, pts/s, etc metrics.  This data isn't easy 
to plot directly, but that's another script/patch.

Other bits:
  - You can run benchmarks in parallel:

    First, in the char.db file, specify the number of processors
    that may be used with the case.  For example, if vmul can only
    be run with 1, 2, and 4 processors (but not 3, for some reason),
    you would say:

	set: vmul
	  pgm: vmul
	  cases: 1 2
	  nps: 1 2 4

    The default for nps is '1'.  If the benchmark works for any number
    of processors, you can set nps to 'all'.

    Second, run char.pl with a number of processors in mpi mode:

	> $srcdir/scripts/char.pl -db ... -mode mpi -np 1,2,3,4 vmul

    This will run each benchmark case for each numbeer of processors in
    the intersection of the 'nps' db entry and the '-np' command line
    option.

    For vmul, this would result in vmul-1-1.dat, vmul-1-2.dat,
    vmul-1-4.dat, vmul-2-1.dat, vmul-2-2.dat, vmul-2-4.dat.

  - You can control the numbers of SPEs used:

    First, similar to 'nps', in the char.db file specify the number of
    SPEs that may be used with the case:

	set: vmul
	  pgm: vmul
	  cases: 1 2
	  spes: 0 8 16

    The default value for 'spes' is "0 1 8 16".

    Second, run char.pl with the number of SPEs in 'cell' mode:

	> $srcdir/scripts/char.pl -db ... -mode cell -spes 1,8,16

    In 'cell' mode, the format of the data file names changes to

	$pgm-$case-$np-$spe.dat

  - You can create macros in the db file to run a group of sets
    under a single set name.

    For example, if vmul had once benchmark case that did not work
    in parallel, you might say:

	set: vmul-ser
	  pgm: vmul
	  cases: 1
	  nps: 1 # only works with 1 processor

	set: vmul-par
	  pgm: vmul
	  cases: 2
	  nps: all

	macro: vmul vmul-ser vmul-par

  - You can run all benchmarks with the '-all' option to char.pl

  - Benchmarks can have "requirements".  For example, if want to add
    a db entry to run the sal/fft benchmark, but only want it to run
    when the library has SAL configured in, you would put a 'req:' entry

	set: sal-fft
	  pgm: sal/fft
	  cases: 1
	  req: sal

    char.pl will only run sal-fft benchmark cases when it is given the
    options '-have sal'.  This is useful when running the '-all' command.

Let me know if you have any questions on using this or ideas on how to 
improve it.  Next I'm going to clean up and post a patch for plotting 
the data files generated by char.pl.

				-- Jules

-- 
Jules Bergmann
CodeSourcery
jules at codesourcery.com
(650) 331-3385 x705
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: char.diff
URL: <http://sourcerytools.com/pipermail/vsipl++/attachments/20070426/e026bc9b/attachment.ksh>