chain

NAME

chain - run a chain of jobs

SYNOPSIS

chain [ DIR ] -ctsrl [-n NN] [-i NN] [-m] [-f] [-v]

chain [ DIR ] -SIGNAL

chain -help

perldoc chain

DESCRIPTION

PURPOSE

CHAIN is a tool to make many runs of a program with different sets of parameters. CHAIN provides a simple way to

o
specify the different parameter combinations

o
automatically create the input files for each run

o
name output files uniquely

.

CHECKLIST

If you want to use CHAIN with a particular code, here is what you have to do:

  1. Make your code read its parameters from files. The input files may have any format you want.

  2. Create format files for the input files which you want to control with CHAIN. To do so, copy these files to unique name (e.g. simply add a ``.fmt'' to the input file name). Edit these format files. Replace all parameters that you want controlled by CHAIN with named diamonds (<NAME1>, <NAME2>, <NAME3>, ...). See below in the documentation for an example.

  3. Use chain -c to create a template of the data file chain.dat. Edit chain.dat. Fill in file names, options, commands, storage names and parameter combinations. If you are not sure how to do this, see below in the documentation.

  4. Use chain -t to check if CHAIN expands your list of parameter combinations and storage names correctly. Also, check the sample input files.

  5. Use chain -s -n 2 to run the first two jobs of the chain. Usually, problems (if any) show up during the first two runs.

  6. Use chain -s & to start the chain.

  7. Check the progress of the chain with chain -l. If you need to kill it, use chain -term or, if necessary, chain -kill. This send the corresponding signal to the whole group of processes related to CHAIN, including the current run of your code.

  8. If someone manages to kill your chain or to crash your computer, chain -r will restart the chain.

.

Format Files

CHAIN uses templates of the jobs input files to create new input files for each individual run. To create these templates, make a copy of any input files. The suggested name for these templates is the input file name with .fmt appended (FORMAT file). In the format files, replace all parameters that should be controlled by CHAIN with named diamonds (e.g <RHO>, ...). CHAIN will produce different input files by replacing these diamonds with current values before each run. If the values need to be especially formatted, the name in the diamond may be followed by a printf format specifier, e.g. <RHO%13.5e>.

> EXAMPLE: > A format file might look like this: > > # Parameter file for job xyz > <OPAC> # this is the file name of some table > density <RHO%10.2f> > particle numbers <NP1> <NP2> > start time <TIME0%10.2e>

Logbook File chain.log

Each run is logged in a LOGBOOK FILE chain.log. This file may be checked at any time to watch progress. It is also used for restarts (chain -r). chain -l produces a pretty report based on this file.

Data File chain.dat

CHAIN uses a DATA FILE chain.dat to determine

  1. the names of input and format files and of parameters

  2. the commands to execute for every run

  3. the combinations of parameters to explore and

  4. how to name the output uniquely.

A template data file chain.dat can be created with chain -c..This file has to be edited:

  1. INPUT FILE NAMES: Enter the names of the input files that your code uses.

    FORMAT FILE NAMES: Specify the names of the corresponding format files, one for each input file. The default can be written as {}.fmt (input file names with .fmt appended).

    OUTPUT FILE NAMES: Enter the names of all output files that have to be saved. If the output has been collected in a directory, the name of the directory can be given here. CHAIN will rename all the files and directories mentioned here to unique names.

    PARAMETER NAMES: Enter the names of parameters as you use them in the format files. The parameters can be referenced under these names in format files, command lines, the TODO list and in Perl code.

    > EXAMPLE: > This example defines two input and three output file names > > INPUT FILE NAMES: in.dat in1.dat > FORMAT FILE NAMES: in.fmt in1.fmt > OUTPUT FILE NAMES: out1.dat out2.dat err.dat > PARAMETER NAMES: RHO NP1 NP2 TIME0 OPAC

  2. DEFAULT MINOR OPTIONS: You may specify a list of minor options that you want to be always in effect. This may be useful if a special chain always needs some switches (in particular -i and -e ). Note, that no major option will be accepted here.

    MAIL ADDRESS: CHAIN will inform you via Email when the chain is completed. You may change the default mailing address here. See also -m option.

    STORAGE NAME PREFIX: A prefix for all storage names - usually only used for specification of a directory.

    STORAGE NAME POSTFIX: A postfix for all storage names. Usually empty, but handy if you decide to compress (.z .Z .gz) or tar (.tar) the output.

    ERROR CASE POSTFIX: A special postfix that will be added to the storage names only if your code or any of the system commands in the script returned a non-zero exit status.

  3. Enter here the commands that run your code. These lines are a shell script which will be executed by /bin/sh for each individual run. If you have specified the -e option, the parameter for the run will be available to the script not only in the input files, but also as shell variables. Thus things like $RHO and ${NP1} will be interpreted correctly (and you should not use parameter names that conflict with important shell variables). Prior to execution, named diamonds with (optional) print formats (e.g. <RHO%13.5e>) will also be interpolated into the script.

    Apart from running your program, the script may do other things, e.g. compressing the output, collecting several files into a tar-file or a directory, making a quick plot and sending it to the printer, etc.

    > EXAMPLE: > Here is a script that collects output into a directory "resdir" > > prg $TIME0 <in.dat >out.dat > rm -rf resdir > mkdir resdir > mv out.dat out1.dat out2.dat resdir

    CHAIN will notice (and say so in the logbook file) when errors occur during the execution of the script.

    Instead of using a shell script, you may actually provide a perl program here. See under ``ADVANCED FEATURES'' futher down in this man page.

  4. Each line in the TODO list specifies the parameters for one or several runs of your code.

    The first fields of each line are the storage names that will be used to save the output files. CHAIN will rename the files mentioned under OUTPUT FILE NAMES to these names. The number of storage names must be consistent with the number of output file names.

    > EXAMPLE: > A TODO list entry appropriate for the output file names given > in the example above (out1.dat out2.dat err.dat) might be: > > run1_out1 run1_out2 run1_err ...

    The actual values of parameters can be inserted into storage names. Any occurrences of named diamonds (e.g. <NAME>) will be replaced by the parameter with that name. As with the diamonds in the format file, you may include format specifiers. In order to avoid white space in file names, CHAIN will discard any whitespace characters introduced by the formatting process. To get fixed-length fields, use zero-padded formats.

    > EXAMPLE: > storage name template: xx_r<RHO>_n<NP1%04d>.dat > parameters of the run: 3 42 > storage name used: xx_r3_n0042.dat

    The rest of each line specifies the parameters to be used in the run in the same sequence as given in the PARAMETER NAMES statement. CHAIN uses these parameters and the format files to put together the input files for a run of your code. Several lines may be used for parameters. When a line ends with a backslash, the next line is treated as continuation line.

    > EXAMPLE of a TODO list: > fn_<NAME1>_<NAME2> par1 par2 par3 par4 ... > fn1_<NAME1>_<NAME2> par1 par2 \ > par3 par4 ... > fn2_<NAME1>_<NAME2> par1 par2 par3 par4 ...

    It is not necessary to create an extra line for each combination of parameters. Instead, several values for any parameter may be specified and CHAIN will go through all possible combinations. Parameters can be specified in three ways:

    a) Plain value or string: e.g. "42" or "test.dat" b) List of values: (val1,val2,val3,val4,...) c) Sequence of values: [start:limit:step]

    Plain values will create only one run. Lists will create a run for each element of the list. Sequences will create a run for each value from start to end in steps of step. step consists of an operator (+-*/) and a number. The default for step is +1. List and sequence specifiers must not contain spaces. When a list or a sequence specifier is used, the storage name MUST have a diamond for this particular parameter (or the storage name will not be unique). Expansion takes also place when the list/sequence is only a part of the parameter.

    > EXAMPLE: > The following TODO list: > > simple a b c d > > test1_<N1>_<N2>_<N3>_<N4> a (b,c) 5 [2:4:*2] > > test a b 1 file(2,3).dat > > will create the following storage names and parameter combinations: > > simple a b c d > > test1_a_b_1_2 a b 1 2 > test1_a_b_1_4 a b 1 4 > test1_a_c_5_2 a c 5 2 > test1_a_c_5_4 a c 5 4 > > test a b 1 file2.dat > test a b 1 file3.dat

    OPTIONS

    MAJOR OPTIONS

    CHAIN will do nothing without a command line option. Exactly one of the following options must be present.

    -c
    Create a template version of the data file ``chain.dat''.

    -t
    Test. CHAIN will check out the specification in ``chain.dat'', expand sequences and lists, report errors.

    -s
    Start the chain. An existing logbook file ``chain.log'' is removed.

    -r
    Restart a chain that has been aborted before completion. With this option, CHAIN will examine the file ``chain.log'' and skip as many parameter combinations as denoted ``finished'' in this file.

    -l
    Create a report based on the logbook file.

    -ll
    Show logbook entries for current run.

    -help
    Print a help message to STDOUT.

    -SIGNAL
    Send a signal to the chain and its children. SIGNAL can be one of (term,kill,stop,cont).

    .

    MINOR OPTIONS

    These options modify the way CHAIN works. Several may be specified. Some minor options may be defined default in the DEFAULT MINOR OPTIONS entry in the parameter file chain.dat)

    DIR
    Run CHAIN in directory DIR instead of the current directory. (e.g. chain ~/work/doit -s).

    -n NN
    Restrict chain to the next NN runs. For test purposes.

    -e
    Export the parameters as scalar variables into the shell script (or perl program). When this is active, a parameter RHO will be available as $RHO etc. Watch out for conflicts between parameter names and environment variables.

    -d
    Debug. For a shell script, this will run the shell with -vx in order to give you better understanding of what is going on. Currently, this option has no effect on perl code supplied instead of the shell script.

    -O N
    Divert the output from the runs of your code. By default, all the output will just go to STDOUT (normally your screen). -O 0 will discard all output. -O 1 will save the output of the current run into the file chain.out, but overwrite it for every new run. -O 2 will put all output from all runs into chain.out. Note that this operation will only divert the output from your script and code, not the output from CHAIN itself.

    -E N
    Like -O , but for STDERR and the file chain.err.

    -m
    Send mail as notification after *each* run. The default is to mail only after the final run.

    -f
    Force: ignore errors in chain.dat.

    -v
    Verbose: copy all logbook entries to STDOUT.

    -i NN
    Make NN extra steps of macro interpolation and list/sequence expansion. By default, one macro interpolation followed by one expansion step is done. -i 1 provides for an additional macro interpolation round, -i 2 adds another expansion step etc. Don't use this unless you know what you are doing.

    .

    ADVANCED FEATURES

    This section covers some extra features for advanced use of CHAIN. They are not required for basic uses. For some of them, knowledge of the Perl language is required.

    Special Characters

    In the interpretation of TODO lists, the following characters have special meanings: {}[](),:\= and SPACE and TAB. If any of these characters has to be part of a storage name or parameter, it can be escaped with a backslash.

    Merged Sequences

    The start specifier in sequences may be a list of values. This can be used to merge several sequences into each other:

    > The sequence [10,20,50:20000:*10] > > is equivalent to the following list: > > (10,20,50,100,200,500,1000,2000,5000,10000,20000)

    Parallel Sequences/Lists

    By default, CHAIN consideres all sequence and list specifiers to be independent. Thus, CHAIN explores all possible combinations when expanding several lists/sequences in the same line (othogonal expansion). It is also possible to expand two or more lists parallel. In this case, CHAIN does not explore all combinations, but puts together only all first elements, then all second elements etc. In order use this feature, the lists to be expanded parallel must be identified and have the same length. You do this by inserting a numbered hash mark (e.g. #2) as the first element after the opening parenthesis. Sequences/lists carrying the same number will be expanded parallel. By chosing several numbers, you define several groups for parallel expansion. Here is an example.

    > EXAMPLES: The following three examples show the difference between > the two methods. The sequences/lists in the first line > are expanded to what follows below. > > (a,b) [1:2:+1] | (#1,a,b) [#1,1:2:+1] | (#1,a,b) (#2,1,2) (#1,X,Y) > ---------------|----------------------|--------------------------- > a 1 | a c | a 1 X > a 2 | b d | a 2 X > b 1 | | b 1 Y > b 2 | | b 2 Y

    File Name Expansion

    Whenever a list of file names is given in the parameter file (i.e. input files, format files, output files and storage names), CHAIN looks for the C-Shell {..,..,..} construct and expands it.

    > The storage name run_<N1>_{out1,out2,err}.dat > > is an abbreviation for the following names: > > run_<N1>_out1.dat run_<N1>_out2.dat run_<N1>_err.dat

    An empty pair of braces ``{}'' has a special meaning. When used in storage names in the TODO list, ``{}'' expands to the list of output file names. Thus

    > OUTPUT FILE NAMES: out1.dat out2.dat err.dat > ... > #--STORAGE-NAME----------INPUT-PARAMETERS----------- > run_<N1>_{} produces the same list of storage names as above.

    Similarily, in the FORMAT FILE NAMES list, ``{}'' expands to the list of input files. Thus

    > INPUT FILE NAMES: in1.dat in2.dat > FORMAT FILE NAMES: {}.fmt > > is equivalent to > > FORMAT FILE NAMES: in1.dat.fmt in2.dat.fmt

    Macros

    C-Preprocessor-like #define macros may be used in the TODO list. Macros are restricted to string replacement (no control structures, no arguments).

    > EXAMPLE: > #define LIST1 (10,20,30,40,50,60,70,80,90,100) > #define LIST2 (1,2,3,10,20,30,\ > 100,200,300) > > #define SEQ1 [100:1000:+100] > #define PAIR (1,2,3) (6,7,8) > > test1_<N2>_<N3> xx.dat LIST1 LIST2 > test2_<N2>_<N3> xx.dat LIST1 SEQ1 > test3_<N2>_<N3> xx.dat PAIR

    Parameter and Storage Name Tests

    The TODO list may contain embedded tests for specific parameters or storage names. This can help to avoid typing errors or to check the existence of files before starting the chain. It can also be misused to change the values themselves. The test definition lines start with ``#test NAME''. NAME is either an output file name or a parameter name, as defined in chain.dat. The rest of the line may be any Perl expression testing $_. CHAIN will assign the value of the corresponding parameter or storage name to $_ and evaluate your expression. Assigning to $_ will change the parameter. The test will be applied to all lines following its definition. To cancel a test, say ``#test NAME reset''.

    > EXAMPLE: > This example defines tests for parameters RHO and OPAC. > It checks if RHO is a positive number and if OPAC is > an existing file in the subdirectory "src/". > > #test RHO $_>0; > #test OPAC -e "src/$_";

    Perlcode instead of a Shell Script

    Instead of a shell script to run your code, you may also use a full fledged Perl program. CHAIN will automatically detect which is being used. The Perl program must define a subroutine RUN, which will be called for each set of parameters. The Perl program will automatically be defined in a special package called user::. The following variables will always contain current values:

    @files the storage names for the current run. Note that the first filename is $files[0] @pars the parameters for the current run. Note that the first parameter is $pars[0]. $irun index of the TODO list entry (0,1,...) Attention: This starts with 0 only when "chain -s" is used. "chain -r" may start with higher values. $NAME1... When running CHAIN with the `-e' option (export), the values of all parameters are directly available as scalar variables of the names given in the PARAMETER NAMES statement. > EXAMPLE: > # Initialization, will be done only once > $my_count = 0; > > # This sub MUST always be defined > sub RUN { > &print_parameters; > } > sub print_parameters { # subroutine definition > $my_count++; > print $my_count,$irun,@files,@pars; > }

    AUTHOR

    (c) 1996 Carsten Dominik dominik@strw.leidenuniv.nl

    CHAIN is free software. It may be distributed under the same terms as Perl.

    FILES

    ./chain.dat ./chain.log ./chain.pid ./chain.out ./chain.err

    BUGS

    When doing several macro interpolation steps (see option -i ), only the first remembers the original order of macro definitions and other lines. Thus, for the second iteration, all macros will be globally active, not just below their definition line. In particular, macros that are defined several times will have the value of their last definition.

    Debug is not implemented for Perl scripts.