To start a computational code, several things need to be done. Where is the code executable? What input data and other files does it need? Where it is started? To simplify tasks connected to these questions, there are classes defined in the pirs.core.scheduler module helping to create directories, set files necessary to start a code, to start it and to wait until it completes.
A directory containing all files necessary to start a code is described in the WorkPlace class. An instance of this class helps to define directory names where a code will be started.
from pirs.core.scheduler import WorkPlace
w = WorkPlace()
w.prefix = 'wp'
# create directory
w.prepare()
print w.report
# create another directory,
# do not interfere with previously
# created
w.prepare()
print w.report
'wp0' created by script source/examples/sched_wp1.py at 2014-08-25 06:52:18.679108
'wp1' created by script source/examples/sched_wp1.py at 2014-08-25 06:52:18.679228
When the WorkPlace.prepare() is called, the current directory is searched for already existing directories starting with string specified in the WorkPlace.prefix attribute, in the example wp. If there are no such directories, the directory wp0 is created. If there already some directories with prefix wp, an integer N is chosen so that directory wpN does not exist yet, and this directory is created. The WorkPlace.report attribute is a string containing the last created directory (see also WrokPlace.lcd) the name of Python script where the method was called, and creation time. The above example results in two newly created empty directories, wp0 and wp1.
To put files in newly created directories, the WorkPlace.files list attribute must be populated with instances of the pirs.core.scheduler.InputFile class. This class describes content and name of the file to be written or copied to the workplace directory. The content can be specified as a string or as a link to existing file.
from pirs.core.scheduler import InputFile, WorkPlace
# MCNP input file
i1 = InputFile()
i1.basename = 'inp'
i1.string = 'c input file'
# Srctp from previous run
i2 = InputFile()
i2.basename = 'srctp2'
i2.exfile = './srctp1'
# shell script to start MCNP
i3 = InputFile()
i3.basename = 'start.sh'
i3.string = '$MCNP inp=inp srctp=srctp2'
i3.executable = True
# workplace
w = WorkPlace()
w.suffix = 'wp'
w.files.append(i1)
w.files.append(i2)
w.files.append(i3)
w.prepare()
print w.report
out = w.run(sec=1)
print out
'wp2' created by script source/examples/sched_if1.py at 2014-08-25 06:52:18.787231
'inp' generated from string
'srctp2' copied from './srctp1'
'start.sh' generated from string
scheduler queued job <job cd 'wp2'; start.sh>
scheduler waits with parameters {'sec': 1}
mcnp ver=5 , ld=09282010 08/25/14 06:52:18
Thread Name & Version = MCNP5, 1.60
Copyright LANS/LANL/DOE - see output file
_
._ _ _ ._ ._ |_
| | | (_ | | |_) _)
|
bad trouble in subroutine pass1 of imcn
unexpected eof in file inp
In this example three input files are created. The first one, i1, will be written to a file in the workplace directory with the name defined by the InputFile.basename. with content defined by the string passed to the InputFile.string argument. The second input file, i2, is a link to an existing file. In this case, the file ./srctp1 will be copied from the current directory to the workplace under the new name, srctp2. The content of the third input file, i3 is defined again as a stging. Additionally, the InputFile.executable attribute is set to True meaning that this file, when written to the workplace, will be added executable mode.
As in the previous example, we create an instance of the WorkPlace class, w, but now we append the defined input files i1, i2 and i3 to the list attribute files. The prepare() method creates a new directory, an we can see in the report that this directory is now called wp2 and has three files: 'inp' and 'start.sh' are generated from string and 'srctp2' is copied from another file. The WorkPlace.run() method called at the last line searches for input files with executable mode set to True and if founds, starts it and waits until particular criteria are satisfied. In the example we specified to wait for 1 second, other criteria are possible, see description of the Scheduler.wait() method. The run() method standard output generated by the script.
TODO: to write.
Defines common directory name and a list of InputFiles to be put to the directory.
Method WorkPlace.prepare() creates folder ‘nameID’, where ‘name’ is the common directory name and ‘ID’ is an unique integer number. The name of the last created folder is saved in the WorkPlace.lcd property.
>>> wp = WorkPlace()
>>> print wp.prefix, wp.nextID, wp.lcd
wp 0 None
>>> wp.prepare()
>>> print wp.report
'wp0' created
>>> print wp.prefix, wp.nextID, wp.lcd
wp 1 wp0
One can specify files to be put to the work place using the WorkPlace.files list:
>>> f1 = InputFile()
>>> f1.basename = 'f1'
>>> f1.string = 'f1 content\n'
>>> f2 = InputFile()
>>> f2.basename = 'f2'
>>> f3 = InputFile()
>>> f3.basename = 'f3'
>>> f3.string = 'f3 content'
>>> wp.files.append(f1)
>>> wp.files.append(f2)
>>> wp.files.append(f3)
>>> wp.prepare()
List of files to be written to the workplace directory.
Elements must be instances of the InputFile class.
The name of the last created directory.
Integer part of the name for the next directory
Directory name prefix
Creates directory with the next available directory name. If necessary, changes the nextID attribute.
Puts input files.
String describing the last created folder: its name and files in it.
Find executable input file in files and starts it using scheduler module.
Assumes that directory containing all input files allready created with the prepare() method.
Keyword arguments are all passed to the scheduler.Scheduler.run() method, see description there.
Default scheduler, used to run executable input file.
File to be written to a workplace.
Content of the file is specified as a string or as a link to existing file.
Keyword arguments can specify all properties of the instance.
If arguments ‘exfile’, ‘string’ and ‘spf’ are specified in kwargs, the order they are set is exactly as specified here.
>>> f = InputFile(basename='if.init', string='string', exfile='_inp')
>>> f.write()
>>> print open(f.basename).read()
>>> f = InputFile(basename='if.init', exfile='_inp', psf=True)
>>> f.write()
>>> print open(f.basename).read()
Basename of the file to be written.
Returns True if string or exfile is specififed.
True this file should be marked executable (necessary for scripts); False otherwise (appropriate for input and output files). Default value is False.
Path to the external file, to be copied to the target file.
Writing mode. Can be ‘w’ or ‘a’.
If mode is ‘w’, the target file is rewritten. In this case, exfile is copied.
If mode is ‘a’, file fileName is appended. In this case, exfile is read and appended to fileName line by line. This might be inappropriate for a binary exfile.
>>> f = InputFile(basename='if.mode', string='Line 1')
>>> f.write()
>>> print open(f.basename).read()
>>> f.mode = 'a'
>>> f.write()
>>> print open(f.basename).read()
The “Prefer String Flag”.
Specifies what to put to the target file if both string and exfile attributes are specified. If True, string is preffered, if False, exfile is preffered.
This property set automatically each time string or exfile is set.
Can be manually set by user.
>>> f = InputFile(string='string', exfile='existting.file', basename='if.psf')
>>> f.psf = True
>>> f.write()
>>> print open(f.basename).read()
>>> f.psf = False
>>> f.write()
>>> print open(f.basename).read()
Report about the last written target file.
String to be written into the target file.
Write file to disk.
Create a new file containing string if string attribute is specified, or copy existing file pointed to by the exfile attribute.
Scheduler for jobs.
A scheduler for running shell jobs. A scheduler consists of a dict of jobs and a queue. The add() method registers a new job, the queue() method adds a job to the queue and the wait() method waits for a queued job to finish. The convenience method run() does both queuing and waiting in one call and returns the result of the wait.
Currently, all jobs will be run in the OS shell using the subprocess module, this may change at some point in the future.
Create a new empty job scheduler.
Add a job to the scheduler. It will be identified by name, any existing job with the same name is replaced.
Get a job that was added to the scheduler, identified by its name. Alternatively just use dictionary syntax (see last example):
>>> s = Scheduler()
>>> j = Job('echo hello world')
>>> s.add('j', j)
>>> s.get_job('j')
<job cd '.'; echo hello world>
>>> s['j']
<job cd '.'; echo hello world>
Add the job identified by name to the current job queue. The same job may be queued several times.
If no job by the given name exists, a ValueError is raised, e.g:
Queue the job identified by name to the job queue wait for it to finish and return the result. For example:
>>> s = Scheduler()
>>> s
{}
>>> s.add('hi', Job('echo hi'))
>>> s.run('hi')
'hi\n'
You can also wait for a number of seconds or for the creation of a file in the execution directory of the Job. You can combine these options:
>>> s = Scheduler()
>>> s.add('t', Job('touch test'))
>>> s.run('t', sec=1, file='test')
''
The kwargs dictionary is passed to the wait() method, see description of allowed keyword arguments there.
Wait for the started job to finish.
Criteria to define whether job is finished or not, depend on the specified keyword arguments:
- sec:
- number of seconds to wait.
- files:
- list of filenames. Wait untill all files specified in the list exist (created by the job). If this argument specified, the sec argument defines the period to check file existence.
- llines:
- list of regexp strings (not regex objects!) to compare with the last string of file specified in the files argument. The llines argument implies that the files argument is given and that they have equal lengths. Criteria meet, if all files exist and their last lines match the regexp strings.
A shell job to be executed by the OS shell. A job is described by a command and a working directory. The working directory is optional, if ommited, the current directory is used. For example:
>>> j = Job('echo hello')
>>> j
<job cd '.'; echo hello>
>>> j = Job('sleep 4', '/home')
>>> j
<job cd '/home'; sleep 4>
If the string you pass as working directory does not describe a directory on your current machine, a ValueError is raised, e.g.
>>> j = Job('sleep 1', '+4w34"\d')
Traceback (most recent call last):
...
ValueError: not a directory: +4w34"\d
To execute a job, add it to a scheduler, queue() it and wait().
Create a new job consisting of a shell command cmd to run and a working directory dir in which to execute.
If dir is not a directory or not accessible by the current user, a ValueError is raised. If no directory is specified, the current working directory is used.
Command to be run in the OS shell.
Must be a string.
Working directory.
Directory where cmd OS shell command is started.