sneak
/
ppss
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity

Edited wiki page through web user interface.

This commit is contained in:
Louwrentius 2009-11-08 18:40:51 +00:00
parent 33cfafefbc
commit b0ebf6b60e
1 changed files with 100 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

100
wiki/Manual2.wiki
View File

@ -12,6 +12,106 @@ To use PPSS in a distributed fasion, The following steps must be performed:
# Deploy PPSS to the nodes. # Deploy PPSS to the nodes.
# Start PPSS on all nodes. # Start PPSS on all nodes.
= A list of all configuration options =
{{{
bash-3.2$ ./ppss.sh --help
|P|P|S|S| Distributed Parallel Processing Shell Script 2.40
PPSS is a Bash shell script that executes commands in parallel on a set
of items, such as files in a directory, or lines in a file.
Usage: ./ppss.sh [ MODE ] [ options ]
Modes are optional and mainly used for running in distributed mode. Modes are:
config Generate a config file based on the supplied option parameters.
deploy Deploy PPSS and related files on the specified nodes.
erase Erase PPSS and related files from the specified nodes.
start Starting PPSS on nodes.
pause Pausing PPSS on all nodes.
stop Stopping PPSS on all nodes.
node Running PPSS as a node, requires additional options.
Options are:
--command | -c Command to execute. Syntax: '<command> ' including the single quotes.
Example: -c 'ls -alh '. It is also possible to specify where an item
must be inserted: 'cp "$ITEM" /somedir'.
--sourcedir | -d Directory that contains files that must be processed. Individual files
are fed as an argument to the command that has been specified with -c.
--sourcefile | -f Each single line of the supplied file will be fed as an item to the
command that has been specified with -c.
--config | -C If the mode is config, a config file with the specified name will be
generated based on all the options specified. In the other modes.
this option will result in PPSS reading the config file and start
processing items based on the settings of this file.
--enable-ht | -j Enable hyperthreading. Is disabled by default.
--log | -l Sets the name of the log file. The default is ppss-log.txt.
--processes | -p Start the specified number of processes. Ignore the number of available
CPUs.
--delay | -D Adds an initial random delay to the start of all parallel jobs to spread
the load. The delay is only used at the start of all 'threads'.
The following options are used for distributed execution of PPSS.
--master | -m Specifies the SSH server that is used for communication between nodes.
Using SSH, file locks are created, informing other nodes that an item
is locked. Also, often items, such as files, reside on this host. SCP
is used to transfer files from this host to nodes for local procesing.
--node | -n File containig a list of nodes that act as PPSS clients. One IP / DNS
name per line.
--key | -k The SSH key that a node uses to connect to the master.
--known-hosts | -K The file that contains the server public key. Can often be found on
hosts that already once connected to the server. See the file
~/.ssh/known_hosts or else, manualy connect once and check this file.
--user | -u The SSH user name that is used when logging in into the master SSH
server.
--script | -S Specifies the script/program that must be copied to the nodes for
execution through PPSS. Only used in the deploy mode.
This option should be specified if necessary when generating a config.
--transfer | -t This option specifies that an item will be downloaded by the node
from the server or share to the local node for processing.
--no-scp | -b Do not use scp for downloading items. Use cp instead. Assumes that a
network file system (NFS/SMB) is mounted under a local mountpoint.
--outputdir | -o Directory on server where processed files are put. If the result of
encoding a wav file is an mp3 file, the mp3 file is put in the
directory specified with this option.
--homedir | -H Directory in which directory PPSS is installed on the node.
Default is 'ppss'.
Example: encoding some wav files to mp3 using lame:
./ppss.sh -c 'lame ' -d /path/to/wavfiles -j
Running PPSS based on a configuration file.
./ppss.sh -C config.cfg
Running PPSS on a client as part of a cluster.
./ppss.sh -d /somedir -c 'cp /some/destination' -s 10.0.0.50 -u ppss -t -k ppss-key.key
}}}
= Preparation of server and nodes = = Preparation of server and nodes =
The following preparations must be made in order to use PPSS in a distributed fasion: The following preparations must be made in order to use PPSS in a distributed fasion: