Edited wiki page through web user interface.
This commit is contained in:
Louwrentius
parent
53bf782dac
commit
6a2fedae41
|
|
@ -11,29 +11,35 @@ The following output is displayed by PPSS when executed without any options:
|
|||
|
||||
{{{
|
||||
|
||||
'Distributed Parallel Processing Shell Script
|
||||
Version: 2.0
|
||||
bash-3.2$ ./ppss.sh
|
||||
|
||||
|P|P|S|S| Distributed Parallel Processing Shell Script 2.40
|
||||
|
||||
usage: ./ppss.sh [ -d <sourcedir> | -f <sourcefile> ] [ -c '<command> "$ITEM"' ]
|
||||
[ -C <configfile> ] [ -j ] [ -l <logfile> ] [ -p <# jobs> ]
|
||||
[ -D <delay> ] [ -h ] [ --help ]
|
||||
|
||||
Examples:
|
||||
./ppss.sh -d /dir/with/some/files -c 'gzip '
|
||||
./ppss.sh -d /dir/with/some/files -c 'gzip "$ITEM"' -D 5
|
||||
./ppss.sh -d /dir/with/some/files -c 'cp "$ITEM" /tmp' -p 2
|
||||
}}}
|
||||
|
||||
If PPSS is executed with -h, more options are displayed.
|
||||
|
||||
{{{
|
||||
bash-3.2$ ./ppss.sh -h
|
||||
|
||||
|P|P|S|S| - Distributed Parallel Processing Shell Script -
|
||||
Version: 2.40
|
||||
|
||||
PPSS is a Bash shell script that executes commands in parallel on a set
|
||||
of items, such as files, or lines in a file.
|
||||
of items, such as files in a directory, or lines in a file.
|
||||
|
||||
Usage: ./ppss.sh MODE [ options ]
|
||||
or
|
||||
Usage: ./ppss.sh MODE -c <config file>
|
||||
This short summary only discuesses options for stand-alone mode. for all
|
||||
options, run PPSS with the options --help
|
||||
|
||||
Modes are:
|
||||
|
||||
standalone For execution of PPSS on a single host.
|
||||
node For execution of PPSS on a node, that is part of a 'cluster'.
|
||||
config Generate a config file based on the supplied option parameters.
|
||||
deploy Deploy PPSS and related files on the specified nodes.
|
||||
erase Erase PPSS and related files from the specified nodes.
|
||||
|
||||
start Starting PPSS on nodes.
|
||||
pause Pausing PPSS on all nodes.
|
||||
stop Stopping PPSS on all nodes.
|
||||
|
||||
Options are:
|
||||
Usage ./ppss.sh [ options ]
|
||||
|
||||
--command | -c Command to execute. Syntax: '<command> ' including the single quotes.
|
||||
Example: -c 'ls -alh '. It is also possible to specify where an item
|
||||
|
|
@ -50,61 +56,24 @@ Options are:
|
|||
this option will result in PPSS reading the config file and start
|
||||
processing items based on the settings of this file.
|
||||
|
||||
--disable-ht | -j Disable hyperthreading. Is enabled by default.
|
||||
--enable-ht | -j Enable hyperthreading. Is disabled by default.
|
||||
|
||||
--log | -l Sets the name of the log file. The default is ppss-log.txt.
|
||||
|
||||
--processes | -p Start the specified number of processes. Ignore the number of available
|
||||
CPUs.
|
||||
CPU's.
|
||||
|
||||
--force | -F Forces PPSS to start even if another instance is already running. This causes all instances
|
||||
of PPSS to keep on running. They will process items, but not stop.
|
||||
|
||||
The following options are used for distributed execution of PPSS.
|
||||
|
||||
--server | -s Specifies the SSH server that is used for communication between nodes.
|
||||
Using SSH, file locks are created, informing other nodes that an item
|
||||
is locked. Also, often items, such as files, reside on this host. SCP
|
||||
is used to transfer files from this host to nodes for local procesing.
|
||||
|
||||
--node | -n File containig a list of nodes that act as PPSS clients. One IP / DNS
|
||||
name per line.
|
||||
|
||||
--key | -k The SSH key that a node uses to connect to the server.
|
||||
|
||||
--known-hosts | -K The file that contains the server public key. Can often be found on
|
||||
hosts that already once connected to the server. See the file
|
||||
~/.ssh/known_hosts or else, manualy connect once and check this file.
|
||||
|
||||
--user | -u The SSH user name that is used when logging in into the master SSH
|
||||
server.
|
||||
|
||||
--script | -s Specifies the script/program that must be copied to the nodes for
|
||||
execution through PPSS. Only used in the deploy mode.
|
||||
This option should be specified if necessary when generating a config.
|
||||
|
||||
--transfer | -t This option specifies that an item will be downloaded by the node
|
||||
from the server or share to the local node for processing.
|
||||
|
||||
--no-scp | -b Do not use scp for downloading items. Use cp instead. Assumes that a
|
||||
network file system (NFS/SMB) is mounted under a local mountpoint.
|
||||
|
||||
--outputdir | -o Directory on server where processed files are put. If the result of
|
||||
encoding a wav file is an mp3 file, the mp3 file is put in the
|
||||
directory specified with this option.
|
||||
--delay | -D Adds an initial random delay to the start of all parallel jobs to spread
|
||||
the load. The delay is only used at the start of all 'threads'.
|
||||
|
||||
Example: encoding some wav files to mp3 using lame:
|
||||
|
||||
./ppss.sh standalone -c 'lame ' -d /path/to/wavfiles -j
|
||||
./ppss.sh -d /path/to/wavfiles -c 'lame '
|
||||
|
||||
Running PPSS based on a configuration file.
|
||||
Extended usage: use --help
|
||||
|
||||
./ppss.sh standalone -C config.cfg
|
||||
|
||||
Running PPSS on a client as part of a cluster.
|
||||
|
||||
./ppss.sh node -d /somedir -c 'cp "$ITEM" /some/destination' -s 10.0.0.50 -u ppss -t -k ppss-key.key'
|
||||
}}}
|
||||
|
||||
A detailed explanation based on examples will follow.
|
||||
|
||||
= How to use PPSS =
|
||||
|
|
|
|||
Loading…
Reference in New Issue