sneak/ppss
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity
8c872255b4
ppss/wiki/Manual2.wiki
Louwrentius 8c872255b4 Edited wiki page through web user interface.
2009-03-13 00:29:25 +00:00

251 lines
11 KiB
Plaintext
Raw Blame History

#summary PPSS Manual (Distributed)
#labels Featured
= Introduction =
To use PPSS in a distributed fasion, The following steps must be performed:
# Setup SSH access on server and nodes.
# Create a list of all nodes.
# Create a configuration file for PPSS, that will be distributed to nodes.
# Deploy PPSS to the nodes.
# Start PPSS on all nodes.
== Preparation of server and nodes ==
The following preparations must be made in order to use PPSS in a distributed fasion:
# Create an unprivileged user 'ppss' on the server.
# Create an unprivileged user 'ppss' on each node.
# Generate a SSH key without a pass phrase.
# Add the SSH key to the authorized_keys file of the 'ppss' user on the server.
# Add the SSH key to the authorized_keys file of the 'ppss' user on the client.
# Place PPSS on the server within the PPSS home directory.
*Security*
Please note that usage of SSH keys without pass phrases may pose a security threat if the machines are shared with other users. You must decide for yourself if the security risk that is associated with this setup is acceptable for your environment. For example, if a node is compromised, the attacker will have (initially unprivileged) access to the server.
== Create a list of nodes ==
A file must be created containing the hostnames (DNS) and/or IP-addresses of all nodes. The file must contain one node per line, such as:
{{{
192.168.0.100
192.168.0.101
host.domain.com
...
}}}
== Create a PPSS configuration file ==
This is the most important part of setting up distributed PPSS. It is exactly the same as setting up a configuration file for standalone mode, except that more options are necessary.
The best way to explain how to create a configuration file for distributed PPSS is to provide an example. In this example, a script is used to encode WAV files to MP3. This script is called 'encode.sh' and takes a filename as an argument.
`./ppss config -C config.cfg -c 'encode.sh ' -d /source/dir -s 192.168.1.100 -u ppss -k ppss-key.key -S ./encode.sh -n nodes.txt -t -o /some/output/dir`
It is quite a long command line, however, it is executed only once. Afther that, the config file config.cfg can be used for all further commands.
*Mode*
The first option sets the mode, in this case 'config' to generate a configuration file.
*Configuration file*
The second option, -C, specifies the name of the configuration file to be created.
*Command*
The third option, -c, specifies the command to be executed. *Please take special note of the single quotes and the space behind the commad.* You can read -c 'encode.sh ' also as -c 'encode.sh "$ITEM"'.
*Source directory*
This option specifies the location on the *server* where the files reside that must be processed. These files will be transfered using SCP to the nodes for local processing.
*Server*
The -s option specifies the SSH server that acts as both fileserver and SSH server for communication between nodes. The SSH server is mainly used for file-locking: nodes know that locked files are already processed or being processed, so another unlocked file must be selected.
If the server acts both as a file server and SSH server, it is not recommended to use it also as a node, in this case for encoding. Filetransers using SSH can take quite some processing power.
*User name*
This is the name of the local system user that is used by the nodes to logon to the server. For deployment, such a user must also be present on the nodes.
*SSH Key*
Scripts using SSH require an SSH key withouth a passphrase. This key must be uploaded to the nodes an the nodes must know which key to use, so it must be specified.
*Script or program that must be uploaded*
The -S option specifies the script or program that should be uploaded to the node because it must be executed by the node for distributed computing. In this case, the encode.sh script must be deployed on all nodes and thus specified.
*List of nodes*
The -n option specifies the file containing all nodes. For every node, PPSS will perform actions such as deploy, start, stop and pause.
*Transfer files to local host*
If this option is specified, the file is copied from the source directory to a local temporary working directory for local processing. This is necessary if SCP is used to access files that must be processed.
If files are distributed over NFS or SMB, the files seem to be present on the local system, because it is just a mount point and thus just a part of the local file system. In this case, the -t option can be omitted, however it it is specified, files are copied to a local directory using 'cp'.
*The output directory*
If the -t option is used, the -o option specifies the destination directory on the server. The results are uploaded to this directory. If the -t option is not specified, the command 'cp' is used to transfer files back to the specified output directory.
== Create a script ==
This step is not mandatory, however it is necessary if you want to be able to upload the results of your commands back to the server. You have to make sure that the output is placed in a specific location.
This specific location is the $PPSS_LOCAL_OUTPUT directory. The result must be stored in a directory that has the name of the item that has been processed.
This is very easily accomplished just by 'sourcing' the just generated PPSS configuration file and use the $PPSS_LOCAL_OUTPUT variable within your script or command. An example script is shown below, that has actually been used to encode 400 GB of WAV files.
{{{
#!/bin/bash
ITEM="$1"
TMP=`basename $ITEM`
source config.cfg
PATH=$PATH:/opt/local/bin:~/bin
lame -a "$ITEM" "$PPSS_LOCAL_OUTPUT/$TMP/$TMP.mp3" --preset standard --quiet
ERROR="$?"
if [ "$ERROR" == "0" ]
then
echo "Encode of $ITEM successful."
exit 0
else
echo "Error when encoding $ITEM."
exit 1
fi
}}}
Take notice of the basename command. Items are provided with full path. Basename strips this path from the filename and uses just the filename in this script.
== Deploy PPSS to nodes ==
Once SSH access is setup and the configuration file is generated, PPSS can be deployed to the nodes. This is very simple, as this example demonstrates:
`./ppss.sh deploy -C config.cfg
During the phase when we generated the configuration file, a nodes file was specified. Thus PPSS knows, just by reading this configuration file, which file contains a list of nodes.
{{{
bash-3.2$ ./ppss.sh deploy -C config.cfg
mrt 12 22:18:22: INFO - ---------------------------------------------------------
mrt 12 22:18:22: INFO - Distributed Parallel Processing Shell Script version 2.03
mrt 12 22:18:22: INFO - Hostname: MacBoek.local
mrt 12 22:18:22: INFO - Deploying PPSS on nodes.
mrt 12 22:18:24: INFO - PPSS installed on node 192.168.1.14.
mrt 12 22:18:28: INFO - PPSS installed on node 192.168.1.12.
mrt 12 22:18:29: INFO - PPSS installed on node 192.168.1.4.
mrt 12 22:18:31: INFO - PPSS installed on node 192.168.1.31.
}}}
Deployment of PPSS is serialised, not parallel. Maybe it should be performed in parallel. This is a to-do.
== Start PPSS on nodes ==
Just as simple as deploying PPSS, PPSS is started on all nodes.
`./ppss.sh start -C config`
{{{
mrt 12 22:21:17: INFO - ---------------------------------------------------------
mrt 12 22:21:17: INFO - Distributed Parallel Processing Shell Script version 2.03
mrt 12 22:21:17: INFO - Hostname: MacBoek.local
mrt 12 22:21:17: INFO - Starting PPSS on node 10.0.0.14.
mrt 12 22:21:17: INFO - Starting PPSS on node 10.0.0.12.
mrt 12 22:21:20: INFO - Starting PPSS on node 10.0.0.4.
mrt 12 22:21:20: INFO - Starting PPSS on node 10.0.0.31.
}}}
== Stop pause and continue PPSS on nodes ==
To stop, pause or continue processing on all nodes, use the following commands:
`./ppss.sh stop -C config.cfg`
`./ppss.sh pause -C config.cfg`
`./ppss.sh continue -C config.cfg`
Please note that nodes will continue processing the current item they are working on, they just stop processing new items if stop or pause is selected.
== Show progress ==
The overall process of the 'cluster' is determined by the number of files present in the input and output directories on the server.
{{{
bash-3.2$ ./ppss.sh status -C config.cfg
mrt 12 21:06:17: INFO - ---------------------------------------------------------
mrt 12 21:06:17: INFO - Distributed Parallel Processing Shell Script version 2.03
mrt 12 21:06:17: INFO - Hostname: MacBoek.local
mrt 12 21:06:17: INFO - 56 percent complete.
}}}
== Logging ==
An important feature of PPSS is its extensive logging. There are two types of log files.
* A single log file created by PPSS itself. This file is found on the local nodes. Using tail -f on these files, it is possible to monitor what PPSS is currently doing.
{{{
Mar 12 22:57:19: INFO - ---------------------------------------------------------
Mar 12 22:57:19: INFO - Distributed Parallel Processing Shell Script version 2.03
Mar 12 22:57:19: INDO - ---------------------------------------------------------
Mar 12 22:57:19: INFO - Hostname: Beest
Mar 12 22:57:19: DEBUG - Found 8 logic processors.
Mar 12 22:57:19: INFO - CPU: Intel(R) Core(TM) i7 CPU 920 @ 2.67GHz
Mar 12 22:57:19: INFO - ---------------------------------------------------------
Mar 12 22:57:19: DEBUG - Job log directory JOB_LOG exists.
Mar 12 22:57:20: INFO - Listener started.
Mar 12 22:57:20: INFO - Starting 8 workers.
Mar 12 22:57:20: INFO - Currently 0 percent complete. Processed 0 of 625 items.
Mar 12 22:57:20: DEBUG - Trying to lock item 20060731.wav.
Mar 12 22:57:20: DEBUG - Item 20060731.wav is locked.
Mar 12 22:57:20: INFO - Currently 0 percent complete. Processed 1 of 625 items.
Mar 12 22:57:20: DEBUG - Trying to lock item 20060801.wav.
Mar 12 22:57:20: DEBUG - Item 20060801.wav is locked.
Mar 12 22:57:20: INFO - Currently 0 percent complete. Processed 2 of 625 items.
Mar 12 22:57:20: DEBUG - Trying to lock item 20060802.wav.
Mar 12 22:57:20: DEBUG - Item 20060802.wav is locked.
Mar 12 22:57:20: INFO - Currently 0 percent complete. Processed 3 of 625 items.
............
mrt 10 23:51:23: DEBUG - Item 20060830.wav is locked.
mrt 10 23:51:23: INFO - Currently 3 percent complete. Processed 23 of 625 items.
mrt 10 23:51:23: DEBUG - Trying to lock item 20060831.wav.
mrt 10 23:51:23: DEBUG - Got lock on 20060831.wav, processing.
mrt 10 23:51:23: DEBUG - Transfering item 20060831.wav to local disk.
mrt 10 23:52:18: DEBUG - Exit code of transfer is 0
mrt 10 23:52:18: DEBUG - Processing item 20060831.wav
}}}
* An individual log file containing information and output of each processed item. these files are uploaded to the SSH server to the 'job_log' directory. For every item, a log file must be present.
{{{
===== PPSS Item Log File =====
Host: MacBoek.local
Item: PPSS_LOCAL_TMPDIR/20060831.wav
Start date: mrt 10 23:52:18
Encode of PPSS_LOCAL_TMPDIR/20060831.wav successful.
Status: Succes - item has been processed.
Elapsed time (h:m:s): 0:5:23
}}}
As you can see, with a few simple grep commands, it is possible to quickly determine which items have failed to process. Also, you can see that my MacBook took 5 minutes and 23 seconds to process this WAV file.
Please note that the "Encode of..." part is output of the script that is executed on an item. The other content is generated by PPSS.
== To wrap it up ==
I am convinced that PPSS is very easy to use and tailored to your needs. If you have questions and/or suggestions, don't hesitate to send an e-mail. If you find bugs, please report them using the issue tracker. Feedback is greatly appreciated.
Reference in New Issue View Git Blame Copy Permalink