Compare commits

...

113 Commits

Author SHA1 Message Date
Louwrentius 0869dc92ac Current version of ppss 2010-01-31 19:39:58 +00:00
Louwrentius 05945843f0 Changed the licence from BSD-style licence to GPL. 2010-01-27 19:45:56 +00:00
Louwrentius a4b5fdc7c1 backup 2009-12-17 18:38:35 +00:00
Louwrentius 43e68825a3 Reworked the ppss-test script to work with the new ppss executable and changed the default workig dir of ppss 2009-12-17 18:16:02 +00:00
Louwrentius 204537cd04 Renamed ppss.sh to ppss, to make it more like a regular unix program. 2009-12-17 18:06:51 +00:00
Louwrentius 297f5eacb8 Fixed issue with master SSH connection on node not being setup. 2009-12-17 15:34:28 +00:00
Louwrentius f3666a49d6 Some 'gui' bugfixes. 2009-12-17 14:45:15 +00:00
Louwrentius 3abe84f218 workdir must be relative to current path. 2009-12-17 01:59:28 +00:00
Louwrentius 24b368e734 Reworked ppss regarding process management.. 2009-12-17 01:52:42 +00:00
Louwrentius 7d0815528e More small fixes. 2009-12-16 20:27:09 +00:00
Louwrentius f2ccafefc6 some rework 2009-12-16 18:29:18 +00:00
Louwrentius 20e520e820 Significant bug fixing and testing. Updated to 2.45 2009-12-16 12:34:03 +00:00
Louwrentius fffccc3e04 Seems to fix some file name issues... 2009-12-15 23:03:23 +00:00
Louwrentius e91f54932c Marking current version as 2.40 2009-11-08 17:54:41 +00:00
Louwrentius fedd72bf1c Marking current version as 2.40 2009-11-08 17:50:49 +00:00
Louwrentius e4bdd59475 More fixes for distributed operation. 2009-11-08 17:50:12 +00:00
Louwrentius 1b16eb62ee More fixes for distributed operation. 2009-11-08 17:14:01 +00:00
Louwrentius d08dc38761 More fixes for distributed operation. 2009-11-08 16:58:06 +00:00
Louwrentius cd4545d0c3 Fixed distributed ppss mechanism. 2009-11-08 15:16:49 +00:00
Louwrentius 05801b6773 Working on distributed home dir. 2009-10-24 20:16:06 +00:00
Louwrentius 722b3b4ea6 Added start-delay for workers to spread the load. 2009-10-24 19:53:35 +00:00
Louwrentius 4d28218ce4 Reworked the percent and job counter. 2009-10-24 19:24:56 +00:00
Louwrentius b1af50040f Rewrote the usage and help screens. 2009-10-24 18:24:45 +00:00
Louwrentius 5b840d33c3 Cleaning things up, like some stuff with PPSS_DIR and PPSS_HOME_DIR 2009-10-24 15:52:40 +00:00
Louwrentius 207e875727 This version is also detecting CPUs on Solaris. 2009-10-21 21:23:36 +00:00
Louwrentius 4a682daaae This version is also detecting CPUs on Solaris. 2009-10-21 20:56:07 +00:00
Louwrentius 276e721c7e This version is also detecting CPUs on Solaris. 2009-10-21 20:43:10 +00:00
Louwrentius 239e422354 PPSS can now be run with multiple instances in stand-alone mode. 2009-10-21 19:44:59 +00:00
Louwrentius fd7f32668e Working on ability to run multiple instances. 2009-10-21 19:35:40 +00:00
Louwrentius e7f158235a Keeping a list of pids 2009-10-21 19:23:18 +00:00
Louwrentius 7a578ff88b Fixed irritating typo that caused the occurance of a file called '1' with some error message within. 2009-10-21 18:48:16 +00:00
Louwrentius e24cf063e9 update 2009-10-21 18:26:21 +00:00
Louwrentius fb3905ad11 Fixed status display of distributed ppss. 2009-09-27 10:51:32 +00:00
Louwrentius 9a880c313d fixed status handling 2009-09-26 20:43:24 +00:00
Louwrentius 3a8257e519 fixed typo 2009-09-26 18:50:24 +00:00
Louwrentius e402fa6936 Major rework, PPSS is now fully asynchronous. There are no while loops polling stuff at an interval. 2009-09-26 18:25:43 +00:00
Louwrentius 24bdb33583 Major rework, PPSS is now fully asynchronous. There are no while loops polling stuff at an interval. 2009-09-26 18:24:54 +00:00
Louwrentius d2e6bf7c54 proper version 2009-09-25 19:30:15 +00:00
Louwrentius 95eb2e0d07 improved some distributed stuff 2009-09-25 19:14:09 +00:00
Louwrentius 1da1de4b18 Improved distributed ppss 2009-09-20 09:04:23 +00:00
Louwrentius 7ca08b87cf bug fix 2.20 2009-08-09 20:12:51 +00:00
Louwrentius 815bb952c9 fixed language typo 2009-08-09 17:17:08 +00:00
Louwrentius 9b0e5fdec5 Improved error handling and messages. 2009-08-09 16:40:09 +00:00
Louwrentius ebd7ca55b7 Improved error handling and messages. 2009-08-09 16:12:23 +00:00
Louwrentius 04847e14db Added improvements regarding multi-user environments as reported by cinly.ooi 2009-08-09 15:49:58 +00:00
Louwrentius 64486cb710 fixed typo (succes) 2009-07-03 22:53:11 +00:00
Louwrentius e6d3539b51 upped version number 2009-05-05 20:30:19 +00:00
Louwrentius a4f1107863 added extra sanitizing of input but it is not perfect. 2009-05-05 20:29:42 +00:00
Louwrentius ac7192cfde 2.18 2009-03-29 21:59:43 +00:00
Louwrentius 7ae3f7c512 version 2.17 with nifty status screen. 2009-03-29 20:48:15 +00:00
Louwrentius 9e4a89e95f Improved escaping of file names 2009-03-27 21:54:15 +00:00
Louwrentius 62059e887e backup 2009-03-27 20:16:00 +00:00
Louwrentius 66e04d8806 PPSS improved... 2009-03-23 23:04:41 +00:00
Louwrentius 113f4d31cb better path handling 2009-03-23 20:24:20 +00:00
Louwrentius 64edfe79c1 Fixes almost complete, more testing needed 2009-03-22 18:06:30 +00:00
Louwrentius be217ac168 Currently fixing file name sanitizing. 2009-03-22 17:15:47 +00:00
Louwrentius 49d54913f7 fixing filename sanitation 2009-03-22 12:14:26 +00:00
Louwrentius 2f5b38aedd fixing filename sanitation 2009-03-22 11:15:13 +00:00
Louwrentius 1e26cef172 fixing filename sanitation 2009-03-22 10:05:36 +00:00
Louwrentius 046ba029dc fixin g filename sanitation 2009-03-22 10:01:36 +00:00
Louwrentius e5dcc81930 fixin g filename sanitation 2009-03-22 09:59:47 +00:00
Louwrentius d523525eca Currently fixing file name sanitizing. 2009-03-22 09:40:31 +00:00
Louwrentius 8b1e5fed9a ppss is still a bit buggy 2009-03-21 23:30:15 +00:00
Louwrentius bd75afbf41 significant bugfixes when using no script 2009-03-21 22:37:38 +00:00
Louwrentius b3ba085fbe 2.11 - Ok version with fixed file handling when using input file instead of dir 2009-03-21 20:50:37 +00:00
Louwrentius d2a8a6e08d 2.11 - Ok version with fixed file handling when using input file instead of dir 2009-03-21 20:45:05 +00:00
Louwrentius 964648de7f 2.10 - Ok version with fixed file handling when using input file instead of dir 2009-03-21 20:34:45 +00:00
Louwrentius 3f6343318a Ok version with fixed file handling when using input file instead of dir 2009-03-21 20:34:05 +00:00
Louwrentius 71e8d829d9 messing around with paths in items 2009-03-21 20:17:01 +00:00
Louwrentius aaae2dc713 fixing download function 2009-03-21 19:57:57 +00:00
Louwrentius 3e5fbfd38f 2.08 fixes 2.07 2009-03-15 21:51:30 +00:00
Louwrentius b1fa580a3e Bug fix, 2.07 is broken. 2009-03-15 20:51:18 +00:00
Louwrentius 954d8632c1 Added config option for known_hosts file. 2009-03-15 00:35:53 +00:00
Louwrentius c6161eb4b4 Forgot to uptdate the version. 2009-03-13 10:47:54 +00:00
Louwrentius 04f13096a4 Parallel deployment of ppss + new mechanism for specifing output files. 2009-03-13 10:12:10 +00:00
Louwrentius 244cfce933 New function display_header, just for better looks. 2009-03-12 20:48:14 +00:00
Louwrentius c8f4a2caa0 version 2.03 2009-03-10 23:00:33 +00:00
Louwrentius 4b45db3169 Cleaned up some logging. 2009-03-10 22:04:13 +00:00
Louwrentius d0a2413f64 Polished logging and cpu/core detection. 2009-03-10 21:32:12 +00:00
Louwrentius c0d58e6752 test 2009-03-10 20:36:33 +00:00
Louwrentius 5317a82ac4 Updated description. 2009-03-09 20:46:46 +00:00
Louwrentius d8c73dba4b Version 2.0. 2009-03-08 22:49:23 +00:00
Louwrentius f6062bfdaf Added proper handling of start stop and continue for distributed processing. 2009-03-08 15:18:15 +00:00
Louwrentius 979b1cc0a4 Working on help screen. 2009-03-08 14:24:28 +00:00
Louwrentius ffeaf1ccf6 Updated help page. Not finished yet. 2009-03-05 23:19:55 +00:00
Louwrentius 808d14e08c Cleaning up code and comments. 2009-03-04 21:18:05 +00:00
Louwrentius 73d49b125b Added logging of processing time of item. If scp is disabled, assume -d directory is local, not remote. 2009-03-02 20:57:28 +00:00
Louwrentius c3975dd64a bug fixes 2009-03-02 02:25:12 +00:00
Louwrentius 0a35341033 Renamed all command line options in long format that is more readable. 2009-03-01 23:25:16 +00:00
Louwrentius 1bdfe61490 This version has been used to encode wavs on 3 different host in parallel. 2009-03-01 22:33:09 +00:00
Louwrentius 02f0d7b4fa update 2009-03-01 21:18:52 +00:00
Louwrentius 21114ec54a The full chain of downloading items to the node, processing and uploading the result back to the server is now functional. 2009-03-01 21:04:16 +00:00
Louwrentius cca01f27fd backup 2009-03-01 20:58:48 +00:00
Louwrentius f83f916c57 backup 2009-03-01 20:57:16 +00:00
Louwrentius 628e93c904 messed up 2009-03-01 20:32:28 +00:00
Louwrentius c231b8e05a Distributed PPSS seems to be working, need more testing. 2009-03-01 17:27:17 +00:00
Louwrentius 6bd4d48a76 Better item logging and error handling. 2009-03-01 16:47:44 +00:00
Louwrentius 66a8983a3e filenames with spaces seem to be handled properly. 2009-03-01 16:10:53 +00:00
Louwrentius aa15c6f868 PPSS can now substitute %ITEM% in commando string for the current item. 2009-03-01 10:17:27 +00:00
Louwrentius f6b9fad83a Distributed PPSS is now able to show overall progress, thus how many items have been processed. 2009-02-28 19:27:30 +00:00
Louwrentius af1eda73d4 Removed usage of bc so this is not a dependency anymore. 2009-02-28 19:11:48 +00:00
Louwrentius 7c16a77860 Bacup 2009-02-28 19:05:20 +00:00
Louwrentius ed55ae5137 Bacup 2009-02-28 18:42:16 +00:00
Louwrentius 39c1ecf542 Reworked OS detection and proces detection, which differs for Linux and Mac OS X 2009-02-28 17:44:02 +00:00
Louwrentius 2edb204c4c This seems the first solid working version of Distributed PPSS. 2009-02-28 16:39:32 +00:00
Louwrentius bca4fda670 just a backup, broken and a mess 2009-02-28 15:51:00 +00:00
Louwrentius cacb489c14 broken en messed up - just a backup 2009-02-28 08:40:11 +00:00
Louwrentius fc5d48b73c got rid of getopts and did it by hand, much better 2009-02-27 19:14:38 +00:00
Louwrentius 2e76e5e34e buggy and broken 2009-02-27 18:17:58 +00:00
Louwrentius 0090a68d70 fixed some cleanup issues with screen 2009-02-08 21:36:40 +00:00
Louwrentius a8bb31a575 backup - broken 2009-02-08 16:41:34 +00:00
Louwrentius 04501ea92a first checkin test 2009-01-31 17:54:45 +00:00
Louwrentius f7c2f34cb5 creating branch for distributed ppss 2009-01-31 17:52:50 +00:00
3 changed files with 3385 additions and 350 deletions

1839
ppss Executable file

File diff suppressed because it is too large Load Diff

223
ppss-test.sh Executable file
View File

@ -0,0 +1,223 @@
#!/bin/bash
DEBUG="$1"
VERSION=2.55
TMP_DIR="ppss"
PPSS=ppss
PPSS_DIR=ppss_dir
cleanup () {
for x in $REMOVEFILES
do
if [ -e ./$x ]
then
rm -r ./$x
fi
done
}
parseJobStatus () {
TMP_FILE="$1"
RES=`grep "Status:" "$JOBLOG/$TMP_FILE"`
STATUS=`echo "$RES" | awk '{ print $2 }'`
echo "$STATUS"
}
oneTimeSetUp () {
NORMALTESTFILES=`echo test-{a..z}`
SPECIALTESTFILES="\'file-!@#$%^&*()_+=-0987654321~\' \'file-/\<>?:;'{}[]\' file-/\/\:\/!@#$%^&*()_+=-0987654321~ file-/\<>?:;'{}[] http://www.google.nl ftp://storage.nl"
JOBLOG=./$PPSS_DIR/job_log
INPUTFILENORMAL=test-normal.input
INPUTFILESPECIAL=test-special.input
LOCALOUTPUT=ppss_dir/PPSS_LOCAL_OUTPUT
REMOVEFILES="$INPUTFILENORMAL $INPUTFILESPECIAL $PPSS_DIR test-ppss-*"
cleanup
for x in $NORMALTESTFILES
do
echo "$x" >> "$INPUTFILENORMAL"
done
for x in $SPECIALTESTFILES
do
echo $x >> "$INPUTFILESPECIAL"
done
}
testVersion () {
RES=`./$PPSS -v`
for x in $RES
do
echo "$x" | grep [0-9] >> /dev/null
if [ "$?" == "0" ]
then
assertEquals "Version mismatch!" "$VERSION" "$x"
fi
done
}
rename-ppss-dir () {
TEST="$1"
if [ -e "$PPSS_DIR" ] && [ -d "$PPSS_DIR" ] && [ ! -z "$TEST" ]
then
mv "$PPSS_DIR" test-ppss-"$TEST"
fi
}
oneTimeTearDown () {
if [ ! "$DEBUG" == "debug" ]
then
cleanup
fi
}
createDirectoryWithSomeFiles () {
A="File with Spaces"
B="File\With\Slashes"
mkdir "/tmp/$TMP_DIR"
for x in "$A" "$B"
do
TMP_FILE="/tmp/$TMP_DIR/$x"
touch "$TMP_FILE"
done
}
testSpacesInFilenames () {
createDirectoryWithSomeFiles
RES=$( { ./$PPSS -d /tmp/$TMP_DIR -c 'ls -alh ' >> /dev/null ; } 2>&1 )
assertEquals "PPSS did not execute properly." 0 "$?"
assertNull "PPSS retured some errors..." "$RES"
if [ ! "$?" == "0" ]
then
echo "RES IS $RES"
fi
grep "SUCCESS" $JOBLOG/* >> /dev/null 2>&1
assertEquals "Found error with space in filename $TMP_FILE" "0" "$?"
rm -rf "/tmp/$TMP_DIR"
rename-ppss-dir $FUNCNAME
}
testSpecialCharacterHandling () {
RES=$( { ./$PPSS -f "$INPUTFILESPECIAL" -c 'echo ' >> /dev/null ; } 2>&1 )
assertEquals "PPSS did not execute properly." 0 "$?"
assertNull "PPSS retured some errors..." "$RES"
if [ ! "$?" == "0" ]
then
echo "RES IS $RES"
fi
RES=`find ppss_dir/PPSS_LOCAL_OUTPUT | wc -l | sed 's/\ //g'`
assertEquals "To many lock files..." "7" "$RES"
RES1=`ls -1 $JOBLOG`
RES2=`ls -1 $LOCALOUTPUT`
assertEquals "RES1 $RES1 is not the same as RES2 $RES2" "$RES1" "$RES2"
rename-ppss-dir $FUNCNAME
}
testSkippingOfProcessedItems () {
createDirectoryWithSomeFiles
RES=$( { ./$PPSS -d /tmp/$TMP_DIR -c 'echo ' >> /dev/null ; } 2>&1 )
assertEquals "PPSS did not execute properly." 0 "$?"
assertNull "PPSS retured some errors..." "$RES"
RES=$( { ./$PPSS -d /tmp/$TMP_DIR -c 'echo ' >> /dev/null ; } 2>&1 )
assertEquals "PPSS did not execute properly." 0 "$?"
assertNull "PPSS retured some errors..." "$RES"
grep -i skip ./$PPSS_dir/* >> /dev/null 2>&1
assertEquals "Skipping of items went wrong." 0 "$?"
rename-ppss-dir $FUNCNAME-1
RES=$( { ./$PPSS -f $INPUTFILESPECIAL -c 'echo ' >> /dev/null ; } 2>&1 )
assertEquals "PPSS did not execute properly." 0 "$?"
assertNull "PPSS retured some errors..." "$RES"
RES=$( { ./$PPSS -f $INPUTFILESPECIAL -c 'echo ' >> /dev/null ; } 2>&1 )
assertEquals "PPSS did not execute properly." 0 "$?"
assertNull "PPSS retured some errors..." "$RES"
grep -i skip ./$PPSS_dir/* >> /dev/null 2>&1
assertEquals "Skipping of items went wrong." 0 "$?"
rm -rf "/tmp/$TMP_DIR"
rename-ppss-dir $FUNCNAME-2
}
testExistLogFiles () {
./$PPSS -f "$INPUTFILENORMAL" -c 'echo "$ITEM"' >> /dev/null
assertEquals "PPSS did not execute properly." 0 "$?"
for x in $NORMALTESTFILES
do
assertTrue "[ -e $JOBLOG/$x ]"
done
rename-ppss-dir $FUNCNAME
}
getStatusOfJob () {
EXPECTED="$1"
if [ "$EXPECTED" == "SUCCESS" ]
then
./$PPSS -f "$INPUTFILENORMAL" -c 'echo ' >> /dev/null
assertEquals "PPSS did not execute properly." 0 "$?"
elif [ "$EXPECTED" == "FAILURE" ]
then
./$PPSS -f "$INPUTFILENORMAL" -c 'thiscommandfails ' >> /dev/null
assertEquals "PPSS did not execute properly." 0 "$?"
fi
for x in $NORMALTESTFILES
do
STATUS=`parseJobStatus "$x"`
assertEquals "FAILED WITH STATUS $STATUS." "$EXPECTED" "$STATUS"
done
rename-ppss-dir "$FUNCNAME-$EXPECTED"
}
testErrorHandlingOK () {
getStatusOfJob SUCCESS
}
testErrorHandlingFAIL () {
getStatusOfJob FAILURE
}
. ./shunit2

1673
ppss.sh

File diff suppressed because it is too large Load Diff