Page 1 of 1
Uploading (compressed) data vie shell script
Posted: Thu Feb 12, 2009 11:29 pm
by Kosh
I am guessing that the audience for a shell script to upload Census results is probably limited, particularly if your OS doesn't have a sufficiently UNIX-like shell environment, but I'll post what i have so far, anyway.
Take the following code and save it as "censusup.sh" (or any name of your choosing) in a convenient directory that you have full permissions to. I put it in my normal "downloads" directory, because I already have a symlink to my CensusPlus.lua file there, but someplace like your home directory would also be fine. Note that you should
not put it somewhere in the WoW directory tree. Oh yes, don't forget to make the script executable (chmod +x censusup.sh)
Code: Select all
#!/bin/sh
myNAME=`basename $0`
cpNAME=**your username**
cpPW='**your password**'
echo "$myNAME: Checking for new version of CencusPlus.lua..."
# change PWD to where we are, for convenience
holdPWD=`pwd`
cd `dirname $0`
if ( [ CensusPlus.lua -nt CensusPlus.lua.gz ] );
then {
echo " lua file newer -- recompressing...";
gzip -c -9 CensusPlus.lua > CensusPlus.lua.gz;
lastERR=$?
if ( [ $lastERR -ne 0 ] );
then {
echo "$myNAME: Error from gzip: $lastERR"
};
else {
echo " ...compressed. Starting upload..."
echo ">>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>"
curl -F [email protected] -F user=$cpNAME -F pw=$cpPW -w "\ncurl: transfered %{size_upload} bytes in %{time_total} seconds (%{speed_upload} b/s)\n" http://www.warcraftrealms.com/uniupload.php
lastERR=$?
echo "<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<"
if ( [ $lastERR -ne 0 ] );
then {
echo "$myNAME: Error from curl: $lastERR"
};
else {
echo " upload successful."
};
fi
};
fi
};
else {
echo " compressed file up to date -- doing nothing.";
};
fi
cd $holdPWD
#eof
(Note that the "curl" line is one long line.)
You will need to put your warcraftrealms.com username and password in the third and fourth lines. Keep the single quotes around the password, if you have any "special" characters in it.
Note that the script creates the gzip-compressed copy of the CensusPlus.lua file in the same directory as the script. It also assumes that the lua file (or a link to it) is also in the same directory. You can change this by hard-coding the path to the file in line 12, but I prefer having the symbolic link in the directory for convenience.
Note that I haven't spent a lot of time on this, so the error handling is rather simplistic (it just stops on errors), and it's fairly verbose. I also haven't gotten around to attempting to parse the response from the php script to look for logical errors. It just dumping out everything that's returned, so you need to look over the response to make sure that the "
your name has been credited" line is there, and that no errors are reported.
Posted: Fri Feb 13, 2009 9:22 am
by Rollie
well isn't that spiffy =)
Posted: Mon Feb 16, 2009 10:44 pm
by Kosh
Yep, I think so
A bonus to me is that I don't have to worry about hitting my head on the size limit (at least for quite a while), since I'm sending it compressed.
Posted: Wed Mar 04, 2009 1:12 am
by Kosh
Well, I put a little more work into my script. In addition to some cleanup, it now remembers if the compress succeeded but the upload failed, so a rerun of the script will retry the upload even if the lua file hasn't changed. It also strips down the server output (via awk) and also detects "logical" errors reported by the server-side script.
BTW Rollie, do any of the possible errors from uniupload.php start out with something other than "Error"? If so, then I will need to tweak the awk script.
Anyway, here's the updated version of censusup.sh:
Code: Select all
#!/bin/sh
cpNAME=**your username**
cpPW='**your password**'
cpFN=CensusPlus.lua
cpFNgz=CensusPlus.lua.gz
#
myNAME=`basename $0`
myREUPFLAG=".$myNAME.reup"
echo "$myNAME: Checking for updated CensusPlus.lua..."
# change PWD to where we are, for convenience
holdPWD=`pwd`
cd `dirname $0`
if ( [ $cpFN -nt $cpFNgz ] ); then
{
echo " lua file newer -- recompressing...";
gzip -c -9 $cpFN > $cpFNgz;
lastERR=$?;
if ( [ $lastERR -ne 0 ] ); then
{
echo "$myNAME: Error from gzip: $lastERR";
needUPLOAD=-1;
};
else
needUPLOAD=1;
fi
};
else
{
# need to force another upload?
if ( [ "$1" == "-f" -o -e $myREUPFLAG ] ); then
{
echo " compressed file up to date, but upload needed";
needUPLOAD=1;
};
else
{
echo " compressed file up to date -- doing nothing.";
needUPLOAD=0;
};
fi
};
fi
if ( [ $needUPLOAD -gt 0 ] ); then
{
echo " Starting upload..."
echo ">>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>"
curl -F CensusPlus=@$cpFNgz -F user=$cpNAME -F pw=$cpPW \
-w "\n!!!curl %{size_upload} %{time_total} %{speed_upload}" \
http://www.warcraftrealms.com/uniupload.php \
| awk \
'/^$/ { next }
/^Checking your/
/^Found / { if( foundLine != "" ) foundLine = foundLine "; ";
if( $NF == "<=" )
foundLine = foundLine $(NF-2) " " $(NF-1);
else
foundLine = foundLine substr( $0, 7 );
next
}
/^Attempting / { if( foundLine != "" )
{ print "Found: " foundLine;
foundLine = "";
}
print $0;
next
}
/^Username/
/ will be credited /
/^!!!curl/ { kb = $2 / 1024;
print "\ncurl: transferred " kb "kb in " $3 " seconds (" kb / $3 " kb/s)";
}
/^Error/ { print "***\n" $0; exit 10 }
'
# copy the array of pipe statuses - $lastERR refs [0] (curl stat)
lastERR=( "${PIPESTATUS[@]}" )
awkERR=${lastERR[1]} # (status of awk)
echo "<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<"
if ( [ $lastERR -ne 0 ] ); then
{
echo "$myNAME: Error from curl: $lastERR"
# flag the failed upload, so we can retry later
touch $myREUPFLAG
};
else
{
if ( [ $awkERR -ne 0 ] ); then
{
echo "$myNAME: Error from server during upload"
# flag the failed upload, so we can retry later
touch $myREUPFLAG
};
else
{
date "+ upload completed at %H:%M:%S"
rm -f $myREUPFLAG
};
fi
};
fi
};
fi
cd $holdPWD
#eof
The script woudl be a bit cleaner if I put the awk script in it's own file, but then another file would have to be floating around. Note that the 22 lines after the fourth line of the "curl" command is one long string constant (the awk command script).
In case you're wondering, here's an example of what the awk-massaged output looks like:
Code: Select all
Checking your file
Found: CensusPlus Database; CensusPlus Info Data; Version: 4.2.2; Locale: US
Attempting to make entry
Username : Kosh<=
Kosh will be credited with this submission!
curl: transferred 1074.76kb in 20.817 seconds (51.6289 kb/s)
Posted: Wed Mar 04, 2009 11:29 am
by Rollie
It should always give an Error if there is a problem
Posted: Thu Mar 05, 2009 11:40 pm
by Kosh
Good deal, thanks
Posted: Sun Mar 08, 2009 10:12 pm
by Ceto
I'll share mine, for comparison. It does not submit compressed data, and it's written in Python, with paths hard-coded for Mac OS X:
http://static.bwerp.net/~adam/2009/03/09/wr-upload.zip
Posted: Tue Mar 10, 2009 12:56 am
by Kosh
Interesting. In my case, my WoW account name isn't the same as my account here, so it would require some tweaking for me. And of course, you can't assume where the game is installed on someone else's machine, but that's configurable (on my machine, it's at /Applications/Games/World of Warcraft/, for example).
Note that you
could compress the file in Python, using the gzip module.
I'll shift into "UNIX security mode" briefly & point out that passing a password on the command-line is not secure, since it will show up in the Process Table (visible using "ps", etc.). Of course, on a machine you have complete physical control over, it's not a big deal. Just pointing it out for future reference.
/UNIX geek mode
Oh, I might as well point out that I discovered a bug in my "version 2" script; it doesn't successfully capture the exit status of both the "curl" and "awk" commands. Instead of reposting the whold script, I will just edit the script above, since there are only a couple of lines that changed (I figure
everyone's eyes will glaze over if I post a "diff"
).
Posted: Thu May 07, 2009 10:18 pm
by Kosh
Well, I've made a couple more tweaks to my upload script. Most notably, it now passes any errors out of the script, so that they can be tested by other scripts (or a "while" loop, when necessary :p ). Other than that, I mainly consolidated the error reporting, including outputting the time on errors.
enjoy
Code: Select all
#!/bin/sh
cpNAME=**your username**
cpPW='**your password**'
cpFN=CensusPlus.lua
cpFNgz=CensusPlus.lua.gz
#
lastERR=0
myNAME=`basename $0`
myREUPFLAG=".$myNAME.reup"
echo "$myNAME: Checking for updated CensusPlus.lua..."
# change PWD to where we are, for convenience
holdPWD=`pwd`
cd `dirname $0`
if ( [ $cpFN -nt $cpFNgz ] ); then
{
echo " lua file newer -- recompressing..."
gzip -c -9 $cpFN > $cpFNgz
lastERR=$?
if ( [ $lastERR -ne 0 ] ); then
{
errTEXT="gzip"
needUPLOAD=-1
}
else
needUPLOAD=1
fi
}
else
{
# need to force another upload?
if ( [ "$1" == "-f" -o -e $myREUPFLAG ] ); then
{
echo " compressed file up to date, but upload needed"
needUPLOAD=1
}
else
{
echo " compressed file up to date -- nothing to do."
needUPLOAD=0
}
fi
}
fi
if ( [ $needUPLOAD -gt 0 ] ); then
{
echo " Starting upload..."
echo ">>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>"
curl -F CensusPlus=@$cpFNgz -F user=$cpNAME -F pw=$cpPW \
-w "\n!!!curl %{size_upload} %{time_total} %{speed_upload}" \
http://www.warcraftrealms.com/uniupload.php \
| awk \
'/^$/ { next }
/^Checking your/
/^Found / { if( foundLine != "" ) foundLine = foundLine "; ";
if( $NF == "<foundLine> 0 )
{ kb = $2 / 1024;
print "\ncurl: transferred " kb "kb in " $3 " seconds (" kb / $3 " kb/s)";
}
}
/^Error/ { print "***\n" $0;
if( index( $0, "nable to connect" ) > 0 ) # no DB connection
exit 221;
if( index( $NF, "uniupload.php" ) > 0 ) # mystery SQL error
exit 222;
exit 220
}
'
# copy the array of pipe statuses - $lastERR refs [0] (curl stat)
lastERR=( "${PIPESTATUS[@]}" )
awkERR=${lastERR[1]} # (status of awk)
echo "<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<"
if ( [ $lastERR -ne 0 ] ); then
{
errTEXT="curl"
# flag the failed upload, so we can retry later
touch $myREUPFLAG
}
else
{
if ( [ $awkERR -ne 0 ] ); then
{
errTEXT="server during upload"
lastERR=$awkERR
# flag the failed upload, so we can retry later
touch $myREUPFLAG
}
else
{
date "+ upload completed at %H:%M:%S"
rm -f $myREUPFLAG
}
fi
}
fi
}
fi
cd $holdPWD
if ( [ $lastERR -ne 0 ] ); then
{
echo -n "$myNAME: Error from $errTEXT: $lastERR"
date "+ [%H:%M:%S]***"
}
fi
exit $lastERR
#eof
Posted: Sat Jun 13, 2009 11:21 pm
by bartman009
thanks a lot for updating the script!
assurance vie
Posted: Thu Jun 18, 2009 1:33 am
by Kosh
You're welcome
Posted: Sat Jun 27, 2009 7:34 pm
by Kosh
After I discovered that the forum software is mangling any text between a "less-than" and a "greater-than" symbol (i.e., angle brackets), I thought I had better check this posting of my script. Sure enough, it is also mangled. Here's another copy with "~{~" and "~}~" substituted for all left and right angle brackets, respectively. To use, just reverse the substitution after copying.
Sorry about the confusion. I wouldn't expect phpBB to do
any kind of (apparent) attempted HTML parsing inside a code block.
Code: Select all
#!/bin/sh
cpNAME=**your username**
cpPW='**your password**'
cpFN=CensusPlus.lua
cpFNgz=CensusPlus.lua.gz
#
lastERR=0
myNAME=`basename $0`
myREUPFLAG=".$myNAME.reup"
echo "$myNAME: Checking for updated CensusPlus.lua..."
# change PWD to where we are, for convenience
holdPWD=`pwd`
cd `dirname $0`
if ( [ $cpFN -nt $cpFNgz ] ); then
{
echo " lua file newer -- recompressing..."
gzip -c -9 $cpFN ~}~ $cpFNgz
lastERR=$?
if ( [ $lastERR -ne 0 ] ); then
{
errTEXT="gzip"
needUPLOAD=-1
}
else
needUPLOAD=1
fi
}
else
{
# need to force another upload?
if ( [ "$1" == "-f" -o -e $myREUPFLAG ] ); then
{
echo " compressed file up to date, but upload needed"
needUPLOAD=1
}
else
{
echo " compressed file up to date -- nothing to do."
needUPLOAD=0
}
fi
}
fi
if ( [ $needUPLOAD -gt 0 ] ); then
{
echo " Starting upload..."
echo "~}~~}~~}~~}~~}~~}~~}~~}~~}~~}~~}~~}~~}~~}~~}~~}~~}~~}~~}~~}~~}~~}~~}~~}~~}~~}~~}~~}~~}~~}~~}~~}~~}~~}~~}~~}~~}~~}~~}~~}~"
curl -F CensusPlus=@$cpFNgz -F user=$cpNAME -F pw=$cpPW \
-w "\n!!!curl %{size_upload} %{time_total} %{speed_upload}" \
http://www.warcraftrealms.com/uniupload.php \
| awk \
'/^$/ { next }
/^Checking your/
/^Found / { if( foundLine != "" ) foundLine = foundLine "; ";
if( $NF == "~{~=" )
foundLine = foundLine $(NF-2) " " $(NF-1);
else
foundLine = foundLine substr( $0, 7 );
next
}
/^Attempting / { if( foundLine != "" )
{ print "Found: " foundLine;
foundLine = "";
}
print $0;
next
}
/^Username/
/ will be credited /
/^!!!curl/ { if( $3 ~}~ 0 )
{ kb = $2 / 1024;
print "\ncurl: transferred " kb "kb in " $3 " seconds (" kb / $3 " kb/s)";
}
}
/^Error/ { print "***\n" $0;
if( index( $0, "nable to connect" ) ~}~ 0 ) # no DB connection
exit 221;
if( index( $NF, "uniupload.php" ) ~}~ 0 ) # mystery SQL error
exit 222;
exit 220
}
'
# copy the array of pipe statuses - $lastERR refs [0] (curl stat)
lastERR=( "${PIPESTATUS[@]}" )
awkERR=${lastERR[1]} # (status of awk)
echo "~{~~{~~{~~{~~{~~{~~{~~{~~{~~{~~{~~{~~{~~{~~{~~{~~{~~{~~{~~{~~{~~{~~{~~{~~{~~{~~{~~{~~{~~{~~{~~{~~{~~{~~{~~{~~{~~{~~{~~{~"
if ( [ $lastERR -ne 0 ] ); then
{
errTEXT="curl"
# flag the failed upload, so we can retry later
touch $myREUPFLAG
}
else
{
if ( [ $awkERR -ne 0 ] ); then
{
errTEXT="server during upload"
lastERR=$awkERR
# flag the failed upload, so we can retry later
touch $myREUPFLAG
}
else
{
date "+ upload completed at %H:%M:%S"
rm -f $myREUPFLAG
}
fi
}
fi
}
fi
cd $holdPWD
if ( [ $lastERR -ne 0 ] ); then
{
echo -n "$myNAME: Error from $errTEXT: $lastERR"
date "+ [%H:%M:%S]***"
}
fi
exit $lastERR
#eof