Uploading (compressed) data vie shell script

Questions and discussions on development tools for WarcraftRealms
Locked
Kosh
Census Taker
Posts: 84
Joined: Sun Jul 01, 2007 2:59 am
Location: Somewhere on or near Earth

Uploading (compressed) data vie shell script

Post by Kosh »

I am guessing that the audience for a shell script to upload Census results is probably limited, particularly if your OS doesn't have a sufficiently UNIX-like shell environment, but I'll post what i have so far, anyway. ;)

Take the following code and save it as "censusup.sh" (or any name of your choosing) in a convenient directory that you have full permissions to. I put it in my normal "downloads" directory, because I already have a symlink to my CensusPlus.lua file there, but someplace like your home directory would also be fine. Note that you should not put it somewhere in the WoW directory tree. Oh yes, don't forget to make the script executable (chmod +x censusup.sh) :)

Code: Select all

#!/bin/sh
myNAME=`basename $0`
cpNAME=**your username**
cpPW='**your password**'
echo "$myNAME: Checking for new version of CencusPlus.lua..."
# change PWD to where we are, for convenience
holdPWD=`pwd`
cd `dirname $0`
if ( [ CensusPlus.lua -nt CensusPlus.lua.gz ] );
  then {
    echo "  lua file newer -- recompressing...";
    gzip -c -9 CensusPlus.lua > CensusPlus.lua.gz;
    lastERR=$?
    if ( [ $lastERR -ne 0 ] );
      then {
        echo "$myNAME: Error from gzip: $lastERR"
        };
      else {
        echo "  ...compressed.  Starting upload..."
        echo ">>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>"
        curl -F [email protected] -F user=$cpNAME -F pw=$cpPW -w "\ncurl: transfered %{size_upload} bytes in %{time_total} seconds (%{speed_upload} b/s)\n" http://www.warcraftrealms.com/uniupload.php
        lastERR=$?
        echo "<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<"
        if &#40; &#91; $lastERR -ne 0 &#93; &#41;;
          then &#123;
            echo "$myNAME&#58; Error from curl&#58; $lastERR"
            &#125;;
          else &#123;
            echo "  upload successful."
            &#125;;
        fi
        &#125;;
    fi
    &#125;;
  else &#123;
    echo "  compressed file up to date -- doing nothing.";
    &#125;;
fi
cd $holdPWD
#eof
(Note that the "curl" line is one long line.)

You will need to put your warcraftrealms.com username and password in the third and fourth lines. Keep the single quotes around the password, if you have any "special" characters in it.

Note that the script creates the gzip-compressed copy of the CensusPlus.lua file in the same directory as the script. It also assumes that the lua file (or a link to it) is also in the same directory. You can change this by hard-coding the path to the file in line 12, but I prefer having the symbolic link in the directory for convenience.

Note that I haven't spent a lot of time on this, so the error handling is rather simplistic (it just stops on errors), and it's fairly verbose. I also haven't gotten around to attempting to parse the response from the php script to look for logical errors. It just dumping out everything that's returned, so you need to look over the response to make sure that the "your name has been credited" line is there, and that no errors are reported.

User avatar
Rollie
Site Admin
Posts: 4783
Joined: Sun Nov 28, 2004 11:52 am
Location: Austin, TX
Contact:

Post by Rollie »

well isn't that spiffy =)
phpbb:phpinfo()

Kosh
Census Taker
Posts: 84
Joined: Sun Jul 01, 2007 2:59 am
Location: Somewhere on or near Earth

Post by Kosh »

Yep, I think so ;)

A bonus to me is that I don't have to worry about hitting my head on the size limit (at least for quite a while), since I'm sending it compressed.

Kosh
Census Taker
Posts: 84
Joined: Sun Jul 01, 2007 2:59 am
Location: Somewhere on or near Earth

Post by Kosh »

Well, I put a little more work into my script. In addition to some cleanup, it now remembers if the compress succeeded but the upload failed, so a rerun of the script will retry the upload even if the lua file hasn't changed. It also strips down the server output (via awk) and also detects "logical" errors reported by the server-side script.

BTW Rollie, do any of the possible errors from uniupload.php start out with something other than "Error"? If so, then I will need to tweak the awk script.

Anyway, here's the updated version of censusup.sh:

Code: Select all

#!/bin/sh
cpNAME=**your username**
cpPW='**your password**'
cpFN=CensusPlus.lua
cpFNgz=CensusPlus.lua.gz
#
myNAME=`basename $0`
myREUPFLAG=".$myNAME.reup"
echo "$myNAME&#58; Checking for updated CensusPlus.lua..."
# change PWD to where we are, for convenience
holdPWD=`pwd`
cd `dirname $0`
if &#40; &#91; $cpFN -nt $cpFNgz &#93; &#41;; then
   &#123;
   echo "  lua file newer -- recompressing...";
   gzip -c -9 $cpFN > $cpFNgz;
   lastERR=$?;
   if &#40; &#91; $lastERR -ne 0 &#93; &#41;; then
      &#123;
      echo "$myNAME&#58; Error from gzip&#58; $lastERR";
      needUPLOAD=-1;
      &#125;;
   else
      needUPLOAD=1;
   fi
   &#125;;
else
   &#123;
# need to force another upload?
   if &#40; &#91; "$1" == "-f" -o -e $myREUPFLAG &#93; &#41;; then
      &#123;
      echo "  compressed file up to date, but upload needed";
      needUPLOAD=1;
      &#125;;
   else
      &#123;
      echo "  compressed file up to date -- doing nothing.";
      needUPLOAD=0;
      &#125;;
   fi
   &#125;;
fi
if &#40; &#91; $needUPLOAD -gt 0 &#93; &#41;; then
   &#123;
   echo "  Starting upload..."
   echo ">>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>"
   curl -F CensusPlus=@$cpFNgz -F user=$cpNAME -F pw=$cpPW \
        -w "\n!!!curl %&#123;size_upload&#125; %&#123;time_total&#125; %&#123;speed_upload&#125;" \
        http&#58;//www.warcraftrealms.com/uniupload.php \
        | awk \
'/^$/           &#123; next &#125;
/^Checking your/
/^Found /      &#123; if&#40; foundLine != "" &#41; foundLine = foundLine "; ";
                 if&#40; $NF == "<=" &#41;
                    foundLine = foundLine $&#40;NF-2&#41; " " $&#40;NF-1&#41;;
                 else
                    foundLine = foundLine substr&#40; $0, 7 &#41;;
                 next
               &#125;
/^Attempting / &#123; if&#40; foundLine != "" &#41;
                    &#123; print "Found&#58; " foundLine;
                      foundLine = "";
                    &#125;
                  print $0;
                  next
               &#125;
/^Username/
/ will be credited /
/^!!!curl/     &#123; kb = $2 / 1024;
                 print "\ncurl&#58; transferred " kb "kb in " $3 " seconds &#40;" kb / $3 " kb/s&#41;";
               &#125;
/^Error/       &#123; print "***\n" $0; exit 10 &#125;
'
# copy the array of pipe statuses - $lastERR refs &#91;0&#93; &#40;curl stat&#41;
   lastERR=&#40; "$&#123;PIPESTATUS&#91;@&#93;&#125;" &#41;
   awkERR=$&#123;lastERR&#91;1&#93;&#125;         # &#40;status of awk&#41;
   echo "<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<"
   if &#40; &#91; $lastERR -ne 0 &#93; &#41;; then
      &#123;
      echo "$myNAME&#58; Error from curl&#58; $lastERR"
# flag the failed upload, so we can retry later
      touch $myREUPFLAG
      &#125;;
   else
      &#123;
      if &#40; &#91; $awkERR -ne 0 &#93; &#41;; then
         &#123;
         echo "$myNAME&#58; Error from server during upload"
# flag the failed upload, so we can retry later
         touch $myREUPFLAG
         &#125;;
      else
         &#123;
         date "+  upload completed at %H&#58;%M&#58;%S"
         rm -f $myREUPFLAG
         &#125;;
      fi
      &#125;;
   fi
   &#125;;
fi
cd $holdPWD
#eof
The script woudl be a bit cleaner if I put the awk script in it's own file, but then another file would have to be floating around. Note that the 22 lines after the fourth line of the "curl" command is one long string constant (the awk command script).

In case you're wondering, here's an example of what the awk-massaged output looks like:

Code: Select all

Checking your file
Found&#58; CensusPlus Database; CensusPlus Info Data; Version&#58; 4.2.2; Locale&#58; US
Attempting to make entry

Username &#58; Kosh<=
Kosh will be credited with this submission!

curl&#58; transferred 1074.76kb in 20.817 seconds &#40;51.6289 kb/s&#41;
Last edited by Kosh on Tue Mar 10, 2009 1:03 am, edited 1 time in total.

User avatar
Rollie
Site Admin
Posts: 4783
Joined: Sun Nov 28, 2004 11:52 am
Location: Austin, TX
Contact:

Post by Rollie »

It should always give an Error if there is a problem
phpbb:phpinfo()

Kosh
Census Taker
Posts: 84
Joined: Sun Jul 01, 2007 2:59 am
Location: Somewhere on or near Earth

Post by Kosh »

Good deal, thanks :)

User avatar
Ceto
Shady Dealer
Posts: 335
Joined: Sun Oct 16, 2005 8:22 pm
Location: Plymouth, NH
Contact:

Post by Ceto »

I'll share mine, for comparison. It does not submit compressed data, and it's written in Python, with paths hard-coded for Mac OS X:

http://static.bwerp.net/~adam/2009/03/09/wr-upload.zip
Image

Kosh
Census Taker
Posts: 84
Joined: Sun Jul 01, 2007 2:59 am
Location: Somewhere on or near Earth

Post by Kosh »

Interesting. In my case, my WoW account name isn't the same as my account here, so it would require some tweaking for me. And of course, you can't assume where the game is installed on someone else's machine, but that's configurable (on my machine, it's at /Applications/Games/World of Warcraft/, for example).

Note that you could compress the file in Python, using the gzip module.

I'll shift into "UNIX security mode" briefly & point out that passing a password on the command-line is not secure, since it will show up in the Process Table (visible using "ps", etc.). Of course, on a machine you have complete physical control over, it's not a big deal. Just pointing it out for future reference. :)

/UNIX geek mode


Oh, I might as well point out that I discovered a bug in my "version 2" script; it doesn't successfully capture the exit status of both the "curl" and "awk" commands. Instead of reposting the whold script, I will just edit the script above, since there are only a couple of lines that changed (I figure everyone's eyes will glaze over if I post a "diff" ;) ).

Kosh
Census Taker
Posts: 84
Joined: Sun Jul 01, 2007 2:59 am
Location: Somewhere on or near Earth

Post by Kosh »

Well, I've made a couple more tweaks to my upload script. Most notably, it now passes any errors out of the script, so that they can be tested by other scripts (or a "while" loop, when necessary :p ). Other than that, I mainly consolidated the error reporting, including outputting the time on errors.

enjoy :)

Code: Select all

#!/bin/sh
cpNAME=**your username**
cpPW='**your password**'
cpFN=CensusPlus.lua
cpFNgz=CensusPlus.lua.gz
#
lastERR=0
myNAME=`basename $0`
myREUPFLAG=".$myNAME.reup"
echo "$myNAME&#58; Checking for updated CensusPlus.lua..."
# change PWD to where we are, for convenience
holdPWD=`pwd`
cd `dirname $0`
if &#40; &#91; $cpFN -nt $cpFNgz &#93; &#41;; then
   &#123;
   echo "  lua file newer -- recompressing..."
   gzip -c -9 $cpFN > $cpFNgz
   lastERR=$?
   if &#40; &#91; $lastERR -ne 0 &#93; &#41;; then
      &#123;
      errTEXT="gzip"
      needUPLOAD=-1
      &#125;
   else
      needUPLOAD=1
   fi
   &#125;
else
   &#123;
# need to force another upload?
   if &#40; &#91; "$1" == "-f" -o -e $myREUPFLAG &#93; &#41;; then
      &#123;
      echo "  compressed file up to date, but upload needed"
      needUPLOAD=1
      &#125;
   else
      &#123;
      echo "  compressed file up to date -- nothing to do."
      needUPLOAD=0
      &#125;
   fi
   &#125;
fi
if &#40; &#91; $needUPLOAD -gt 0 &#93; &#41;; then
   &#123;
   echo "  Starting upload..."
   echo ">>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>"
   curl -F CensusPlus=@$cpFNgz -F user=$cpNAME -F pw=$cpPW \
        -w "\n!!!curl %&#123;size_upload&#125; %&#123;time_total&#125; %&#123;speed_upload&#125;" \
        http&#58;//www.warcraftrealms.com/uniupload.php \
        | awk \
'/^$/           &#123; next &#125;
/^Checking your/
/^Found /      &#123; if&#40; foundLine != "" &#41; foundLine = foundLine "; ";
                 if&#40; $NF == "<foundLine> 0 &#41;
                    &#123; kb = $2 / 1024;
                      print "\ncurl&#58; transferred " kb "kb in " $3 " seconds &#40;" kb / $3 " kb/s&#41;";
                    &#125;
               &#125;
/^Error/       &#123; print "***\n" $0;
                 if&#40; index&#40; $0, "nable to connect" &#41; > 0 &#41;  # no DB connection
                    exit 221;
                 if&#40; index&#40; $NF, "uniupload.php" &#41; > 0 &#41;    # mystery SQL error
                    exit 222;
                 exit 220
               &#125;
'
# copy the array of pipe statuses - $lastERR refs &#91;0&#93; &#40;curl stat&#41;
   lastERR=&#40; "$&#123;PIPESTATUS&#91;@&#93;&#125;" &#41;
   awkERR=$&#123;lastERR&#91;1&#93;&#125;         # &#40;status of awk&#41;
   echo "<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<"
   if &#40; &#91; $lastERR -ne 0 &#93; &#41;; then
      &#123;
      errTEXT="curl"
# flag the failed upload, so we can retry later
      touch $myREUPFLAG
      &#125;
   else
      &#123;
      if &#40; &#91; $awkERR -ne 0 &#93; &#41;; then
         &#123;
         errTEXT="server during upload"
         lastERR=$awkERR
# flag the failed upload, so we can retry later
         touch $myREUPFLAG
         &#125;
      else
         &#123;
         date "+  upload completed at %H&#58;%M&#58;%S"
         rm -f $myREUPFLAG
         &#125;
      fi
      &#125;
   fi
   &#125;
fi
cd $holdPWD
if &#40; &#91; $lastERR -ne 0 &#93; &#41;; then
   &#123;
   echo -n "$myNAME&#58; Error from $errTEXT&#58; $lastERR"
   date "+ &#91;%H&#58;%M&#58;%S&#93;***"
   &#125;
fi
exit $lastERR
#eof

bartman009
Posts: 1
Joined: Sat Jun 13, 2009 11:12 pm

Post by bartman009 »

thanks a lot for updating the script!
assurance vie

Kosh
Census Taker
Posts: 84
Joined: Sun Jul 01, 2007 2:59 am
Location: Somewhere on or near Earth

Post by Kosh »

You're welcome :)

Kosh
Census Taker
Posts: 84
Joined: Sun Jul 01, 2007 2:59 am
Location: Somewhere on or near Earth

Post by Kosh »

After I discovered that the forum software is mangling any text between a "less-than" and a "greater-than" symbol (i.e., angle brackets), I thought I had better check this posting of my script. Sure enough, it is also mangled. Here's another copy with "~{~" and "~}~" substituted for all left and right angle brackets, respectively. To use, just reverse the substitution after copying.

Sorry about the confusion. I wouldn't expect phpBB to do any kind of (apparent) attempted HTML parsing inside a code block.

Code: Select all

#!/bin/sh
cpNAME=**your username**
cpPW='**your password**'
cpFN=CensusPlus.lua
cpFNgz=CensusPlus.lua.gz
#
lastERR=0
myNAME=`basename $0`
myREUPFLAG=".$myNAME.reup"
echo "$myNAME&#58; Checking for updated CensusPlus.lua..."
# change PWD to where we are, for convenience
holdPWD=`pwd`
cd `dirname $0`
if &#40; &#91; $cpFN -nt $cpFNgz &#93; &#41;; then
   &#123;
   echo "  lua file newer -- recompressing..."
   gzip -c -9 $cpFN ~&#125;~ $cpFNgz
   lastERR=$?
   if &#40; &#91; $lastERR -ne 0 &#93; &#41;; then
      &#123;
      errTEXT="gzip"
      needUPLOAD=-1
      &#125;
   else
      needUPLOAD=1
   fi
   &#125;
else
   &#123;
# need to force another upload?
   if &#40; &#91; "$1" == "-f" -o -e $myREUPFLAG &#93; &#41;; then
      &#123;
      echo "  compressed file up to date, but upload needed"
      needUPLOAD=1
      &#125;
   else
      &#123;
      echo "  compressed file up to date -- nothing to do."
      needUPLOAD=0
      &#125;
   fi
   &#125;
fi
if &#40; &#91; $needUPLOAD -gt 0 &#93; &#41;; then
   &#123;
   echo "  Starting upload..."
   echo "~&#125;~~&#125;~~&#125;~~&#125;~~&#125;~~&#125;~~&#125;~~&#125;~~&#125;~~&#125;~~&#125;~~&#125;~~&#125;~~&#125;~~&#125;~~&#125;~~&#125;~~&#125;~~&#125;~~&#125;~~&#125;~~&#125;~~&#125;~~&#125;~~&#125;~~&#125;~~&#125;~~&#125;~~&#125;~~&#125;~~&#125;~~&#125;~~&#125;~~&#125;~~&#125;~~&#125;~~&#125;~~&#125;~~&#125;~~&#125;~"
   curl -F CensusPlus=@$cpFNgz -F user=$cpNAME -F pw=$cpPW \
        -w "\n!!!curl %&#123;size_upload&#125; %&#123;time_total&#125; %&#123;speed_upload&#125;" \
        http&#58;//www.warcraftrealms.com/uniupload.php \
        | awk \
'/^$/           &#123; next &#125;
/^Checking your/
/^Found /      &#123; if&#40; foundLine != "" &#41; foundLine = foundLine "; ";
                 if&#40; $NF == "~&#123;~=" &#41;
                    foundLine = foundLine $&#40;NF-2&#41; " " $&#40;NF-1&#41;;
                 else
                    foundLine = foundLine substr&#40; $0, 7 &#41;;
                 next
               &#125;
/^Attempting / &#123; if&#40; foundLine != "" &#41;
                    &#123; print "Found&#58; " foundLine;
                      foundLine = "";
                    &#125;
                  print $0;
                  next
               &#125;
/^Username/
/ will be credited /
/^!!!curl/     &#123; if&#40; $3 ~&#125;~ 0 &#41;
                    &#123; kb = $2 / 1024;
                      print "\ncurl&#58; transferred " kb "kb in " $3 " seconds &#40;" kb / $3 " kb/s&#41;";
                    &#125;
               &#125;
/^Error/       &#123; print "***\n" $0;
                 if&#40; index&#40; $0, "nable to connect" &#41; ~&#125;~ 0 &#41;  # no DB connection
                    exit 221;
                 if&#40; index&#40; $NF, "uniupload.php" &#41; ~&#125;~ 0 &#41;    # mystery SQL error
                    exit 222;
                 exit 220
               &#125;
'
# copy the array of pipe statuses - $lastERR refs &#91;0&#93; &#40;curl stat&#41;
   lastERR=&#40; "$&#123;PIPESTATUS&#91;@&#93;&#125;" &#41;
   awkERR=$&#123;lastERR&#91;1&#93;&#125;         # &#40;status of awk&#41;
   echo "~&#123;~~&#123;~~&#123;~~&#123;~~&#123;~~&#123;~~&#123;~~&#123;~~&#123;~~&#123;~~&#123;~~&#123;~~&#123;~~&#123;~~&#123;~~&#123;~~&#123;~~&#123;~~&#123;~~&#123;~~&#123;~~&#123;~~&#123;~~&#123;~~&#123;~~&#123;~~&#123;~~&#123;~~&#123;~~&#123;~~&#123;~~&#123;~~&#123;~~&#123;~~&#123;~~&#123;~~&#123;~~&#123;~~&#123;~~&#123;~"
   if &#40; &#91; $lastERR -ne 0 &#93; &#41;; then
      &#123;
      errTEXT="curl"
# flag the failed upload, so we can retry later
      touch $myREUPFLAG
      &#125;
   else
      &#123;
      if &#40; &#91; $awkERR -ne 0 &#93; &#41;; then
         &#123;
         errTEXT="server during upload"
         lastERR=$awkERR
# flag the failed upload, so we can retry later
         touch $myREUPFLAG
         &#125;
      else
         &#123;
         date "+  upload completed at %H&#58;%M&#58;%S"
         rm -f $myREUPFLAG
         &#125;
      fi
      &#125;
   fi
   &#125;
fi
cd $holdPWD
if &#40; &#91; $lastERR -ne 0 &#93; &#41;; then
   &#123;
   echo -n "$myNAME&#58; Error from $errTEXT&#58; $lastERR"
   date "+ &#91;%H&#58;%M&#58;%S&#93;***"
   &#125;
fi
exit $lastERR
#eof

Locked