Uploads

Isaura · Post by **Isaura** » Fri Nov 02, 2007 6:15 am

Is there something wrong with the uploading?
The data from yesterday and the day before yesterday aren't processed.
I update on the site, not through uniuploader or something
So I was wondering if something is wrong?

DM. · Post by **DM.** » Fri Nov 02, 2007 7:24 am

Yup looks to be stopped,

http://www.warcraftrealms.com/dbstats.php

Last processed file was uploaded on 2007-10-31 10:34:27.
Current server time is 2007-11-02 08:23:45

Hybuir · Post by **Hybuir** » Fri Nov 02, 2007 9:08 am

/RollieSignal

Balgair · Post by **Balgair** » Fri Nov 02, 2007 10:31 am

Yep noticed that myself, it's moving but very very slowly and stops for a few hours at a time. Plus even when files are processed my total updates count isn't going up - it's stayed the same for about 3 days now, and I have had a few files processed in that time

Post by **Rollie** » Fri Nov 02, 2007 10:36 am

Digging

Post by **Rollie** » Fri Nov 02, 2007 10:52 am

Well, looks like it's mainly just that the update process is slowing again. Moving to more powerful hardware helped out for nearly a year, but I knew it wouldn't be a permanent fix.

At first glance, I am faced with the same decision I had to make back in January.

1) I can either start pruning/archiving characters (cheap and not too hard)
2) I can move to even more powerful hardware (expensive and only a stop gap until we reach the next level)
3) I can move to server clusters that could constantly expand to meet demand (very expensive and basically out of the question)

At this point, I'm leaning towards #1. What I could potentially do is move all characters not seen in XX months into an archive table that is not looked at for updates which means smaller number of characters to check against when doing updates. I could also add maybe a once per month process that can check all latest added characters against the archive and if a match is found, it can 'un-archive' those characters. This adds some complexity rather than just archiving the data for good, but probably the best bet to be honest.

Post by **Rollie** » Fri Nov 02, 2007 11:03 am

Upon further investigation it appears that it is particular uploads from the same individual that is prematurely killing the process. Digging further.

One thing of note, if I looked to start doing archives, here's a few stats:

5M Chars not seen in 24 months
15M Chars not seen in 18 months
26M Chars not seen in 12 months

I think 12 months would be too soon, and 24 months wouldn't give enough benefit. Maybe 18 months is not a bad amount of time for a character to be dormant before being archived?

Post by **Rollie** » Fri Nov 02, 2007 11:08 am

Okay, it was WyriHaximus! Let's get him!!! Wyri, please PURGE or at least PRUNE your file. Also, can you check your UU settings as I am getting an upload from you every 30 minutes and surely the file shouldn't be changing that often?

Balgair · Post by **Balgair** » Fri Nov 02, 2007 11:32 am

Phew, was wondering if it was me at first

18 months sounds a fair compromise, especially if you can run a check now and again to make sure they haven't become active again (maybe not even as often as monthly, every 2-3 months should be enough).

DM. · Post by **DM.** » Fri Nov 02, 2007 12:04 pm

LOL Wyri!!

And yea 18 months does seem reasonable.

Hybuir · Post by **Hybuir** » Fri Nov 02, 2007 1:17 pm

Rollie wrote:Okay, it was WyriHaximus! Let's get him!!! Wyri, please PURGE or at least PRUNE your file. Also, can you check your UU settings as I am getting an upload from you every 30 minutes and surely the file shouldn't be changing that often?

Wyri's secrets come out! *scrambles for UU*

Why don't we just do both prune & archive

DM. · Post by **DM.** » Fri Nov 02, 2007 1:55 pm

Wyri's been getting a string of 0's all the time he submits.....

Post by **Rollie** » Fri Nov 02, 2007 4:18 pm

Because his files have been unparsable.

WyriHaximus · Post by **WyriHaximus** » Sat Nov 03, 2007 4:13 pm

Rollie wrote:Okay, it was WyriHaximus! Let's get him!!! Wyri, please PURGE or at least PRUNE your file. Also, can you check your UU settings as I am getting an upload from you every 30 minutes and surely the file shouldn't be changing that often?

/me starts running for his life!

Anyway this is what happend:

Yeah it was a realy stupid error. Using a script to check and if changed upload the lua save file. After rollie poked me I trew away the lua file on my server and the check file with the md5 hash of the lua file. The system I'm using is kind simple. Got my WS and my server. My WS copies the lua files each minute to my server. My server checks every 15 minutes if the lua file is update by using a md5 hash. If changed the script updates the md5 check file and uploads the lua file to WCR. Somewhere along the way something went wrong (we still don't know what exactly) but by trewing away both files the problem got fixed .

xpolockx · Post by **xpolockx** » Sat Nov 03, 2007 7:21 pm

Heh Wyri, that's pretty funny

Don't forget to turn on aspect of the cheetah while you run...

I think 18 months is a pretty good marker - most people that quit for longer than a year don't end up playing that same character when they come back (not everyone, obviously, but most of the people I've talked to who come back after being gone for a year or more roll new characters to get back into the game.) Of course, having a monthly check to make sure that archived characters haven't been seen recently is also a good idea. Anything to help out your poor servers

Balgair · Post by **Balgair** » Sat Nov 03, 2007 10:43 pm

Also a point on archiving characters, perhaps have an even longer threshold on archiving ones on servers which consistently have very low submissions - it's perfectly possible for a server with average 1-5 submissions per month to just never see a character that only logs on for a few hours a week

WyriHaximus · Post by **WyriHaximus** » Sun Nov 04, 2007 9:46 am

xpolockx wrote:Heh Wyri, that's pretty funny Don't forget to turn on aspect of the cheetah while you run...

Sprint + Preparation + Sprint + Vanish ftw

!

Skyfire · Post by **Skyfire** » Sun Nov 04, 2007 9:48 pm

Much as I hate to say it (I would hate to lose my sig ;( ), 18 months sounds good. Of course, that leaves me set until this June, so who knows.

Post by **Rollie** » Mon Nov 05, 2007 2:28 pm

Well, appears the process has caught up already so after Wyr's files that were choking the system were cleaned out, we're back where we are supposed to be =)

Balgair · Post by **Balgair** » Mon Nov 05, 2007 2:32 pm

Good to know - I noticed it seemed to be catching up fast

Btw Rollie, could you take a look at this server and see if there's anything wrong with the display there, or if it genuinely is just zero entries on both sides? I'd kinda hoped it'd say 0:0 on the entries space if it was, the blank spaces leave me unsure if it's really 0:0 or just having issues