HELP! F@h: Attempting to get work packet

Stevo@ARM · Post by **Stevo@ARM** » Thu Feb 05, 2004 9:52 pm

Hey guys...

HELP!

I looked around the forums for a while but didn't see an answer to anything like this and every minute I spend looking for an answer we are loosing precious cycles

I checked the Stanford F@h site and I guess the servers ALL my ARM clients here at the shop are talking to are DOWN!

We are on track to post a lot of points today and tonight and we really need to keep up our team momentum we have all been building.

Do any of you guys know how I can get my folders back in action quickly so the SPCR team doesn't loose any more points?

Thanks,
Stevo

Member Name: ARM_Systems:_Home_of_the_StealthPC

haysdb · Post by **haysdb** » Thu Feb 05, 2004 10:16 pm

Stevo,

I just checked my own logs - no problems getting work today. I don't know if I would be using different servers than you or not.

Here is a page that shows server status:
http://folding.stanford.edu/serverstat.html

Only one server currently shows "DOWN"

David

Stevo@ARM · Post by **Stevo@ARM** » Thu Feb 05, 2004 11:04 pm

haysdb wrote:Stevo,

Only one server currently shows "DOWN"

David

I am not sure what happened, but this is what one of my log files says:

[06:35:04] + Attempting to send results
[06:35:04] - Connecting to assignment server
[06:35:25] - Couldn't send HTTP request to server
[06:35:25] - Couldn't send HTTP request to server
[06:35:25] + Could not connect to Assignment Server
[06:35:25] + Could not connect to Work Server (results)
[06:35:25] (171.67.89.151:8080)
[06:35:25] - Error: Could not transmit unit 03 (completed February 6). Keeping unit in queue.
[06:35:25] - Successful: assigned to (171.67.89.151).
[06:35:25] + News From Folding@Home: v.4 client available
[06:35:25] Loaded queue successfully.
[06:35:28] - Couldn't send HTTP request to server
[06:35:28] (Got status 500)
[06:35:28] + Could not connect to Work Server
[06:35:28] - Error: Attempt #1 to get work failed, and no other work to do.
Waiting before retry.

Not sure what status 500 is. Anyone got any ideas, I have stored up WUs waiting to get logged now. And a bunch of clients now in idle

Stevo

Zyzzyx · Post by **Zyzzyx** » Thu Feb 05, 2004 11:08 pm

Its annoying. It happens. You just have lots of systems affected by it.

Should be cleared up by morning.

Stevo@ARM · Post by **Stevo@ARM** » Thu Feb 05, 2004 11:12 pm

Zyzzyx wrote:Its annoying. It happens. You just have lots of systems affected by it.

Should be cleared up by morning.

Morning!

Oh man....

Hmm...

is there anyway I can get my idle clients folding on another server then?

(Edit: yep now I just know I have the Folding Madness

)

Zyzzyx · Post by **Zyzzyx** » Thu Feb 05, 2004 11:15 pm

Dunno. Its the assignment server that determines what work server you get. There might be a way to get around that, but not that I can figure out after a 14 hour day.

Oh, I will be glad when class is over...

haysdb · Post by **haysdb** » Thu Feb 05, 2004 11:19 pm

Stevo, I just posted your log over at folding-community.org, in their Data Server Status forum. Here is the link:

http://forum.folding-community.org/viewtopic.php?t=7022

I took the liberty of doing it for you since it will take you a bit to sign up, get your password and such. Sometimes they don't know the servers are having problems until someone tells them.

I can 'ping' the server, so it's "there".

David

haysdb · Post by **haysdb** » Thu Feb 05, 2004 11:46 pm

Stevo,

If you haven't already, try stopping and restarting F@H on those machines and see if they try a different server. I'm not sure it works that way, but it's worth a try.

Some WU's may remain in the queue, but the results will eventually get sent to Stanford. I think F@H only tries to send results every 6 hours. The real problem is not getting new work and therefore having cpu's sitting idle.

That server (.151) is staying busy according to the stats. There is no indication that it's down or not accepting connections. Net Load is 21, which means it's moderately heavily loaded. CPU load is 1.66 and 1.55, which are reasonable numbers, not abnormally high or low.

David

Wrah · Post by **Wrah** » Fri Feb 06, 2004 4:07 am

Statuscode 500 means "Internal Error", "The server encountered an unexpected condition which prevented it from fulfilling the request."

It looks like a problem over at Stanfords. However fah should try another server after a few attempts. Can you check a log to see if it has tried to contact other assignment servers?

You can also try restarting the clients without the -advmethods parameter. It might try to get a Tinker WU then and this might make it connect to a different server. Better Tinkers then nothing at all.

(Don't forget to put the -advmethods back later on!

)

Wrah · Post by **Wrah** » Fri Feb 06, 2004 4:54 am

And perhaps Miker can shed some light on this problem, he has some experience

:

http://forum.folding-community.org/viewtopic.php?t=3948

Wrah · Post by **Wrah** » Fri Feb 06, 2004 9:21 am

I see in the stats that your computers have resumed folding, so I guess the problem has disappeared.

dukla2000 · Post by **dukla2000** » Fri Feb 06, 2004 9:51 am

Are you OK now?

My thoughts that your router/firewall was blocking (especially if you changed something). The clients will automatically try a few different servers for new work.

dasman · Post by **dasman** » Fri Feb 06, 2004 9:55 am

FWIW, I had the same problem with a few boxes yesterday -- problem was auto-magically taken care of and I'm not having problems any more.

Dave

ArtCubed · Post by **ArtCubed** » Fri Feb 06, 2004 11:52 am

dasman wrote:FWIW, I had the same problem with a few boxes yesterday -- problem was auto-magically taken care of and I'm not having problems any more.

Dave

Had the problem yesterday also...I just kept quitting and reopening the client, and after 5 minutes, it picked up a WU. It didn't send the one I had stored for another few hours tho.

Stevo@ARM · Post by **Stevo@ARM** » Fri Feb 06, 2004 2:04 pm

Wrah wrote:I see in the stats that your computers have resumed folding, so I guess the problem has disappeared.

dukla2000 wrote:Are you OK now?

My thoughts that your router/firewall was blocking (especially if you changed something). The clients will automatically try a few different servers for new work.

ArtCubed wrote:
dasman wrote:FWIW, I had the same problem with a few boxes yesterday -- problem was auto-magically taken care of and I'm not having problems any more.

Dave
Had the problem yesterday also...I just kept quitting and reopening the client, and after 5 minutes, it picked up a WU. It didn't send the one I had stored for another few hours tho.

Thanks for the support guys

No, unfortunately the problem isn't fixed yet, and the numbers you saw posted throughout last night were for work units in the hopper already from other units attached to other servers. I still have 16 - 18 clients that are still not picking up new work units and have been idle for anywhere from 12 - 24 hours now.

That's a LOT of points our team could use right now.

No changes in our firewalls, both port 80 and 8080 are open. So Pat and I are going to keep trying to noodle through what's going on.

I'll keep you all posted in this thread on our progress.

Thanks,
Stevo

-------------------
Well.... I'm trying to Fold... Fold... Fold...

Stevo@ARM · Post by **Stevo@ARM** » Fri Feb 06, 2004 7:55 pm

Strange stuff guys

None of our clients under our Member name are connecting for new work units and all are still getting the (status 500 error). All the point gains you guys have been seeing from ARM Systems were pent up points. As each client completed it's work unit that unit would get submitted and then not be re-assigned any additional work. Now where is Sherlock Holmes when I need him

So here comes part of the mystery, if I take that very same computer, change only the Member name in the config file to something else, the client connects right up and grabs a work unit. (yes, all other command line parameters are the same)

So, I close that client down again, change the name back to ours, and whammo! Get the same 500 error and no ability to connect.

My theory is that maybe Stanford shortened the allowable name length or changed the rules of what a valid name can be and somehow our long and wacky member name

ARM_Systems:_Home_of_the_StealthPC

Is now no longer being accepted by their assignment servers as valid. Any ideas? Maybe some of you Veteran Folders can tell me who to contact at Stanford, or anything to get past this road block? (EDIT: continue reading more posts in this thread for the answer to the Error 500 WU Killer Mystery....)

Who knows, maybe the KWSN are behind this in some sort of electronic siege to block ARM from from getting our points through

Stevo
---------------------
Wishing I was Folding.... Error 500... Cannot Connect....

haysdb · Post by **haysdb** » Fri Feb 06, 2004 8:09 pm

Stevo, it's a suboptimal solution, but why don't you go with a shortened name for now, so you can get those machines working again, and then arm-wrestle with Stanford over the name?

David_Hays will work.

David

Stevo@ARM · Post by **Stevo@ARM** » Fri Feb 06, 2004 9:18 pm

haysdb wrote:Stevo, it's a suboptimal solution, but why don't you go with a shortened name for now, so you can get those machines working again, and then arm-wrestle with Stanford over the name?

David_Hays will work.

David

Hi David, more clues to solving the mystery of The Error 500 WU Killer! New evidence has come to light that rules out any name changes guidelines from Stanford as the WU killer... (insert dramatic theme music here).

I checked the post you made for us on Folding-community.org and noticed that halo there had posted a response. His clients running -advmethods were not picking up new WU either and he was getting error 502.

Based upon that clue, I reactivated some of my windows CLI clients without the -advmethods argument, but most importantly, with our correct Member name. And Viola! I picked up a work unit

but wait... its a... Tinker!

Well, it happens I guess, so I fire up a total of 20 clients the same way, and what do I get? 20 more Tinkers!

Also to further the plot of the mystery, my Windows GUI clients refused to pick up new WU even with removing -advmethods. However, the very same machine will get a work unit with the CLI DOS windowed client. Oh well, it may be a very long slog then, but at least ARM's clients are Folding again.

So that leads me to wonder, are the -advmethods projects currently out of new WU to assign, or do the servers RATIO out which units are assigned to which Member names based upon how many of each type are worked on?

Any thoughts? Our systems can crunch on Tinkers if that's all I'm going to get, but I just want to make sure our Folders are generating the most points possible for team SPCR

Stevo
----------------
Currently folding-away exclusively in TINKER-Land

ArtCubed · Post by **ArtCubed** » Fri Feb 06, 2004 9:38 pm

No relation as far as I can tell...I've picked up a few Tinkers, but not too many. And I'm also having connection problems on everything but the Macs, so you're definately not alone.

Threads on folding-community.org on the problem:
http://forum.folding-community.org/viewtopic.php?t=7032
http://forum.folding-community.org/viewtopic.php?t=6971

haysdb · Post by **haysdb** » Fri Feb 06, 2004 10:48 pm

Stevo,

Perhaps .151 doesn't like the long name, but some of the other servers don't mind? Removing -advmethods likely caused you to be assigned to a different server.

Have you posed the name question over at folding-community.org ?

I'm going to do a bit of research and experimentation of my own.

David

haysdb · Post by **haysdb** » Fri Feb 06, 2004 11:17 pm

The LENGTH of the name is not the problem:

Code: Select all

ARM_Systems:_Home_of_the_StealthPC
University_of_Alabama_Computer_Science

My bet is on the ":". The rules for choosing a username at the bottom of the download page just say to use letters, numbers, and the underscore, but some other characters are obviously ok too, including: []()-?@!/=*+&|.

Only three special characters are specifically prohibited:

We strongly recommendation to stick to just letters, numbers and underscore. Right now, we reserve the characters # ^ ~ |. # is used for firewall differentiation (see above). We want to save ^ | and ~ for other problems which might come up. Also, don't put spaces in your username; please use some character like "_" instead.

No specific mention of : anywhere yet.

Edit: I have found some names containing :

Code: Select all

Walter:-)
Vrykyl's_Girlfriend:_Anneliese
http://rich.2y.net
Christophe_FENDER_(France:Strasbourg)
[Impact]_..::SkyzO::..

It's concievable that none of these users use -advmethods, so it doesn't prove anything conclusive.

David

Stevo@ARM · Post by **Stevo@ARM** » Sat Feb 07, 2004 1:45 am

haysdb wrote:The LENGTH of the name is not the problem:
Code: Select all
ARM_Systems:_Home_of_the_StealthPC
University_of_Alabama_Computer_Science
David

Thanks David! Your detailed research gives me peace of mind.

Whew! I really didn't want to have to start totaling points for ARM all over again. Well, chalk this bit of Folding-Panic up to me still being a Folding-Newbie

Anyway we lost about 24 hours worth of about 25 of our folding clients, but I have them back in action now

I really need to get my network-wide folding management tools installed and learn how to use them. I'm hoping I'll be able to better manage the collective here then. Plus I'll get more attuned to the various pitfalls of hardcore folding as I get more experience. So with the support of the team, and some Mental-Folding of my own

I'll figure these things out.

Actually I was most rattled during this first setback because I didn't want to slowdown the excellent momentum the SPCR team is building now. From what I can see, it seems everyone has been pouring on the steam and we've even gained some more new team members too. I tell ya guys, it is a blast watching us catch the MacOS X guys and putting us on track down the road to take down the Knights.

So all you guys who quit Folding for the team better get back into the game now cause SPCR has got the 'Big Mo' and you might as well get back on the train so you can completely enjoy the conquests up ahead. We have just begun to fight the good fight here at ARM for team SPCR and you can rest assured those teams on the conquest radar are Goin' Down

Warning! Warning! - Beware of Folding Trash Talk Up Ahead - Terminating Maturity Threads... Initializing Adolescent-Humor language sub-routines...

Oh yeah... uh-huh, uh-huh, uh-huh, don't ya just know it y'all, we still haven't brought all of our Folding-Firepower online yet

Yep, our crew hasn't even started going through our older Athlon and P3 motherboards from our spare parts pile to start building our Blade Farm yet.

Ahhhh, yessss - I can smell something cooking now... What's that?... Ohhh eegggsellent... It's Rabbit Stew roasting nicely over some burning shrubberies...

Terminating Adolescent-Humor Threads... Re-Initializing Maturity Sub-Routines....

ur... um... eh...

You still wanted some more trash talk right David? 'Cause with the Knights average mental maturity being about age 12 the tauntings above just had to be translated into their native KWSN tongue

Stevo
------------------------
Powering up the WARP Folding Drives --- Engage!

haysdb · Post by **haysdb** » Sat Feb 07, 2004 3:26 am

Stevo,

F@H LogStats will be the easiest way to monitor your bazillion clients. Adding a new clients is simple because the configuration is all in one "file", so you can cut-paste and edit the names and such very easily. Perhaps my favorite feature is it will tell you at a glance your current PPD and PPW based on current work units.

Its major shortcoming is that it doesn't give you any indication when a client is not running. However, you would see the problem you had yesterday, where 20 machines were waiting for work. All the progress bars would show "empty" and you'd know something was up.

I'm still "concerned" with why you were getting the status 500 errors. It may be an issue with the server software, but the chances are "somewhere between slim and none" that any change would be made on your behalf, so you may yet be forced to change your folding name, IF indeed that is the problem with server 151.

I do have another trick up my sleeve, a way to "hide" one or more of the Stanford servers. It's considered boarderline "unethical" since it has the potential to "rig the game" in favor of certain servers which may be giving out more "desireable" work units, but if server 151 just won't cooperate with you, and another server will, then it might be worth investigating. But I am getting ahead of myself, since the problem with 151 is still unknown, and it has NOT been proven that any other server will dispense Gromacs to you either. Of course, you WERE getting Gromacs, so it's still a question of "what changed?"

David

haysdb · Post by **haysdb** » Sat Feb 07, 2004 3:40 am

OMG Stevo, 1550 points yesterday!

ARM Systems is already at 76th and y'all will be in the top 50 in just 5 more days.

Your "graph" is going to look really interesting, with a relatively "down" day today and then OMG on Sunday and Monday when all those Tinkers get turned in. 20 Tinkers x 70.9 points = 1418 points.

Oh, and aren't our Mac friends going to be pleasantly surprised?

David

haysdb · Post by **haysdb** » Mon Feb 09, 2004 2:09 am

What is SUPPOSED to happen when you can't get work from one server, is that after so many tries, it should try a different server. The following is snipped from one of my own logs. It tries to connect six times to .149, but then tries .150 and is able to get work.

Something I wonder about: I tend to think of Gramacs as the "advanced methods," but on the server status page, in the PRO column, there is gah, fah, adv, and bet. Does fah mean Tinkers only, or does fah mean Tinkers and Gromacs? And if the later, what does "adv" mean? On Friday when I looked, only two servers were "serving" adv, with .151 being one of them. Now there are several. I'm just wondering why Stevo's machines didn't try any other server than .151?

Code: Select all

[00:41:12] + Attempting to get work packet
[00:41:12] - Connecting to assignment server
[00:41:12] Connecting to http://assign.stanford.edu:8080/
[00:41:12] Initial: 43AB; - Successful: assigned to (171.67.89.149).
[00:41:12] Loaded queue successfully.
[00:41:12] Connecting to http://171.67.89.149:8080/
[00:41:13] Initial: 0000; + Could not connect to Work Server
[00:41:13] - Error: Attempt #1  to get work failed, and no other work to do.
             Waiting before retry.

<snip>

[00:45:34] + Attempting to get work packet
[00:45:34] - Connecting to assignment server
[00:45:34] Connecting to http://assign.stanford.edu:8080/
[00:46:00] Initial: 43AB; - Successful: assigned to (171.67.89.149).
[00:46:00] Loaded queue successfully.
[00:46:00] Connecting to http://171.67.89.149:8080/
[00:46:00] Initial: 0000; + Could not connect to Work Server
[00:46:00] - Error: Attempt #6  to get work failed, and no other work to do.
             Waiting before retry.
[00:48:41] + Attempting to get work packet
[00:48:41] - Connecting to assignment server
[00:48:41] Connecting to http://assign.stanford.edu:8080/
[00:49:06] Initial: 43AB; - Successful: assigned to (171.67.89.150).  <-- Different server
[00:49:06] Loaded queue successfully.
[00:49:06] Connecting to http://171.67.89.150:8080/
[00:49:11] Initial: 0000; - Receiving payload (expected size: 376325)
[00:49:12] - Downloaded at ~367 kB/s
[00:49:12] - Averaged speed for that direction ~287 kB/s
[00:49:12] + Received work.

David

Wrah · Post by **Wrah** » Mon Feb 09, 2004 3:44 am

haysdb wrote:I'm just wondering why Stevo's machines didn't try any other server than .151?

We don't know if they did or didn't.

I did ask.

It can also look more like this:

Code: Select all

[18:08:56] + Attempting to get work packet
[18:08:56] - Connecting to assignment server
[18:08:56] - Couldn't send HTTP request to server
[18:08:56]   (Got status 502)
[18:08:56] + Could not connect to Assignment Server
[18:08:56] - Couldn't send HTTP request to server
[18:08:56]   (Got status 502)
[18:08:56] + Could not connect to Assignment Server 2
[18:08:56] + Couldn't get work instructions.
[18:08:56] - Error: Attempt #3  to get work failed, and no other work to do.
             Waiting before retry.

No IPs, just Assignment Server and Assignment Server 2. This is what you get if you can't connect at all to Stanford's servers. So this is clearly not the case here.

Wrah · Post by **Wrah** » Mon Feb 09, 2004 5:32 am

Another idea to try out: connect one of your pc's via a regular modem to the internet and use a different internet provider. If you still can't get gromacs then the problem is fah or the stanfords servers. If you can get gromacs now then the problem lies either at how you connect your LAN to the internet (proxys/firewalls/servers), or your ISP. You can test this further by dialing in at your regular ISP, if possible.

Stevo@ARM · Post by **Stevo@ARM** » Mon Feb 09, 2004 5:14 pm

Wrah wrote:Another idea to try out: connect one of your pc's via a regular modem to the internet and use a different internet provider. If you still can't get gromacs then the problem is fah or the stanfords servers. If you can get gromacs now then the problem lies either at how you connect your LAN to the internet (proxys/firewalls/servers), or your ISP. You can test this further by dialing in at your regular ISP, if possible.

Thanks Wrah, we thought about that too, and I did test this theory out by taking my Centrino laptop home to see if I could get a WU there. This is one of our many clients that could not connect here at ARM Systems to get a new work unit. At home, it also failed to get a new work unit on my DSL line. However, also just like here at the office I killed the Windows GUI client and fired up a fresh DOS CLI without -advmethods and bingo I picked up a Tinker work unit instantly.

I am hoping that whatever was just changed at Stanford now will help prevent this from happening in the future. In any event we will keep on working on tuning up the process here so we can post better numbers for the team. I tell you, I am getting more and more worked up at seeing Team MacOS X, which is just within our grasp and somehow has ever so slowly managed to inch a little further ahead. That just frost's my eyeballs

Stevo
---------------------
More CPU's on the Folding Fire - It's not hot enough yet!

mas92264 · Post by **mas92264** » Mon Feb 09, 2004 5:39 pm

Stevo@ARM wrote:I tell you, I am getting more and more worked up at seeing Team MacOS X, which is just within our grasp and somehow has ever so slowly managed to inch a little further ahead. That just frost's my eyeballs

That makes 2 of us!

Hey, you guys have done a heck of a job so far; keep up the good work!

Not sure if this will help much, but, the only time I've had trouble sending/getting work was when I had an internal network problem that clogged up communication with the outside world. For me, it was obvious tho, as I had a locked up folding computer.

Weird deal with your folders. Sorry that I can't help any more than this.

M

Beyonder · Post by **Beyonder** » Mon Feb 09, 2004 6:10 pm

FYI - I had some weird connection issue with my work computer (P4 2.66), and it couldn't get another workload. It was finished. After about six hours and me trying hourly, I managed to pick up another wu. Seems like maybe their servers are just spotty or something like that....