I posted this on the Alpha mailing list, those that have 4 and 8 core systems may want to consider how your systems handle workloads in light of these observations ... if you are only running one or two projects these notes may not apply ... comments, as always, welcome ...

One of my long standing complaints about BOINC is with the CPU scheduler and the fact that the more CPUs you have the less optimal the choices seem to be as far as scheduling work. The CPU scheduler seems to be modeled and tested primarily on single CPU systems with occasional duals. Most of the problems I have noted are usually only readily apparent when you look for them on 4 CPU systems or better. My first trip down this rabbit hole was over 4 years ago when I had one of the first 4 CPU systems and JM VII was lead on the development of the CPU scheduler.

To date little has changed and perhaps I am one of the few to note these issues, but that does not mean that they do not exist.

For example, I installed 6.6.20 and noted in prior posts that I had a long standing count of "waiting" tasks to run and I thought that it was unusual. For other reasons I down-leveled to 6.5.0 and less than 8 hours after doing that I have only three ...

More troublesome is the question of 6.6.20 not scheduling correctly to the point that it actually causes GPU Grid tasks on multi-GPU systems to take excessive amounts of time to complete. Yet the actual run times may only reflect the "normal" 6 hour run time. I can't recall if I posted about that here ... and I still have pending inquiries on GPU Grid, but, since the down level I seem to be back to executing normally. With 6.6.20 I was seeing at least two and as many as all 4 tasks taking 24 or more hours to process, yet, as I said, only logging 6 hours in the internal time. Since time accounting is still not properly recorded and validated I do not know what to trust though clock watching does tell me that there is a problem ... so does the drastic reduction in throughput (cut in half) ... (Also for the first time I noted GPU tasks being suspended, though none of them were actually in deadline trouble) ...

My first posts here indicated that I felt that 6.6.20 was not honoring the task switch interval ... but I think that the major conceptual flaw is almost simpler than that, (though there still may be issues with the TSI mechanism).

The CPU scheduler is really based on a single CPU (or GPU) model with the assumption that what works for a single processing stream will scale to multiple streams. This has not been the case, though those of us that have brought it up in the past (usually by pointing out issues that arise as a consequence of this design deficiency; such as large numbers of "waiting" tasks or large memory footprints, or tasks failing because of being suspended or halted too many times, etc.) have largely been ignored.

Lets talk cash register lines... in the old days when 1 register was open there was one line, when 4 were open, there were 4 lines ... and if you got in the "wrong" line you got screwed by the person counting out $2,259.69 ... all in change, mostly pennies ...

Modern stores have one line that feeds the "bank" of registers and thus the one slow person does not impact those behind that one slow person... because they have access to all the other open registers...

BOINC Should work the same way, but does not. When it recalculates the work schedule it does so even for those tasks that are being processed. And, as often as this calculation is done, the old execution plan is discarded.

What SHOULD happen is that the queue should be examined and scheduling within that queue should be re-ordered if needs be, but the currently executing tasks should not be abandoned or suspended willy-nilly ...

One or more tasks actually being processed should be interrupted only when there is an emergency or the project's time slice is filled. Just as we would not pull a person out of line that was already being rung up at the cash register we should not stop processing an executing task.

And this, is why I think that I was seeing problems with 6.6.20 (I probably am one of the few that already have 4 GPU cores in one system) in that BOINC was changing its mind so often that the tasks were not being allowed to run ... I posted a log a few post ago here and the clue may be in there ...

Last note, if the task in potential deadline trouble can be satisfied by being run after the completion of the next task to be suspended because of switching or completion then that task still should not pre-empt running tasks.

I think you are being a bit vague about the problems you are experiencing due to this scheduling issues. It's only a guess that the recent GPU-Grid issue is caused by the BOINC scheduler.

Regarding the many WUs in flight: you're probably right that this is not ideal. However, they may just say "well, just increase the time between app switches or switch off 'leave in memory'".
One problem I see with your approach: very long tasks like CPDN could throw the system off balance, i.e. run for far too long. This could be stopped, though, as soon as a certain amount of debt has built up.

MrS

Now I have a better understanding of what you mean. And I agree, with 6.5.0 or previous versions I have not seen a suspended GPU task either. And yes, it doesn't make any sense to suspend a GPU-Grid task (if no other CUDA project is present).

And I also leave apps in memory, mainly so that I don't waste computation time. But I don't have as many projects as you do, so this is not as much of an issue for me.

Now I have a better understanding of what you mean. And I agree, with 6.5.0 or previous versions I have not seen a suspended GPU task either. And yes, it doesn't make any sense to suspend a GPU-Grid task (if no other CUDA project is present).

And I also leave apps in memory, mainly so that I don't waste computation time. But I don't have as many projects as you do, so this is not as much of an issue for me.

I'm running v6.6.20 on 7 machines (5 of which are quads) and have seen no problem with suspending tasks excessively. I have seen a GPUGRID task suspended in favor of running a new one but that's because the newly downloaded task had an earlier due date than the one already running. It seems due dates have been shortened. That's a choice by the GPUGRID admins and doesn't reflect a BOINC problem.

6.6.20 has other issues.

it stopped downloading milkyway and lattice cpu workunits almost entirely reporting that no cpu tasks were selected but cpu tasks were availible. even on my crappy laptop with no cuda gpu.

it did seem to handle gpugrid and seti on the same machine better than 6.5.0 for me at least. and i never saw the scheduler swap apps every 3 seconds like i get with 6.5.0 occasionally.

ive reverted back to 6.5.0. hopefully the next release is a little better.

Paul, could you switch the i7 back to 6.6.20 and see if you can get the hanging tasks again? If you could observe this we'd know there's a serious problem. Otherwise we can only speculate so much.

Well, I jut got done making a 800K+ log file to demonstrate the poor performance of the CPU scheduler in electing which tasks to run and noting that on that system I have seen it change its mind in less than 30 seconds as to what is desperately needed to be done ...

Well, I jut got done making a 800K+ log file to demonstrate the poor performance of the CPU scheduler in electing which tasks to run and noting that on that system I have seen it change its mind in less than 30 seconds as to what is desperately needed to be done ...

i don't think thats limited to 6.6.20. when i downgraded to 6.5.0 my cpu switched projects about 1 a sec for a couple of minutes till i suspended everything but one cpu project. turning them back on 1 at a time let everything resume normally.

Paul,

The behavior you describe... sheesh, it's so correct I've always thought that was the *intended* behavior. BOINC has always behaved like that, as you say.

From what I can tell, BOINC re-calculates its short term debts (and its work scheduling) whenever various events occur, and a lot of things that it does incur these recalculations.

Under lots of circumstances, it also appears to override the 60 minute run-time preference (or 720 in your configuration).

This is one of the reasons all of my computers now remove tasks from memory when they are suspended. It's not the only reason, however: Some tasks use extremely large amounts of RAM and need to be removed if anyone is using the computer.

I've found that by letting the tasks get removed from memory that the footprint of having *many* suspended tasks is eliminated, and the overhead from reloading the tasks is actually insignificant.

It also greatly improves system latency for anyone who is using the computer.

Of course, this only works with applications that checkpoint. Then again, I consider not checkpointing to be an excellent reason to detach from a project.

Bottom line is that I have always seen this behavior in BOINC, but it's simply not an issue if you're not leaving tasks in memory. Yes, you lose some efficiency, especially with taks that have long times between checkpoints.

BTW... the WORST behavior I've observed from BOINC with unnecessary swapping is when it runs short of ram. Here's a real life example I observed recently. A single task (a huge Lattice GARLI WU) was using 1.2 gig of ram. The computer was set to limit BOINC RAM use to 1.0 gig when the computer is in user. If that GARLI is running and I start using the computer (it's a quad core), you would expect BOINC to suspend the GARLI task, right? It HAS to be removed, period, because it alone uses more ram than it is allowed.

But, of course, it doesn't work that way. BOINC decided somehow to remove a smaller task. That left a free core, so it started up a new task on that core -- from the same project as the task it just suspended!!! It's still over committed on memory, so a few seconds later (after that new task initializes), it suspends the brand new task -- and starts yet another new task from the same project.

Rinse and repeat until every single task from that project moved from the "Ready to start" status to the "Ready to Run" status with 5 to 10 seconds of run time on each task.

Eventually, of course, after trying to run nearly every unstarted non-GPU task on the system, it got around to actually suspending the 1.2 gig GARLI task that was causing the problem. But by that time I had 30+ suspended tasks.

Stupid, stupid, stupid.

BTW, the main reason I run BOINC as 'remove from memory' is to leave more ram for users. I'd rather lose credits than have my kids complaining about the computers being slow.


____________
Want to find one of the largest known primes? Try PrimeGrid. Or help cure disease at WCG.

Michael,

So far on the mailing list the reception is that BOINC is fine and I am making a big deal out of nothing. Sadly, this seems to be the first reaction anytime an issue is reported...

In essence every event that changes the state of the system in relationship to a task does it. Start an upload, recalculate, finish an upload, recalculate, finish a task, recalculate (this makes sense actually), finish a download, recalculate (this ALMOST makes sense, unless you just recalculated a minute or so ago); oh well, there are other actions that trigger ... and this is again my point that on "larger" systems these triggers come so fast that it is lunacy... on a single threaded machine yes these triggers make sense... but not when you have many cores ... I guess that on the up and coming 16 CPU machines it is going to be really insane ...

The reason I leave in memory is that many applications, though they should be "suspendible" and removable from memory don't always react well with that setting. Thus you can have tasks that are suspended and crash costing you the time.

Heck there is evidence that using the restriction on allowing 100% of computer CPU use (putting it at 98% for example) causes some tasks to abort (Einstein and Rosetta).

Have you posted this behavior on the mailing lists? Will you, if you are not interested, may I summarize it and post it?

Feel free to summarize and repost. I'm not up to leading this crusade right now.

I'm kind of tied up recovering data and migrating to a new computer -- the main disk in my primary system died yesterday.

Between recovering data, setting up raid on the new system, migrating data, installing apps, physically moving hardware, etc., I'm feeling kinda swamped right now.

Mike

____________
Want to find one of the largest known primes? Try PrimeGrid. Or help cure disease at WCG.

I started a 6.6.21 thread in that there is a version of that name available...

6.6.20 has replaced 6.4.7 as the official BOINC release.

Yes, and there are a couple potential major issues that may bite us in the butt too ...

Not to mention the annoyances I am trying to get some attention on ...