Advanced search

Message boards : Multicore CPUs : File uploads / downloads

Author Message
Profile Krunchin-Keith [USA]
Avatar
Send message
Joined: 17 May 07
Posts: 512
Credit: 111,288,061
RAC: 0
Level
Cys
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 270 - Posted: 30 Jul 2007 | 0:42:25 UTC

I made a change in my cc_config.xml file which seems to improve file uploads.

I set the max to 1 at a time

Previously when I had it at the default of 2 per project it took about 12-13 minutes for the largest file to complete, while the shorter ones completed during that time.

With setting it to 1 max, all the files went through in about 9 minutes total.

Host Project ID Date Message
PS301 PS3GRID 23 7/29/2007 8:23:13 PM [file_xfer] Started upload of file Ny23493-WF5-3-nodelete_0_0
PS301 PS3GRID 67 7/29/2007 8:30:48 PM [file_xfer] Finished upload of file Ny23493-WF5-3-nodelete_0_0
PS301 PS3GRID 68 7/29/2007 8:30:48 PM [file_xfer] Throughput 54063 bytes/sec
PS301 PS3GRID 69 7/29/2007 8:30:48 PM [file_xfer] Started upload of file Ny23493-WF5-3-nodelete_0_1
PS301 PS3GRID 70 7/29/2007 8:31:06 PM [file_xfer] Finished upload of file Ny23493-WF5-3-nodelete_0_1
PS301 PS3GRID 71 7/29/2007 8:31:06 PM [file_xfer] Throughput 51779 bytes/sec
PS301 PS3GRID 72 7/29/2007 8:31:06 PM [file_xfer] Started upload of file Ny23493-WF5-3-nodelete_0_2
PS301 PS3GRID 73 7/29/2007 8:31:24 PM [file_xfer] Finished upload of file Ny23493-WF5-3-nodelete_0_2
PS301 PS3GRID 74 7/29/2007 8:31:24 PM [file_xfer] Throughput 51067 bytes/sec
PS301 PS3GRID 75 7/29/2007 8:31:24 PM [file_xfer] Started upload of file Ny23493-WF5-3-nodelete_0_3
PS301 PS3GRID 76 7/29/2007 8:31:27 PM [file_xfer] Finished upload of file Ny23493-WF5-3-nodelete_0_3
PS301 PS3GRID 77 7/29/2007 8:31:27 PM [file_xfer] Throughput 1385 bytes/sec
PS301 PS3GRID 78 7/29/2007 8:31:27 PM [file_xfer] Started upload of file Ny23493-WF5-3-nodelete_0_4
PS301 PS3GRID 79 7/29/2007 8:31:50 PM [file_xfer] Finished upload of file Ny23493-WF5-3-nodelete_0_4
PS301 PS3GRID 80 7/29/2007 8:31:50 PM [file_xfer] Throughput 52684 bytes/sec

Lines I added to cc_config.xml
<max_file_xfers>1</max_file_xfers>
<max_file_xfers_per_project>1</max_file_xfers_per_project>

these go in the between the <options> tags

____________
Alpha Tester ~~ BOINCin since 10-Apr-2004 (2.28) ~~~ Join team USA

Profile [AF>HFR>RR] Jim PROFIT
Send message
Joined: 3 Jun 07
Posts: 107
Credit: 31,331,137
RAC: 0
Level
Val
Scientific publications
watwatwatwatwatwat
Message 271 - Posted: 30 Jul 2007 | 12:24:57 UTC
Last modified: 30 Jul 2007 | 12:27:58 UTC

For my last WU, result files for upload are very huge:

>20 Mo
This is this one http://www.ps3grid.net/result.php?resultid=6171

Is this normal?

And why i can\'t have more than 1 WU crunching, and no one in my waiting task?

I check other users, and they have one WU crunching, and another waiting for calculations.



Profile GDF
Volunteer moderator
Project administrator
Project developer
Project tester
Volunteer developer
Volunteer tester
Project scientist
Send message
Joined: 14 Mar 07
Posts: 1957
Credit: 629,356
RAC: 0
Level
Gly
Scientific publications
watwatwatwatwat
Message 272 - Posted: 30 Jul 2007 | 13:08:01 UTC - in response to Message 271.

For my last WU, result files for upload are very huge:

>20 Mo
This is this one http://www.ps3grid.net/result.php?resultid=6171

Is this normal?

And why i can\'t have more than 1 WU crunching, and no one in my waiting task?

I check other users, and they have one WU crunching, and another waiting for calculations.





Hi Jim,
file size is ok. It has been like this for a while now. The app needs to return the MD trajectories which I have tried to keep at the minimum size. Also, files upload are concurrent with calculations, so no time is lost.

Because WUs are chained in a workflow the server maintains a limit of 2 per host such that there is a sufficient buffer but they return as quickly as possible.

You have only one because there are few wus left in this batch of runs. Every wu finished creates another one and you have been unlucky so far.

I am preparing a new large set of runs. It should be out after testing in a few days.

g

Profile [AF>HFR>RR] Jim PROFIT
Send message
Joined: 3 Jun 07
Posts: 107
Credit: 31,331,137
RAC: 0
Level
Val
Scientific publications
watwatwatwatwatwat
Message 273 - Posted: 30 Jul 2007 | 16:33:12 UTC - in response to Message 272.

For my last WU, result files for upload are very huge:

>20 Mo
This is this one http://www.ps3grid.net/result.php?resultid=6171

Is this normal?

And why i can\'t have more than 1 WU crunching, and no one in my waiting task?

I check other users, and they have one WU crunching, and another waiting for calculations.





Hi Jim,
file size is ok. It has been like this for a while now. The app needs to return the MD trajectories which I have tried to keep at the minimum size. Also, files upload are concurrent with calculations, so no time is lost.

Because WUs are chained in a workflow the server maintains a limit of 2 per host such that there is a sufficient buffer but they return as quickly as possible.

You have only one because there are few wus left in this batch of runs. Every wu finished creates another one and you have been unlucky so far.

I am preparing a new large set of runs. It should be out after testing in a few days.

g


Ok, i just notice the size yesterday, when i saw that i don\'t have another WU to crunch, and saw the WU finished and then uploading.

And yes i was not lucky yesterday, no WU before the end of the morning.

And may the force be with you until September!

Profile Krunchin-Keith [USA]
Avatar
Send message
Joined: 17 May 07
Posts: 512
Credit: 111,288,061
RAC: 0
Level
Cys
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 350 - Posted: 3 Sep 2007 | 19:05:56 UTC

I had a download problem today.
It started about 7 hours ago, I just noticed it.
My PS3 had no work except for one it was trying to download for the past 7 hours.
It had 3 files named input.pdb, input.psf and input.txt. Log is showing temporarily failed, http error every time it tried these downloads.
I exited the core client and restarted but this did not help.
I finally just aborted the task and the downloads.
Then did update and got another task, which downloaded without problem, this one downloaded 10 files, but none of them were named input.anything

Heres the wu for the bad one, maybe there is something wrong with it ?
http://www.ps3grid.net/PS3GRID/workunit.php?wuid=5515

Looking back in the log the problem actually started before that another 7 hours, it actually shows at 1:53AM local time that 2 files, named input.coor and input.vel downloaded OK, the next file input.pdb, then input.psf and then input.txt did not download and show the http error.

Log in reverse order:
Host Project ID Date Message
PS301 PS3GRID 2799 9/3/2007 1:53:42 AM Backing off 1 min 0 sec on download of file input.txt
PS301 PS3GRID 2798 9/3/2007 1:53:42 AM [file_xfer] Temporarily failed download of input.txt: http error
PS301 PS3GRID 2797 9/3/2007 1:53:39 AM [file_xfer] Started download of file input.txt
PS301 PS3GRID 2796 9/3/2007 1:53:39 AM Backing off 1 min 0 sec on download of file input.psf
PS301 PS3GRID 2795 9/3/2007 1:53:39 AM [file_xfer] Temporarily failed download of input.psf: http error
PS301 PS3GRID 2794 9/3/2007 1:53:35 AM [file_xfer] Started download of file input.psf
PS301 PS3GRID 2793 9/3/2007 1:53:35 AM Backing off 1 min 0 sec on download of file input.pdb
PS301 PS3GRID 2792 9/3/2007 1:53:35 AM [file_xfer] Temporarily failed download of input.pdb: http error
PS301 PS3GRID 2791 9/3/2007 1:53:33 AM [file_xfer] Started download of file input.pdb
PS301 PS3GRID 2790 9/3/2007 1:53:33 AM [file_xfer] Throughput 148013 bytes/sec
PS301 PS3GRID 2789 9/3/2007 1:53:33 AM [file_xfer] Finished download of file input.vel
PS301 PS3GRID 2788 9/3/2007 1:53:27 AM [file_xfer] Started download of file input.vel
PS301 PS3GRID 2787 9/3/2007 1:53:27 AM [file_xfer] Throughput 122640 bytes/sec
PS301 PS3GRID 2786 9/3/2007 1:53:27 AM [file_xfer] Finished download of file input.coor
PS301 PS3GRID 2785 9/3/2007 1:53:19 AM [file_xfer] Started download of file input.coor




Profile GDF
Volunteer moderator
Project administrator
Project developer
Project tester
Volunteer developer
Volunteer tester
Project scientist
Send message
Joined: 14 Mar 07
Posts: 1957
Credit: 629,356
RAC: 0
Level
Gly
Scientific publications
watwatwatwatwat
Message 351 - Posted: 4 Sep 2007 | 7:42:03 UTC - in response to Message 350.

Yes, this is a test wu which did not work.
We are working hard for the next big step of PS3GRID.

zombie67 [MM]
Avatar
Send message
Joined: 16 Jul 07
Posts: 207
Credit: 1,666,901,456
RAC: 4,301,034
Level
His
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 352 - Posted: 8 Sep 2007 | 22:51:56 UTC

I also had a WU with this download problem two days ago. I aborted the transfer files, and it successfully downloaded another WU. Just FYI.

____________
Reno, NV
Team: SETI.USA

Profile Bender10
Avatar
Send message
Joined: 3 Dec 07
Posts: 167
Credit: 8,368,897
RAC: 0
Level
Ser
Scientific publications
watwatwatwatwatwatwat
Message 1041 - Posted: 31 Mar 2008 | 21:13:48 UTC - in response to Message 270.

Where do I find this cc_config file? It does not appear in my BOINC directory....


I made a change in my cc_config.xml file which seems to improve file uploads.

I set the max to 1 at a time

Previously when I had it at the default of 2 per project it took about 12-13 minutes for the largest file to complete, while the shorter ones completed during that time.

With setting it to 1 max, all the files went through in about 9 minutes total.

Host Project ID Date Message
PS301 PS3GRID 23 7/29/2007 8:23:13 PM [file_xfer] Started upload of file Ny23493-WF5-3-nodelete_0_0
PS301 PS3GRID 67 7/29/2007 8:30:48 PM [file_xfer] Finished upload of file Ny23493-WF5-3-nodelete_0_0
PS301 PS3GRID 68 7/29/2007 8:30:48 PM [file_xfer] Throughput 54063 bytes/sec
PS301 PS3GRID 69 7/29/2007 8:30:48 PM [file_xfer] Started upload of file Ny23493-WF5-3-nodelete_0_1
PS301 PS3GRID 70 7/29/2007 8:31:06 PM [file_xfer] Finished upload of file Ny23493-WF5-3-nodelete_0_1
PS301 PS3GRID 71 7/29/2007 8:31:06 PM [file_xfer] Throughput 51779 bytes/sec
PS301 PS3GRID 72 7/29/2007 8:31:06 PM [file_xfer] Started upload of file Ny23493-WF5-3-nodelete_0_2
PS301 PS3GRID 73 7/29/2007 8:31:24 PM [file_xfer] Finished upload of file Ny23493-WF5-3-nodelete_0_2
PS301 PS3GRID 74 7/29/2007 8:31:24 PM [file_xfer] Throughput 51067 bytes/sec
PS301 PS3GRID 75 7/29/2007 8:31:24 PM [file_xfer] Started upload of file Ny23493-WF5-3-nodelete_0_3
PS301 PS3GRID 76 7/29/2007 8:31:27 PM [file_xfer] Finished upload of file Ny23493-WF5-3-nodelete_0_3
PS301 PS3GRID 77 7/29/2007 8:31:27 PM [file_xfer] Throughput 1385 bytes/sec
PS301 PS3GRID 78 7/29/2007 8:31:27 PM [file_xfer] Started upload of file Ny23493-WF5-3-nodelete_0_4
PS301 PS3GRID 79 7/29/2007 8:31:50 PM [file_xfer] Finished upload of file Ny23493-WF5-3-nodelete_0_4
PS301 PS3GRID 80 7/29/2007 8:31:50 PM [file_xfer] Throughput 52684 bytes/sec

Lines I added to cc_config.xml
<max_file_xfers>1</max_file_xfers>
<max_file_xfers_per_project>1</max_file_xfers_per_project>

these go in the between the <options> tags


____________


Consciousness: That annoying time between naps......

Experience is a wonderful thing: it enables you to recognize a mistake every time you repeat it.

Profile Bender10
Avatar
Send message
Joined: 3 Dec 07
Posts: 167
Credit: 8,368,897
RAC: 0
Level
Ser
Scientific publications
watwatwatwatwatwatwat
Message 1043 - Posted: 1 Apr 2008 | 0:31:31 UTC


Never mind.

____________


Consciousness: That annoying time between naps......

Experience is a wonderful thing: it enables you to recognize a mistake every time you repeat it.

Post to thread

Message boards : Multicore CPUs : File uploads / downloads

//