does gpugrid really need the full 16x or 8x bandwidth of pci-e 3?

or can i also build a system looking like mining rig?
what i mean using special mining motherboard with lots of 1x lane pcie-3, connecting pcie-3 x16 riser to mainboard via usb for data transfers and putting gpu-card into pcie-16x riser.
should this work?
because the limiting factor here is pci-1x lane connection on motherboard

You can certainly try it to see what GPU usage you get with SWAN_SYNC on Linux. Without SWAN_SYNC I notice about 30% PCIe usage on an 8x link, but with SWAN_SYNC on Linux I notice only about 2% usage.

It might be possible, but only with SWAN_SYNC on Linux, though I have never tried it.

I wouldn't go out and buy mining specific hardware until you have tested it out. Keep in mind you should want at least 1 thread free per GPU for science workloads. Let us know what you find!

You can certainly try it to see what GPU usage you get with SWAN_SYNC on Linux. Without SWAN_SYNC I notice about 30% PCIe usage on an 8x link, but with SWAN_SYNC on Linux I notice only about 2% usage.

It might be possible, but only with SWAN_SYNC on Linux, though I have never tried it.

I wouldn't go out and buy mining specific hardware until you have tested it out. Keep in mind you should want at least 1 thread free per GPU for science workloads. Let us know what you find!

He kind of did answer it. 30% usage at 8x. Theoretically one could run 3 GPUs across an 8x link but I'd wager there would be some performance losses at 90% utilization. 1x would not be enough in that case.

With SWAN_SYNC the tasks use a full CPU core and you'd be limited by CPU threads before PCI-E 1x lanes on a mining board.

Mining programs are small and fit in GDDR memory so there isn't a lot of traffic across the PCI-E links. GPUGrid computing requires more GDDR and CPU computations.

it is my purpose to "mine" for gpugrid -:)
no, just kidding....i do not want to mine any crypto at all.

my question was:
setting up gpu machine for only 1 purpose = gpugrid, (in a way as miners do). a lot of gpu cards in 1 case, connected to 1x lane pcie-3.
is 1 lane of pcie-3 enough to use gpugrid? or i need at least 8 lanes or 16 lanes pcie-3??

if my question or idea not clear enough, plz ask me

He kind of did answer it. 30% usage at 8x. Theoretically one could run 3 GPUs across an 8x link but I'd wager there would be some performance losses at 90% utilization. 1x would not be enough in that case.

With SWAN_SYNC the tasks use a full CPU core and you'd be limited by CPU threads before PCI-E 1x lanes on a mining board.

Mining programs are small and fit in GDDR memory so there isn't a lot of traffic across the PCI-E links. GPUGrid computing requires more GDDR and CPU computations.

it is my purpose to "mine" for gpugrid -:)
no, just kidding....i do not want to mine any crypto at all.

my question was:
setting up gpu machine for only 1 purpose = gpugrid, (in a way as miners do). a lot of gpu cards in 1 case, connected to 1x lane pcie-3.
is 1 lane of pcie-3 enough to use gpugrid? or i need at least 8 lanes or 16 lanes pcie-3??

if my question or idea not clear enough, plz ask me

I apologize if my answer was not clear enough. Basically at 8x I use 30% of the 8x bandwidth. But with SWAN_SYNC on Linux I only use 2% of the bandwidth of 8x. So I would imagine you could see less than 80% usage on 1x pcie. I have not tested this myself but I would imagine it would still work just fine with minimal loss. Keep in mind you need 1 CPU thread per GPU unlike mining which is almost 0% reliant on CPU threads.

30% on an 8x pcie is without SWAN_SYNC. With SWAN_SYNC on Linux I get 2% on 8x. As long as you use SWAN_SYNC it might be theoretically possible.

He kind of did answer it. 30% usage at 8x. Theoretically one could run 3 GPUs across an 8x link but I'd wager there would be some performance losses at 90% utilization. 1x would not be enough in that case.

With SWAN_SYNC the tasks use a full CPU core and you'd be limited by CPU threads before PCI-E 1x lanes on a mining board.

Mining programs are small and fit in GDDR memory so there isn't a lot of traffic across the PCI-E links. GPUGrid computing requires more GDDR and CPU computations.

this whole mining sh*t is making me crazy.....I am not talking about mining....I am talking about building a system like miners do.

putting a lot of nvidia gtx-cards in 1 case, connecting them to pcie-3 x1 lane. and then only running boinc for gpugrid, using windows 10 with swansync enabled.

no mining program at all

should gpugrid run OK on pcie-3 x1 lane?

He kind of did answer it. 30% usage at 8x. Theoretically one could run 3 GPUs across an 8x link but I'd wager there would be some performance losses at 90% utilization. 1x would not be enough in that case.

With SWAN_SYNC the tasks use a full CPU core and you'd be limited by CPU threads before PCI-E 1x lanes on a mining board.

Mining programs are small and fit in GDDR memory so there isn't a lot of traffic across the PCI-E links. GPUGrid computing requires more GDDR and CPU computations.

this whole mining sh*t is making me crazy.....I am not talking about mining....I am talking about building a system like miners do.

putting a lot of nvidia gtx-cards in 1 case, connecting them to pcie-3 x1 lane. and then only running boinc for gpugrid, using windows 10 with swansync enabled.

no mining program at all

should gpugrid run OK on pcie-3 x1 lane?

I was comparing how mining is different from BOINC crunching and in particular GPUGrid. How mining can get away with using just a 1x slow and how GPUGrid cannot. There ya got it. It won't work. You want a Black and white answer without any understanding, here it is. No

Just test it. It's not that hard.

In the past I have used a usb cable connected riser to a 1x pci-e slot using a GTX750ti for a few months on GPUgrid. (I no longer use it)
It was on a Linux host set to BLOCK mode.
The GPUgrid output was lower by around 15% (approx). I am guessing a faster card could suffer a larger speed reduction.
I found the GPUgrid task would randomly pause for no reason, but I could manually start the task again. (I had a script that would check the task status and start it again if necessary)
Not sure why tasks would pause, as multiple GPU cards work fine on rigs with multiple 8x/16x slots, I assumed it was just a poorly designed/implemented riser.

The people in my team at Seti who are using mining hardware to support multiple high gpu count hosts (7-12 cards) find the risers and particularly the USB cables have to be high quality and shielded to crunch without constant issues of cards dropping offline.

Many hosts are using PCIe X1 slots. But the Seti task requirement is a lot less than what either Einstein or GPUGrid need for speed and bandwidth.

Answer is try it and see if it will work.

The people in my team at Seti who are using mining hardware to support multiple high gpu count hosts (7-12 cards) find the risers and particularly the USB cables have to be high quality and shielded to crunch without constant issues of cards dropping offline.

Many hosts are using PCIe X1 slots. But the Seti task requirement is a lot less than what either Einstein or GPUGrid need for speed and bandwidth.

Answer is try it and see if it will work.

The people in my team at Seti who are using mining hardware to support multiple high gpu count hosts (7-12 cards) find the risers and particularly the USB cables have to be high quality and shielded to crunch without constant issues of cards dropping offline.

Many hosts are using PCIe X1 slots. But the Seti task requirement is a lot less than what either Einstein or GPUGrid need for speed and bandwidth.

Answer is try it and see if it will work.

thanks for your reply.

to try I firstly need to buy gtx1070 or gtx1080 cards, used ones. the price here for used cards are around 250-300 euro;s.
spending 1000 euro for 4 cards to try something is odd for me.
that is why asked here before I spend 1000 euro.
and I would by 4 gpu's just only for gpugrid, no other intended use.

so now I know it will not work because of limitation of speed of x1 lane.

thanks all who replied

Yes, any HEDT motherboard should be able to support 4 gpus natively. I have an Intel X99 board that supports 4 gpus at PCIe 3.0 X16 speeds and one X399 board that supports 2 gpus at PCIe 3.0 X8 speeds along with 2 gpus at PCIe 3.0 X16 speeds. As long as the gpus are no wider than two slots, even air cooled cards fit. Water cooling or hybrid cooled cards keep the cards cool so they clock well with the best performance.

Hybrid or custom cooling loops is best options for multi card systems. Prevent throttling of the cards if the are all in 1 box. However, if you go the route of hanging them from a support beam above the Mobo, then you could probably get away with air cooled as long as you have proper ventilation.

Keith is correct in that using a higher PCI-e bandwidth is preferable if you are looking to crunch the fastest. If not then, then yes using risers is a viable option.

As you noted in the other thread, you have to keep in mind how many threads are on the CPU, how many are available to the GPUs, are those threads going to be shared with the Sata's or M2? The more lanes you can devote to the cards, the faster they will finish the work.

Good luck

Z
____________

my idea is to build a setup with mining rig frames, like this one

https://images.app.goo.gl/F8vavF34ggADQoiC9

with asus b250 mining motherboard, intel i7-7700 or i7-7700T (4 cores, hyperthreading to 8 cores/threads), 4 - 6 gpu's (gtx1070), some fans of 120x120x38mm. and of course using pcie-x16 risers connected to motherboards x1-slots

This software is fundamentally different than mining software and requires more resources. You will need at least 1 CPU core per GPU and I highly doubt PCIe 1x is enough bandwidth to supply a fast GPU like a 1070 (PCIe 16x riser on a PCIe 1x slot is the same thing as a 1x riser on a 1x slot.) You will need minimum 4x and recommended 8x PCIe per GPU.

I have used PCIe 1x risers in the past and I can tell you they are an absolute nightmare. Most notably if the USB connection between PCIe connectors isn't perfect you have enormous difficulty getting the Operating system to recognize the GPU.

Zoltan and I have found that CPU frequency also plays a large role in GPU usage, but you should be fine with an i7-7700. You might be better off having 3-4 GPUs per system with 16x risers and having two cheap (but high frequency) systems.

Yes, any HEDT motherboard should be able to support 4 gpus natively. I have an Intel X99 board that supports 4 gpus at PCIe 3.0 X16 speeds and one X399 board that supports 2 gpus at PCIe 3.0 X8 speeds along with 2 gpus at PCIe 3.0 X16 speeds. As long as the gpus are no wider than two slots, even air cooled cards fit. Water cooling or hybrid cooled cards keep the cards cool so they clock well with the best performance.

intel x99 boards have mostly 16x/8x/4x modus.
probably you are talking about asus ws-serie mainboards when you say 4 gpu's natively x16. but do you realize about the presence of PLX pcie-switches on those ws-boards? the net result is much lesser bandwidth

so, you recommend to get a motherboard with plx-swtich? is this better than b250 mining bord with x1 lanes? do i understand you correct? a motherboard with plx will crunch faster than x1-lanes?

I'm saying, PCIe speed does make a difference. You are better off with higher than PCIe x1. What level of PCIe is up to you. X16 will process the data faster than x8 and so forth.

PLX boards are extreme examples but not necessary for this project. You can get by with lot of different boards as long as you take into account PCIe speeds and total number of lanes of the CPU.

Seti has a good example of someone using a mining rig, but the applications there have been refined over the years and the data packets are small enough that PCIe aren't a factor. This project, like Einstein perform better with larger PCIe lanes.

my idea is to build a setup with mining rig frames, like this one

https://images.app.goo.gl/F8vavF34ggADQoiC9

with asus b250 mining motherboard, intel i7-7700 or i7-7700T (4 cores, hyperthreading to 8 cores/threads), 4 - 6 gpu's (gtx1070), some fans of 120x120x38mm. and of course using pcie-x16 risers connected to motherboards x1-slots

so, you recommend to get a motherboard with plx-swtich?

is this better than b250 mining bord with x1 lanes?

do i understand you correct? a motherboard with plx will crunch faster than x1-lanes?

edit...deleted

ok, ppl. today i have been testing asus b250 mining board.
crap. those x1 ==> x16 risers are so sensitive and so crappy that i decided not to use any mining board.

so, i have decided to build my 3rd Threadripper build with threadripper 1920x cpu. but which motherboard has the best pcie-slots orientation for 3 gpu setup when you look at this site?

it is in dutch, but click on pictures for judgement of pcie-slots.

https://tweakers.net/categorie/47/moederborden/producten/#filter:TcvBCsIwEATQf5lzhCzFNuwHFDx46lE8hHSRlWBDUjxY8u9NEcTTMI-ZDUueJY8qcQYjZX0WmC9OS16b-RJ-kiRc2u5EBsk_ZNKPgMlaczyDXPUFbqW03ahxlVzAG7rBHvH2EXwDkXNn3KsB9d2_D65prTs

any advice??

I would go with either of the Asrock boards on that site. The Taichi has four X16 PCIe slots so you could fit four gpu cards. The Pro Gaming 6 satisfies your minimum 3 gpu requirement.

I have the Asrock X399 Fatality Professional Gaming motherboard with a 2920X and I really like it. There are some sweet deals on the 1950X now as AMD tries to reduce inventory in advance of Zen 2. Surprised you can even find a 1920X anymore as I thought all its stock disappeared last year when the TR2 models came out and retailers were blowing the 1920X out the door for less than $250.

ok, little update on my build project.

https://imgur.com/a/7dfC8ym

what i have done:
- asrock x399 taichi <<=== motherboard
- TR 1950x <<=== cpu (with noctua 120mm TR4 cooler)
- pcie-riser cables from x16 to x16 (25cm long, https://www.highflow.nl/hardware/videokaarten/li-heat-pci-e-gen-3.0-ribbon-flexible-riser-cable-v2-black.html
- 4 msi gtx 1070, 8gb, itx cards
- gpu load around 85%
- pcie interface load around 20% on x16-slot, around 30% on x8-slot (app_config.xml with cpu usage at 0.975)
- all gpu-cards undervolted with msi afterburner to 50% of power limit (with disadvantage lower core clock. but i dont mind that).
- crunching time according boinc around 10 hours


my next step:
building another system with a mining frame for 5 gpu cards (msi gtx 1070, 8gb, itx). this one:
https://imgur.com/a/U5gP4tW
purpose is to still using pcie riser cables x16 >> x16 to connect so much gpu's to mainboard. i will not use regular computer case, i like to have good airflow, so i choose for mining frame. and therefore using x16 >> x16 riser cables.

so, i need your advice (or have questions).
- which mainboard has more than 5 pci-e slots? (intel, amd, old, new platform doesnt matter). but it must be ATX form factor
- is it true that windows 10 64bit supports maximal 6 gpu's?

thanks in advance
erik

ok, little update on my build project.

https://imgur.com/a/7dfC8ym

what i have done:
- asrock x399 taichi <<=== motherboard
- TR 1950x <<=== cpu (with noctua 120mm TR4 cooler)
- pcie-riser cables from x16 to x16 (25cm long, https://www.highflow.nl/hardware/videokaarten/li-heat-pci-e-gen-3.0-ribbon-flexible-riser-cable-v2-black.html
- 4 msi gtx 1070, 8gb, itx cards
- gpu load around 85%
- pcie interface load around 20% on x16-slot, around 30% on x8-slot (app_config.xml with cpu usage at 0.975)
- all gpu-cards undervolted with msi afterburner to 50% of power limit (with disadvantage lower core clock. but i dont mind that).
- crunching time according boinc around 10 hours


my next step:
building another system with a mining frame for 5 gpu cards (msi gtx 1070, 8gb, itx). this one:
https://imgur.com/a/U5gP4tW
purpose is to still using pcie riser cables x16 >> x16 to connect so much gpu's to mainboard. i will not use regular computer case, i like to have good airflow, so i choose for mining frame. and therefore using x16 >> x16 riser cables.

so, i need your advice (or have questions).
- which mainboard has more than 5 pci-e slots? (intel, amd, old, new platform doesnt matter). but it must be ATX form factor
- is it true that windows 10 64bit supports maximal 6 gpu's?

thanks in advance
erik

forgotten to say: under windows 10 is Swan_sync enabled (set to 1). but having 80% gpu load, I don't mind that. I think the reason for such a low gpu load is my cpu. because my cpu is partially broken. (but this is another issue, asked rma at AMD)

now for testing....connected 1 gpu to pcie-x1 v2 slot, gpu load at 68-70%, bus interface load 65-67%.

is a PLX-switch not becoming bottleneck for gpugrid-calculations when I make a 7-gpu-setup?

and...any info about maximal supported gpu in windows 10??

It's hard to see but is there support under those GPUs or are they just hanging by the PCI bracket? I have this one that 'supports' 6x GPUs but you can easily drill more holes for the PCI brackets. Not even $30 and it comes with fan mounts.

https://www.amazon.com/gp/product/B079MBYRK2/

There are server boards that have more than 4x PCI-E 16/8x slots. Some have 7 but may not be ATX. I've hard some people using a Rosewell cases for higher count GPU setups.

No need to limit yourself to an ATX board for either type of case. Those open air mining rigs are just made out of aluminum t-slot pieces. https://8020.net/shop

It's hard to see but is there support under those GPUs or are they just hanging by the PCI bracket?

I have this one that 'supports' 6x GPUs but you can easily drill more holes for the PCI brackets. Not even $30 and it comes with fan mounts.

No need to limit yourself to an ATX board for either type of case. Those open air mining rigs are just made out of aluminum t-slot pieces. https://8020.net/shop

Ah missed the 1st link. 2nd one with fewer images are darker.

With a couple of 120mm fans along the length of the rack it would become like a wall of air, esp if there was a top/sides to force the air along the GPUs.

To get that many PCI-E slots you'll need a single socket WS board/CPU as you mentioned that came with more lanes, a 2P board or an AMD TR/Epyc setup. Even if you don't need all those lanes for GPUGrid, those are the type of systems where the slots will be available. Z10PE-D8 WS, EP2C621D12 WS or even X9DRX+-F with ten 8x 3.0 slots.

https://edgeup.asus.com/2018/asus-h370-mining-master-20-gpus-one-motherboard-pcie-over-usb/could this board working with gpugrid? because data connection is usb 3.1 gen 1 (5gbps speed). is this enough data speed (usb 3.1) for gpugrid?
i want to connect 6-8 gpus.

if one lane pci-e v3.0 has around 1gbps speed and x16-slot has around 16gbps speed, then one usb 3.1 gen1 will be comparable with pci-e v3.0 x4-slot.
is my calculation rigth? or am i missing something?

now for testing....connected 1 gpu to pcie-x1 v2 slot, gpu load at 68-70%, bus interface load 65-67%.

now for testing....connected 1 gpu to pcie-x1 v2 slot, gpu load at 68-70%, bus interface load 65-67%.

That is not as bad as I thought, though you did mention you lowered the power limit of the cards a lot. What GPU clockspeed are they running at? I would imagine with a faster card you would get less GPU utilization.

I too have a mining-esque system with hopefully 5-6 gpus. The case is already designed for open air 6 GPUs. My current and hopefully only problem is the cards I currently have require two 6 and 8 pin 12v plugs and I've run out of cables from my power supply. I should have done a bit more research before buying!

At first I had severe issues getting the GPUs to be recognized so if you ever have this problem, try updating the BIOS, this is what fixed it for me.

I too have a mining-esque system with hopefully 5-6 gpus. The case is already designed for open air 6 GPUs. My current and hopefully only problem is the cards I currently have require two 6 and 8 pin 12v plugs and I've run out of cables from my power supply. I should have done a bit more research before buying!

At first I had severe issues getting the GPUs to be recognized so if you ever have this problem, try updating the BIOS, this is what fixed it for me.

for psu you can use atx-psu with enough power, starting at 1200 watt. preferably full modular, so dont want to have molex or sata power connectors. or use hp server psu with special modules for 12x 6pin pcie-power-connector.https://tweakers.net/aanbod/1983990/mining-starters-kit-benodigdheden.html
this site is in dutch. but you can check the pictures about ho server psu.

for your build...what kind of motherboard are you using? and cpu?

does gpugrid really need the full 16x or 8x bandwidth of pci-e 3

i am considering this setup:

https://imgur.com/a/e03wZMM

mainboard: Onda B250 D8P (not D3-version)
support for socket 1151 intel cpu (6th gen, maybe even 8th gen)
sodimm upto 16gb ddr4 (laptop ram-modules, sodimm)

That CPU only has 16 lanes on the CPU so it would be pretty pointless to have a full PCIe connector for each GPU. It would only make sense if there were enough PCIe lanes to go around

so, your better choice is using mainboards with PLX-switches?

or....making this

https://www.asrockrack.com/general/productdetail.asp?Model=EPYCD8-2T#Specifications mainboard of cost at 480 euro
with x8-slots open-end, so using my x16 ==>> x16 pcie-risers to setup a system with maximal 7 gpu
with cpu epyc 7251 [535 euro, 8 cores] or epyc 7281 [720 euro, 16 cores]

That CPU only has 16 lanes on the CPU so it would be pretty pointless to have a full PCIe connector for each GPU. It would only make sense if there were enough PCIe lanes to go around

ok, yet another update, using x1-slot (on asrock x399 taichi mainboard with threadripper 1950x)

https://imgur.com/a/VV6CdUB

little explanation about the above image...
values from 2 to 20 are under power limit of 50%
values from 21 to 38 are under no power limit (so power is 100%)

changing the power limit has only effect on core clock speed, meaning 100% power will finish WU faster while gpu-temperature will grow with 4-5 degree Celsius.

another note...with or without power limit nothing changed regarding gpu-load and pcie-bus-load. these both values are around 74-76, with or without power limited (or undervolted).

my another card in real x16-slot (electrical x16) has gpu-load around 88-90%.

so, my performance loss using x1-slot vs x16-slot is from 88-90% to 70-72% gpu-load.
my conclusion: 95% chance I am going to choose for mining board for my multi-gpu setup with 6 gpu's...much cheaper that buying mainboard with 7 slots of x16-lanes (intel platform and PLX-switches).

this is just sharing info with you.
have fun !!

Turing RTX 2080 PCIe 2.0 x1 performance 50% of PCIe 3.0 x4 on Primegrid n=20 Genefer.
GPU power went from 205W to 133W - same as the GTX 1070 on PCIe 2.0 x1.

Pascal's x1 performance loss around 8-17%.Genefer a heavy on PCie bandwidth.

GPUGRID can have high PCIe bandwidth usage with Multi GPUs motherboard.

Turing PCIe x1 performance worse than Pascal / Maxwell / Kelper x1.

My Zotac mini GTX 1080 died yesterday after 28 months 24/7 service.
I returned a RTX 2070 and kept the RTX 2060.
2060 limited to 160W compared to 210W where the 2070 operated OCed for 12-20% more performace.

Today I purchased my first ever ti GPU: 2080ti: GPU Asus ROG strix COD edition (700$ open box Microcenter.)
Boosted out of the box at 1970Mhz with 280W power on n=20 WU.

With the new 2080ti I decided to test Turing on PCIe x1.

Also my 2013 z87 Haswell showing age with the OCed RTX 2080ti and 2080 on primegrid PPSsieve.
Runtimes are slower then the RTX 2080ti & 2080 skylake / coffeelake CPUs combos.

CPU speed with GPU overclocking on Primgrid (PPSsieve) scales very well as does (AP27).
These two programs require minimal PCIe bandwidth.

Higher clocked (+3.7GHz) CPUs help the OCed GPU finish the WU faster.
This is similar to GPUGRID.

Today I purchased my first ever ti GPU: 2080ti: GPU Asus ROG strix COD edition (700$ open box Microcenter.)

i will not use pcie-x1 slot.
i have decided to use mainboards with plx8747 pcie-switches, mostly they have 7 pcie-slots. fully loading mining frame with 7 gpus. i think i am going for gtx 1070, but not for sure yet.
and i am always undervolting all my gpu's to power limit 50% because of the heat.

rigth now running 4 systems. 3 for gpugrid (with totall 6gpu's), 1 system with 6 gpus for Folding@Home.

in the planning. add 1 more system for gpugrid, add 1 more system for FAH, 1 more system for WCG and FAH.


but not yet decided which gpu to choose.

Today I purchased my first ever ti GPU: 2080ti: GPU Asus ROG strix COD edition (700$ open box Microcenter.)

Wow I think you got the deal of the century, I bought a new EVGA 2080ti from Microcenter for $1015 and I thought that was a pretty good deal. The difference is, my max boost speed seems to be 1800 using 275 watts according to GPU-Z. Pretty amazing you can get 1970Mhz with only 280 watts.