this copper mod might help ukposts.info/have/v-deo/nmiXapB_eoaH0as.html

So did you go the water-cooled option or just multi-fans experiments?

+1

My pleasure! Thanks!

I guess you must be misguided by this guy. Don't buy 3060, for this price range, the memory is not important anymore, get a 3070 or 3080, much much faster than 3060.

@@kilosierraalpha I have a 2080 ti and a 3060 in my computer, they work good. The 3060 is not horribly slower than my 2080 ti, so... plz dont make it sound like the 3060 is not suitable for ML. You can use overclocking on the buswidth btw, I did it aswell and nothing bad happened yet :D

wolfram?

@@thewizardsofthezoo5376 may you add more information?

Working on that now, actually.

@@HeatonResearch Thank you for your work, I would want to know which plan is better for GAN project, 4 3080Ti or 2 3090? Thannk you.

Could you figure out the answer?

I have the same problem

Hey man, I'm currently running a 3080. I know you said pricing aside, but the 3090 has come down to the same price as 4070's so I would strongly consider that. I have the 10GB model and would kill for the extra VRAM. Creating a convolutional neural network I ran out of VRAM pretty fast when trying to expand my model. So I either had to split my model among different GPU's or go with a smaller model. Thats why you want to try for more VRAM on a single GPU. That was also on a dataset with 510 classes for classification, which isn't the easiest. I recommend spending what you would on a 4070 or 4070ti and getting a used 3090 for the VRAM. Barring that, I would consider trying to get a used 3080 12GB and saving up for a second.

That is a 3-slot GPU, so make sure there is enough space and that you can fit it and have at least decent airflow. This is an area where the gamer recommendations on dual 3090 would apply directly to machine learning, and I've seen YT videos on dual 3090.

Did you use nvlink?

Dual use behalves the PCI bus?

Did the dual 3090 (NVLink) actually give you double the vram seamlessly?

Yes, I had pretty good luck with nvlink, more here ukposts.info/have/v-deo/oHJ8l4JvnYSLkmw.html

You can't "join all 6" together like you suggest. If you just plug in all 6 you can use them in parallel for machine learning but then they don't share any memory (aka there is no memory pooling). You can get nearly linear speedup as long as the model type you are training is parallizable and no other pc component is creating a bottleneck. You can typically expect 1.92 for two cards, 3.84 for 4 cards so for 6 identical gpus you will get your near linear scaling. However, the rtx3090 does not support nv-bridge etc. What you can (and should) do is get 3x nv-link which allows you to bundle two of them always together. By doing that you can effectively use 48 instead of 24gb memory allowing for bigger models and larger batch sizes. So you can both get a nice speedup (large batchsizes are tyically much faster for transformers etc) and you can play around with larger models. Some software like video editing often times does not support nvlink, but tensorflow and pytorch do (what you are probably using).

That's exactly what nvlink is for, this is correct

@@andreas7278 I'm still confused, does that mean nvlink provides a 48 GB unified vram but it's not a drop-in replacement and we still need to write some acrobatic code to run models larger than vram of a single card?

It is indeed a drop-in replacement if you want to call it like that, i.e. 2x rtx3090 (same goes for 2x nvidia titan rtx from the previous generation) connected via nvlink indeed provide you with one unified 48gb vram memory pool which allows you to train larger models and use larger batch sizes. As long as the library you are using supports unified memory you don't need to do any additional trickery or coding, e.g. pytorch or tensorflow will handle this automatically if you use multi gpu mode so no further coding is needed. However, other math libraries such as numpy won't make use of memory pooling. For modern deep learning this is sufficient though since most people will only need the high vram amounts for deep learning. This is what made these dual cards being so popular for machine learning researchers. A lot of scientific ML papers have been using one of these two setups (with the exception of the big players with their gigantic server farms out there like OpenAI, DeepMind, GoogleResearch etc). It was a very economic way to get nearly twice the performance of the corresponding quadro 48gb card (2 cards mostly end up with like 1.92x performance over a single one in pytorch, taking into consideration that quadra cards with their ECC memory are usually a little bit slower you end up at roughly twice the throughput) at the same memory size for an extremely competitive price. Now we finally have the rtx4090, which pushes linear algebra calculations further at a larger generational jump than ever before. But the reason why the generational jump is higher is that they cut out the nvlink memory controller and used that space for more cuda units. This means that the rtx4090 has a larger generational jump over the rtx3090 than the rtx3090 over the titan rtx at a very competitive price. Also, it means that the rtx4090 in comparison to their rtx4070 and rtx4080 delivers exceptional value for money (just look at the total cost for proper water cooling, energy consumption and ML throughput for an rtx4090 compared to like an rtx4080, it's not just much faster, it's a better deal even though it's the highend card). But if you work with any type of transformer models which are very common right now, 24gb is kinda a very low boundary. Often you may only choose the small models and then in combination with ridiculously small batch sizes (not just making training slower but also resulting in other final network results due to maximum likelihood estimation being applied on too few samples for each epoch). More reasonable SOTA models require 50-60gb upwards and 48gb vram provides you with way better options. There are crazy models out there by the likes of OpenAI which literally needs hundreds of gb of vram but well, ... you can't have everything and you would only analyse or downstream train them anyways. If the rtx4090 would allow for nvlink we could get a reasonably prized 48gb setup but as it stands, you need to buy the rtx6000 ada lovelace which will cost a lot more and you also will only be able to leverage your single card throughput. Furthermore, going to 96gb will be impossible with quadro cards now since also these ones don't allow for memory pooling via nvlink any more. So you will have to get tesla cards which are a whole price tier higher. Basically, this new generation is a disappointment for ML researchers if we take reasonable setups into consideration. Other than that the new generation is pretty amazing.

@@andreas7278 thank you for this detailed explanation, very appreciated ! I’m extremely disappointed that Ada Lovelace 40 series cards have no Nvlink anymore, not even the top end RTX 6000 (Ada). Surely anyone who needs more than 48 GB will go with a last gen RTX A6000 setup. Maybe thats another one of Nvidias ways to get rid of Ampere oversupply? What really surprises me, is that Nvlink is supposed to be removed from Ada Lovelace cards on silicon design…yet the new Nvidia L40 datacenter card, which has an Ada Lovelace chip, does have Nvlink according to their website. I guess that makes it the “cheapest” card for ML with >48 GB requirement.

@@andreas7278 You're awesome, man. Just to be specific, to train large models on nvlinked 2x 3090, all I have to do is enable ddp in pytorch, no need for any model parallelization code right? Looks like nvidia is not going to make any relatively cheap card that has vram more than 48 GB, I'm definitely considering picking up another 3090. Having done two research projects on bert-scale model, I'm fed up with not being able to lay my hands on SOTA, mid-size models. My guess is they might ramp up next-gen 5090 cards to 32 GB, but that is not going to bridge the gap between the demand anyway.

AMD needs more cloud support, the day I can start to get AMD AWS instances, I will start to consider them. I like my local to mirror what I use in the cloud. I am excited about the 4000 series as well, all the rumor mills that I follow suggest 4K series will be out next year this time.

yes

@@danielwit5708 how? NV Link appears to only connect 2x RTX 3090's but not 4? I have 6x RTX 3090s 😛

@@sigma_z your question didn't specify that you asked about nvlink bridge lol I thought you just asking about more than 2 cards 😅

People speak at different speeds. Often highly analytical people speak at a slower pace.