AMD says these are the same... We DISAGREE. - Testing 12 of the same CPUs for Variance
Переглядів 1,278,901
День томуRemove your personal information from the web at joindeleteme.com/LinusTechTips and use code LTT for 20% off.
NexiGo is hosting product giveaways every month! Enter for your chance to win here: viralnation.link/LTTxNexiGoGi... (Contest is open to anyone 13+ in the United States)
Do you really know if your CPU is performing the same as the ones we review? We don’t know. But we know that if we want to increase our testing capacity, we need to PARALLELIZE. But that means we need nearly identical test benches. And trying to make that happen sent us down a far deeper rabbit hole than we could have anticipated.
Testing Process Doc and Forum Post: linustechtips.com/topic/15546...
Buy an AMD Ryzen 7 7800X3D CPU: lmg.gg/s8Y1K (Canada: lmg.gg/yZnVo ) (Micro Center: lmg.gg/EkzYG )
Buy a NVIDIA GeForce RTX 4090 FE Graphics Card: geni.us/Won6A
Buy a GIGABYTE X670E AORUS XTREME Motherboard: geni.us/FADHuEG
Buy G.Skill Trident Z5 NEO 6000MT/s CL30 2x16GB DDR5 RAM: geni.us/TYRaJf
Buy a Samsung 980 Pro 2TB NVMe M.2 SSD: lmg.gg/aL0BL
Buy a Noctua NH-D15 CPU Cooler: geni.us/lK6GI9O
Buy a MSI MEG Ai1300P PSU: geni.us/NRCy
Purchases made through some store links may provide some compensation to Linus Media Group.
Purchases made through some store links may provide some compensation to Linus Media Group.
► GET MERCH: lttstore.com
► GET EXCLUSIVE CONTENT ON FLOATPLANE: lmg.gg/lttfloatplane
► SPONSORS, AFFILIATES, AND PARTNERS: lmg.gg/partners
► EQUIPMENT WE USE TO FILM LTT: lmg.gg/LTTEquipment
► OUR WAN PODCAST GEAR: lmg.gg/wanset
FOLLOW US
---------------------------------------------------
Twitter: / linustech
Facebook: / linustech
Instagram: / linustech
TikTok: / linustech
Twitch: / linustech
MUSIC CREDIT
---------------------------------------------------
Intro: Laszlo - Supernova
Video Link: • [Electro] - Laszlo - S...
iTunes Download Link: itunes.apple.com/us/album/sup...
Artist Link: / laszlomusic
Outro: Approaching Nirvana - Sugar High
Video Link: • Sugar High - Approachi...
Listen on Spotify: spoti.fi/UxWkUw
Artist Link: / approachingnirvana
Intro animation by MBarek Abdelwassaa / mbarek_abdel
Monitor And Keyboard by vadimmihalkevich / CC BY 4.0 geni.us/PgGWp
Mechanical RGB Keyboard by BigBrotherECE / CC BY 4.0 geni.us/mj6pHk4
Mouse Gamer free Model By Oscar Creativo / CC BY 4.0 geni.us/Ps3XfE
CHAPTERS
---------------------------------------------------
0:00 Intro
2:25 Why Same Model ISN'T Same Performance
5:32 Sources of Variance
8:21 Gaming Results
10:43 CS:GO is Wonky
11:41 Gaming Results Cont.
12:39 Productivity Results
13:59 Our Selections
15:15 Testing Mobos and RAM
17:16 Final Discussion
21:26 Outro
@XenFPV 3 місяці тому
This is LTT using their higher budget compared to most tech channels for something genuinely useful. This is really good to see! Well done LTT, this deserve genuine praise.
@zypher5876 3 місяці тому
Gamers nexus did the same thing a while ago
@joaomendes3907 3 місяці тому
@@zypher5876 Gamer's Nexus not listed as other media. Funny
@raawesome3851 3 місяці тому
@@zypher5876 yeah, true. But Linus has a far greater scale than Gamers Nexus, as of now, reviewing everything from CPUs, GPUs, laptops, etc, so it's cool to see that level of standardization on a large scale to make sure the testing is consistent.
@frankwong9486 3 місяці тому
Well , those overclocker do lots of binning and probably run through more samples than the average LTT benchmark review
@raawesome3851 3 місяці тому
@@joaomendes3907 I'm pretty sure that's not out of malice, since Linus' not the editor for it 🤷♂️
@Joshfarmpig 3 місяці тому
glad to see factorio used as a gauge for cpus
@phooogle 3 місяці тому
As it should be lol it's mental intense on ticks. Causes some real aggro on multiplayer systems with different spec.
@Hugs288 3 місяці тому
haha, guage, i get it
@demondoggy1825 3 місяці тому
Factoriohno.
@whatistruth_1 3 місяці тому
The issue is all of these reviewers are doing the factorio benchmark wrong. The game has a 60 tps cap as that's what the engine runs at. What needs to be tested are heavier maps that push the cpu BELOW 60. These above 60 tps benchmarks are as good as a synthetic benchmark in that they're not realistic. And yes, results absolutely do change and so does the cpu hierarchy.
@nepnep6894 3 місяці тому
@@whatistruth_1not to mention Factorio scales more with memory performance than core perf or cache.
@spacen3rd950 3 місяці тому
As a data scientist, I absolutely love this video and greatly appreciate this perspective. The silicon lottery is absolutely real, and sample size of one is far from sufficient. Glad to see someone doing more in depth analysis. I don't expect the methodology of ltt labs to dive into frequentist vs bayesian stats, but it would be interesting to use public benchmarks (for an incredibly nerdy perspective/audience) as a prior distributions and see how the results differ. Regardless of the depth of it, just seeing more stats in hardware/software benchmarks is a fantastic breath of air. Keep up the good work!
@ChristianRoseroF 2 місяці тому
La muestra es real, en el consumidor final la variación será más grande pero le darán la culpa a los otros componentes como cantidad de ram, ssd u otros factores, AMD solo garantiza ka velocidad del reloj en ciertss condiciones, en el hogar la temperatura ambiente también ocasiona ruido en los datos. Lo cierto es que no hay un tercero que obligue a cumplir un estándar legal porque es complejo medir que los productos sean iguales
@kytechnelson 2 місяці тому
Agreed! Would also be interested in seeing a bayesian analysis with public benchmarks as a prior (though I'm also one of those nerds who would love more of this kind of stuff). I also appreciate how they detailed their similarity metric of using euclidean distance, along with explaining what that looks like in a higher-dimensional space for those not accustomed to seeing it used that way since it's such a common approach for any sort of clustering problem.
@jannejohansson3383 Місяць тому
When public is put in test, there's always possible cheating or possible than manufacturer goes in game too.. if you understand what I mean..
@diegoberan7883 Місяць тому
LTT should hire a data team to take care of all this data. It would be great to have it properly stored and analyzed, maybe you guys could ge into testing neural network performance of GPUs too. I find it hard to find data on how well GPUs run AI tasks.
@kytechnelson Місяць тому
@@diegoberan7883I second the request for GPU benchmarks for AI tasks. Looking at some of what Puget Systems has done for their AI/ML benchmarks would probably give a good starting point for the LTT team.
@4RILDIGITAL 3 місяці тому
I never realized how much work goes into ensuring consistency in benchmark testing. It's worrying to think about the lack of oversight in computer hardware compared to other industries and the issues that could ensue.
@the_undead 3 місяці тому
Now imagine real scientific testing and trying to get perfectly consistent results and not within 0.25%
@michaelbailey9857 2 місяці тому
As nice as it'd be, it kind of admittedly makes sense to an extent - Cars aren't tested and regulated because they're expensive, they're tested and regulated because they can kill people
@pixels_per_inch 2 місяці тому
Actually it's not an oversight. Just like Nvidia GPUs, they boost automatically to the max possible on the silicon. Though AMD should've never advertised the max (up to) clock speeds, because 90% of people won't get it. Nvidia did it right by advertising the base/lowest possible clocks, Intel tells you both the lowest turbo boost and the max with TVB.
@smayaan2479 3 місяці тому
Genuine praise to linus for taking time to address this and how confusing and weird these companies are becoming launching their cpus and gpus in today’s market
@Blast_HardCheese 3 місяці тому
The silicon lottery's always been a thing. I didn't think it would still be this bad though, and it's nice to see it tested
@FcoEnriquePerez 3 місяці тому
Yeah, this video was like the old LTT content we used to get.
@n_core 3 місяці тому
@@Blast_HardCheese I think it's always this bad but the manufacturers just locked the maximum potential performance so then it won't vary once it reaches the end consumer (so your typical consumer won't feel aggrieved because of it,) and you can always unlock it through overclocking. But nowadays they use a different sophisticated method which almost makes overclocking obsolete and it is already unlocked for the end consumers, so we can see the various results between each chip.
@alexandruilea915 3 місяці тому
@@Blast_HardCheese It's not that bad, all they had to do was find the maximum frequency that all 3 CPUs could run at and fix them at that speed. Then the testing should result in minimal difference. For customers, you should be glad if you get a golden sample and don't care if you didn't because you got what you paid for which are the guaranteed speeds. Everything over the guaranteed speeds is a bonus that you may or may not get.
@Blast_HardCheese 3 місяці тому
@@alexandruilea915Agreed, I just assumed they would have it fully reined in by now
@y_is_everywhere_lol 3 місяці тому
The silicone lottery strikes again, It's crazy how much variation there can be between CPUs, especially within the same model
@2021videowatcher2021 3 місяці тому
Yep, Linus dose all these videos just to keep the best stuff for himself.....You think the stats is for us but it's not.
@nonamenameless5495 3 місяці тому
Disagree, it s more like: it s crazy how good the production process for such delicate tech has become that there s so little deviation and the results being so insanely close!
@NightKev 3 місяці тому
@@2021videowatcher2021 l o l
@BoujaXD 3 місяці тому
Theres not Much variance its +-2%.
@renchesandsords 3 місяці тому
especially for more bottom bin chips, since the premium ones are all reserved for the higher end chips
@thegamechanger7352 3 місяці тому
Seems like Gamers Nexus did not make the list of media outlets in 17:24
@Capellix0001 3 місяці тому
LTT Saltmine be like
@Not.Your.Business 3 місяці тому
LTT staying "classy"
@yutt 3 місяці тому
Why would they? Gamers Nexus is a backpack warranty review channel, not technical.
@firstnameIastname 3 місяці тому
Wow what a shocker 🙄 Lineypoo seems like the kind of guy that holds a grudge even when he's the one who's wrong lol It is cute though that they still do all this testing when anyone with half a brain knows not to trust their data, it's like he has to justify spending all that $$$ on a team & equipment somehow. So funny lol
@nomadicdragon7157 3 місяці тому
Did I miss something between Gamers Nexus & LMG?
@cheredia13 3 місяці тому
Statistically,from experience in semiconductor industry, It will take at least a sample size of 30 cpus to establish proper sigma. Ideally you would want to do this experiment with 1 motherboard first and repeat with other boards to gauge variation. The next step would be to test about 100 cpus from different time stamps (hopefully getting different lots),to gauge the variability of their process.
@vyor8837 3 місяці тому
Also at JEDEC to remove differences in memory controller overclock quality.
@sindrisuncatcher653 3 місяці тому
Of course those hundred CPUs are gonna cost about $35,000... for every model of CPU that you test.
@TechDaddyFr 3 місяці тому
Looks like the transparency we wanted. You clearly stated the challenges, how you worked around them and the variables you can't really control. It was very interesting, at least to me. Good job guys!
@brendan9335 3 місяці тому
Incredible leveraging youtube videos to finance an independent lab to preform tests on all kinds of tech hardware and then making vids showing and explaining all of the data collected which in turn can finance the next round of tests. Bangin job LTT I love it.
@Lurch-Bot 3 місяці тому
Yeah, they have the equipment but no clue how to use it properly, as shown in this video. They pulled 11 tray CPUs, probably from the same tray. Does not reflect the consumer experience at all. Of course enterprise hardware will be more consistent, even if it is going in a consumer product. Imagine if a company like HP had people doing side by side tests of the same model and getting a 20% variance in benchmark scores... In other words, the major manufacturers of complete systems get the best and what goes in the consumer boxes is like a random grab bag of everything else that wasn't good enough for HP or Dell. This is why I benchmark test new CPUs and GPUs and have and will send them back solely for unreasonably low benchmark scores. If everyone did that, it would be a completely different market and the variance would become statistically insignificant for everyone, not just HP, Dell and LTT. Oh, and don't think you can get around it buying a tray CPU from AliExpress. Those will be bottom of the barrel, just consistently so. You're a dreamer if you think that was some leftover from HP's order. Those were ordered by some no name company with the name produced by randomly mashing a few keys on the keyboard and were binned and priced accordingly. I'm sure you can buy 5600X CPUs there all day long that are really a 5500. And it is AMD that is the one running the scam there.
@ElNeroDiablo 3 місяці тому
@@Lurch-Bot Did you miss the part where 11 of the CPU's came from 11 different stores across 3 countries (Canada, USA, and looked like UK)? These 12 chips (including LTT's pre-launch test sample) are going to be from different batches across the 6-month window they were collected, which will emphasis the differences between wafers let alone between chips on the same wafer.
@treborrrrr 3 місяці тому
@@Lurch-Bot 22 seconds from the start of the video, that's how long it took for them to tell you how the CPUs were acquired, and you somehow still managed to make this comment. Wow.
@danieloberhofer9035 3 місяці тому
@@Lurch-BotYour comment shows an astonishing lack of knowledge about silicon manufacturing, distribution and basic economic dependencies. You also show a lack of text and information comprehension that might explain most of the rest. If I were you, I'd scale down the confidence level of what I'm posting lenghtily and take a step back to evaluate what I actually know and what not.
@MariuszChr 3 місяці тому
Ummm, that's literally how businesses work. Spend money to make something, sell it so you can make more, and sell more.
@Grandwigg 3 місяці тому
I love seeing this kind of testing across multiple channels. And another shining example of why you should always check across multiple review sources. (Loved the Pokemon referencing of the processors)
@Grandwigg 3 місяці тому
(and I've only known a couple years now that Raikou is long o sound (ish). And i have the gbc Crystal bundle. Best Christmas ever, lol)
@kaizen_unknown 3 місяці тому
I'm glad to see linus and his team working towards doing better, especially after that whole thing where they were admitting lack of accuracy and that they failed their community. Seeing them be so passionate about this really brings a smile to my face.
@the_undead 2 місяці тому
You say that like they're the only ones that don't have perfect accuracy, You say that like putting in the tens of thousands of dollars in equipment and man hours is something that they can just do overnight, You say that like gamers Nexus was at the time he made that disgusting disingenuous video was in the habit of retesting products to use that data in comparison charts for new product launches. But no, none of those things are true and I still don't think gamers Nexus retests. Although if you can provide me with real evidence that he does retest stuff I will happily change this opinion
@squatzandoatz92 3 місяці тому
i love the fact that you come out and say "yes, our tests are inconsistent, and here's why" in such a fascinating, entertaining fashion. there's a reason i always watch every LTT video that pops up in my feed.
@AK-tf3fc 3 місяці тому
you do know gamer nexus is better then this
@lilcheaty 3 місяці тому
@@AK-tf3fcnot comparable lmao, this video specifically shows us a big difference in gamers nexus testing vs ltt
@pituguli5816 3 місяці тому
@@lilcheatyIn that case lets throw all reviewers performance results out the window because of a few percent variance, I do agree with you though there is a huge difference in GN and LTT testing and that difference is GN can at least be trusted with their numbers but LTT cannot. "Look guys there is a variance so its not our fault we put out shonky reviews and fudged numbers". Silicon lottery has always been a thing but of course Loonus spins into this to obfuscate the fact their testing methodology is bogus at best. Wanna see a good correction video? HUB's latest video where Steve apologised for his mistakes and moved on, LTT "Silicon lottery cause number to go boom".
@the_undead 3 місяці тому
@@GsrItaliabecause gamers Nexus has to the best of my knowledge never came out and explicitly confirmed their testing method and if they have, it's not even comparable to this, no one else in the industry is doing what LTT is doing because they don't have the money to just buy 11 top of the line. CPUs and at least 10 4090s just for data accuracy reasons. This isn't necessarily a knock against anyone else on this platform, they just don't have the money to do this
@Lurch-Bot 3 місяці тому
Yeah, how many times are they gonna have to say it before they realize they don't understand scientific method?
@dahahaka 3 місяці тому
you can maybe work around the inconsistencies in red dead 2 by using cheat engine with the unrandomizer, it forces the random number generator to always return the same values, so you could end up with a deterministic benchmark run :)
@aaaafireball9322 3 місяці тому
that's a great idea, do it for every game if possible or matters
@danymend5797 3 місяці тому
I don't think they can automate it
@Deveyus 3 місяці тому
I came down here to check to see if someone had suggested similar. As a former game dev (not for RDR2): This will have a minor impact on performance of games that utilize system random number generators especially (because of the syscall overhead) but importantly the result should be changed the same (incredibly tiny) amount. In exchange, you will get drastically more consistent runs from the behaviors in game. We actually had a debug build we did that replaced all the calls to random things like this with static values, so that we could apply internal test harnesses, especially during building our tutorials. I am not intimately familiar with how Cheat Engine goes about derandominzing, but all the sane ways I can think of it would do it, give you more in repeatability than you lose in cost to run it, or in deviance from player system behavior. (You're simply selecting one player system behavior and repeating it)
@dieterblancke1756 3 місяці тому
@@Deveyus this may be a stupid question, but would the (albeit minor) performance difference be mitigated if you leave in the random number generation but just don’t use it and use the static value instead (therefore still creating the “load” of generating a random number while also keeping your consistency)
@ErulianADRaghath 3 місяці тому
I am not sure if a result gathered with the default randomized behavior turned off is representative of the average gamer experience. Since the goal of the labs is to provide statistically significant testing for the masses, introducing irregularity is maybe not the right solution.
@Lmmolsen 3 місяці тому
@17:24 discretely showcasing bad blood between Gamers Nexus and LTT 😅😂
@mikebutterface8583 2 місяці тому
I thought the same thing
@minijag972 2 місяці тому
Lol, pretty sad LTT cant be a professional
@GULIwer1980 2 місяці тому
@@minijag972Never trust pirate w/o ship. Such a big company, such a small sample.
@jjlortez 2 місяці тому
@@minijag972 Yeah I always recommend you check out my competitor's shop after he smashes my windows 😂
@andersonmcintosh7214 2 місяці тому
Gamers Nexus incident made this video possible IMO. LTT forced to up their game
@timduncan6750 3 місяці тому
6:50 Smiled when I saw the only thing you add is Notepad++ because this is truly the only thing I miss since going from Windows to Mac on my work computer...
@mjdevlog 3 місяці тому
Notepad++ is great for it's autosave and multiple tabs, though now the regular notepad (WIndows 11) has the same feature🤔
@Seed 3 місяці тому
Same lol.. vscode isn’t nearly as good
@K.Parth_Singh 2 місяці тому
@@Seedheh really?
@rikuleinonen 2 місяці тому
@@Seed sure wish np++ had support for practically any language.
@KeinNiemand 2 місяці тому
Vscode is way better the only thing notepad++ does better is beeing a bit faster@@Seed
@Just_another_tom 3 місяці тому
As PhD researcher, this is my favorite video that you guys have published in a while. Love the data and scientific process used here guys. Keep it up, go labs go!!
@zaxbit 3 місяці тому
Waiting for the first LTT Labs paper to hit Science
@shinjisan2015 3 місяці тому
It's great to see. All this work will make sure their internal comparisons are reliable. You can look at all LTT Labs testing when you want to compare DIFFERENT products for performance and know it's just the products you're looking at that vary, then look across different reviewers testing for the SAME products to get a glimpse the actual market quality variances for that one product. LTT Labs is actually benefiting the whole techtuber community, not just LMG.
@chairface1859 3 місяці тому
lol "PhD researcher"
@TheRockybulwinkle 3 місяці тому
@@chairface1859yes, one who researches PhD candidates 🧐
@grumpyratt2163 3 місяці тому
@@chairface1859 now you look a bit daft. See working in a University I know this is a proper job title. Just because you have little understanding of job roles within a University doesn't make a job title any less real. I have a very unassuming job title I work within the Estates/Operations department of the University that I work at my job title is Building Support. But dig a little deeper and Im actually part of a team that run three of my uni's flagship research buildings one doing research into neuroscience ( like finding a cure for dementia Huntington's) and that has the Compound Semiconductor research Institute ( yes we have the ability to fab our own silicon wafers) also in the same building is the Chemical Catalysts Institute. This build brings together the school of chemistry school of physics and school of engineering. The final building houses the Social sciences with departments within it that research help form public policies which work with The British Government also the business side of the university. Out of the 5000plus staff at my university about 1000 are academics
@jamesmatthews291 3 місяці тому
Well done. Possibly the most thorough and well-explained video on the effects* of the Silicon Lottery I've ever seen. *Not covering much on the causes, or it would've been a two hour video! In fact, can we have an updated video on that please?
@winebartender6653 3 місяці тому
It's a 1 minute explanation: Each transistor within the silicon will be electrically different. Differences in needed gate voltage (what vCore is) for current to pass, differences in leakage (the amount of current loss through electron migration, aka heat), differences in output capacitance (the inevitable result of having current paths in parallel), etc. With these differences, we have a different required voltage to drive the transistor to operate properly given a clock speed, different heat output given a voltage and different power usage given both. Each modern clockspeed boosting algo for Intel and AMD take these into account to be both stable and maximize performance (aka, clockspeed) given the current load on the CPU. So with all of that in mind, you can see why CPU A and CPU B will perform slightly differently given the same workload. Their clockspeeds will differ because of power, voltage and temperature differences down to a transistor level.
@the_undead 3 місяці тому
@@winebartender6653You're 1 minute explanation assumes a fairly deep level of knowledge about how a CPU works and seeing as LTT's target audience is not scientists and engineers where these kinds of assumptions could be made, your 1 minute explanation with likely turn into a five or 10 minute exclamation if they were being insanely information dense, at the average information density of an LTT video, I would say the hour-long prediction is not unreasonable given all the other stuff they would have to then cover to make it a complete video
@winebartender6653 3 місяці тому
@@the_undead ? Absolutely not. A simple preface of "A CPU is made up of billions, or even trillions, of transistors, which are essentially tiny electrical switches". You don't need to know anything about voltage, current, capacitance or anything else that you think is a low level concept. The point of the 1 minute explanation is to touch on the fact a transistor will always be different when made in a CPU, that's it. You could absolutely expand it out, but it is unnecessary to get the point across
@mikemotorbike4283 3 місяці тому
Thanks for the breakdowns; appreciate the work, team! Can the audio tech please level the volume on his expostulations? I'm listening on headphones because others need silence.
@AaronShenghao 3 місяці тому
This is absolutely what Gamers Nexus loved to see LTT's testing become, even though LMG seems forgot to mention them at 17:24. (It can be just oversight, but not including Gamers Nexus, the most through on testing tech reviewer before LMG have the Lab, seems a bit intentional.)
@Hathos9 2 місяці тому
The issue is that GN isn't trustworthy.
@martine5923 2 місяці тому
@@Hathos9how, cause they called out your best boy linus? go toutch grass please
@Hathos9 2 місяці тому
@@martine5923 That is the issue. GN has a legion of toxic sheep that love drama and attack whoever GN says to. I like technology, not drama.
@BoraHorzaGobuchul Місяць тому
@@Hathos9 that was very funny, please continue. People who love tech-not-drama don't watch ltt much, the whole point of this show is ~ 70% entertainment 30% info. It's basically a discovery channel tv show talking to the lizard brain off its TA. Nothing bad in it per se, some videos are pretty entertaining, but that's what it is primarily - a tech-related entertainment business.
@Hathos9 Місяць тому
@@BoraHorzaGobuchul I'm fine with entertainment/tech. It is better than the non-stop drama and negativity of GN. LTT seeks views by making fun videos. GN seeks views by attacking others.
@thedorsetflyer9113 3 місяці тому
As a professional test engineer, I am truly impressed with the level of detail, thought and effort that has gone into the LTT Labs. 👏👏👏👏 Now if you could just coming and explain this level of dedication to some of my colleagues in other disciplines of engineering that would be great. They do seem to think that everything will just work, and don’t appreciate that us test professionals have to think around corners sometimes…
@raawesome3851 3 місяці тому
@@terrylyn what's your issue?
@FlagerMiszcz 3 місяці тому
@@terrylyn yeah its pretty terrible
@ncg8224 3 місяці тому
Majority of the population do engineering for money, not because they are overly passionate. It would be useless to explain
@Daunlouded 3 місяці тому
Your comment is not constructive and is worthless. Please tell your own brilliant idea to improve the testing methods. Else don't type anything.@@terrylyn
@danniemck 3 місяці тому
What's pretty Terrible about it? @@FlagerMiszcz
@hardyhousinger7426 3 місяці тому
Always excited for labs stuff! That was exceptionally well presented and explained. Really great how you adressed inconsistencies and the multitude of possible causes. Probably one of the best ways I've seen to explain testing methodology, the reasoning behind it and the resulting discrepancies when compared to real world applications. Loved it!
@luisjalabert8366 3 місяці тому
The thing I'm not sure about, is why they couldn't just completely fix the clock frequency to a set value, instead of letting the CPU run AFAP. Sure, you won't get the best performance out of it, but if you want to use them to test a GPU, then it shouldn't be part of the bottleneck either...
@MikkoRantalainen 3 місяці тому
Superb work! The only thing missing was repeating the testing on another motherboard to check if the CPU performance depends on individual CPU or the combination of motherboard power delivery and CPU. Would the same CPUs perform the best in any motherboard or did those CPUs simply happen to work the best with that specific motherboard?
@XTRLFX 3 місяці тому
I never got lucky in silicon lottery but once. That was an Northwood P4 that can be pushed from 1.8 to 2.8 Ghz. For RAM I'm usually happy if I get the values printed on the sticks to run stable.
@peperoni_pepino 3 місяці тому
For my favourite brand, the average jar of peanut butter actually contains about 6% too much, to make sure their average variance doesn't put too many jars below the stated amount (quantité estimée). There will still be some with too little, that is just how the variance works out. Apparently trying to reduce that margin of error would be more expensive than giving the average customer 6% more. I sort of assume the same applies for many processes, so a 4% difference between your best and worst chip does not surprise me at all.
@username8644 3 місяці тому
It's much more complicated for processors. The difference in performance is due to the fabrication process. All model processors come from the same silicon wafer. And after cutting the individual chips, they bin them by how they perform. Hence why there are performance differences. They just have a given range where a certain model should perform and categorize the silicon from that. Edit: they can't even make chips that all perform identically the same, it's not possible. Each processor has its own unique characteristics.
@greggv8 3 місяці тому
I spotted some crazy poor quality control in Paul Newman Italian salad dressing. Not only was every bottle on the shelf filled to a different level, the amounts of the various ingredients of the settled out dressing varied wildly. I lined them all up sideways on the store shelf, took some pictures and sent them off to the company so they could identify where those bottles came from and when they were filled. Someone at the company did thank me for that. Crappy consistency like that is not good in a food product. Apparently they *don't* whip up a huge batch and keep it stirred constantly while filling the bottles. It must be a one ingredient at a time deal with poorly regulated flow, likely based on time instead of by actual measurement of weight or volume. As long as the finished weight is at least x, it's good, for low values of good. The crazy thing is, I don't like Italian or Vinaigrette salad dressing. Oily vinegar with little hard seeds sliming all over the lettuce? EWWW. Hidden Valley Ranch is my favorite. The *original* ranch dressing.
@goshoko8835 3 місяці тому
Yep. I work at a cereals factory. Our machinery drops into a package x grams of product. However it's got 5% margin of error. So you can get either 5% too much or 5% too little. We cannot send out a product with 2% less cereals than what the pacage says. So we calibrate the machinery to package at least 3% more cereals than it's supposed to. Later down the production line we have weight checkers. Packages too light are reworked. Packages way too heavy - I'm talking >5% also reworked. We considered different solutions. Get more precise machinery. Nope - bettter machinery costs around 1.5 MILLION £. For that money we can put in entire brand new packing line. Some dumbass whatever manager recommended hiring additional people to rework all those packaging. Nah, not cost effective. And the volumes we would make would go significantly down. So we just keep on sending out nice little bonus :D
@UncleButterworth 3 місяці тому
@@greggv8 Hidden valley is mostly soybean oil, so you're definitely still sliming all over your "healthy" salad.
@Genetic17 3 місяці тому
@@goshoko8835 Very similar process for bread manufacturing. We have 3 legal requirements: the lot average (lot is defined as the entire days production) needs to be meet or exceed the declared weight, and there are tolerances for T1 and T2 (+/- 15g, and +/- 30g) that have to be met as well. Every single loaf passes over a checkweigher where that information is stored for those averages, and if the weight is crazy out of spec, it gets rejected. We also do rework in the form of throwing finished loaves back into the dough, which was a weird experiment at first but ultimately made no difference whatsoever on the quality of the product and results in less food waste.
@eaturfeet653 3 місяці тому
When presenting Euclidean distance data you can use a dendrogram to present the pairs of CPUs that are closest to one another. Might help for intuitive visual interpretation of the data
@frederikex4545 3 місяці тому
Or using something like PCA plots or tsne bonus points for showing the audiance how to read scientific plots made for this kind of data ^^
@moonik665 3 місяці тому
@@frederikex4545 This, or the new kid on the block - UMAP. Makes graphing any kind of multidimensional data a breeze
@VuLamDang 3 місяці тому
@@moonik665 or their older more linear cousin, tSNE
@Solocoyote49 3 місяці тому
@@frederikex4545PCA also would potentially highlight which benchmarks/games have significant impacts on differences between performance. Add in a clustering algorithm like DBSCAN that can catch outliers and you're set
@ColaBeGaming 3 місяці тому
This guy diagrams
@muizzy 3 місяці тому
Small point of contention: Usually for distances of similarity like this it's better to use the minimum squared distance to reduce the effect of outliers. I'm sure someone thought of that already, but figured I'd mention it just in case.
@alexr1969 3 місяці тому
you can isolate a lot of the problems with pbo by setting the clock rates and voltages, then you can measure a lot of the inconsistent impacts of software (OS and benchmarks) and then you can give a reasonably confident statistical deviance due to those factors and find the actual impact of binning
@MrThevirus512 3 місяці тому
HUGE respect to all members of the team. It must have been really exhausting finishing all these tests.
@MegaLokopo 3 місяці тому
It is mostly automated at this point.
@robertt9342 3 місяці тому
How many months did they work on it?
@sierra2632 3 місяці тому
@@MegaLokopo it's like you didn't even watch the video and decided to comment because you heard the word automated and thought you knew what's happening. Amazing.
@AK-tf3fc 3 місяці тому
@@sierra2632 it's like you are a naive who watch cringe linus instead of gamer nexus
@the_undead 3 місяці тому
@@AK-tf3fcYou say that like gamers Nexus is dramatically better, does gamer is Nexus upload videos showing their processes that they used to get the performance specs. Has gamers Nexus spent over a million dollars on scientific equipment and learning how to use said equipment to get better more accurate numbers, I don't f****** think so, feel free to say he makes more entertaining videos all you want, because that is an entirely subjective point. But the purpose of this video was never entertainment, It was to show their processes and the struggles they are having getting data accuracy to a point where they are happy. Where as far as I can tell gamers Nexus doesn't even care about accuracy
@theftking 3 місяці тому
17:24 THAT GN SNUB
@sandmaster4444 3 місяці тому
I'm glad others noticed! 😂😂😂
@AshtonCoolman 3 місяці тому
I noticed too
@uchiaslaya7 3 місяці тому
I was looking for this comment lol
@ark_knight 3 місяці тому
Its been a thing for a long time since the warranty criticism long time ago. They dont name GN unless they exclusively need to. I think last time they did, Linus thanked Steve because he was the one who first noticed channel hack and called Linus to inform about the hacked channel. So its a very professional relationship now.
@jeremyO9F911O2 3 місяці тому
Oh we all noticed.
@GT5Player 3 місяці тому
From collab to no mention at all... Where is GamersNexus?
@dimitrimichaux461 3 місяці тому
It look like you missed something 😅
@Not.Your.Business 3 місяці тому
the saltmine runs deep at LTT
@aaronhopkins7354 3 місяці тому
A histogram could be a useful way to show the CPUs side-by-side. Perhaps 11 was too few to properly flesh out a histogram, but I know that I struggled to look at the column charts and really get a good feel for who, if anyone, was an outlier.
@claytonstanley2416 3 місяці тому
Loved this video as a statistician, and really like the approach you’re taking. It may be a bit much, but if you did an equivalence check pre/post for any gpu lineup you’re reviewing, that would be pretty solid evidence that any differences you found in the tests across the gpus is due to differences in the gpus. The other option would be to model the specific test rig in the regression, but then you’d need to put each gpu in each test rig, which would defeat the purpose of parallelizing the test in the first place.
@falcie7743 3 місяці тому
Inb4 copyright strike by Nintendo for using Pokemon names
@XxZannexX 3 місяці тому
Nahhh, Linus's 4D chess move of that Corsola pronunciation will evade the Nintendo ninjas.
@kuolettavaVids 3 місяці тому
They're currently overburdened with Palworld, so nows the time to do this.
@Respectable_Username 3 місяці тому
Nah wouldn't be a copyright strike. A trademark violation cease-and-desist maybe though!
@GeorgeJFW 3 місяці тому
@@XxZannexX😂
@quantum5661 3 місяці тому
nah nintendont is busy with palworld this month
@brundeasie_OG 3 місяці тому
This is the type of analysis I'm looking for! Very good experimental design and good presentation of these results. Are you going to publish this in a peer review journal?
@TurtleKwitty 3 місяці тому
Could be fun to see how things perform under a veryyyy paired down linux install and running the games/tests in wine/proton so that you can have near 0 background tasks and just the game running see how close things get then
@duckbilldaniel 3 місяці тому
(Tech horror story) There's an Azure server out there with a haunted CPU. We had a lot in the cloud, on one cluster we always had more errors with exactly the same deployments. We got around it for months by updating to a new cluster. Just a few months later the errors were back. I think that the old compute server got moved to the new cluster when the old cluster got decommissioned. And to this day.... I moved to ARM. Just because i know that x86 box is going to get me.
@Jimmy_Jones 3 місяці тому
Change to a different availability zone?
@duckbilldaniel 3 місяці тому
@@Jimmy_Jones company policy had reasons I can't remember for keeping availability zones the same. (I think there was a pre purchasing compute agreement ) Fortunately I moved companies and it hasn't found me yet.
@tiagobelo4965 3 місяці тому
Try getting some techpriests to help
@lapislazuli2896 3 місяці тому
@@duckbilldaniel"it hasn't found me yet" lol
@oblivion_2852 3 місяці тому
Would it be more expense than its worth to run a profiler on the containers to check performance? Can your company seek compensation for being placed on faulty hardware?
@Pete292323 3 місяці тому
Oh boy, people are going to be loosing their minds again for the average 1% differences.
@tcolec540 3 місяці тому
@@jake20479I have a 7800x3d. Am I the minority here for buying a chip, and if it runs my programs and games at an acceptable speed that I wouldn't notice a difference unless I lined it up with 11 other chips, that I'm just happy with my purchase and enjoy my games? (Run on sentence, I ran out of breath trying to read it. I'm also not fixing it though.)
@DanVibesTV 3 місяці тому
@@jake20479 you know that that is basically impossible, in almost every industry. no one can build 100% the same product every time
@NikolaguitarREZB 3 місяці тому
@@jake20479 Except you'd be comparing that 'any percentage less' to reviewers who most certainly will not share the same test bench, software variables, temperatures, humidity and several other variables. So, this is good for LTT to keep data similar, but for your average consumer, the rule of 'within margin of error' very much applies.
@manunen00 3 місяці тому
@@jake20479 I mean, I'd much rather a cheap cpu than a consistent one. Do you have any idea how expensive it would be for chip makers to get their consistency down under 1%? I guarantee whenever you buy anything at a gorocery store theres at least a 1% variance to someone else who bought that same thing, it's just pointless to make stuff that consistent.
@glenwaldrop8166 3 місяці тому
@@jake20479if you want the best you buy the best. We all know they bin the chips, we all know they don't all turn out identical. This has been the norm for decades.
@s3rit661 3 місяці тому
1:00 yes, silicon lottery, btw it happened the same with 5K series with AM4, they simply overclock themselves until 90c
@LordMoriancumer 3 місяці тому
Thanks, enjoyed the detailed description of the testing methodology.
@alexlowe2054 3 місяці тому
This is industry defining data. Genuinely. Anyone notice at ~9:50 when the charts fly by, that Corsola goes from being in the top 3 CPUs for cyberpunk at 1080p, to suddenly being the worst CPU in the test by a wide margin at 1440p. I expected different games to stress the CPUs in different ways, but I did NOT expect a single game to completely change the order of a single CPU. I guess this goes back to the performance controversy with Halo Infinite that Hardware Unboxed covered, where they found that the specific part of the game you benchmarked could completely change the order of the top performing GPUs. I suppose the same logic applies within the same game, for each resolution. That's crazy. Also, good catch with CS GO. Showing the differences between CPUs is great, but at ~700 frame per second, a microscopic difference in latency has an outsized effect on the FPS value. That's because the time between frames is actually the inverse of the FPS count. Which is fine when you're roughly around AAA game refresh rates between 30 and 120, but that inverse relationship starts skewing your data more and more the higher your FPS number is. High FPS games always have a larger impact on average performance, just because of the way math works. I'd like to see someone normalize using "time between frames" to measure those differences instead of the actual FPS number. That would probably give a number that better represents how different it feels to use two CPUs. It would also give more weight to games with lower FPS values, which is actually where performance matters the most. Almost no one can notice the difference between 400 or 500 FPS, but people can definitely notice the difference between 40 and 50 FPS, even though the "percent difference" is exactly the same. Absolute values matter, and the percentage difference calculation everyone uses can obscure very important data that actually matters. When I'm looking at buying parts, I don't care if a CPU gets 5% more FPS in a game that's already maxing out my monitor's refresh rate. Worst case games generally matter more, since the esports titles with high FPS counts are always going to be easy to run.
@wohnai 3 місяці тому
People dog on UKposts comments... But then sometimes you get good ones like this.
@Renegade605 3 місяці тому
Uhh... Except, because of the way math works, the inverse relationship between FPS and frame time doesn't affect the percent difference at all, and regardless of which way you show it our perception is a logarithmic scale. ie. The difference 40 to 50 FPS is 25%, and the difference 400 to 500 FPS is also 25%. Invert that and the difference 20 to 25 ms is still 25%, and the difference 2.0 to 2.5 ms is even still (surprise) 25%.
@ridwaan2443 3 місяці тому
@@Renegade605you're getting it but you're not getting it. Say an anomaly increased latency in testing by 2.5 ms. The 20ms case would increase to 22.5ms, i.e. 12.5% and the 2ms case would increase to 4.5ms, i.e. 125%
@Renegade605 3 місяці тому
@@ridwaan2443 what you just said is the exact opposite of the point OP was making. You're correct that a 2.5ms hitch will feel much worse at 2ms average frame times (400 fps) than at 20ms, except that OP said small differences in frame rate matter more at low average frame rates. Both are true, but for very different reasons in very different cases. You just can't simplify the data that much and at some point you have to trust that your audience is capable of understanding when and why the different numbers matter.
@alexlowe2054 3 місяці тому
Getting "um acktuallyed" in youtube comments by someone who repeated exactly what I said. Perfection. @@Renegade605 For reference, I said "the percent difference is exactly the same". That's a verbatim quote from my unedited post. Check yourself before you wreck yourself. We agree, so I'm not sure why you're telling me I'm wrong about the percent difference. On my other points, I'll clarify. When I mentioned "High FPS games always have a larger impact on average performance", I was specifically talking about average performance charts like the one at 8:37. Charts that take the FPS values from each game, and average the result. That's a graph where the inverse relationship skews the data. Specifically, it heavily weights the graph towards the result of the game with the highest FPS values, and reduces the weight of games with low FPS values. You can compare the graph at 8:34 that included CS GO and the graph at 11:41 which excluded CS GO, to see how much a single outlier game affected the entire average result. That inverse relationship definitely skews the results when you average the FPS values between games, as basically every reviewer does. That's a big problem, because an extra 3fps in CS GO mean basically nothing when it's already maxing out your monitor, but 3fps more in cyberpunk at 4k is a pretty big change. However, both those changes are averaged out on the same graph. That's why the inverse relationship is a problem. Rewatch the 10:43 "CS:GO is Wonky" segment to have Linus explain why games with framerates that high aren't valuable for benchmarking. There's some CS GO specific stuff there, but there's also a lot of statements that can be generalized to all high FPS games. If anyone is still confused, I can clarify further.
@iaxyz 3 місяці тому
One of the best videos you've made in a long time. I really like the labs content and the transparancy that you're trying to give. There is a bunch to learn about testing to learn in this video. Really looking forward to more of this content.
@Corn_DOG 3 місяці тому
Hey Euclidean Distance! I remember that gem from my grad school days. We used it for satelight to classify pixels as land types/change detection
@magnutron 3 місяці тому
I have personally experienced this with the machines I have built as a system integrator. On top of this, I have also seen varying degrees of stability and ability to PBO.
@Byefriendo 3 місяці тому
11:53 Colours on 1% low and Avg are inverted
@-Ninnux- 3 місяці тому
Gamers Nexus about to make a third rant video
@niikon 3 місяці тому
Triggered my OCD - Thanks LTT.
@ikillyou96 3 місяці тому
Throw in a benchmark running under a minimal Linux install for good measure, will give a good indication of raw performance without many background tasks.
@thegeekno72 3 місяці тому
How many of us game on Linux ? They run tests under windows because that's what the humongous majority of us use to play
@rigf1997 3 місяці тому
@@thegeekno72 But they remove a lot of the bloat anyway, they try to get the raw performance of the CPU's. Not the performance of the CPU on Windows. I'm not sure why they don't use Linux, maybe they've tried and something didn't work out?
@thegeekno72 3 місяці тому
@@rigf1997 Linux raw performance and Windows raw performance are not comparable things, they're software that have wildly different operation, it could be analog to comparing X86 and ARM, it's just not the same in the way it behaves
@Apeeksiht 3 місяці тому
@@thegeekno72we are talking about cpu performance variance doesn't matter which platform. linux might have given better consistent results because of less bloatware. Regarding arm and x86 they are different cpu architecture you comment doesn't make sense.
@adrian2ka46 3 місяці тому
@@rigf1997linus said that he wants to be relevant for the general user and that pretty much explains it all. Yes maybe using linux will give interesting results, not gonna lie but things change once you change use cases and os and the difference might become minimal for general users tasks.
@KarsonNow 3 місяці тому
In comparison of the results in some sports - they get also scores from the jury + the results itself for example in ski jumping. They always cut the best and worst Jury scores out to prevent bias/prejudice.
@Olimorveu 3 місяці тому
About the car-industry comparison. I'm not sure if the car industry is much different. I do believe that there is some batch-to-batch variance in engines etc. The quoted horsepower can differ, and some manufacturers provide an underestimated specification to the customers, such that any car is guaranteed to meet it (same as with the CPUs). I think that there is no problem as long as the quoted specs are met. If we get into supported memory speeds vs. the overclocked memory speeds that are sometimes used in their marketing, then we run into a zone that I personally like much less.
@romannasuti25 3 місяці тому
I’d say the variance is actually higher in autos. Everything is tolerances, and a piston on the lower end of it’s accepted tolerance and the cylinder being on the high end would mean a slightly larger than normal gap around the piston, which can introduce some oil burn if the car is meant to take thinner oil. This isn’t a huge deal at low performance, but high output engines can end up with problems from something as innocuous as this. Hell, a piston being at its upper tolerance and the piston head being as it’s lower tolerance could increase the risk of seizing.
@romannasuti25 3 місяці тому
One way to think about it: 99% of V6 Toyota 4Runners in the late 90’s were perfectly fine and even famously reliable, but some minor flaws in the design in that specific era of V6 meant that, if you were unlucky with a marginal tolerance engine, you could end up with cooling issues that could blow it up.
@ZimmMaster 3 місяці тому
I loved how much meme-age they worked into their BSOD in the intro. They even got the QR code go to the LTT store lol
@BoraHorzaGobuchul Місяць тому
Absolutely no surprise re marketing included.
@TeunSegers 3 місяці тому
I just imagined Steve in the frame looking over his shoulder critically all the time. ... and Gamers Nexus wasn't among the "multiple outlets". 🤔
@pitmaster226 3 місяці тому
Gamer’s Nexus used to be referred to regularly when Linus mentioned other outlets (check old wan shows). Steve repeatedly set fire to the bridge that they had. The exposé was when he took a wrecking ball to it. My personal largest gripe: Steve took issue with LTT saying in a video “Our numbers seemed completely wrong, so we checked in with AMD to make sure there wasn’t something screwy” Then, in the next AMD Card review, Steve says “We had weird numbers in this benchmark, so we checked in with AMD and received this response”
@megapro125 3 місяці тому
I mean I like the GN content generally but I can see why LMG wouldn't exactly advertise that channel after some of the less than comradely stuff they pulled.
@megapro125 3 місяці тому
@@tappy8741 well if you literally got the other guys phone number I would have at least shown them the courtesy to make a call to make my point before pouring gas into a public shitstorm but maybe that's just me.
@rawhide_kobayashi 3 місяці тому
kinda funny that GN literally did this video a couple months ago, with derbauer, and actually hardcore overclocking, and got pretty close to the same variance... and they were all just like "yup, seems about right" rather than "government pls help, my test bench isn't as easy as I want it to be"
@TeunSegers 3 місяці тому
@@pitmaster226 yeah, that's a fair point. I get both perspectives. I hope they can work it out in the future.
@ramtinnazeryan 3 місяці тому
You guys doing amazing R&D on this bench mark. Was wondering if you have a plan to publish detailed experimental design in a scientific paper or something to help others catch up?
@zachfosterWeeklyVlog 3 місяці тому
been looking at the labs webpage the last few weeks wondered why so little was there expected there must be limitations on what youve been working on looks like im right glad yall are taking the time to do it right!
@Substant_ 3 місяці тому
Respect for the Vegemite shout out. Love, Australia
@bruceanderson8588 3 місяці тому
expect thats not vegemite ,real stuff is black not brown
@coolgirladrian 3 місяці тому
If you ever name something similar to like you did with the Pokemon, I would consider doing it alphabetically. So the first one starts with A, second starts with B, third starts with C and so on and so forth. It's just one of those nice-to-haves that doesn't change much but makes the presentation a bit easier to digest
@custard131 3 місяці тому
lots of interesting info about how 2 of the same model cpus can perform differently, i do have 2 main thoughts though firstly, if the end goal isnt to test the cpus themselves, but just to have cpus that perform the same to test gpus, would there be any value in setting them to a fixed clock speed of say 5ghz rather than letting them turbo? similar to the point about older cpus/ram, would that not remove the factors of how willing a particular unit is to turbo (and maybe also remove factors like the thermal compound application) secondly about the point of comparing to automotive industry, i do agree that there could be some benefit in an official standards agency, but im not sure it would help too much with the issues described here, those official tests that are performed on cars are generally only checking that the car as at least as good as some regulation/manufacturer claim, cars dont fail the tests because they performed better than the manufacturer said it would, and generally i dont think consumers are going to be upset if the product they purchased is better than the manufacturers claims (whether thats a car or a computer). tbh i think the reason its not actively regulated is just because the stakes really just dont matter in the grand scheme of things, the consequences of how a car performs on different road conditions or in a crash is literally life and death, how many fps a gaming pc can deliver probably isnt even life and death for the virtual character you are controlling nvm any real people.
@cricketerrushi 3 місяці тому
Possibly the most thorough and well-explained video on the effects of the Silicon Lottery I've ever seen. The only thing I found missing in the video was the thermal performance of these chips and how they maintain their clock speeds. I have been there when I compared my Ryzen 5 3600 with another R5 3600 of my friend with the same mobo, RAM, and Storage. The difference in thermal performance and the clock speeds was noticeable... that was the day I realized silicon lottery is a thing too.
@tbrooke3016 3 місяці тому
This video feels like it would be great for teaching high school science students about isolating variables and the impact it can have on results! Such an interesting video!
@GiacomoDose 3 місяці тому
Awesome video! Lab is gaining a lot of credibility after delivering such great and in depth analysis. Impressive! I must admitt however that is a bit sad to see at 17:24 no mention of gamer nexus.. I guess the community would love to see a reconciliation... I'll keep dreaming
@pitmaster226 3 місяці тому
The ball’s in Steve’s trunk. “If an article contains personal or serious allegations or claims against an individual, it may be appropriate and necessary to give that individual an opportunity to respond to these claims, or to deny them if they wish” - Independent Press Standards Organization But they’re UK based, so why should Steve care (:
@ionutpogacean 3 місяці тому
yeah, a bit of a shame. especially since GN, HUB and Der8auer did a video on this exact thing ukposts.info/have/v-deo/iIWWjodqp6Gqjqs.html - GN ukposts.info/have/v-deo/nHeTi22mqZCEjnU.html - Der8auer
@aaronmorrow4957 3 місяці тому
I noticed the GN omission too. Really sad to see.
@backinbusiness95 3 місяці тому
I don't get why you don't show the clock speeds, which are most of time the first indicator for different results. Der8auer did something similar.
@elatedmaniac 3 місяці тому
I know this was extremely time-consuming and detailed, but it gives me confidence in watching your channel and the results you obtained. Just want to stop and appreciate the hard work everyone is doing at LTT.
@Bugattiboy912 3 місяці тому
Anyone notice how they didn't show Gamer's Nexus when mentioning "other outlets"?
@etherealmistlc 3 місяці тому
I noticed a bunch of channels that I sub to that they didn't put in there. Very angry right now.
@joog79 3 місяці тому
Not surprises though after the nutpunch GN delivered to LTT. Eventhough it made LTT force to do beter.
@John120196 3 місяці тому
I literally scrolled to find this comment as soon as I saw it :P
@MarioGoatse 3 місяці тому
Lmao. Yes. Very clearly on purpose. I knew he wouldn’t show it as soon as I what they were saying. Hilarious
@spdcrzy 3 місяці тому
TBF, GN also isn't doing the same exhaustive kind of testing in the same space. Their niche, and by definition the type of testing they do, is completely different.
@joao-o-santos 3 місяці тому
Amazing work @LinusTechTips! Your discussion on sources of variability really got my brain going. As someone who works with psychological data, this is something we often discuss (inter-trial vs inter-participant). Have you looked at using linear mixed models (aka hierarchical models, aka multilevel models)? They could allow you to control for multiple sources of variability without having to aggregate data so much.
@mjouwbuis 2 місяці тому
Nice test and as far as i can tell statistically sound... But I think it would also benefit from a raw benchmark without operating system influence to establish some kind of base line. After all, you're comparing CPU's in the first place and CPU's running applications in the second place - at least in this specific test.
@nikeshkunwar24 24 дні тому
crazy to think that the 1-2 % sillicon lottery difference could be the power of thousands of old 6502 cpus
@perplexedon9834 3 місяці тому
Yeah after you explained it I am not nearly as mad about this as I thought I'd be. If they design it and manufacture it to achieve a standard, but include features that may be able to get even MORE than advertise, then I don't really care about the variability of how much more it gets. It's kind of like rock climbing gear, where a carabiner will be rated to 22kN of force. 99.7%+ of the time it'll be stronger than this, and it could easily vary from 23kN to over 30kN. That's be like a CPU having a performance spread of nearly 40%. The thing is, the lowest number of 22kN is still 5x stronger than any realistic safety situation, and so maybe we should think about CPUs in terms of the demand we expect to place on the. Eg. "This CPU will have a 99.7% chance of running *game* at more than 60fps/144fps/250fps" For your testing, maybe Linux with Wine/Proton would be better in terms of minimisation of background tasks? The lack of oversight vs automotives and, indeed, rock climbing gear kind of checks out because if your CPU is 5% slower, you aren't going to find yourself in an accident where your brakes fail and you, your partner, your two kids and another random family all fucking die. It is definitely worth doing in a decentralized, nongovernmental fashion though. Maybe a consumer union could be made where manufacturers have to provide representative samples in order to get a trademarked seal of independent testing.
@themisterx8660 3 місяці тому
Yeah I think you are right. There are margins in probably every industry so I think the consumer has to be fine when the product reaches its rated performance. When we talk about cars for example, there you can often measure differences between the power of engines. But then the average consumer doesn’t care about fuel quality, tires and regular oil services that much, so probably those have more impact on the actual performance on the road than the engine itself. The same is probably true for pc part’s performance in different countries with different average temperatures. So in the end I guess you should buy a product by it’s confidential rated speed and be happy if you get more than you paid for but not be necessarily sad if you don’t gat that much of a benefit. Your friend might have ended up with a better chip itself but maybe he has a low quality power supply, which makes the cpu perform worse than yours. Or maybe there are also margins in psus?
@Dr.Spatula 3 місяці тому
@themisterx8660 the margins are in EVERY industry. If it's not federally regulated, it's financially regulated or consumer regulated... excerpt vitamins and supplements...
@memethief4113 3 місяці тому
the problem with using Linux is that it's just not a realistic OS to use for benchmarks. Linux has many problems working with NVIDIA GPUs, but the main thing is that most people just don't run Linux so it's better to just use Windows, even if it's bloated and makes getting reliable test data harder
@Winnetou17 3 місяці тому
@@memethief4113 True, but it can be used for non gaming tests. As it stands out, Linux will have to be used in specific tests. Like you said, having Linux be usable for ALL tests, we might get there one day, but not today. Not this year either. Maybe next year, in the most optimistic timescale, but realistically I think it's more 5 years, IF it continues to grow like in the last 2 years.
@perplexedon9834 3 місяці тому
@@memethief4113 I guess the argument would be that it would be fore controlled testing. If one chip ran a game with proton on linux at 100fps and another at 105, then you could meaningfully say the second is 5% faster, even if on people's actual systems the real world framerate would be 45 or 145 or whatever. The problem with testing in realistic scenarios is that realistic scenarios have a lot of uncontrolled variables. If you tested on windows you could get 145fps or you could get 95 depending on what windows fuckery is happening in the background and you couldn't know without doing 100s of tests and averaging it. Also Linux gaming, even with NVIDIA, is my personal main use case :P.
@maxgarcia9829 3 місяці тому
as someone who works in big data and analysis of it the way you guys described all your statistic analysis in a way easy to digest for regular people was amazing 10/10 vid
@vincenterrickitucal6265 22 дні тому
Kudos to the use of euclidiean distance in determining similarity. Other measure of similarity can be used is cosine similarity, jaccard, etc. Its nice to see LTT actually doing Data Science work. Maybe could use Principal Component Analysis (PCA) to see which of the features is the main contributor to the difference between the highest and lowest rated CPU's
@Victus-ih7tx 3 дні тому
As a computer engineer student myself, I want to say something to help Linus while he creates these type of videos. those processor's are exactly the same but they aren't exactly the same at the same time and there is a multiple reason but ill just tell eligible ones so you guys understand. First one is DIE-Connecter contact with the motherboard, which is sometimes even the slight changes on contact surface between CPU die and motherboard pins are eligible to create those type of problems, that's why we don't have CPU extension units but we have PCIe extensions? we do but it doesn't mean they don't take any performance cuz they do take some performance. and the second thing is when you test CPU the neutrino's (beams come from sun that goes through everything) is effecting CPU itself. yes i know its so rare or sometimes nearly impossible but its not as you might think. the reason behind this is neutrinos can change bits on registers or even ram. so there is a lot to say about it. We can trust about CPU manufactories about those type of things since it provides enough performance (nearly)…
@richardwiech 3 місяці тому
Love the update on testing methodology! Very detailed and I do appreciate that! I did notice that Gamer's Nexus didn't get a spot in the B-roll alongside the other creators, I honestly hope you two make up and I strongly recommend looking to them every now again for sanity checks because they are really great and highly respected.
@Martycrane 3 місяці тому
I noticed that as well. The Steve's are my go to reviewers for almost all purchases on parts
@the_undead 3 місяці тому
I lost all respect for that pile of garbage when he made that hour-long video, as someone who knows a great deal about human psychology, it is obvious to me that that video was not made for any reason other than revenge. And more specifically revenge about an LTT labs employee calling him out specifically on his data testing procedures, more specifically calling into question how accurate gamers Nexus testing is. And seeing as Steve's response was that hour-long video instead of an hour-long video detailing his processes or the updates he's made to his processes, I'm going to suspect those same concerns are still valid today, and I feel this is partially confirmed by the lack of gamers Nexus in that b-roll shot, Yvonne and Luke would not allow Linus to hold a grudge for this long, also, linus isn't even the CEO anymore, so I suspect there was a reason other than the controversy for the lack of a gamers Nexus in that shot
@Spawn- 3 місяці тому
@@the_undead "as someone knows a great deal about being a fanboi" If my firm would be slandered like it happened I'd do exactly the same. And oh btw to clear your delusional POV even more... Surely LLT just issued apologies and shut down testing because there wasn't any truth about the claims... surely! If you fire a gun you better be goddamn ready to get backfire. Back to School or where ever you gained that dubious knowledge about human psychology.
@jackalovski1 3 місяці тому
As someone who used to do statistical process control for safety critical electrical components in the automotive industry, I approve of the comprehensiveness of this. Now that you have the process laid out for doing this, it will only get faster and more routine.
@stupidscholars9842 3 місяці тому
I think I speak for everyone here that this is EXACTLY what we want. We love tests like these, in depth, practical and perfectly executed. Thank you so much.
@LeitoAE 3 місяці тому
I think it would be great if you would take those worst performing samples and try to bring them back to stack by changing Pbo settings. If they are falling behind due to thermals and thick IHS, it might be a good tuttorial for people that ended up with preety non - gollden samples. I mean that even if particular chip might not perform well in stock, might be better for OC or maybe will be a king of efficiency and allow for a high voltages drops? Even if somebody is totally unlucky, maybe at least you will come up with a few setup tricks to increase it's efficiency.
@robertpatton26 3 місяці тому
Love the focus on consistency and variance, and would love to see error bars on future comparisons to quantify this!
@mikerosato83 3 місяці тому
Great job guys! It would be nice to see another video the same detail on the latest 13th gen and 14th gen Intel CPUS
@Donivar 3 місяці тому
Problem is: Silicon lottery can be in the cores, in any of the caches, or some other component of the CPU. Also, when benchmarking games, you can't just trust the game to choose the "best cores" to run. Stuff that uses all the cores should provide a smaller variance between runs.
@zebkayne4253 3 місяці тому
#1 - this is the labs content I am here for #2 - I hope you swapped the CPUs between the benches they were on, to ensure the issue is on the CPU and not something else on the bench.
@vyor8837 3 місяці тому
They didn't do the second, indeed they tested with EXPO which is an *extra* variable that AMD does not list as a stock configuration.
@joaopedrosa9060 3 місяці тому
honestly, HUGE respect for doing this, first of all, the video is hella interesting and second of all knowing that you guys do this is extremely reassuring. great job dudes.
@joaopedrosa9060 3 місяці тому
@@boscotheman82 thank you! but I did not ask
@Ceelvain 3 місяці тому
I love this kind of videos when you show the behind the scene of the lab. This is the most literal "computer science" that could be done. ^^ (Usually, "computer science" is not about computers as Edsger Dijkstra once said.) As a totally unsolicited advice: I might just suggest considering the Chebyshev distance instead of the Euclidean distance since you are interested in avoiding having a sample be an outlier in one of the tests. (Other high-order Minkowski distance would trade the exact rejection of outliers with some "average closeness".)
@nollandvoid 3 місяці тому
It's so crazy seeing LTT do what's my job. Great to see the attention to detail. (Having to keep myself from doing uncertainty calculations right now)
@riddixdan5572 3 місяці тому
love to see factorio used as a benchmark, been playing that game non-stop for a while now :D
@madeintexas3d442 3 місяці тому
This is some incredible research. Not many, if any review channels that do testing would go to this level of effort to obtain the results you have here. I think the amount of effort that it takes to obtain these different samples is probably a reason why. I have always heard of the silicon lottery but did not think about that the lottery may be based on the piece of silicone being used were the environments of the factory. As an afterthought it seems obvious but of course you would need to obtain units from different manufacturing runs to get samples that would perform differently. I appreciate the effort that it took to do this and this demonstrates the true value of The LTT lab.
@AveryChow 3 місяці тому
8:31 honestly really disappointed to see that diabetes didn't win out over the other CPUs. I had my bets on this one :(
@accordingone3598 3 місяці тому
Well, it makes progress in the real world tests.
@-.-Elle-Marie-.- 3 місяці тому
10 years watching LTT, it keeps getting better! Great job team
@rafaelUSP 3 місяці тому
It's soooo cool to see you guys using math that I learned getting my bachellor in system information. Keep up the excellent work!
@TheTakenKing999 3 місяці тому
Intriguing, might have some better results with some other metrics like Mahalanobis distance instead of Euclidean to account for more correlations in the variance trends. (Mahalanobis is just Euclidean distance in PCA space). Loved this detailed video! Hope LTT does more videos like this!
@njhdunn 3 місяці тому
I had a similar thought, there is likely significant correlation between the dimensions in game testing space. My brain went a linear algebra route - the axes in game testing space are (probably) not orthogonal, so you would want a metric tensor to allow for proper definition of distances. The Mahalanobis distance route you propose is probably more realistic to actually use here, but I'd love to see an exploration of how to treat correlations in one way or another. Or to just have access to the dataset to fiddle around with it myself, now that my interest has been piqued. I also loved this video and I look forward to seeing more Labs content as it develops.
@bulletflight 3 місяці тому
I was wondering what Mahalanobis distance meant, then I realised it used standard deviation. Then some of my high school math came flooding in.
@clayhudson7104 3 місяці тому
Freaking amazing video, well done everyone. It would be cool to see if the results are still as varied if the clock speed was locked across all of the cpus. I assume this wasn’t looked at because that’s not as equivalent to the real world use of the chips, but still could have been an interesting add.
@cartman20000 10 днів тому
I wonder what the average tempurature was for these cpu's, as that can have an effect on clockrate and voltages being applied to the cpu's.
@Hellsfoul 3 місяці тому
Maybe also measure the power consumption for each CPU. Der 8auer did that and it was very interesting.
@ramih43082 3 місяці тому
Very useful video. Interesting that you didn't list Gamers Nexus despite them making an even bigger 68-CPU test back in August.
@Nekudza 3 місяці тому
Also noted that, too bad if they are not friends anymore
@Shapershift 3 місяці тому
Linus' ego is probably still bruised.
@skak3000 3 місяці тому
@@ShapershiftIt's Steve's ego, he was insulted big time when an LTT employee on a tour criticism he's way of testing. That was started it all... 😂
@soul_slayer7760 3 місяці тому
@@skak3000 Dude, if someone criticises your work, would you do nothing? Well, he did, and he also proved with data and examples that actually his testing was better, and more reliable than the LTT ones, which were ridden with errors. Everyone knows Linus is a little narcisistic, he said so himself, and he apologised. That said, I watch them both and other outlets to make my decisions, I just don't understand why people side with anyone in cases like this, research everywhere and make your own decisions. Watching only one outlet and trying to protect them like they are your friend is just the stupidest parasocial shit. Fuck GN, fuck LTT, they are means to an end. Stop thinking emotionally and start thinking rationally.
@yutt 3 місяці тому
@@soul_slayer7760 People who say they are thinking rationally are cringingly hilarious - and peak irrational. Anyone who thinks they or other people "think rationally" are so ignorant and arrogant it isn't worth discussing.
@LightningTechNL 3 місяці тому
This is a good investigation! I've been waiting for someone on this investigation to come out because, as a system integrator for CAD pc's I thought I was losing my mind! Ever since the 13th gen intel and the X3D line that AMD brought out, I've been seeing these inconsistencies. And I was really worried that something in our testing wasn't right! Glad to see my sanity is in check... But I REALLY think you should also take a look at intel! The 13th gen was already a real 'silicon lottery' thing, but the 14th gen intels are... WAY out of spec now and then! PLEASE PLEASE check intel too!!!
@ABaumstumpf 3 місяці тому
", but the 14th gen intels are... WAY out of spec now and then!" But in general that is not the CPU but the motherboards. Sadly intel allows the boardpartners to run insanely bad automatic overclocks as the default settings -.-
@LightningTechNL 3 місяці тому
@@ABaumstumpf OK! Nice to know! But the thing is. If I swap the same cpu with the same cooler between different motherboards (of the same type that is, we use the b760m aorus elite a lot and the b760 tomahawk) I don't see as many differences as when I change the cpu's out... When I only swap mobo, I see a difference of about 2% max in cinebench. But between different cpu's on the same motherboard I can get up to 7% difference! So, what am I doing wrong in testing?
@ABaumstumpf 3 місяці тому
@@LightningTechNL "So, what am I doing wrong in testing?" Pretty much nothing. Getting an identical CPU in the same motherboard with Intel normally stays within 1-2% of each other, at least according to the 3 companies i know the hardware off. If you see bigger differences then first thing to check would be temperatures and powerconsumption as many boards increase Voltage and thus powerdraw even without affecting clockspeeds which can lead to shorter boost-durations. This is done excessively with Z-chipsets but i had seen that with B boards too. Had built a 12600K for a relative (for broswing and looking at pictures :P ) and disabling all the "enchancements" and running it according to specs has dropped the powerdraw to sub 100W at full boost. there are always some differences in how efficient each CPU is and the boards increasing powerdraw for no reason will exacerbate those differences greatly.
@thezaret639 3 місяці тому
I had a 4790k that could overclock way over spec kinda sad to let it go but needed an upgrade but it served me well for many years
@CarlizWang 3 місяці тому
I lost the silicone lottery with my 7950X3D, out of the box it performs 10% worse than most of the major reviews online. It doesn't like undervaluing, runs hotter than expected and even after much tinkering it barely matches the performance of most with stock settings. Worst of all, sometimes the system refuses to post at all during start up even though it's completely stable in all my stability tests.😢
@rafal5863 3 місяці тому
That is what makes synthetic benchmarks more consistent and deterministic. But are less relevant to real life performance which is not deterministic.
@emmamitchell1582 3 місяці тому
It has been shown that degradation will cause these chips to clock lower over time. I would be interested in a retest of the test benches after they have been used extensively. Both to validate that they are still identical, and to quantify the degradation. Edit: what about the ssd performance influencing performance because assets are streamed while gaming?
@aser6494 3 місяці тому
Exelente video, se valora muchísimo el esfuerzo en hacer pruebas tantas veces y transmitir la información de manera tan clara
@Lampe2020 3 місяці тому
Well, when the OS is doing random stuff you can either use minimized Window$ or even a highly-configurable Linux distro to prevent as many random factors.
@BoraHorzaGobuchul Місяць тому
On the other hand, that may provide "spherical x in vacuum" results which have little bearing on real-life where peculiarities of an os may result in skewing performance in a particular direction. A very complex issue, hard to see which approach is best
@Dan-vl6fk 3 місяці тому
0:52 someone’s got a good taste in keyboards, i love seeing the moon drop dash getting some love!
@alaingraham 3 місяці тому
Glad to see a local chain on your comparison (UK - Computer Orbit)
@low-key-gamer6117 2 місяці тому
As a microelectronics student, larger the wafer larger the variation among chips in general. Larger the chip, larger the variation inside the chip, like variation between different blocks.
@jarnom85 3 місяці тому
I noticed this in 3 7600X's i have had, one did 5.15Ghz all core (was also hottest running by 5celcius), one did 5.50Ghz all core and the third did 5.45Ghz all core with just PBO enabled. Also 5.45Ghz one seemed to have best IO die as it did 6400Mhz with just enabling EXPO on my 32Gb DDR5 6400 CL32 memory kit and 2133 on FCLK without any kind of stability issues. 5.5Ghz one did only 2067 FCLK and 6200Mhz CL32 on memory at best, it would boot and work with 2133 FCLK but would also show high latencies in AIDA memory test and performance was slower, this was not the case with 5.45Ghz one. All were tested on same mainboard. Silicone lottery is crazy.
ТРІНЧЕР: чи був се*с після розлучення? Я намагалась врятувати ШЛЮБ.Чоловік має заробляти БІЛЬШЕ мене
Okay Eva
Переглядів 366 тис.
Thebox - о технике и гаджетах
Переглядів 22 тис.