funny seeing you here, love your vids

Easier to digest and more accurate statistics content on PrimerBlobs's channel And the currently 1.7 million subscribers agree

I thought he was just nervous or trying to fit in in a small time lol

Wait what do you mean "normal"? Does he have a turbo mode???

@@0netwoguy54 GAS GAS GAS

Where did they say this? I’m having a hard time finding it.

@@Sarmachus Sorry I can't remember exactly. I saw it in a game development video a year or so ago, and I believe he flashed a tweet from one of the MC devs on the screen. Regardless of it's authenticity it still serves as a good example and valuable lesson.

@@_lime. Thanks for clarifying

@CruzZ fake news

@CruzZ what are you talking about. This guy provided a ton of real world examples where statistics could help solve a problem. That doesn’t mean people will always use statistics for good though, he even mentions that in the presentation with an example. Just because big gaming companies suck at stats doesn’t mean his presentation wasn’t phenomenal!

@@dontfk Big gaming companies most definitely do not suck at stats, if anything, that's the one thing they master above all else. It's just that most statistics are not relevant to the players enjoyement. They are very relevant to shareholders tho.

@@ailurusfulgens1849 You're right, I used poor word choice there. What I meant by that was that they don't always use their stats for good intentions

Extremely simplified. Statistics is, like, a whole field of mathematics.

It matters a lot for procedural generation, as statistical distribution is a huge part of random number generation. It can also be used to approximate various things... replacing physics in some cases. Sometimes called an "analytical" solution, you can see this show up on some games oceans, etc. The ocean is based on statistical analysis of real oceans, instead of trying to actually simulate fluid dynamics. I'm sure there are more uses than that, especially outside the game.

Yeah, that thing he did was called hypothesis testing. That took me a good 20 hours during a single week to figure out how to do it by hand at school. Finding out that it could be done in a minute in excel blew my mind.

As a data scientist... it's a LOT. But really, you don't need all the math to do it practically. You really just need to know the basic definitions, and what the test does. And there you go you got analysis. If you're a game dev, assuming you got some programming experience, you can already do a lot of these things in the language R, with very little effort, and even very easily build some machine learning models.

I essentially got most of this stuff during my semester of statistics class. As he said, he pretty much blazes through it, you mostly need time to understand when what is used, why to use it, what the downsides of using it are, etc and lastly of course, HOW to use it.

Eiði

Chik-fil-a is not a five-star establishment, people >_>

Bayesians can play around with their Bayes factors all they like, but at the base, they’re still operating under a frequentist model if theyre gunna do any form of null hypothesis testing. Without a criteria to reject the null (p val), you can’t falsify a hypothesis. So collect all the data you want and build up those Bayes factors, but you’re not escaping the problem of induction. :) Frequentists of the world, unite (and not be undermined by a single black swan)!

@@hamm8934 The belief that you can "reject" the null hypothesis based on a single yes/no measurement IS THE PROBLEM. (Sorry, got a little loud there.) Look at the PDF. Draw conclusions about underlying behaviors. Make better predictions and test again. Do not pretend that "there's a 96% probability in this case" and "there's a 94% probability in this case" are vastly different, binary outcomes.

@@jonwatte4293 what statistician or scientist worth their salt believes that a single positive or negative outcome is sufficient? That’s a bit of a straw man. Of course you either (1) directly replicate the result or (2) perform an extension with a different operationalization of the same hypothesis. If it isn’t replicating approximately 95% of the time, it’s quite safe to say the effect isn’t there (assuming adequate power). If it is replicating approximately 95% of the time, it’s quite safe to say the effect is there. The point I (and other frequentists) make is you have to have a criteria of falsification for null hypothesis testing. If you don’t, the very logic of hypothesis testing collapses as you are no longer able to discern a success from a failure. You have to make a judgement call for null hypothesis testing to exist. This whole notion that Bayesian stats somehow avoids or overcomes this judgement call is a complete failure to acknowledge that you are still making a judgement call, just with a different threshold. (See chp 1 and 2 of The Logic of the Scientific Discovery). Get those Bayes factors as juicy as you want. It just takes 1 falsification for them to be undone. We’ll see which method is more fruitful :)

​@@hamm8934 bruh I feel like you're still being incredibly disingenuous about this whole thing. The key issue with NHST is that a p-value *only* tells you p(Data | H0 = TRUE)-that's it, full stop. The far more interesting question is p(H | D), and that's entirely beyond the realm of classical frequentist methods. 'Rejecting the null' with p < .05 doesn't mean that there's a 95% chance the null is indeed false, or that the alternative is actually true. What we should be doing is systematically pitting models against each other, and this, I think, is something Bayesian methods are exquisitely well-suited for. And sure, there are some rules of thumb when you're doing Bayesian model comparison and trying to figure out how 'meaningful' the difference between models is, but it's a laughably false equivalence to say that the process of multi-model inference (literally comparing the evidence in favor of competing models) is anything close to a binary NHST decision based on differences in means or a correlation. Not to mention you can compare models based not only on the parameters you include, but on your priors, or the underlying likelihood function... Shit, you don't even need to use Bayes Factors-it's super trivial to compare models via their posterior predictive densities using Bayesian cross-validation with PSIS-LOO. All of this ranting is basically just to say that 'all models are wrong, but some are useful'-and I think if we really want to find the best models that explain (or even better, can *generate*) our data, you're gunna have a bad time with frequentist NHST.

@@neur0leptic782 Preach

This depends on assumptions; the assumption here probably is "I am optimizing my game for ability for at least most of the initial group to make it through".

That wasn't actually the point he was trying to make. Especially with a small sample size, outliers greatly skew the mean. As for the point of a few dedicated players willing to spend money, that only works if it's a game that does not depends on having an active online community.

I know I'm really late to finding your comment, but I thought the same thing! Also, Mark Rosewater (of Magic: The Gathering) has a presentation on UKposts about Game Design and in HIS opinion, that highly polarized distribution is better. It's better to make something that SOME people love even if some other people hate it, instead of something that everyone gives a 'meh' to. In game design, I think it's the difference between "cult classic that some people love and play forever" and "totally forgettable game that disappears in two weeks". If at least some group loves it, it can spread by word-of-mouth and certain reviews. Provided your budget was appropriate to build a niche game, you can have a success... while some game that everyone merely tolerates probably makes no impact and loses money.

@@stuartconrod8364 exactly :) Better to be hated by 90% ignored by 5% and loved by 5% Then hated by 20% ignored by 80% and loved by none. Who is going to spend money on a product they don’t love when they have so many alternatives. Plus all those haters are free press too! I think we should lean into the fans more, look at dark souls, its brutal unforgiving and very niche, but clearly doing fine. League of legends, unforgiving, brutal player interactions, but doing fantastically well. Counter strike, same thing. Yes i do think we should keep games accessible, but Not at the cost of what the fans love. I think for example what halo infinite is doing is great, bringing back bots to practice offline before going into the fray. Allows the multiplayer to be as cutthroat and great as it always was, not with unlockable weapons thate give you an edge at the start of the round, No everyone starts with the same weapons, and you need to earn and fight over better ones, so its a true skill matchup. Thats why ists so unforgiving to new players, but also why its so incredibly good.

@@FreekHoekstra it really REALLY depends on how you make money off your game. If it's some recurring revenue, then you need to retain a decent number of players. If it's a game which has interaction between users, then you need a decent player pool. If it's a one-time purchase, you can keep it mediocre across the board. If it's for e-sport publicity, you better make that as balanced as possible. Make the goals easy to understand and controls simple enough to get players pouring in. Overall, no matter the game, a larger pool of players will bring more potential spenders, and of those players only 20% of them will be providing your entire income. Money keeps a business going. So making a game for only 2 people is a ridiculous endeavor unless each piece of content is a guaranteed buy and they cannot continue into the new 'season' without making their purchases... Though if only two people are playing they'll need to be spending hundreds of thousands each time you release content. in a free to play game competition drives purchases. You need some fodder for the big spenders to show off their purchases/power to, or they have no reason to buy the newest released item/cosmetic the day it comes out.

it is.. but its is not its a video on how poor ftp gamers flock because lack of money... and how to get them to spend more.... and about how bad ssd are... in 2016 but are now 40-60% cheaper per gigabyte and much much faster... bravo to skipping the description and basic computer imp in the last 5-6 years....

@@simlife445 Moron alert. You confirmed that it is indeed a good presentation then went on some personal rant of the content you didn't like? I don't give a damn sheesh.

He needed to prove SSDs had any improvement at all first. After that he had a good idea on how much it improved, and eventually he proved Fred right. It would take too many daily builds for SSDs to be worth it. But before that, he needed to know what the difference even was, and after that he used a simple formula to see how much money it saved. Poor Fred just had some words put in his mouth to make the presentation go a little smoother at the beginning.

To be fair, that wasn't daily builds, it was total builds, since SSDs are a one-time investment. Getting even the lowball estimate of 210 builds out of the lifetime of the SSD is probably easily achievable, so SSDs would be a worthwhile investment.

He covered that briefly with the discussion of dev time cost and how many builds you'd have to do for the SSD to pay for itself. You have to have a null hypothesis to test, and "X isn't worth it" isn't possible, IIRC. It's been a while, but I think your test *has* to basically 'touch zero'; either x=0, x>0, etc. An "even if does save time, does it save *enough* time" hypothesis requires a test that is basically "is x >= y" (where y is the 'threshold' where SSDs pay for themselves). It's either easier to first prove that there *is* a time difference, then calculate the 'value' of the time difference, or it's not even possible to do it the other way (or at least not with 101 statistics).

@@donanderson3653 Also, SSDs can speed up OS and App boot times as well as many other tasks, so it's ignoring a lot of the other benefits they give.

In fact, his own first example should have been a one-tailed test.

Under what circumstances would a 1 tailed test be more useful? The 2 tailed t test is more stringent and the test statistic will tell you the direction of the effect.

@@richardsejour7731 A 2 tailed test splits your significance level on both tails, so it's only half as strong as a one tailed test when showing a difference between groups IN A SPECIFIC DIRECTION. Frankly, a 2-tailed test is a sloppy but acceptable way to test, but it really shouldn't be used when you have a specific direction of difference between the groups in mind. A 1 tailed test has more power at the same alpha level. It's basically weakening your hypothesis to hedge your bets by using a 2-tailed test when you should be using one. That's why I don't like this lecture. It's a computer programmer with a SINGLE statistical tool he knows, so everything looks good to apply that tool on. It's like that old adage that if you have only a hammer, everything looks like a nail. If he were a statistician, he'd know better. But he's sitting there spouting off like he does, when in fact he's dead wrong.

@@andrewneedham3281 I didn't interpret it as such. He was trying to create an antagonistic force between him (supporting the alternate hypothesis) and the...wtf animal that was (supporting the null). He was more interested in testing the efficacy of the ssd which is tested using a 2 tail test. The 2 tailed test is more conservative in general and will give you the direction of the effect which is why it's generally more preferred. One sided tests are rarely used, and are often associated with p hacking because there is rarely a scenario to assume the directionality of effect. Yes the one tailed test gives more statistical power, but that's only if you are certain that you won't see any effect in the opposite tail, which is incredibly rare. His wording was off because he should have never assumed that the ssd can only be better or the same, when the ssd could be worse. However, his approach to use the 2 tailed test was spot on for this type of question.

@@richardsejour7731 Sure. I never said that he shouldn't use a 2-tailed test in that situation. I merely said that it's foolish to say "Always use the 2-tailed value." Edit: In science, if you have a hypothesis, your hypothesis generally has directionality to it, or you've written a piss-poor hypothesis. So, frankly, I'm often using 1-tailed tests to show that X is strictly less than/strictly greater than, on some real life data, such as, "Are female babies truly smaller than male babies?" or "Did the biodiversity index for the Upper Nooksack area truly increase due to our conservation measures?" In those cases, as a scientist trying to get published in a peer reviewed paper, I'd get laughed right out of publication for trying to use a 2-tailed test in those or many other situations where I find myself relying on statistical inference. Just saying.

That works if you want all 1 vs 1 fights to be mostly fair. Think of something like Street Fighter where you can't change your character mid-match. A rock paper scissors relationship would be fair but then if you are playing rock and the opponent is paper then the match isn't a good test of skill, the game was over at the character select screen. Depending on your context (something like Team Fortress or StarCraft) you might need to instead find the Nash equilibrium to make sure all units have their niche. But looking purely at win rates might mislead you if your player base is not playing optimally. Even if you can trust your win rate statistics, finding the Nash equilibrium is NP complete, meaning that each new character class exponentially increases the complexity of the problem. And there's probably units like the SCV where the kill death ratio is exceedingly bad but you can't win without them because their role is non-combat. Or a unit like the carrier (maybe? I'm not a pro) that isn't resource efficient but is a way to force the game to end if you are already ahead in resources and tech. If that's true and you analyze the carrier per unit, it might look overpowered, if you look at it per resource it might look underpowered, but it still has a niche. I guess that all I'm saying is that it's a hard problem, and game theory might be useful, but could still be difficult to apply if you have a game that is interestingly complex.

Yes.

Fair question. His reasoning was pretty sound. He would want the one-tailed t-test if it were a safe assumption that SSDs are always either faster or the same (an assumption about the underlying distribution). Making that assumption (which is a bad assumption) is not the same as being mostly interested in finding out if they are faster (which is valid, but does allow for them being slower). His test concluded that they were different distributions, and he could also see that the difference was to SSDs' benefit.

​@@ArsenicDrone The boss was specifically asking if SSDs were worth it (i.e. sufficiently faster that their mean speeds come from a different, faster, population than HDD mean load speeds). Wouldn't it be a mistake to intentionally test a broader hypothesis than you require just to verify your actual, narrower hypothesis by observation at the end?

@@mrichards Ah, one of many not-so-intuitive things about statistics. It really comes down to only making the assumptions that you can justify. What the boss was interested in doesn't determine what's possible to test or what assumptions are valid. Notice that his p-value is half as large for the one-tailed test (the result is even more significant). The test got substantially more powerful, but that power doesn't come for free, it comes by making this unjustified assumption. (It's not justified because before he runs the test, he really doesn't know which outcome will happen, and it could actually be slower.)

​@@ArsenicDrone No, he really is mistaken. Whether or not it is a safe assumption that SSDs are always faster is actually irrelevant. What is relevant is that the hypothesis he's testing is a one-sided hypothesis--that SSDs are faster. If he had measured SSDs to be slower, by any magnitude, the hypothesis would have been rejected.

If your game has a shop you've already failed.

@@MrDavidCollins that's just objectively wrong lots of incredibly good and fun games have shops

The test was simply to determine whether difficulty had an effect on time played in either direction greater than the margin of error for the sample size. These are great as backup tests to ensure the results aren't just a fluke without a unreasonably large sample size.

@@jacobb5484 Doesn't matter. If I'm a tester and only testing the game for 10-15 minutes if its too hard I'm going to report that it's too hard. If the game gets made easier and released and I pick it up and find that 30 mins in it's too easy, I'm going to get bored and quit. I think he oversimplifies the situation.

@@lushen952 Its a simple example of a T test on a paired sample. this isn't for small engaged focus groups with detailed subjective data, but rather big data statistics such as the example of a sub mode being beta tested. The situation in this example the T test gives a percentage chance of either: A. The change had the effect of either increasing OR decreasing what's being measured by a notable amount. B. the data is probably skewed due to bad sampling and falls within the margin of error. once you rule that out, you can make further changes and run detailed tests to actually make an improvement.

If you have 20% of something... let's say all IBM shares, and you increase your holdings by 5% = now you have 22%. But when you say you have increased it by 5% percentage POINT you went from 20%=>25%.

​@@dezimal9143 I bamboozled myself. meant to say 21% sry. How is 5% of 20 = 2 ?

@@YT775 Actually it isn't 2% I didn't check the math xD. And you are right it should be 21 vs 25%.

Thanks, so I guess theres no hidden meaning, it was just a minor error/inaccuracy of the speaker. :)

lots of practice by making mods, level design, digital modeling, etc.?

I took statistics and didn't become a game developer (at a company, I just make it all myself now). College costs too much

Kryptonite.

or that his is 5 year old gdc session(read the discription) so this data is insanely old ssd are 60% cheaper per gig and much faster

Yeah and long build times is actually one of the biggest blockers to ci/cd, which the lack of is usually the best indicator of long lead time which is the best indicator for slow development trapping more resources inside the system, increasing the number of bugs, less feedback, less data, less experimentation and less revenue. Overall meaning slower delivery and lower quality product and/or requiring more resources to deliver. And in the end you should not generally build on your local machine but do it automatically on a build server.

If only that was the only cringe part... His explanation of p-values is statistical illiteracy 101. I was really surprised when I heard him making that mistake, interpreting p-values as Pr(H0). I thought that this statistical concept entered pop culture (kind of like "correlation is not causation" already did)... Amazing that people like him have the confidence to give talks on statistics.

You can tell the direction and magnitude of the effect using a 2 tailed t test. The limitations of a 1 tailed t test is that it is less stringent and in most cases you don't have any justification to assume that an effect is greater or lower.

@@f.p.5410 what did he say wrong? I missed it.

@@richardsejour7731 he always says that the p-value is the probability of the boss (can't remember the name) being right. In other words, the probability of H0 being true. So that p

@@f.p.5410 I see what you mean. In a lot of schools and even on some online sources, the p value is described as the probability of incorrectly rejecting your null hypothesis, which some people can misconstrue as "proving" whether the null is correct or not. In practice, you can never really prove a null, or really any hypothesis. I think that his attempt at making the presentation lighthearted muddied the waters a little, but the general premise is still there. Speaking of which, I was sloppy in my wording. The 2 sided test is more conservative since the cutoff is 0.025 in either tail instead of 0.05 in 1 direction. Usually, when a two tailed test is statistically significant, a one tailed test is significant in that direction, but there are many cases when a one tailed test is significant, but a two tailed test won't be significant. There are very few circumstances to assume directionality before doing a statistical tests, which is why one sided tests are so rare. Most researchers will just do a 2 sided test to be conservative and then calculate effect sizes, and will consider one sided tests as p hacking. For the research question that he was asking (the efficacy of the ssd) he can't assume that the ssd can only be better out the same, when it's entirely possible that it could be worse. From his presentation of the question, he understands this, but used sloppy wording to make things simple. Regardless, the 2 tailed test is the best approach here.

If you were to perform a test 1000 times, then wouldn't the distribution of the p value be skewed to the right assuming that you reject the null hypothesis? If this is the case then you are bound to have some p values that are not as extreme as others.

@@richardsejour7731 I'm not sure, but I don't think the distribution of the p value is affected by rejecting or assuming the null hypothesis is correct. And of course you will have values that are less extreme. The first point in the clarifications en.wikipedia.org/wiki/Misuse_of_p-values is basically what I tried to get through, p-values aren't about probablities.

@@andrewandersson p values are about probabilities because they are tied to a normal distribution. More specifically, p values are supposed to represent the probability of getting a result at least that extreme assuming that you accept the null. My confusion was that I wasn't sure if p values are supposed to reflect an expected outcome if you repeated it multiple times. I understood p values as a test of the null hypothesis for that specific test/data. If you did repeat that test 1000 times, you would see a range of p values, but I don't think you could determine where most of the values would lie.

@@richardsejour7731 it depends on the data and test, not all of them have normal distributions. If you aaaume a different model you get a different distribution but i guess you are right but it is important to distinguish the probability of the hypothesis being true (which frequentist statistics does not address) and the probability of seeing a result as extreme or more extreme assuming the hypothesis is true.

@@andrewandersson you are right, the p value is not about whether a null or alternate hypothesis is true, or wrong. It's about the probability of getting at least your observed value assuming if there was no difference between the means of your groups (aka the null hypothesis). Under this definition, you have to apply your p value in context to the distribution of a null hypothesis. So if you get a small p value, then that means that the likelihood of your data supporting the null hypothesis is so low that it cannot be an accurate representation of the null hypothesis, and thus you would reject the null since it conflicts with your data. Conversely, if you have a high p value this finding would be acceptable and not uncommon if you follow the distribution of a null hypothesis, or in other words, the null hypothesis doesn't disagree with your data, so you fail to reject the null. Or at least that is how I interpret the p value as.