title

Why Dividing By N Underestimates the Variance

description

This is the follow up video to:
Statistics Fundamentals: The Mean, Variance and Standard Deviation
https://youtu.be/SzZ6GpcfoQY
In it, we show exactly why, when we estimate the variance, dividing by 'n' underestimates the value we are interested in. It also describes why we square each term instead of taking the absolute value. The visuals used in this StatQuest make it easy to remember why we should divide by n-1, and this will save us from falling into a very common pitfall.
If you'd like to support StatQuest, please consider...
Support StatQuest by buying The StatQuest Illustrated Guide to Machine Learning!!!
PDF - https://statquest.gumroad.com/l/wvtmc
Paperback - https://www.amazon.com/dp/B09ZCKR4H6
Kindle eBook - https://www.amazon.com/dp/B09ZG79HXC
Patreon: https://www.patreon.com/statquest
...or...
YouTube Membership: https://www.youtube.com/channel/UCtYLUTtgS3k1Fg4y5tAhLbw/join
...a cool StatQuest t-shirt or sweatshirt:
https://shop.spreadshirt.com/statquest-with-josh-starmer/
...buying one or two of my songs (or go large and get a whole album!)
https://joshuastarmer.bandcamp.com/
...or just donating to StatQuest!
https://www.paypal.me/statquest
Lastly, if you want to keep up with me as I research and create new StatQuests, follow me on twitter:
https://twitter.com/joshuastarmer
Corrections:
3:23 I should have said "To understand why dividing by n underestimates the variation around the population mean".
3:40 The estimated mean was switched with the population mean.
#statquest #variance

detail

{'title': 'Why Dividing By N Underestimates the Variance', 'heatmap': [{'end': 339.676, 'start': 289.01, 'weight': 0.708}, {'end': 725.091, 'start': 677.423, 'weight': 0.977}, {'end': 904.692, 'start': 888.308, 'weight': 0.725}], 'summary': 'Explains why dividing by n underestimates population variance, showing that using sample mean and dividing by n consistently underestimates variances compared to population mean. it also discusses finding minimum variance using derivatives and estimating population variance through sample mean, leading to better variance estimates.', 'chapters': [{'end': 431.735, 'segs': [{'end': 167.145, 'src': 'embed', 'start': 116.632, 'weight': 0, 'content': [{'end': 125.135, 'text': 'However, since we rarely, if ever, have enough time and money to measure every single thing in a population,', 'start': 116.632, 'duration': 8.503}, {'end': 128.216, 'text': 'we almost always estimate the population mean.', 'start': 125.135, 'duration': 3.081}, {'end': 137.447, 'text': 'The estimated population mean x-bar equals the sum of the measurements, divided by the number of measurements,', 'start': 129.841, 'duration': 7.606}, {'end': 140.309, 'text': 'and that equals the average measurement x-bar.', 'start': 137.447, 'duration': 2.862}, {'end': 145.614, 'text': 'And we estimate the variation and standard deviation.', 'start': 142.211, 'duration': 3.403}, {'end': 155.582, 'text': 'The estimated population variance is the sum of the squared differences divided by n minus 1.', 'start': 147.395, 'duration': 8.187}, {'end': 159.325, 'text': 'And the standard deviation is just the square root of the variance.', 'start': 155.582, 'duration': 3.743}, {'end': 167.145, 'text': 'And since the standard deviation is in the same units as the original data, we can draw it on the graph.', 'start': 161.061, 'duration': 6.084}], 'summary': 'Population mean and standard deviation are estimated using sample measurements for statistics.', 'duration': 50.513, 'max_score': 116.632, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/sHRBg6BhKjI/pics/sHRBg6BhKjI116632.jpg'}, {'end': 339.676, 'src': 'heatmap', 'start': 289.01, 'weight': 0.708, 'content': [{'end': 290.771, 'text': 'And here are a few more points.', 'start': 289.01, 'duration': 1.761}, {'end': 301.537, 'text': 'Bam! the point with the smallest variance corresponds to the sample mean x-bar.', 'start': 292.492, 'duration': 9.045}, {'end': 309.614, 'text': 'And this point, with a slightly larger variance, corresponds to the population mean.', 'start': 302.99, 'duration': 6.624}, {'end': 323.803, 'text': 'So, in this case, when we plug in the population mean and divide by n, we get a larger variance than when we plug in the sample mean and divide by n.', 'start': 311.135, 'duration': 12.668}, {'end': 330.487, 'text': 'In other words, when we use the sample mean, we underestimated the variance we got with the population mean.', 'start': 323.803, 'duration': 6.684}, {'end': 339.676, 'text': "Bam! Now let's get five new measurements from the same population and see if the same thing happens.", 'start': 332.068, 'duration': 7.608}], 'summary': 'Comparing sample mean and population mean variances for 5 measurements', 'duration': 50.666, 'max_score': 289.01, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/sHRBg6BhKjI/pics/sHRBg6BhKjI289010.jpg'}, {'end': 431.735, 'src': 'embed', 'start': 402.779, 'weight': 1, 'content': [{'end': 410.263, 'text': 'we get a larger variance than when we plug in the sample mean and divide by n.', 'start': 402.779, 'duration': 7.484}, {'end': 419.148, 'text': 'So, again, when we used the sample mean and divided by n, we underestimated the variance calculated around the population mean.', 'start': 410.263, 'duration': 8.885}, {'end': 431.735, 'text': 'Bam. So far we have seen two simple examples where, using the sample mean and dividing by n, underestimated the variances we got with the population mean.', 'start': 420.089, 'duration': 11.646}], 'summary': 'Using sample mean and dividing by n underestimated variances around population mean.', 'duration': 28.956, 'max_score': 402.779, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/sHRBg6BhKjI/pics/sHRBg6BhKjI402779.jpg'}], 'start': 0.925, 'title': 'Dividing by n underestimates variance', 'summary': 'Explains why dividing by n underestimates population variance, providing examples and mathematical proofs, demonstrating that using the sample mean and dividing by n consistently underestimates variances compared to using the population mean.', 'chapters': [{'end': 431.735, 'start': 0.925, 'title': 'Dividing by n underestimates variance', 'summary': 'Explains why dividing by n underestimates population variance, providing examples and mathematical proofs, demonstrating that using the sample mean and dividing by n consistently underestimates variances compared to using the population mean.', 'duration': 430.81, 'highlights': ['The population mean, mu, equals the sum of the measurements divided by the number of measurements, which equals the average measurement, mu.', 'The estimated population variance is the sum of the squared differences divided by n minus 1.', 'When using the sample mean and dividing by n, the variance around the population mean is consistently underestimated, as demonstrated in multiple examples.']}], 'duration': 430.81, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/sHRBg6BhKjI/pics/sHRBg6BhKjI925.jpg', 'highlights': ['The estimated population variance is the sum of the squared differences divided by n minus 1.', 'When using the sample mean and dividing by n, the variance around the population mean is consistently underestimated, as demonstrated in multiple examples.', 'The population mean, mu, equals the sum of the measurements divided by the number of measurements, which equals the average measurement, mu.']}, {'end': 709.405, 'segs': [{'end': 582.202, 'src': 'embed', 'start': 555.949, 'weight': 0, 'content': [{'end': 563.113, 'text': 'Bam! Just to remind you, the derivative corresponds to the slope of the purple line.', 'start': 555.949, 'duration': 7.164}, {'end': 573.137, 'text': 'And we want to find the value for v such that the slope of the purple line equals zero, because that is where we will find the minimum variance.', 'start': 564.352, 'duration': 8.785}, {'end': 582.202, 'text': 'To make this as clear as possible, we will find where the derivative is zero and the variance is minimized three different ways.', 'start': 574.958, 'duration': 7.244}], 'summary': 'Finding v to minimize variance using three methods', 'duration': 26.253, 'max_score': 555.949, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/sHRBg6BhKjI/pics/sHRBg6BhKjI555949.jpg'}, {'end': 709.405, 'src': 'embed', 'start': 648.226, 'weight': 1, 'content': [{'end': 650.568, 'text': 'Now solving for v is super easy.', 'start': 648.226, 'duration': 2.342}, {'end': 663.008, 'text': 'V is the average of the five measurements, which is the sample mean x-bar.', 'start': 656.685, 'duration': 6.323}, {'end': 670.351, 'text': 'And thus, V equals x-bar, which equals 17.6.', 'start': 664.068, 'duration': 6.283}, {'end': 677.423, 'text': 'Thus, the derivative is zero when V equals x-bar, which equals 17.6.', 'start': 670.351, 'duration': 7.072}, {'end': 684.348, 'text': 'and the variance is minimized when v equals x-bar, which equals 17.6.', 'start': 677.423, 'duration': 6.925}, {'end': 692.474, 'text': 'This is why, given this data, the value around the sample mean is less than the value around the population mean.', 'start': 684.348, 'duration': 8.126}, {'end': 703.102, 'text': 'In other words, the differences between the data and the sample mean tend to be smaller than the differences between the data and the population mean.', 'start': 693.815, 'duration': 9.287}, {'end': 709.405, 'text': 'Thus, the differences around the population mean will result in a larger average.', 'start': 704.363, 'duration': 5.042}], 'summary': 'The variance is minimized at v=x-bar=17.6, indicating smaller differences around the sample mean than the population mean.', 'duration': 61.179, 'max_score': 648.226, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/sHRBg6BhKjI/pics/sHRBg6BhKjI648226.jpg'}], 'start': 433.336, 'title': 'Finding minimum variance', 'summary': 'Discusses using derivatives to find the minimum variance, demonstrating the process and providing an example where v equals x-bar, which equals 17.6.', 'chapters': [{'end': 709.405, 'start': 433.336, 'title': 'Finding minimum variance with derivatives', 'summary': 'Discusses finding the minimum variance by using derivatives and demonstrates how to find the value for v such that the slope of the purple line equals zero, resulting in minimized variance, with an example showing v equals x-bar, which equals 17.6.', 'duration': 276.069, 'highlights': ['Using derivatives to find where the slope of the purple line equals zero, resulting in minimized variance, with an example showing v equals x-bar, which equals 17.6.', 'Demonstrating the process of plugging in the data, setting the derivative equal to zero and solving for v to find the minimized variance, with a specific example showing V equals x-bar, which equals 17.6.', 'Explaining that the differences around the population mean result in a larger average variance compared to the differences around the sample mean.']}], 'duration': 276.069, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/sHRBg6BhKjI/pics/sHRBg6BhKjI433336.jpg', 'highlights': ['Using derivatives to find where the slope of the purple line equals zero, resulting in minimized variance, with an example showing v equals x-bar, which equals 17.6.', 'Demonstrating the process of plugging in the data, setting the derivative equal to zero and solving for v to find the minimized variance, with a specific example showing V equals x-bar, which equals 17.6.', 'Explaining that the differences around the population mean result in a larger average variance compared to the differences around the sample mean.']}, {'end': 1030.13, 'segs': [{'end': 887.107, 'src': 'embed', 'start': 784.147, 'weight': 2, 'content': [{'end': 794.334, 'text': 'And just like before, we multiply both sides by 5 divided by negative 2 to cancel out the negative 2 divided by 5.', 'start': 784.147, 'duration': 10.187}, {'end': 796.736, 'text': 'Now solving for v is super easy.', 'start': 794.334, 'duration': 2.402}, {'end': 801.879, 'text': 'V is the average of the five measurements, which is the sample mean x bar.', 'start': 796.756, 'duration': 5.123}, {'end': 818.139, 'text': 'So, no matter what five measurements we start with, the value that gives us the minimum variance is x-bar.', 'start': 811.053, 'duration': 7.086}, {'end': 828.448, 'text': "Bam! Now let's see what happens when we have any size sample, i.e., a sample with n measurements.", 'start': 819.42, 'duration': 9.028}, {'end': 833.392, 'text': "So, let's plug the unknown data into the derivative.", 'start': 830.109, 'duration': 3.283}, {'end': 841.116, 'text': 'Now, instead of replacing n with a number, we just leave it since we have n measurements.', 'start': 835.154, 'duration': 5.962}, {'end': 854.5, 'text': 'Now we plug in all n measurements and set the derivative equal to 0 and solve for v.', 'start': 843.057, 'duration': 11.443}, {'end': 861.843, 'text': 'First, we multiply both sides by n divided by negative 2 to cancel out the negative 2 divided by n.', 'start': 854.5, 'duration': 7.343}, {'end': 866.774, 'text': 'Now solving for v is super easy.', 'start': 864.372, 'duration': 2.402}, {'end': 875.94, 'text': 'v is the average of the n measurements, which is the sample mean x bar.', 'start': 869.415, 'duration': 6.525}, {'end': 884.585, 'text': 'So, no matter how many measurements we start with, the value that gives us the minimum variance is x bar.', 'start': 877.521, 'duration': 7.064}, {'end': 887.107, 'text': 'Double bam!.', 'start': 885.986, 'duration': 1.121}], 'summary': 'Multiplying sides by n/-2, v is the sample mean x-bar for n measurements.', 'duration': 102.96, 'max_score': 784.147, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/sHRBg6BhKjI/pics/sHRBg6BhKjI784147.jpg'}, {'end': 909.875, 'src': 'heatmap', 'start': 888.308, 'weight': 0.725, 'content': [{'end': 898.39, 'text': 'Thus, when we divide by n, the value around the sample mean is always less than the value around the population mean.', 'start': 888.308, 'duration': 10.082}, {'end': 904.692, 'text': 'unless the sample mean is the exact same as the population mean, and that pretty much never happens.', 'start': 898.39, 'duration': 6.302}, {'end': 909.875, 'text': 'Triple bam! P.S.', 'start': 906.013, 'duration': 3.862}], 'summary': 'When dividing by n, value around sample mean is less than population mean, unless they are exactly the same.', 'duration': 21.567, 'max_score': 888.308, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/sHRBg6BhKjI/pics/sHRBg6BhKjI888308.jpg'}, {'end': 995.486, 'src': 'embed', 'start': 938.405, 'weight': 0, 'content': [{'end': 946.011, 'text': 'And since we have this sharp angle at the minimum value and derivatives do not exist at sharp angles like this,', 'start': 938.405, 'duration': 7.606}, {'end': 952.972, 'text': 'then finding the minimum value is much harder with the absolute value than with the square.', 'start': 947.688, 'duration': 5.284}, {'end': 966.043, 'text': 'Bam! In summary, when we only divide by n, we underestimate the variation in the data around the population mean.', 'start': 954.614, 'duration': 11.429}, {'end': 976.552, 'text': 'This is because the differences between the data and the sample mean tend to be smaller than the differences between the data and the population mean.', 'start': 967.825, 'duration': 8.727}, {'end': 982.636, 'text': 'Thus, the differences around the population mean will result in a larger average.', 'start': 977.571, 'duration': 5.065}, {'end': 987.201, 'text': 'And the larger average is what we are trying to estimate.', 'start': 984.058, 'duration': 3.143}, {'end': 995.486, 'text': 'So, if you are estimating the population variance, divide by n minus 1.', 'start': 988.902, 'duration': 6.584}], 'summary': 'Estimate population variance by dividing by n minus 1 to account for variation around population mean.', 'duration': 57.081, 'max_score': 938.405, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/sHRBg6BhKjI/pics/sHRBg6BhKjI938405.jpg'}], 'start': 710.706, 'title': 'Estimating averages and variance', 'summary': 'Discusses estimating larger averages using derivatives and population variance estimation through sample mean, emphasizing on techniques such as plugging unknown values into the derivative and using square differences to estimate population variance, ultimately leading to better estimates for variance.', 'chapters': [{'end': 854.5, 'start': 710.706, 'title': 'Estimating larger averages with derivatives', 'summary': 'Discusses estimating larger averages using derivatives, including plugging unknown values into the derivative, solving for v where the slope equals 0, and concluding that the value giving the minimum variance is x-bar.', 'duration': 143.794, 'highlights': ['We plug unknown values x sub 1, x sub 2, x sub 3, x sub 4, and x sub 5 into the derivative, then solve for v where the slope equals 0. The value giving the minimum variance is x-bar, the average of the measurements.', 'When dealing with any size sample with n measurements, we plug the unknown data into the derivative, leave n as it is, plug in all n measurements, and solve for v where the derivative equals 0.']}, {'end': 1030.13, 'start': 854.5, 'title': 'Population variance estimation', 'summary': 'Explains the concept of estimating population variance using sample mean and the reason for using square differences instead of absolute value, concluding that for estimating population variance, divide by n-1 gives a better estimate.', 'duration': 175.63, 'highlights': ['Dividing by n-1 gives a better estimate for population variance, as it accounts for the larger average of differences around the population mean (3 bam!)', 'Using square differences instead of absolute value makes it easier to find the minimum value, as derivatives do not exist at sharp angles, providing a simplified way to calculate variance (2 bam!)', 'Solving for v is super easy as v is the average of the n measurements, which is the sample mean x bar, giving the value that gives us the minimum variance (1 bam!)']}], 'duration': 319.424, 'thumbnail': 'https://coursnap.oss-ap-southeast-1.aliyuncs.com/video-capture/sHRBg6BhKjI/pics/sHRBg6BhKjI710706.jpg', 'highlights': ['Dividing by n-1 gives a better estimate for population variance, accounting for larger average of differences (3 bam!)', 'Using square differences instead of absolute value simplifies variance calculation (2 bam!)', 'Solving for v is super easy as it is the average of the n measurements, giving the minimum variance (1 bam!)', 'Plugging unknown values into the derivative and solving for v gives the minimum variance as x-bar, the average of measurements', 'When dealing with any size sample with n measurements, plugging unknown data into the derivative and solving for v yields the minimum variance']}], 'highlights': ['Dividing by n-1 gives a better estimate for population variance, accounting for larger average of differences (3 bam!)', 'Using square differences instead of absolute value simplifies variance calculation (2 bam!)', 'Solving for v is super easy as it is the average of the n measurements, giving the minimum variance (1 bam!)', 'Using derivatives to find where the slope of the purple line equals zero, resulting in minimized variance, with an example showing v equals x-bar, which equals 17.6.', 'When using the sample mean and dividing by n, the variance around the population mean is consistently underestimated, as demonstrated in multiple examples.', 'Explaining that the differences around the population mean result in a larger average variance compared to the differences around the sample mean.', 'The estimated population variance is the sum of the squared differences divided by n minus 1.', 'Plugging unknown values into the derivative and solving for v gives the minimum variance as x-bar, the average of measurements', 'When dealing with any size sample with n measurements, plugging unknown data into the derivative and solving for v yields the minimum variance', 'The population mean, mu, equals the sum of the measurements divided by the number of measurements, which equals the average measurement, mu.']}