Probability & Statistics in Practice: Inference Problems from the ML Trenches

Applying Bayes' theorem:

$P(\text{Spam} \mid \text{"free"}) = \frac{P(\text{"free"} \mid \text{Spam}) \cdot P(\text{Spam})}{P(\text{"free"})}$

Computing the denominator (law of total probability):

$P(\text{"free"}) = 0.6 \times 0.2 + 0.05 \times 0.8 = 0.12 + 0.04 = 0.16$

$P(\text{Spam} \mid \text{"free"}) = \frac{0.6 \times 0.2}{0.16} = \frac{0.12}{0.16} = 0.75$

Conclusion: An email containing "free" is spam with 75% probability. The prior (20%) has been significantly updated to a posterior of 75%.

ML connection: The Naive Bayes classifier applies this exact principle independently to each word to classify text.

1. No: Correlation ≠ causation. Ice cream does not cause drowning.

2. Confounding Variable: "Summer temperature" is the common cause.

$\text{Temperature} \rightarrow \text{Ice cream sales}$ $\text{Temperature} \rightarrow \text{More swimmers} \rightarrow \text{Drowning incidents}$

Once we condition on temperature, the two variables become independent: $P(\text{Drowning} \mid \text{Ice cream}, \text{Temperature}) = P(\text{Drowning} \mid \text{Temperature})$

3. ML Problems: A model that learns spurious correlations will fail on test data from a different distribution (Distribution Shift). It can still make predictions, but its reasoning about causal interventions will be wrong.

A (House prices): Positive skewness → right-skewed (heavy right tail). The mean is much higher than the median, so the median is the better representative value. Recommend log-transforming and fitting a normal distribution, or modeling directly with a log-normal distribution.

B (Exam scores): Skewness ≈ 0, kurtosis ≈ 0 → approximately normal distribution. Mean and median are similar, so either can be used. A normal distribution fits directly.

C (Click counts): Both skewness and kurtosis are very large → extreme right skew with many outliers (viral content, etc.). Use the median. Since values are discrete, consider a Negative Binomial distribution or a power-law distribution.

1. Correlation coefficients:

$r_A = \frac{500}{\sqrt{10000 \times 40000}} = \frac{500}{20000} = 0.025$

$r_B = \frac{100}{\sqrt{100 \times 200}} = \frac{100}{141.4} \approx 0.707$

2. Scenario B has a much stronger linear relationship: The raw covariance looks larger in A, but the variables are on very different scales, making direct comparison meaningless. The correlation coefficient normalizes for this.

3. Zero correlation but clearly related: $Y = X^2$, $X \sim U(-1, 1)$. X and Y are obviously related, yet their linear correlation coefficient is 0. → Correlation only captures linear relationships. Non-linear relationships require measures like Mutual Information (MI).

Key differences:

LLN holds but CLT does not — example: The Cauchy distribution has undefined expectation, so the LLN doesn't even hold there. More generally, distributions with infinite variance (e.g., Pareto with certain parameter values) satisfy the LLN but violate the CLT.

ML applications:

• LLN → theoretical justification for mini-batch SGD: a sufficiently large batch approximates the full gradient
• CLT → uncertainty estimation for ensemble predictions; distributional assumptions for test statistics in hypothesis testing

1. MLE: Maximizing the log-likelihood of the binomial distribution gives:

$\hat{\theta}_\text{MLE} = \frac{8}{10} = 0.8$

2. MAP with Beta(3,3): The posterior is $\text{Beta}(3+8, 3+2) = \text{Beta}(11, 5)$

$\hat{\theta}_\text{MAP} = \frac{\alpha - 1}{\alpha + \beta - 2} = \frac{10}{14} \approx 0.714$

3. Comparison:

With little data, MLE over-trusts the accidental outcome (8/10). The MAP prior encodes the reasonable belief that the coin is "probably near fair," stabilizing the estimate. This is mathematically equivalent to L2 regularization.

Answer: b

Why a is wrong: The true parameter (accuracy) is a fixed value — it is not a random variable. The interval is what varies across repeated experiments.

Why c is wrong: A CI describes the range of a parameter, not the spread of individual data points (that would be a prediction interval).

Why d is wrong: This confuses a CI with a Prediction Interval. A CI estimates the range of a parameter; a prediction interval gives the range for a new observation. (Prediction intervals are always wider than CIs.)

1. Hypothesis formulation:

• $H_0$: $p_\text{drug} \leq p_\text{placebo}$ (no effect or harmful)
• $H_1$: $p_\text{drug} > p_\text{placebo}$ (effective)
• The direction is clear (we expect the drug to be better), so use a right-sided one-tailed test.

2. Type II error (False Negative) is more serious: Concluding that an effective drug "has no effect" means patients lose access to treatment. Type I errors can be caught with additional trials. (However, if side effects are severe, Type I errors become critical too.)

3. Ways to increase power:

• Increase sample size: The most direct approach. $n$↑ → SE↓ → greater sensitivity to detect a difference.
• Relax the significance level $\alpha$: Larger $\alpha$ expands the rejection region → power↑ (but Type I error↑).
• Use a one-tailed test: Switching from two-tailed to one-tailed increases power.
• Target a subgroup with a larger effect size: Focus on subpopulations where the treatment effect is expected to be larger.
• Reduce measurement error: More precise measurement → lower variance → lower SE.

1. Paired t-test: Before/after measurements on the same users → paired data. Removes individual differences, increasing power.

2. Two-sample proportion test (z-test): Binary outcome (click / no click) → comparing proportions, not continuous means.

3. One-sample t-test: Comparing our model's sample accuracy against a single known value (89.3%).

4. One-Way ANOVA: Comparing means across three or more groups with repeated t-tests inflates the Type I error rate (multiple comparisons problem) → use ANOVA. Follow up with a post-hoc test (e.g., Tukey HSD) to identify which pairs differ.

1. CI interpretation: The existing model's 91.5% falls within the new model's CI of [91.3%, 92.9%]. This means the difference between the two models may not be statistically significant. We cannot conclusively say the new model is better. → A formal hypothesis test or a CI for the difference should be computed separately.

2. Why the sample size must be fixed in advance: Peeking at results mid-experiment and stopping early inflates the Type I error rate well above the nominal 5% due to the multiple comparisons problem. Committing to a predetermined sample size ensures a single test is performed.

3. p-value misconception: A p-value of 0.03 means "if the null hypothesis were true, there would be a 3% chance of observing results this extreme." It is not the probability that the null hypothesis is true. Saying "97% chance the new model wins" is a posterior probability — a Bayesian statement that requires defining a prior distribution. It cannot be derived from a frequentist p-value.