Question 1

What does a p-value actually tell you?

Accepted Answer

A p-value is the probability of seeing a result at least as extreme as yours, purely by chance, if the null hypothesis (the "nothing's going on" assumption) were actually true. A small p-value means your data would be quite unusual under that assumption — which is the evidence used to argue against it.

Question 2

How do I choose between a Z-test and a T-test?

Accepted Answer

Use a Z-test when you know the population's true standard deviation or your sample is large (typically 30+), since the sampling distribution reliably approximates a normal curve at that point. Use a T-test — which needs degrees of freedom, usually sample size minus 1 — when the population standard deviation is unknown and your sample is smaller, since the t-distribution has heavier tails that account for that extra uncertainty.

Question 3

What's the difference between one-tailed and two-tailed?

Accepted Answer

A two-tailed test asks whether your result is unusually extreme in either direction — bigger or smaller than expected — and is the standard default when you don't have a specific direction predicted in advance. A one-tailed test (left or right) only checks one direction, which requires you to have committed to that direction before seeing the data; picking one-tailed after peeking at your result to get a smaller p-value is a well-known way to fool yourself.

Question 4

What does it mean when p is below my alpha level?

Accepted Answer

It means your result falls in the rejection region you set up in advance — conventionally α = 0.05, meaning you're willing to accept a 5% chance of wrongly rejecting a true null hypothesis. Falling below that threshold is typically reported as "statistically significant," though a low p-value alone says nothing about how large or practically meaningful the effect actually is.

Question 5

Is a smaller p-value always better?

Accepted Answer

Not automatically — statistical significance and practical significance are different questions. A very large sample can produce a tiny p-value for an effect too small to matter in the real world, while a genuinely important effect can fail to reach significance in a small sample simply from lack of statistical power.

P-Value Calculator — Z-Test & T-Test

P-Value Calculator

Reading Your Result

📏 Z-Test

📐 T-Test

🎯 Choosing Alpha

↔️ Picking a Tail

Frequently Asked Questions

Related Calculators