Misleading Medical Language. 2 0 What does medical significance mean?
It's significant but is it important?
Statistical significance ≠ clinical importance.
Medical studies are framed in the most favourable way to the sponsors or the authors. After all, they have spent a lot of time doing the study and they want to get something out of it. In most studies they are looking for the ‘magical’ goal of achieving statistical significance or ‘p less than or equal to 0.05 (two-sided). The editors and reviewers will not crown the results as being statistically significant. Tada. Major journal. Kudos.
What does that mean? It is not what you think. I bet if you asked most people they would say that there is 1 in 20 chance of ‘being wrong’. They are wrong. A p-value ≤ 0.05 indicates strong evidence against the null hypothesis, as there is less than a 5% probability the null is correct (and the results are random) if there is truly no difference. Therefore, we reject the null hypothesis, and accept the alternative hypothesis. However, this does
notmean that there is a 95% probability that the alternative hypothesis is true. The p-value is conditional upon the null hypothesis being true, but is unrelated to the truth or falsity of the alternative hypothesis.Does it mean the difference is important? No, not at all. There is the old rube: ‘A difference is a difference only if it makes a difference'; in other words what is important is a magnitude of difference that patients would find clinically meaningful. Statistical significance
does not equalclinical importance.How replicable are studies with different p-values? In general, the smaller the p-value, the stronger is the evidence to reject the null hypothesis. Therefore, p=0.001 is stronger evidence than p=0.049, although both as ‘statistically significant.’ But replication is key in science. So what is the probability of doing a study getting a significant result and repeating the study to get a similar result. It is not as good as one might guess, meaning that p-values don’t always tell you the strength of the finding.
These number are not exact but are ballpark figures.
P=0.05 replication 50%
P=0.01 replication 70%
P=0.001 replication 90%
Bottom Line: Statistical significance is not the same as clinical importance. Be careful not to conflate the two. There are different degrees of statistical significance so that a 1 size fits all approach where a 5% threshold is the goal is, in some circumstances, not good enough.
Note: this is not meant to be an exhaustive discussion on p-values. There are numerous excellent discussion in PubMed and on-line.