P-values
How likely is it that our sample (the descriptive statistic or measure of association observed in our sample) could have come from a population with characteristic, X, just by the luck of the draw? (X is a specific descriptive statistic or measure of association). For example, how likely is it that we could have gotten a sample with a relative risk of 2 by the "luck of the draw" from a population with relative risk of 1.0?
The basic question we are pursuing here is: How likely is it that this particular sample could come from this particular population? Occasionally, especially when we are dealing with descriptive statistics, we may know the value of the descriptive statistic for the population (such as mean height). Typically, however, we are dealing with measures of association and we have no idea what the value is for the population. This is why we usually begin with an assumption: that there is no connection between the two variables under study in the population and that the value for the measure of association reflects this (we commonly call this the "null hypothesis"). In other words, if we were studying the association between benzene exposure (exposed/unexposed) and leukemia (get it/don't get it) we would use relative risk (RR) as our measure of association. If there were no connection between benzene and leukemia the RR would equal 1. Thus RR=1 is our "null hypothesis" and we begin with the assumption that this is true for the population.
Say that we then observe a RR of 2 in the sample of subjects that we are studying. How likely is it that we could get a RR of 2 just by the "luck of the draw" if our population RR was 1? More precisely, how likely is it that, "just by the luck of the draw" we could get a RR that deviates at least this much from the null value of 1?
To answer our question, we use the appropriate Chance Assessment Tool (also known as "inferential statistic") to derive a p-value. In this case, since we are dealing with two dichotomous variables, the appropriate Chance Assessment Tool would be the Chi-square test. Let's assume that it identified a p-value of 0.06. This means that there is a 6% likelihood that we could get, by the "luck of the draw", a RR of 2 in our sample if the population from which the sample was drawn had a RR of 1. Another way of looking at it is that if we drew 100 samples (of the same sample size as the one we are studying) from a population with RR=1, only 6 of those samples would have an RR of 2 or more. A final way to interpret the p-values is that there is a 6% probability that chance could be responsible for the observed deviation from RR=1.
This suggests that it is relatively unlikely that we could have drawn our sample from a population with RR=1. In other words, our assessment of chance suggests that our original assumption about the population (our "null hypothesis") is wrong.
A final note on p-values. In the past, statisticians called p-values of 0.05 or less "statistically significant" and suggested that this meant you could reject your null hypothesis. The folly (and danger) in this mindless approach to decision-making should be apparent. The 0.05 level is strictly arbitrary. Why should a p-value of 0.055 force you to act as if there were no association between your variables? More importantly, rejecting your null hypothesis on the basis of a p-value alone makes sense only if you ignore the possible role of bias and confounding in explaining your results, which would be a foolish thing to do! Fortunately, researchers are slowly abandoning this myopic view of p-values.
Have I Grasped the Key Concepts Here?