Suppose we have some population about which we are interested in the values of a particular numerical measurement. If we pick individuals from the population at random, then we can think of the values of this measurement as a quantitative random variable $X$. Suppose we are interested in the population mean, $\mu$, of $X$, although somehow we know the population standard deviation, $\sigma$. We are interested in the question of whether $\mu$ is equal to some specific value $\mu_0$ — maybe the value it used to have before some intervention, or the value someone says it should have — but since the population size $N$ is large, it is impractical simply to go out and collect all the data to compute $\mu$ and see if the null hypothesis $$H_0:\quad \mu=\mu_0$$ is true. Here, $\mu_0$ is always a specific number and we are wondering if the population mean $\mu$, which is out there in the universe but which we have trouble finding out exactly, is equal to that number $\mu_0$.
Perhaps we have reasons to think that if the null hypothesis is not true, then maybe a particular alternative hypothesis $H_a$ might instead be true. The alternative hypothesis can have one of several forms:
To see which hypothesis, $H_0$ or $H_a$, is supported by the data, we take a sample of size $n$ and compute the sample mean $\overline{X}$. You might think that if $\overline{X}<\mu_0$, that would tell us that $\mu$ must also be less than $\mu_0$ ... but $\overline{X}$ varies a bit every time we take an SRS, so unless $\overline{X}$ is significantly less than $\mu_0$, we can't take this as evidence that $\mu<\mu_0$. Well, we might be able to take $\overline{X}<\mu_0$ as good evidence that $\mu<\mu_0$, if $\overline{X}$ is enough smaller than $\mu_0$ so that this is unlikely to happen just by chance.
In fact, as we have done before, $\overline{X}$ should be significantly smaller than $\mu_0$, in terms of how much variation we typically see in $\overline{X}$. Fortunately, the CLT helps here!
Here's how this works:
Let's assume the null hypothesis $H_0$ is true. Then we can use the
full CLT, which tells us that $\overline{X}$ is distributed like
$N(\mu_0,\sigma)$, to make a $z$-score out of $\overline{X}$. This number is
called the test statistic, or, for this particular test (being about
$\mu$ and with $\sigma$ known), it is called $z$-statistic:
$$z=\frac{\overline{X}-\mu_0}{\sigma/\sqrt{n}}\ .$$
We then compute the probabilty on the standard Normal distribution that the test statistic has values like that, or more in the direction of the alternative hypothesis. This probability is called the $p$-value of the test. Here is how we compute it depending upon the particular alternative hypothesis:
What does the $p$-value tell us? Remember, to compute it, we assumed that $H_0$ was true. Therefore, the $p$-value is the conditional probability
The reasoning here is a little strange, but can be understood! E.g., suppose I told you that I had a fair coin (that's playing the role of the null hypothesis). If I flipped it 100 times and got 53 heads, you would say that is not exactly 50/50, but it is close enough the it could easily have happened just by the randomness of the flips — i.e., the probability of seeing something like what you saw, given that the coin was fair (the $H_0$), is not too small. So you would fail to reject the null hypothesis, meaning you had no statistical evidence to reject the idea that this is a fair coin. If, instead, I got 97 heads, you would say that is so unlikely to happen if the coin were really fair, that since it happened, you don't believe any more that the coin is fair. In other words, since the probability of seeing what you saw, or something more extreme, given that the $H_0$ is true, is so small, but you did see it, so you take that as statistical evidence that the assumption $H_0$ is true must be false. In other words, the $p$-value being small, you reject the null hypothesis.
How small is small enough? Sometimes you pick a value $\alpha$, called the significance level, and then we if $p<\alpha$, you say We reject the null hypothesis at the $\alpha$ significance level. (The courts in the United States use $\alpha=.05$.) So, for example, if you did a hypothesis test and got a value of $p=.07$, you might say "We reject the null hypothesis at the 10% signifcance level, but fail to reject at the 5% significance level."
A more modern approach to hypothesis testing does not simply stop at particular significance levels. So, if the $p$-value were $.02$, rather than saying "We reject the null hypothesis at the 5% level," you would probably say "We reject the null hypothesis ($p=.02$)." This gives much more information! For example, two tests that gave $p$-values of $.02$ and $.0007$ might both cause you to reject at the 5% significance level, but the rejection with $p=.0007$ should be a much stronger one, as it will be if you report the $p$-value as well as the simple conclusion.
One final comment. If the $p$-value of a test is $.05$, that means you would expect to see values like the ones you saw, or more extreme, in 5% of the SRSs you might take from that population. In other words, if you do 20 SRSs or more, it would not be suprising to see one corresponding $\overline{X}$ which gives a test statistic like the one you saw. For example, if you tested 30 medicines for some disease, and all were actully equally ineffective, it would still not be surprising to see that one of the tests showed that it was effective, in the appropriate hypothesis test, at the 5% significance level, just due to randomness of the SRSs you were taking.