Some Pre- and Post-Test Data from the CC Cert Course, December 2020

Caveat: Not the most recent version of this study!

What follows is the original version of this study, dating from December of 2020. But a newer version, dating from January, 2022, is now available, please enjoy that one instead!

First, the over-all pre- vs post-test scores for all participants:

Next, the pre- vs post-test scores for all participants, for each Unit of the course, separately:

Finally, how about disaggregating the participant populations?

You may ask if the participants who choose to take the Librarian course — presumably, mostly librarians — come to the course with a different knowledge base from those who choose to take the Educator course — presumably, mostly faculty (with a few other professions thrown in)?

The answer to that is: these two subsets are statistically indistinguishable.

Why? Well, first lets look at the over-all scores, disaggregated by Librarian and Educator sections:

Two histograms of percentage correct of the whole test (which had 59 points of questions), for participants in the Librarian section, one histogram for before the course and one for after. Both show somewhat symmetric and very roughly bell-shaped distributions. The histogram for before the course has a median of 58.5, an x-bar of 57.366, a sigma of 13.118, and an n of 26. The histogram for after the course has a median of 79.7, x-bar=78.501, sigma=6.1, and n=19.

Two histograms of percentage correct of the whole test (which had 59 points of questions), for participants in the Educator section, one histogram for before the course and one for after. Both show fairly symmetric and roughly bell-shaped distributions. The histogram for before the course has a median of 50.8, an x-bar of 53.107, a sigma of 13.773, and an n of 21. The histogram for after the course has a median of 77.1, x-bar=76.412, sigma=7.305, and n=12.

Come on, you say, those librarians are starting with a higher pre-test score and ending with a higher post-test score.

There are two problems with that:

  1. There is variability in the scores, so the slightly higher average and median for librarians might just be due to this random variation. We can do a T-Test of Independent Population Means to test this, and it determines that:
    These samples provide insufficient evidence to reject the null hypothesis that the pre-test averages of these two groups are the same in favor of the alternative hypothesis that the librarians' average is higher, with a p-value of .1492.
    [And in fact the p-value is .2984 when using the alternative hypothesis that the two groups simply have a different average.]
    [That's a statisticians' way of saying that there is very, very little statistical evidence of this difference being anything other than random variation in these particular samples. The quantification of how weak the evidence is comes from that p-value, which is a number in the range from 0 to 1 and the larger it is, the weaker the evidence. A typical cut-off for a result to be considered the typical cut-off for a result to be statistically significant. is p=.05, and we're well above that here.]
  2. But even the post-test scores are higher, you cry. Well, after both groups go through the cert course, they are no longer independent: they've both had the same treatment (the cert course) and have even interacted with each other in the common course Slack channel. The T-Test only applies to independent samples, so we cannot use it again. And, in fact, if everyone does better after taking the course, then we would expect that if a random group of librarians just happened to start out with a higher average score than another random group of folks taking the educator version, then of course they will both have scores that go up after taking the course, probably by around the same amount, so that random group of librarians will end up with a higher average post-test score. [Whew!]

Let's look at the disaggregated participant populations (librarians' section and educators' section), disaggregated by each unit!

Maybe there is a statistically significant difference in these two population at least with respect to how they do on particular units.... [Also, I just like saying the word disaggregated.]