Teaching students to talk about data science

Last summer (😬), I was invited to give a talk at the Joint Statistical Meetings on data science education. The session I was invited to join included Shannon Ellis, Stephanie Hicks, and Michael Love as the other speakers, and I immediately knew what I wanted to talk about!

Figure 1: L to R: Shannon Ellis, Stephanie Hicks, Michael Love, me

Teaching students to talk about data science

One of the things I spent a lot of time thinking about in my 12 years as a professor was how to teach students how to communicate about data science with other human beings. It is sometimes all too easy to focus on “just” the practical skills like being able to wrangle, explore, analyze, and visualize data (preferably using code). But, teaching students to be able to describe, document, plan, collaborate, question, and communicate effectively about data science? That is way harder than it sounds.

For example, you may know learners like this:

You can ask them focused questions on an assignment, and they do fine. Maybe even great.

Figure 2: A+ writing

Then you may ask them to present their work. Also fine. Maybe even great.

Figure 3: A+ presenting

Then comes the Q&A, and maybe a colleague you invited or another student asks a good question…and your star student freezes. Or maybe they tell you about a job interview they had, and they fumbled it.

Figure 4: Oh no

As a former language researcher, I like to compare this to learning the semantics versus pragmatics of language. So semantics is concerned with getting vocabulary and meaning correct. But pragmatics is all about language use in real life. And to get a job, to engage with and contribute to the data science community, learners need pragmatics.

Figure 5: The semantics & pragmatics of data science communication

So how did I start getting them there? Sort of by accident! One day, I happened to stumble upon a syllabus by Hal Daumé from his machine learning class. I noticed something intriguing in there called “self-assessments.”

Figure 6: From Hal Daumé's syllabus

This slide was all I could find out about it, but it was enough to get my wheels turning right away. Having a Ph.D. in developmental psychology, I’d spent a lot of time studying the literature on the science of learning, and the idea of a self-assessment made me think:

Why might this actually work? It reinforces some tried and true self-directed learning strategies that are actually evidence-based like:

• retrieval practice!
• distributed practice!
• elaborative interrogation!
• self-explanation!

All that to say, self-assessment would help students talk to themselves and to me about data science- in a way that I hadn’t enabled before.

I had also had this little bee in my bonnet for awhile, and that was that I spent an inordinate amount of time crafting the most beautiful homework answer keys for my students…that no one seemed to read. No one used my answer keys for learning. It bothered me because I knew there was value in reviewing these answer keys, but I hadn’t made them into an actual learning tool. When I saw Hal’s slide, it dawned on me: the solution to my problem was to give my answer keys away, and make them into a usable learning tool.^[I later found out that my palmerpenguins co-author Dr. Allison Horst does this too! https://www.allisonhorst.com/post/share-keys/]

The truth is, providing the answer keys or feedback at the end is not actually helpful for learners. Students have high motivation and set aside time for what matters for their grade, not necessarily their learning. So it’s no surprise that motivation was low and time was limited for students to devote to “extracurricular” learning.

What if the homework was less about how well students could do with little support, and more about how much they could learn with all my support?

I decided to find out. The next time I taught my statistics course, I began requiring students to turn in self-assessments. Here is my version of Hal’s slide with my rough plan:

Figure 7: Self-assessment plan

If you look carefully there, you’ll see that there are in fact two ways to succeed while really trying in my class, and I was very proud of this. At the beginning of class, I would tell students that some over-achievers might find this annoying at first, but that for me, my ultimate goal was that every single person learned. I didn’t care whether you got it right off the bat, or whether you needed more time and resources to get it right. Either way, my grades mapped onto actual learning.

Also note that the 0/1/2 points for the initial submission had nothing to do with accuracy of their initial solutions. And the accuracy for the self-assessment had only to do with the accuracy of their self-assessment! So really, we washed away the grading effect of getting it right in 1 week alone or in 2 weeks with the key. The students who got it right on the initial submission bought themselves every other week free and clear from assignments, though.

How to succeed while really trying

Submit a perfectly accurate initial solution.

If your initial submission was a good faith effort (2 points) and happened to be perfectly accurate when you reviewed the answer key (3 points), you got a 5/5 for that question.

“I found the same results as the answer key. Bada bing, bada boom.”

Submit a perfectly accurate self-assessment.

If your initial submission was a good faith effort (2 points) but you did a bang-up job analyzing all of your mistakes and misunderstandings after thoroughly reviewing the key (3 points), you got a 5/5 for that question.

“Well, well, well. I was really confused about interaction effects.”

“Gah! I didn’t actually understand the null hypothesis here.”

“I forgot to set a seed for my permutation test, so now I see why my results kept changing!”

Sample self-assessments

“Most of my code matches the answer key - there were only one or two sections where I didn’t make the correct plot or write the correct code for the desired output, but I think a good portion of my narrative was accurate, which is encouraging.”

“After doing this self-assessment, I’m realizing that I had a better grasp on this analysis than I had previously thought. I made a couple small errors, but after seeing the answer key, I think I understood why I made them and how to avoid those errors in the future. "

“I liked that the self-assessment allowed me to see where I was close and what parts I was really far off.”

“I found that, while generating this model was straightforward, interpretation of the results was more difficult than I expected.”

Logistics

OK, at this point, you may also be credibly curious. But you are probably wondering, how on earth did this actually work?

Turning it in

We advised students to color their text in R Markdown files knitted to HTML. The self-assessment portion always needed to be colored. You can find guidance on how to do that in the R Markdown cookbook: https://bookdown.org/yihui/rmarkdown-cookbook/font-color.html

Trade-offs

Yes, I did have to cut back significantly on my homeworks. Whereas I would have anywhere from 6-8 homeworks before, I went down to four per quarter (plus a final project). This was because I typically allowed 1 week for the initial submission and another for the self-assessment, with no overlap. So each homework took 2 weeks. I trimmed homework length down, and I cut out anything that wasn’t essential.

Time spent by TAs

Honestly, I would have loved for self-assessments to have saved myself and my TAs time. I cannot say this was my experience. It was about the same amount of time overall.

But, what our teaching team did notice was that we felt better with our own grade assignments, and we enjoyed the grading process more. Rather than feeling like you were searching for mistakes to remove points, you were searching for true insights gained to add points. Small code errors or typos were usually caught by the students themselves, so TAs felt less personally responsible for making sure students were made aware of everything they did wrong. Also, student self-assessments were funny, charming, and allowed us to get to know them better. We also were better able to identify students who could use 1-on-1 help…because they would tell us that.

And that brings me to the most common question I get: how did I assign grades? Truth be told, grading was my absolute least favorite part of teaching in academia. But self-assessments actually helped me get to a place where I felt like my grades mapped onto my goals. Here is an excerpt from my syllabus that describes my system:

Homework

A total of 4 homeworks will be assigned; your lowest score will be dropped at the end of the quarter. Some homeworks will require you to use R to analyze data. Although no prior R experience is required for this course, be prepared to do a lot of self-guided learning. Students are expected to run R on their own computer or a computer they have plenty of access to and control over. Please attempt to do all homeworks on your own, but you may work with other students. However, you may not submit homework assignments as a group. You should submit your own original work.

You will have 1 week to complete each homework assignment, and your initial solutions must be submitted to Sakai by 2pm on the due date (at the start of class on Thursdays). Late homeworks will not be accepted.

Homework self-assessment

After the initial solutions are due, you will be provided a solution key. Using that key, you will be asked to assess your own initial solutions for accuracy and thoroughness; where you made mistakes, you must discuss and analyze where you went wrong, and correct them without copying/pasting directly from the key (this typically means that you need to include more narrative than we provide in the key). A good self-assessment will include:

• Assessment of the accuracy and completeness of your “initial solutions”
• Correct worked solutions with some discussion and analysis of why your initial solution was incorrect, and reflection on the source of your confusion (if you got an answer correct, this is not necessary)
• Attributions as appropriate to other students who helped you, or other sources such as lecture notes, readings, online resources, etc. that helped you

Homework grades

Homework grades will be based on:

• Was your initial solution a good faith effort?
• Did you catch all of your own errors in your self-assessment?
• Is your updated solution correct?

Each homework includes 3 questions, worth 5 points per question, scored as follows:

2 points for each initial solution being “in-good-faith”.

• 2 (Strong attempt): answer reflects strong independent problem solving, with clearly thought out attempts to approach the problem and a diligent and honest effort to find the solution

• 1 (Weak attempt): answer reflects some attempt to approach the problem, but approach appears to be superficial and lacks depth of analysis

• 0 (No attempt)

3 points for the quality of the final answer / discussion.

• 3 (Exceptional): answer is thorough, concise, and clearly demonstrates ability to analyze and interpret statistics as well as theoretical understanding of statistical concepts

• 2 (Adequate): answer addresses the question with moderate inaccuracies in analysis and/or interpretation, or offers a correct but incomplete answer

• 1 (Inadequate): answer attempts to address question with substantial inaccuracies in analysis and/or interpretation

• 0 (Insufficient): answer does not attempt to address question or answer is insufficient to grade

This means that:

• You can get 100% of the points if you either:
  • Submit perfectly accurate initial solutions, or
  • Submit a perfectly accurate self-assessment,
• If you simply cannot submit any homework solutions on time, after the homework due date, you will receive the solutions key and can submit a self-assessment for a max score of 60% (3 out of 5 points per problem). We feel this is fair given that:
  • You did not attempt a good faith effort, and
  • We will drop your lowest homework grade.

What is a “good faith effort”?

Simply submitting nonsense or saying “I can’t do this” for each problem will not meet our criteria for a good faith effort, because there is no attempt on your part to show us why you are struggling, what you tried but didn’t work, what specific part of the problem you got stuck on, which other examples in the text/lecture you tried to work through to get a grasp on the problem, etc. ^[This is especially true if you don’t attend any office hours, post on Sakai, or otherwise seek out help before the due date.] The good faith effort is just that- we are interested in seeing evidence of a diligent and honest effort on your part, made with deliberate intention, to understand the problem and attempt an answer.

In the talk I gave at JSM, I talked about self-assessments as just one out of three strategies I used to get students talking about data science. My students often had a final project and/or presentation at the end of a course, which I also think helps them to think out loud and on their feet. I sincerely hope other educators try these kinds of methods out! I can honestly say I wouldn’t teach a university-level course any other way.

Figure 8: All three strategies