As the term has progressed, my LSM2241 lectures are getting more consistent. I’m aiming to use 45 slides for what is officially a two hour lecture, although in reality it lasts about 90 minutes. We take a break at about 50 minutes, and the students leave at about 11:40 in order to make any noon lectures elsewhere on campus.

Feedback on each lecture is now given on-line, with four numeric rating questions, and three free-form questions. The inspiration for this weekly feedback was from the story of Muhammed Zaman in this New York Times article, and in fact I’m using Zaman’s questions, which he provided to Paul Matsudaira. The questions are:

1. How would you rate the professor so far? (1-5)
2. Has the professor improved in his teaching since last time we had this survey? In particular, has he incorporated your suggestions? (1-5)
3. How satisfied are you with the teaching so far? (1-5)
4. How would you rate this course so far? (1 being difficult, boring and useless and 5 being interesting and useful)
5. How can the professor improve your learning of the material? (free text)
6. How can the delivery of the material be altered to enhance your understanding of the material? (free text)
7. Is there anything else you would like to convey to the professor? (free text)

The feedback has been very useful, and I’ve tried, with mixed success, to incorporate the feedback in to each subsequent lecture. To manage the process, I now have an Org template that munges the results downloaded from the on-line form in a CSV file and produces a PDF report. The numeric ratings are presented in bar charts, with trends over multiple lectures shown in line charts. This makes the feedback itself a series of reproducible reports, much, much deeper than I could possibly get with conventional course feedback. I present all the comments unfiltered, and upload the report within a day of the lecture. (It actually takes about 5 minutes to generate, but the students don’t submit the feedback until late in the afternoon). In each lecture, I’ll open with a few slides with highlights from the feedback and how I intend to use it.

I also use the feedback slides to poke fun at myself a little bit, which breaks the ice with Singaporean students who tend to be – at least to my face – a bit overly deferential.

On the technical side, there are also things to be learned. For starters, the ob-R.el library, which handles in interaction between Org mode and R, exhibits some odd behaviors. I still prefer it to knitr, for a whole host of reasons, but when I posted a bug report to the Org email list, I found out that there is no active maintainer of ob-R.el. So I may wind up patching it myself.

Here’s one such oddity:

#+begin_src R :results value list 
c("a","b","c")
#+end_src

which results in

• (“a”)
• (“b”)
• (“c”)

That’s not what should happen. The :results value list directive should create an org mode list. What should happen, instead, can be achieved by some additional processing in R:

#+BEGIN_SRC R :results output raw
a <- c("a","b","c")
cat(paste("-", a), sep="\n")
#+END_SRC

• a
• b
• c

All of the free text feedback is reported as bulleted lists, so I may end up hacking the elisp rather than continuing this workaround in R. The ob-R.el part of Org-babel is just so useful it really needs to be maintained.

An example of the Org file (with R code blocks, and with results included) can be found here, and a PDF of the associated report is here.

I’ve also been toying with sentiment analysis for the free text question. For this I find the recent blog posts by Schaun Wheeler very useful, especially the bulk run through a spell checker with automated fixes. Unfortunately, sentiment analysis is easily fooled by specialized language. For example, the most recent lecture in LSM2241 covered the BLAST tools for searching sequence databases. I rushed the end of the lecture, and the numerical scores dropped a bit from the previous week. This didn’t surprise me, but I was surprised when my attempt to classify_emotion (from the sentiment package) showed a strong amount of “joy” for all questions, while the classify_polarity was less positive than it seemed from my reading of the text, particularly in the most recent lecture. What was going on?  I looked at word frequencies associated with the sentiment classifications.

Word cloud for feedback responses for question 5 of the BLAST lecture

One of the most common student requests (for every lecture) is “please slow down”, a request I always take seriously at the beginning of a lecture before I rush towards the end. But “please” in the lexicon used by classify_emotions signals joy, not pleading. Similarly, the word “blast” shows up in the polarity lexicon as negative sentiment in classify_polarity.  I hope to get some of this cleaned up and useful in the next couple of weeks, so the amount of data I’m collecting has a decent analysis to accompany it.