• The Ecstasy and Agony of Primitive Learning Analytics

    I’m awake and trying to be productive (for me) early in the day. I’m technically on a medical leave of absence but I’m not very good at doing nothing. I therefore promised to coordinate and edit the efforts of four moderators to produce a cohesive TT284 moderators’ report and I have some work ahead contributing my share to one for T320 too. This led to some musing about the primitive learning analytics I like to collect based on forum participation and the difficulties in obtaining them.

    Forum Statistics for OU Courses

    One thing I like to do is track forum usage statistics, a primitive form of learning analytics. Since we changed to Moodle from FirstClass, I don’t find this very easy. In FirstClass, not only could you do standard types of search on message data, but the read history of each message was also searchable. Combine that with a built-in way to restrict the search to specific conferences, sort the output by conference, user, or date, and group by conference or user, and you could determine all kinds of things. Some of my favourites were:

    • Total # of messages posted.
    • Total # of unique posters.
    • Total # of unique readers contrasted with enrolled students.
    • Percentage of posts that were moderators/course news versus students.
    • Top ten student posters and % of overall posts they contributed.
    • A breakdown of posting activity by logical parts and subparts, e.g. “Block 1″ overall but also “Block 1: Software Support” and “Block 1: Discussion”.

    The last one was useful to examine between different presentations when combined with knowledge of total number of students enrolled. It permitted me to see where students had the most problems and collect evidence if, when changes had been made for the following presentation, changes were having a positive effect. You could also see the trends in posting behaviour across cohorts.

    Getting at the Data

    In theory, some of this information is available in the Moodle logs. I just downloaded the log for one of my past courses I chaired and was surprised to note I could see “add reply” buried amongst the many “view forumng” entries. It’s downloadable as a CSV, so you’d have to roll your own data analysis tools to pull out the relevant bits. There are built-in statistics analysis facilities but they always seemed to be disabled on my courses, making download logs the only real option.

    The problem is access to those logs isn’t always available. As a course chair on Moodle 1.x, if the course was “editable”, then the admin tools were visible and the logs could be accessed. My last presentation (2012B, ending May 2012) somehow got into LTS’s update loop and the status/workflow changed back to needing to request access, so the admin links aren’t visible. I was able to hack the URL based on access to another course and get at it but that’s a bit of a pain.

    On my Moodle 2.x version course, I can see “Reports” but not a link to logs anywhere. I could edit the course site and back up the content, but perhaps I don’t have the permissions to access the logs. Certainly a typical moderator likely wouldn’t.

    What I Do in Moodle

    My approach generally in Moodle, regardless of the version, has therefore been very simplistic. I discovered that if I used Safari (but not Firefox) and copied the table listing the threads in a given forum and then pasted that into a spreadsheet, the HTML table’s columns were preserved. I could then have it sum the total number of messages per forum as one of the columns was number of thread posts. This isn’t very automated. I have to do it per forum and copy the totals into an appropriate place and most forums have multiple pages, each of which has to be handled separately.

    To Automate Or Not

    This is ripe for automation because certain actions are predictable, repeatable, and tedious. It’s the classic story though: do I spend the time trying to write something to automate it or just do it? Which will take less time? In the long run, if you do this yearly and across many courses, then automating it will save you time but there’s that up-front cost.

    A tool would also need to have a settings file, probably listing the module’s base URL and containing a list of the forum ID numbers/URLs and names. These are required because every presentation has a different ID and every forum has its own unique ID used to access it. Most modules don’t maintain a page that solely lists only the forums and the number/structure of those forums would vary between different modules. I suggested including names—or at least names I’d like to use to refer to them in reports—because otherwise you have to scrape that off the forum pages too and I’d find shorter ones more useful than the full, formal names.

    Another issue to contend with is authentication. I don’t already have code that can sign into the OU and maintain authentication for the session, although I know some people must. Before we had the “Dashboard”, one T320 AL wrote a tool to scrape metadata from the VLE and stored it in a local MySQL database. He then had an interface producing a dashboard for him that was something more than just a list of forums per course with an unread message indicator. I’ve recently heard, however, he gave up on his tool because VLE changes kept breaking it.

    Conclusion

    Here I am writing about what I should be doing rather than doing it, but the process of thinking about it is always useful. Perhaps someone’s already done some of or all of this? My bet would be on Tony Hirst, but LTS colleagues may have some tools and I just don’t know about them.

     
  • Research Journal Tools Reflection: NotePad Deluxe

    Image of several open paper notebooks open on top of one another
    Photo by Abizem / CC BY-NC

    My friend Howard Rheingold was asking on Twitter recently about how people use DEVONThink, a personal information manager for the Macintosh.  While Howard’s relatively new to DEVONthink, I’ve been using DEVONthink Pro since 2005.

    DEVONthink Pro includes all kinds of import and file-use features to help people put all their information in one place.  It can import or natively use PDFs, e-mail archives, Microsoft Office files, OPML (like from OmniOutliner, for example), iWork documents, log files for instant messenger clients or from SecondLife, BibTeX, XML, and address book.  The Pro edition can also interface with a scanner to import your printed documents into the application and even OCR them.  I don’t use most of these features.  I don’t believe there’s a one-size fits-all information management tool, so I stick to dedicated applications for managing my e-mail, bibliographies, and chat logs.  The primary use I make of DEVONthink in a research context is as my ongoing research journal.

    This is the first of a series of posts reflecting on using personal information management tools, like DEVONthink, for maintaining a research journal while doing my Ph.D.  Before I talk about DEVONthink, I think it will be helpful to go back in time and look at how my research journal began.  This will help explain why my research journal is structured the way that it is today as well as illuminate the path that led me to DEVONthink. I start off the series looking at NotePad Deluxe.

    Read the rest of this entry »

     
-->