• The Ecstasy and Agony of Primitive Learning Analytics

    I’m awake and trying to be productive (for me) early in the day. I’m technically on a medical leave of absence but I’m not very good at doing nothing. I therefore promised to coordinate and edit the efforts of four moderators to produce a cohesive TT284 moderators’ report and I have some work ahead contributing my share to one for T320 too. This led to some musing about the primitive learning analytics I like to collect based on forum participation and the difficulties in obtaining them.

    Forum Statistics for OU Courses

    One thing I like to do is track forum usage statistics, a primitive form of learning analytics. Since we changed to Moodle from FirstClass, I don’t find this very easy. In FirstClass, not only could you do standard types of search on message data, but the read history of each message was also searchable. Combine that with a built-in way to restrict the search to specific conferences, sort the output by conference, user, or date, and group by conference or user, and you could determine all kinds of things. Some of my favourites were:

    • Total # of messages posted.
    • Total # of unique posters.
    • Total # of unique readers contrasted with enrolled students.
    • Percentage of posts that were moderators/course news versus students.
    • Top ten student posters and % of overall posts they contributed.
    • A breakdown of posting activity by logical parts and subparts, e.g. “Block 1″ overall but also “Block 1: Software Support” and “Block 1: Discussion”.

    The last one was useful to examine between different presentations when combined with knowledge of total number of students enrolled. It permitted me to see where students had the most problems and collect evidence if, when changes had been made for the following presentation, changes were having a positive effect. You could also see the trends in posting behaviour across cohorts.

    Getting at the Data

    In theory, some of this information is available in the Moodle logs. I just downloaded the log for one of my past courses I chaired and was surprised to note I could see “add reply” buried amongst the many “view forumng” entries. It’s downloadable as a CSV, so you’d have to roll your own data analysis tools to pull out the relevant bits. There are built-in statistics analysis facilities but they always seemed to be disabled on my courses, making download logs the only real option.

    The problem is access to those logs isn’t always available. As a course chair on Moodle 1.x, if the course was “editable”, then the admin tools were visible and the logs could be accessed. My last presentation (2012B, ending May 2012) somehow got into LTS’s update loop and the status/workflow changed back to needing to request access, so the admin links aren’t visible. I was able to hack the URL based on access to another course and get at it but that’s a bit of a pain.

    On my Moodle 2.x version course, I can see “Reports” but not a link to logs anywhere. I could edit the course site and back up the content, but perhaps I don’t have the permissions to access the logs. Certainly a typical moderator likely wouldn’t.

    What I Do in Moodle

    My approach generally in Moodle, regardless of the version, has therefore been very simplistic. I discovered that if I used Safari (but not Firefox) and copied the table listing the threads in a given forum and then pasted that into a spreadsheet, the HTML table’s columns were preserved. I could then have it sum the total number of messages per forum as one of the columns was number of thread posts. This isn’t very automated. I have to do it per forum and copy the totals into an appropriate place and most forums have multiple pages, each of which has to be handled separately.

    To Automate Or Not

    This is ripe for automation because certain actions are predictable, repeatable, and tedious. It’s the classic story though: do I spend the time trying to write something to automate it or just do it? Which will take less time? In the long run, if you do this yearly and across many courses, then automating it will save you time but there’s that up-front cost.

    A tool would also need to have a settings file, probably listing the module’s base URL and containing a list of the forum ID numbers/URLs and names. These are required because every presentation has a different ID and every forum has its own unique ID used to access it. Most modules don’t maintain a page that solely lists only the forums and the number/structure of those forums would vary between different modules. I suggested including names—or at least names I’d like to use to refer to them in reports—because otherwise you have to scrape that off the forum pages too and I’d find shorter ones more useful than the full, formal names.

    Another issue to contend with is authentication. I don’t already have code that can sign into the OU and maintain authentication for the session, although I know some people must. Before we had the “Dashboard”, one T320 AL wrote a tool to scrape metadata from the VLE and stored it in a local MySQL database. He then had an interface producing a dashboard for him that was something more than just a list of forums per course with an unread message indicator. I’ve recently heard, however, he gave up on his tool because VLE changes kept breaking it.

    Conclusion

    Here I am writing about what I should be doing rather than doing it, but the process of thinking about it is always useful. Perhaps someone’s already done some of or all of this? My bet would be on Tony Hirst, but LTS colleagues may have some tools and I just don’t know about them.

     

    4 comments on “The Ecstasy and Agony of Primitive Learning Analytics”

    • Trandafir says:

      I think the better qsuteion to ask is, What do the developers feel they can support as they move to Moodle 2.0? It seems to me that your asking the qsuteion means you want to move in a new direction because of some technical issues you are encountering. I am new to poodLL application and I am very interested in getting this setup on my 2.1 box. Setting up RED5 is not that bad, but I think if you could do without it you might pick up some additional users. I would say stop and think about what would be best long term. What is going to allow for poodLL to grow as needs or requirements change? If not going with Red5 gets you access to use this on an iPad, I would say DROP Red5. My district is buying iPads left and right for classrooms instead of netbooks

      • Eingang says:

        Hi Trandafir.

        The Open University’s Moodle user base is huge. They’re supporting tens of thousands of students and staff and they’ve invested substantial effort into linking everything up with their other backend systems. Some courses, like the one I was talking about, were deployed on an earlier version of the Open University’s Moodle system. That’s based on Moodle 1.x. Some newer courses, which I’m also involved in, are based on the newer Moodle 2.x.

        My role is in presenting courses and/or working with students enrolled in courses that are hosted via Moodle, not in deciding IT strategy or in installing and supporting the IT. I don’t really know much about the innards of Moodle 1.x versus Moodle 2.x and why it would be better to migrate or work with Moodle 2.x. I’m also not familiar with RED5 or PoodLL or what they can or can’t offer me as an end user.

        The Open University has been making efforts to make their content more mobile-device accessible. I think that’s mostly done, however, through the design of the site and not through additional plug-ins. That is, AJAX, HTML, and CSS are being used to make a “responsive design” rather than needing to either dynamically stream the content in an appropriate format by running it through a converter or caching different versions.

        Going back to the idea of data analysis, one of my frustrations is that, because the OU is such a large organization and there’s a tendency to lock things down, getting at the kinds of data for performing useful analytics of the behaviour of students on course forums or on course pages can be quite difficult. I do have access to logs on Moodle 1.x-hosted courses I work on at our organization but not on the newer Moodle 2.x one. Even when you have access to the logs, sometimes the detail you want isn’t there. For example, I can tell someone downloaded a file, but the Moodle 1.x logs I get don’t tell me which file the student downloaded. I know a student added a reply in a forum but I can’t tell to which thread from the log entries. That’s not to say the log files I have are useless. I certainly can (and have been) determining all kinds of things. I’m hoping to write more about these topics in later entries!

        Thanks for stopping by.

    • Hi Michelle

      Thanks for writing this, since as analytics begin to roll out we really need feedback from end-users such as yourself on what’s working and what’s not.

      I can’t comment on what’s coming next on the frontline from the OpenU, but you can see from some of these links what various folks around the university are working on for the future…

      http://www.educause.edu/ero/article/improving-retention-identifying-and-supporting-risk-students
      http://www.slideshare.net/fullscreen/sbs/ascilitewebinaroct2012/49

      Simon

      • Eingang says:

        Thanks for stopping by, Simon. The same weekend I wrote this piece, I did automate my process somewhat to scrape things. I intended to write a follow-up but it’s on my to-do list. I got distracted by… delving into automating parsing of OU Moodle logs, importing them into a SQLite database, and then seeing what kinds of useful queries I could run on that. That has a planned blog post as well. If I have time today, I’ll try to squeeze in at least one of those before tomorrow’s #SolarSLA learning analytics symposium (http://stadium.open.ac.uk/2065)

-->