• The Ecstasy and Agony of Primitive Learning Analytics

    I’m awake and trying to be productive (for me) early in the day. I’m technically on a medical leave of absence but I’m not very good at doing nothing. I therefore promised to coordinate and edit the efforts of four moderators to produce a cohesive TT284 moderators’ report and I have some work ahead contributing my share to one for T320 too. This led to some musing about the primitive learning analytics I like to collect based on forum participation and the difficulties in obtaining them.

    Forum Statistics for OU Courses

    One thing I like to do is track forum usage statistics, a primitive form of learning analytics. Since we changed to Moodle from FirstClass, I don’t find this very easy. In FirstClass, not only could you do standard types of search on message data, but the read history of each message was also searchable. Combine that with a built-in way to restrict the search to specific conferences, sort the output by conference, user, or date, and group by conference or user, and you could determine all kinds of things. Some of my favourites were:

    • Total # of messages posted.
    • Total # of unique posters.
    • Total # of unique readers contrasted with enrolled students.
    • Percentage of posts that were moderators/course news versus students.
    • Top ten student posters and % of overall posts they contributed.
    • A breakdown of posting activity by logical parts and subparts, e.g. “Block 1″ overall but also “Block 1: Software Support” and “Block 1: Discussion”.

    The last one was useful to examine between different presentations when combined with knowledge of total number of students enrolled. It permitted me to see where students had the most problems and collect evidence if, when changes had been made for the following presentation, changes were having a positive effect. You could also see the trends in posting behaviour across cohorts.

    Getting at the Data

    In theory, some of this information is available in the Moodle logs. I just downloaded the log for one of my past courses I chaired and was surprised to note I could see “add reply” buried amongst the many “view forumng” entries. It’s downloadable as a CSV, so you’d have to roll your own data analysis tools to pull out the relevant bits. There are built-in statistics analysis facilities but they always seemed to be disabled on my courses, making download logs the only real option.

    The problem is access to those logs isn’t always available. As a course chair on Moodle 1.x, if the course was “editable”, then the admin tools were visible and the logs could be accessed. My last presentation (2012B, ending May 2012) somehow got into LTS’s update loop and the status/workflow changed back to needing to request access, so the admin links aren’t visible. I was able to hack the URL based on access to another course and get at it but that’s a bit of a pain.

    On my Moodle 2.x version course, I can see “Reports” but not a link to logs anywhere. I could edit the course site and back up the content, but perhaps I don’t have the permissions to access the logs. Certainly a typical moderator likely wouldn’t.

    What I Do in Moodle

    My approach generally in Moodle, regardless of the version, has therefore been very simplistic. I discovered that if I used Safari (but not Firefox) and copied the table listing the threads in a given forum and then pasted that into a spreadsheet, the HTML table’s columns were preserved. I could then have it sum the total number of messages per forum as one of the columns was number of thread posts. This isn’t very automated. I have to do it per forum and copy the totals into an appropriate place and most forums have multiple pages, each of which has to be handled separately.

    To Automate Or Not

    This is ripe for automation because certain actions are predictable, repeatable, and tedious. It’s the classic story though: do I spend the time trying to write something to automate it or just do it? Which will take less time? In the long run, if you do this yearly and across many courses, then automating it will save you time but there’s that up-front cost.

    A tool would also need to have a settings file, probably listing the module’s base URL and containing a list of the forum ID numbers/URLs and names. These are required because every presentation has a different ID and every forum has its own unique ID used to access it. Most modules don’t maintain a page that solely lists only the forums and the number/structure of those forums would vary between different modules. I suggested including names—or at least names I’d like to use to refer to them in reports—because otherwise you have to scrape that off the forum pages too and I’d find shorter ones more useful than the full, formal names.

    Another issue to contend with is authentication. I don’t already have code that can sign into the OU and maintain authentication for the session, although I know some people must. Before we had the “Dashboard”, one T320 AL wrote a tool to scrape metadata from the VLE and stored it in a local MySQL database. He then had an interface producing a dashboard for him that was something more than just a list of forums per course with an unread message indicator. I’ve recently heard, however, he gave up on his tool because VLE changes kept breaking it.


    Here I am writing about what I should be doing rather than doing it, but the process of thinking about it is always useful. Perhaps someone’s already done some of or all of this? My bet would be on Tony Hirst, but LTS colleagues may have some tools and I just don’t know about them.

  • Pigeonholing the Sample

    Photo of many coloured marbles
    Credit: Photo by Marsha Brockman (whodeenee) under an Attribution-NonCommercial-NoDerivs 2.0 Generic license

    Image: Marbles, many marbles. I think I have lost mine in a sample of many marbles.

    I’ve been re-running analyses today on my population of survey responses. I decided to remove some more responses to eliminate some the scatteredness in the population. The majority of responses were from European PvE (player versus the environment) realm players, so I removed the four American realm players and then the five non-PvE players, leaving me with a sample of 30.

    The more I read about sampling, the more confused I am.

    Read the rest of this entry »

  • Coding It Wrong on the Right Side of Town

    Photograph of Elephant and Castle on a rainy day in London through rain-streaked window
    Credit: Photograph by Keven Law under an Attribution-ShareAlike 2.0 Generic license

    Image: Photograph of street near Elephant and Castle on a rainy day in London through rain-streaked window

    I’m about halfway through my initial coding of the motivation essays collected last April.  I should have been done this months ago, but I’ve somehow been scared to do it.  I think the big reason behind that is I’m afraid that I’m doing it or will do it incorrectly.  As I am going through and creating codes, I cannot help but feel that I am not always focussing on the motivation issue, which is the primary question. I am generally coding for content or themes I see appearing in the essays.  As an example, an essay may express that the author is more likely to assist someone else if they feel that other person has put some effort and thought into their character.  That is not their motivation for playing, but I have still created a code for it as “assist others”.  When I get to the end and review the list, I will not be able to tell which ones refer to motivation.  Some probably are where a participant has expressed it as a motivation, but other instances, even of the same code, might just be a theme that was raised.

    Read the rest of this entry »

  • Hermeneutics as Methodology

    I was reading through Chapter 4 of Silverman’s (2010) Doing Qualitative Research.  This chapter looks at the methodological approaches that different students take.  This is, of course, an important part of having a framework from which to hang your analysis.  There are so many choices.  He starts off with some descriptions of students describing their work as discourse analysis, narrative, analysis, and hermeneutics.  At first I thought this was related to something I’d looked up earlier in the month, heutagogy, but it’s just that they both start with “he”.  Wikipedia defines hermeneutics like this:

    Hermeneutics (English pronunciation: /hɜrməˈnjuːtɨks/) is the study of interpretation theory, and can be either the art of interpretation, or the theory and practice of interpretation. Traditional hermeneutics — which includes Biblical hermeneutics — refers to the study of the interpretation of written texts, especially texts in the areas of literature, religion and law. Contemporary, or modern, hermeneutics encompasses not only issues involving the written text, but everything in the interpretative process. This includes verbal and nonverbal forms of communication as well as prior aspects that affect communication, such as presuppositions, preunderstandings, the meaning and philosophy of language, and semiotics.[1] Philosophical hermeneutics refers primarily to Hans-Georg Gadamer’s theory of knowledge as developed in Truth and Method, and sometimes to Paul Ricoeur.[2] Hermeneutic consistency refers to analysis of texts for coherent explanation. A hermeneutic (singular) refers to one particular method or strand of interpretation.
    Wikipedia (2010)

    It’s apparently related to computational semiotics or used in computational semiotics.  That reminds me of James Paul Gee again because he talks about the semiotics of things in his What Video Games Have To Teach Us about Learning and Literacy (2007).  Is it another sign that I need to be looking at Gee’s book on discourse analysis (Gee 2011)?


    Gee, J.P. (2007) What Video Games Have To Teach Us About Learning and Literacy, 2nd edition, New York, NY, United States, Palgrave Macmillan.

    Gee, J.P. (2011) An Introduction to Discourse Analysis Theory and Method, 3rd edition, Abingdon, United Kingdom, Routledge.

    Silverman, D. (2010) Doing Qualitative Research: A Practical Handbook, 3rd edition, London, United Kingdom, Sage Publications Ltd.

    Wikipedia. (2010) Hermeneutics, [online] web page, Wikipedia. Available from: http://en.wikipedia.org/wiki/Hermeneutics (Accessed September 21, 2010).

  • Quantitative or Qualitative: The Eternal Question

    Doing Qualitative Research: The Book

    Doing Qualitative Research: The Book

    Chapter 2 of David Silverman’s Doing Qualitative Research:  A Practical Handbook (2010, p.16) asks students to consider why they believe a qualitative approach is appropriate for their possible research topics.  In fact, I had not initially considered a qualitative approach at all.  With my background in artificial intelligence, software engineering, and information retrieval, I was tending towards quantitative methodologies.  Information retrieval is very much about calculations and measurement, so that was a natural fit. Wikipedia (2010) describes the qualitative method as one that “investigates the why and how of decision making, not just what, where, when.”

    Read the rest of this entry »

  • How To Track People Anonymously Across Multiple Studies

    Image of Zul'Aman Dragonhawk boss fight
    Image: Elsheindra and Team Pink tackle the Dragonhawk Boss in Zul’Aman back in 2008. As a healer, Elsheindra has to make difficult decisions about who will live and who will die, in her role as main healer.  Being a researcher and maintaining anonymity is, I’ve discovered, a lot easier.

    Back in April, I posted my first preliminary study to look at motivation, community formation, and learning in World of Warcraft.  When I was crafting my ethics approval for that study and future studies, I was very concerned with maintaining the privacy of the individuals participating.  The first survey was designed specifically to not require any personally identifiable information, although participants did have the option of giving an e-mail address if they wanted to participate in future studies or if they did not mind being contacted for any follow-up questions.

    A problem arises, however, in following participants across multiple studies.  This is somewhat related to longitudinal studies where repeated observations are collected over long periods of time from the same participants.  The purpose of such studies is to help distinguish actual effects from short-term causes.  However, longitudinal studies aren’t the only time researchers may want to track participants across time and across multiple studies.  That would also be useful to help me build a more complex, detailed picture of participants, even though I intend to be asking different questions in different surveys.

    Read the rest of this entry »

  • The Great Date Night Experiment

    When I last saw J, my supervisor, we were disagreeing about how to do the motivational essay coding for my first World of Warcraft survey.. My plan was to go through the essays first to come up with some themes. Then Basil and I would independently code them for theme. My reasoning was I wanted the coding to be free from subjective bias. If two of us agreed independently, then that would be better than just my assessment of the data. J. thought it was unlikely Basil and I would agree, so she set me the “Great Date Night Experiment.” In this experiment, Basil, my partner, and I would sit down on “date night” and test out my theory on a small scale. Basil would read one essay and summarize the main themes or ideas he thought were represented in the essay. I would independently do the same. Then I would report back to J.

    Read the rest of this entry »

  • OU in the Cloud: The Q&D Results


    I know people are very curious about the results of my recent E-Mail in the Cloud: An Open University Survey. Time is a bit short for me, so I decided to write up this quick and dirty post outlining the key result. An analysis of the comments people left about why they made the choice they did will be covered in a later posting, as those comments proved to be extremely interesting.

    In a more formal report, the order of detail presented would be different. I’ve started with the results first, as that’s likely to be of interest to most people, and then discussed the methodology, survey deployment, and motivation.

    Read the rest of this entry »