Forum Statistics for OU Courses

One thing I like to do is track forum usage statistics, a primitive form of learning analytics. Since we changed to Moodle from FirstClass, I don’t find this very easy. In FirstClass, not only could you do standard types of search on message data, but the read history of each message was also searchable. Combine that with a built-in way to restrict the search to specific conferences, sort the output by conference, user, or date, and group by conference or user, and you could determine all kinds of things. Some of my favourites were:

• Total # of messages posted.
• Total # of unique posters.
• Total # of unique readers contrasted with enrolled students.
• Percentage of posts that were moderators/course news versus students.
• Top ten student posters and % of overall posts they contributed.
• A breakdown of posting activity by logical parts and subparts, e.g. “Block 1″ overall but also “Block 1: Software Support” and “Block 1: Discussion”.

The last one was useful to examine between different presentations when combined with knowledge of total number of students enrolled. It permitted me to see where students had the most problems and collect evidence if, when changes had been made for the following presentation, changes were having a positive effect. You could also see the trends in posting behaviour across cohorts.

Getting at the Data

In theory, some of this information is available in the Moodle logs. I just downloaded the log for one of my past courses I chaired and was surprised to note I could see “add reply” buried amongst the many “view forumng” entries. It’s downloadable as a CSV, so you’d have to roll your own data analysis tools to pull out the relevant bits. There are built-in statistics analysis facilities but they always seemed to be disabled on my courses, making download logs the only real option.

The problem is access to those logs isn’t always available. As a course chair on Moodle 1.x, if the course was “editable”, then the admin tools were visible and the logs could be accessed. My last presentation (2012B, ending May 2012) somehow got into LTS’s update loop and the status/workflow changed back to needing to request access, so the admin links aren’t visible. I was able to hack the URL based on access to another course and get at it but that’s a bit of a pain.

On my Moodle 2.x version course, I can see “Reports” but not a link to logs anywhere. I could edit the course site and back up the content, but perhaps I don’t have the permissions to access the logs. Certainly a typical moderator likely wouldn’t.

What I Do in Moodle

My approach generally in Moodle, regardless of the version, has therefore been very simplistic. I discovered that if I used Safari (but not Firefox) and copied the table listing the threads in a given forum and then pasted that into a spreadsheet, the HTML table’s columns were preserved. I could then have it sum the total number of messages per forum as one of the columns was number of thread posts. This isn’t very automated. I have to do it per forum and copy the totals into an appropriate place and most forums have multiple pages, each of which has to be handled separately.

To Automate Or Not

This is ripe for automation because certain actions are predictable, repeatable, and tedious. It’s the classic story though: do I spend the time trying to write something to automate it or just do it? Which will take less time? In the long run, if you do this yearly and across many courses, then automating it will save you time but there’s that up-front cost.

A tool would also need to have a settings file, probably listing the module’s base URL and containing a list of the forum ID numbers/URLs and names. These are required because every presentation has a different ID and every forum has its own unique ID used to access it. Most modules don’t maintain a page that solely lists only the forums and the number/structure of those forums would vary between different modules. I suggested including names—or at least names I’d like to use to refer to them in reports—because otherwise you have to scrape that off the forum pages too and I’d find shorter ones more useful than the full, formal names.

Another issue to contend with is authentication. I don’t already have code that can sign into the OU and maintain authentication for the session, although I know some people must. Before we had the “Dashboard”, one T320 AL wrote a tool to scrape metadata from the VLE and stored it in a local MySQL database. He then had an interface producing a dashboard for him that was something more than just a list of forums per course with an unread message indicator. I’ve recently heard, however, he gave up on his tool because VLE changes kept breaking it.

Conclusion

Here I am writing about what I should be doing rather than doing it, but the process of thinking about it is always useful. Perhaps someone’s already done some of or all of this? My bet would be on Tony Hirst, but LTS colleagues may have some tools and I just don’t know about them.

Image: Marbles, many marbles. I think I have lost mine in a sample of many marbles.

I’ve been re-running analyses today on my population of survey responses. I decided to remove some more responses to eliminate some the scatteredness in the population. The majority of responses were from European PvE (player versus the environment) realm players, so I removed the four American realm players and then the five non-PvE players, leaving me with a sample of 30.

The more I read about sampling, the more confused I am.

When we read the Oliver and Carr (2009) WoW communities of practice and learning paper the other day, we were somewhat dismissive because it only had five couples. However, the paper also mentioned that it used “theoretical sampling”, which we had not heard of. Someone looked it up quickly on Wikipedia and it sounded like you chose your sample based on it generating the features you wanted to look at. Now the description in the Oliver and Carr paper sounds more like “purposive sampling”, because they described their sampling in a way that seemed to fit with Cohen et al’s description of “…qualitative researchers handpick the cases to be included in the sample on the basis of their judgement of their typicality or possession of the particular characteristics being sought” (2007, p. 114-115):

Players were recruited through online guilds and real-world social networks. The first two sets of participants were sampled for convenience (two heterosexual couples); the rest were invited to participate in order to broaden this sample (one couple was chosen because they shared a single account, one where a partner had chosen to stop playing and one mother–son pairing).
Oliver and Carr (2009, p. 446).

I was browsing through Research Methods in Education today and it specifically mentions theoretical sampling as a feature of grounded theory and the sample size is immaterial. The important part is that you have enough data to saturate the categories in your theory. You collect more and more data until the acquisition of more data does not advance or modify the theory developed. It suggests that the size of the data set may be fixed by the number of people to whom one has access but you have to consider that it may be necessary to seek further data (Cohen et al. 2007, 116-117). A sample of five couples would then possibly be acceptable. Although I am taking a grounded theory approach, this does not feel quite like what I am doing, although I do have the intention of generating the theory from the data I have and then pursuing a larger-scale study later.

Another possibility is volunteer sampling. This is apparently different than convenience sampling. I suppose in a convenience sample, you have more control over how many people respond, e.g. a class of students, and you are directly asking them. In volunteer sampling, you rely on volunteers, like personal friends or friends of friends, although it can also be via, for example, a newspaper advertisement (Cohen et al. 2007, p. 116). This sounds similar to the approach that I took. I already knew I had to be careful about making generalizations and certainly the representativeness of the sample is lacking. This is probably acceptable, provided the lack of typicality is made clear.

Real World Research describes a convenience sample as one of the most widely used (Robson 2002, p. 265). Sensible uses of convenience samples, Robson suggests, are for piloting a proper sample survey or getting a feeling for the issues involved. This too feels like what I was doing, since I designed the study originally to be the starting point for a future, larger study. Providing a springboard for future research is also described as being acceptable by Bryman (2008, p.183) in Social Research Methods.

My section describing the survey distribution currently reads as follows:

A blog site was created for the overall project and readers invited to participate (Hoyle 2010) through an initial posting. Readers were given a brief explanation of the survey’s purpose, contact details for the author, and an explanation of the rules and time and effort expected. The page explained that there would be an opportunity to enter an optional draw to win a virtual in-game pet as a reward. This page also contained a link to the survey, hosted on SurveyMonkey, a third-party commercial web survey site.

At a minimum, 25 to 30 participants fully completing the survey were required and more than 50 to 75 would be burdensome. Advertising was therefore not  ambitious or comprehensive. Short messages were broadcast periodically on a European (player versus environment) game realm to a text communication channel shared by members of five allied guilds. A month before the survey, allied guild leaders were questioned about their current membership numbers. This information is available in the game and reflects the number of individual accounts that belong to a given guild. Total number of player accounts was 437. That count includes inactive players and players belonging to more than one allied guild. It is also possible for players to have more than one account, if they are willing to pay for it, resulting in the same person being counted more than once. However, after discussion with the guild leaders, the number of people with multiple accounts or multi-guild membership was believed to be small; the number of people reported is therefore probably fairly close. However, it is difficult to estimate what proportion would be active players or would have seen the periodic messages.

In addition to the in-game messages, the study was also advertised numerous times via the author’s main Twitter account and an account dedicated to news for the allied guilds. This resulted in a number of rebroadcasts as other researchers and followers tried to assist by passing along the message. Twitter messages, by their nature limited to 140 characters, were very brief, basically a tease along with the survey blog posting URL containing more information and the actual survey link. Finally, there was some promotion and requests for participation on guild forums belonging to the allied guild members, but not on the official Blizzard World of Warcraft forums, Elitist Jerks, Joystiq, or other large WoW community forums. Most participants would therefore be recruited from a community of people who knew of the author. This was intentional to benefit from social capital gained already by being a guild leader and co-leader of the allied guild group, especially as participants were expected to engage in a non-trivial task.

The study was designed as the first of a series investigating factors contributing to players persisting in learning and working in massively multiple online games, like World of Warcraft. Solicitation for participation was deliberately low-key to make the analysis of discursive responses manageable. Themes derived from the discursive responses could then be used to design a larger scale survey in the future. In this study, I particularly wanted to start collecting data on the following six research questions from a combination of qualitative and quantitative questions:

1. What motivates people to play World of Warcraft?
2. What motivates people to persist in playing?
3. Is there a relationship between gender and stated motivations?
4. Is there a relationship between age and stated motivations?
5. Is there a relationship between nationality and stated motivations?
6. Is there a relationship between character roles and classes and motivation?

In keeping with the overarching theme of learning, I hoped to see some evidence of learning behaviour or practices, prompting the most important research question:

7. What, if anything, are people learning in World of Warcraft?

The question therefore remains: convenience sample, volunteer sample, theoretical sample, or a mixture? I originally thought it was a convenience sample, but now I do not feel confident in that at all. Oliver and Carr describe two of the couples in their theoretical sample as being convenience samples. Are mixtures “acceptable”? I am leaning now strongly towards labelling it a volunteer sample. What have I done? Help!

Sincerely,
Confused in London

References

Bryman, A. (2008) Social Research Methods. 3rd edition. Oxford, United Kingdom, Oxford University Press.

Cohen, L., Manion, L. & Morrison, K. (2007) ‘Chapter 4: Sampling’, in Research Methods in Education, 6th edition. Milton Park, United Kingdom, Routledge UK.

Hoyle, M.A. (2010) WoW Learning: A Study of Learning in World of Warcraft by Michelle A. Hoyle, [online]. (Accessed June 24, 2010).

Oliver, M. & Carr, D. (2009) ‘Learning in Virtual Worlds: Using Communities of Practice to Explain How People Learn From Play’, British Journal of Educational Technology, 40 (3), pp:444-457. Also available from: http://doi.wiley.com/10.1111/j.1467-8535.2009.00948.x (Accessed June 14, 2011).

Robson, C. (2002) Real World Research: A Resource for Social Scientists and Practitioners-Researchers. 2nd edition. Oxford, United Kingdom, Blackwell Publishing.

Image: Photograph of street near Elephant and Castle on a rainy day in London through rain-streaked window

I’m about halfway through my initial coding of the motivation essays collected last April.  I should have been done this months ago, but I’ve somehow been scared to do it.  I think the big reason behind that is I’m afraid that I’m doing it or will do it incorrectly.  As I am going through and creating codes, I cannot help but feel that I am not always focussing on the motivation issue, which is the primary question. I am generally coding for content or themes I see appearing in the essays.  As an example, an essay may express that the author is more likely to assist someone else if they feel that other person has put some effort and thought into their character.  That is not their motivation for playing, but I have still created a code for it as “assist others”.  When I get to the end and review the list, I will not be able to tell which ones refer to motivation.  Some probably are where a participant has expressed it as a motivation, but other instances, even of the same code, might just be a theme that was raised.

At the moment, I have the following free nodes in NVivo:

Feeling a little insecure, I thought it might be time to consult a book I bought late last year but had yet to open: The Coding Manual for Qualitative Researchers by Johnny Saldaña (2009).  While I have many books now on research methods and specifically on qualitative analysis, I have found it difficult to get a grasp on the mechanics of coding.  I am somewhat reassured to read in the first chapter that “Rarely will anyone get coding right the first time” (p.10).

Saldaña differentiates between themes and codes, based on work of Rossman & Rallis: “think of a category as a word or phrase describing some segment of your data that is explicit, whereas a theme is a phrase or sentence describing some more subtle and tacit processes.” (Saldaña 2009, p. 13, his emphasis).  He goes on to say that “SECURITY can be a code, but A FALSE SENSE OF SECURITY can be a theme.”   He recommends avoiding coding thematically initially and to instead note potential themes down in an analytic memo.

In examining my list, aren’t most of my existing codes themes rather than categories, even if they’re a single word?  Maybe not necessarily.  If an essay’s author says they play World of Warcraft as stress relief, “stress relief” is an explicit thing.  That’s a category?   I am still unsure.  For the moment, I think I will continue on as I am.  This is only the first iteration and I can always improve it later.  However, I think I should start explicit coding some passages as “motivation” to delineate it from other points of interest that may also arise within a given essay and then go back and do the same for essays prior to case S1-028.

I suspected I was deviating from the main goals of the survey while doing my coding.  Saldaña addresses this by supporting the recommendation of Auerbach & Silverstein to make a one-page  summary of your research concerns, central research question, theoretical framework, goals of the study, and any other major issues (Saldaña 2009, p.18).  Then, keep that in front of you to aid you in maintaining your focus during coding.  Some questions were suggested as being applicable to coding field notes for all research by Emerson, Fretz, & Shaw (quoted in Saldaña 2009, p. 18):

1. What are people doing?
2. How, exactly, do they do this?  What specific means and/or strategies do they use?
3. How do members talk about, characterize, and understand what is going on?
4. What assumptions are they making?
5. What do I see going on here?  What did I learn from these notes?
6. What did I include them?

I have trouble seeing the applicability of those questions to my current task.  I do, however, agree with Saldaña’s addition of “What strikes you?”, suggested by Creswell (Saldaña, 2009, p.18).  I suspect it is that question that helps save all my existing work from having been useless, even if I did forget the purpose behind the study at times.

One thing I know I have not done is be rigorous about the codebook or code list.  MacQueen (quoted in Saldaña 2009, p. 21) recommends that a codebook entry should contain “the code, a brief definition, a full definition, guidelines for when to use the code, guidelines for when not to use the code, and examples.”  As I have created codes, I usually have not done any of that, although the odd one here or there has a brief description.  I have a plan to go back and “clean up” the codes.  For example, some codes need to be merged, like “exploration” and “exploring”.  Perhaps I can review how the codes have been used and write up descriptions for them at that point as well.

At the moment, I feel very much like the person looking through a rain-streaked window: everything is distorted and unclear. If I persevere, the hope is eventually the rain will stop and the streaks will fade away.

References:

Saldaña, J. (2009) The Coding Manual for Qualitative Researchers, London, United Kingdom, Sage Publications Ltd.

Hermeneutics (English pronunciation: /hɜrməˈnjuːtɨks/) is the study of interpretation theory, and can be either the art of interpretation, or the theory and practice of interpretation. Traditional hermeneutics — which includes Biblical hermeneutics — refers to the study of the interpretation of written texts, especially texts in the areas of literature, religion and law. Contemporary, or modern, hermeneutics encompasses not only issues involving the written text, but everything in the interpretative process. This includes verbal and nonverbal forms of communication as well as prior aspects that affect communication, such as presuppositions, preunderstandings, the meaning and philosophy of language, and semiotics.[1] Philosophical hermeneutics refers primarily to Hans-Georg Gadamer’s theory of knowledge as developed in Truth and Method, and sometimes to Paul Ricoeur.[2] Hermeneutic consistency refers to analysis of texts for coherent explanation. A hermeneutic (singular) refers to one particular method or strand of interpretation.
Wikipedia (2010)

It’s apparently related to computational semiotics or used in computational semiotics.  That reminds me of James Paul Gee again because he talks about the semiotics of things in his What Video Games Have To Teach Us about Learning and Literacy (2007).  Is it another sign that I need to be looking at Gee’s book on discourse analysis (Gee 2011)?

References

Gee, J.P. (2007) What Video Games Have To Teach Us About Learning and Literacy, 2nd edition, New York, NY, United States, Palgrave Macmillan.

Gee, J.P. (2011) An Introduction to Discourse Analysis Theory and Method, 3rd edition, Abingdon, United Kingdom, Routledge.

Silverman, D. (2010) Doing Qualitative Research: A Practical Handbook, 3rd edition, London, United Kingdom, Sage Publications Ltd.

Wikipedia. (2010) Hermeneutics, [online] web page, Wikipedia. Available from: http://en.wikipedia.org/wiki/Hermeneutics (Accessed September 21, 2010).

Doing Qualitative Research: The Book

Chapter 2 of David Silverman’s Doing Qualitative Research:  A Practical Handbook (2010, p.16) asks students to consider why they believe a qualitative approach is appropriate for their possible research topics.  In fact, I had not initially considered a qualitative approach at all.  With my background in artificial intelligence, software engineering, and information retrieval, I was tending towards quantitative methodologies.  Information retrieval is very much about calculations and measurement, so that was a natural fit. Wikipedia (2010) describes the qualitative method as one that “investigates the why and how of decision making, not just what, where, when.”

Much of my survey data, like population demographics, is very amenable to quantitative methods to usefully describe the types of people and characters who participated in the first survey.  However, the core questions I was interested in were more what some people would call “touchy-feely” or how and why questions:

1. How do people describe the guilds they belong to.
2. What motivated people to play World of Warcraft initially.

While the first of those questions could be approached in a quantitative way by coding each 140-character response into one of a number of categories, I found that approach unsatisfying.  Even in such short responses, there was more nuance than I could easily accommodate in a simple, quantitative coding scheme.  For the second question, which I had not yet even attempted to analyze, I knew the number of game players saying the same thing was not the important part; the variety was important because I was interested in the underlying themes being expressed and, because I gave survey participants the space to write an essay, one or two categories was definitely not going to capture the detail.  Traditional quantitative analysis tools would not easily allow me to explore and group themes dynamically either, which is why I started investigating NVivo, a qualitative analysis tool.

So for this study, I am looking at mixed methods research.  I will be using quantitative analysis for the demographic details and qualitative analysis for analyzing the content of free-form responses.  The moral of the story, and one which David Silverman tries to get across right at the start, is that you need to choose your methods based on your data and what you want to discover.  Don’t be wed to a methodology just because it is familiar to you or even necessarily just because it has always been done that way.

Silverman, D. (2010) Doing Qualitative Research: A Practical Handbook, 3rd edition, London, United Kingdom, Sage Publications Ltd.

Wikipedia. (2010) Qualitative Research, [online] web page, Wikipedia. Available from: http://en.wikipedia.org/wiki/Qualitative_research (Accessed September 14, 2010).

Share/Save

Image: Elsheindra and Team Pink tackle the Dragonhawk Boss in Zul’Aman back in 2008. As a healer, Elsheindra has to make difficult decisions about who will live and who will die, in her role as main healer.  Being a researcher and maintaining anonymity is, I’ve discovered, a lot easier.

Back in April, I posted my first preliminary study to look at motivation, community formation, and learning in World of Warcraft.  When I was crafting my ethics approval for that study and future studies, I was very concerned with maintaining the privacy of the individuals participating.  The first survey was designed specifically to not require any personally identifiable information, although participants did have the option of giving an e-mail address if they wanted to participate in future studies or if they did not mind being contacted for any follow-up questions.

A problem arises, however, in following participants across multiple studies.  This is somewhat related to longitudinal studies where repeated observations are collected over long periods of time from the same participants.  The purpose of such studies is to help distinguish actual effects from short-term causes.  However, longitudinal studies aren’t the only time researchers may want to track participants across time and across multiple studies.  That would also be useful to help me build a more complex, detailed picture of participants, even though I intend to be asking different questions in different surveys.

While looking at other projects investigating World of Warcraft and motivation, I came across Nick Yee’s Daedalus Project, his old research project, and PlayOn, his new research project investigating social dimensions of virtual worlds.  I was quite surprised that, in at least one of his previous studies, he invited people to identify themselves by their e-mail addresses so that they could be tracked across his multiple studies.  Although I like Nick Yee’s work, I thought this approach was ethically incorrect.  The question is: how do you do it in a way that does not compromise the participants’ anonymity or their rights to privacy?

I got an answer to this recently from an unexpected place: the virtual common room of associate lecturers at The Open University where the topic was anonymous feedback from students being used potentially as a performance measurement mechanism.  Many of the lecturers felt that anonymous data collection wasn’t reliable.  Fellow IDEAs Lab alumna  Diane Brewster chipped in to say that a large quantity of research data is collected anonymously.  I got in touch with her via Twitter and she gave me the following tip: ask participants to identify themselves using a combination of specific letters from their month of birth and digits from their mobile telephone number.

Depending on the size of your participant pool, there might be some duplication.  However, if you choose your identifier tokens well, you can minimize that and still retain the desired anonymity.  Great tip, Diane.  Thanks a lot.  I will be putting this idea to use in my future survey work.

In the actual experiment, I gave Basil the following three essays:

Essay 1:
At first it was a way of keeping in touch with friends after I’d moved away. But I made more new friends throughthe online gaming community that occurs around the game. I’ve met a good number of my fellow guild members, including my guild leader and most of the other officers. To me, game has always been about exploring, storylines and the exotic locales presented therein. That’s all secondary to killing bosses, and trudging through raids really.

Essay 2:
I play WoW and other MMORPGS for the simple reason that I’m intrigued by the online community and game play aspects. WoW is my particular favourite that I return to again and again. I believe the reasoning behind this is the friendly community that has matured to quite a size over the number of years I’ve been playing. In addition to the community I find the story lines within the game interesting, challenging and sometimes, dare I say it, exciting. By exciting I mean, that like a good book, you want to see what is going to happen next!

Originally I started playing WoW for the simply reason it was an MMORPG. I was intrigued by the genre and WoW was really one of the first to be highlighted through the media, etc. As I progressed in the game, I discovered that it was a great way to relax after a busy day. As a form of escapism, it helped with relieving stress.

Now I rarely get to play WoW or any other MMORPG for that matter, however, for the same reasons of relaxation, online community, exciting stories, I still try to play as regularly as I can.

Essay 3:
Originally I moved to WoW simply because the majority of my guild had moved from DAoC, when WoW was released it was the next game that the existing guild members were collecting in. Ironically even though I followed my guild to the game I am actually motivated by the personal achievement.

I am the kind of player that likes to explore every location, complete every quest before moving on to the next zone and maximise trade skills. With each expansion, I spent most of the time solo’ing to the level cap, then exploring group content with my guild or raiding alliance.

With access to the raiding alliance I get to try challenging content which often require a level of skill and co-ordination. Currently I am motivated with the challenges of raiding with the aim to have completed as much as possible before the next content patch.

I know there is a sigma [sic] attached with gamers, but when you consider some people will return from work and just sit passively in front of a TV for 5hours. Similarly you see people sit all night on online chat channels. Given how some spend their time, how can spending your time problem solving and socialising with others with similar interests be so wrong.

Basil was asked to summarize the main ideas that occurred in each essay. Unfortunately, he was somewhat influenced by the question and noted down what people said their initial impetus for playing World of Warcraft was and then why they continue to play. I had to send him off to do it again. Table 1 illustrates our responses.

When I looked at essay 1, there was a question about things being “secondary to killing bosses , and trudging through raids…” Secondary implies that the other things were of lesser importance, but the negative tone implicit with words like “trudging” would seem to bely that, so I didn’t include the raiding. In talking to Basil, I know he had the same problem, because he asked me about it and I told him I would not give him an answer. As a result, he included raiding, whereas I did not.

On the whole, we don’t seem that different. If we had gone through the essays in advance together and agreed on some themes, I suspect the coding would have been similar. What do you think?

I know people are very curious about the results of my recent E-Mail in the Cloud: An Open University Survey. Time is a bit short for me, so I decided to write up this quick and dirty post outlining the key result. An analysis of the comments people left about why they made the choice they did will be covered in a later posting, as those comments proved to be extremely interesting.

In a more formal report, the order of detail presented would be different. I’ve started with the results first, as that’s likely to be of interest to most people, and then discussed the methodology, survey deployment, and motivation.

Table of Contents

• The Respondents
• Key Findings
• The Specifics
• Caveats
• Motivation
• Methodology
• Conclusions

The Respondents

533 people participated in the week-long survey. This is broken down visually in Figure 1. Of those:

• 71.1% declared themselves as students (379 people)
• 22.5% declared themselves as associate lecturers, academic conference moderators, or script markers (120 people)
• 3.4% declared themselves as permanent members of staff, either academic or support (18 people).
• 3.0% chose the “other” category (16 people).

Figure 1: Graph representing numbers and percentages of respondents, broken down by role

Of the 16 others, 7 were alumni. 3 others should probably have been in the AL category but politically considered themselves permanent members of staff. 3 were combinations of ALs/students, 1 was an AL/external contractor, 1 was a student but hoping to become an AL, and 1 claimed to belong to all three categories.

In this quick and dirty analysis, I have not assigned the “others” to appropriate existing categories, so their input is being omitted for the moment. I’ll leave that for a subsequent post.

Key Findings

1. Microsoft Live@edu is the preferred choice of very few people overall (11.63%)
2. A large number of people don’t know enough to make a choice between the two (36.21%)
3. An even larger number of all surveyed respondents (43.52%) would choose Google Apps Eduction Edition.
4. If a choice had to be made, Google Apps Education Edition was the most preferred by at least 40% of the respondents of a given role, with the exception of the 16 “Other” respondents.
5. If the “don’t care either way” respondents (46) are considered, Google Apps Education Edition would be the choice of 50.28% of all respondents and Microsoft Live@edu 20.26%.
6. If Microsoft Live@edu was chosen, it was by a student, far above any other respondent role (14.78% vs the next closest of 6.25%).

The Specifics

The following data table and graphic illustrates the specific choices of different respondents by role. If you’re examining Table 1 visually, bolded cells indicate that the majority of respondents in that row choose that option. For example, in the first row, which is Google Apps Education Edition, the cells for students, permanent staff, and response totals are all bolded, indicating those groups preferred Google Apps Education Edition over the other choices available.

Figure 2: Graph representing the preferences for a system by role.

Figure 2 shows a cylinder for each role in the survey. Each cylinder shows the percentage of respondents who chose Google Apps Education Edition, Microsoft Live@edu, don’t care either way, and don’t really know enough to make a choice with different colours. Google is red, Microsoft is blue, don’t know is yellow, and don’t care is green. While specific numbers aren’t shown on this graph, the total number of respondents in that category is indicated at the bottom, so you can either consult Table 1 for the number of respondents or do a quick calculation yourself.

Caveats

This was an unofficial survey that was designed and released on very short notice. Although I made a good effort to advertise it widely, the number of respondents is relatively low when compared with the Open University’s population of associate lecturers, permanent staff, and students.

While I specifically advertised in places where I knew Open University community members would see the information, I cannot guarantee that everyone who responded was associated with the Open University. I cannot see a reason why external people would participate, but I cannot preclude the possibility.

SurveyMonkey attempts to prevent the same person from completing the survey multiple times. However, that is based on the respondents’ IP addresses. Therefore, if a respondent changed location or has changing dynamically assigned IP addresses, it is possible they could have completed the survey more than once. This could have been avoided by collecting unique Open University identification information for each participant, but that would also have meant needing more stringent data handling and an increased reluctance to participate.

The rest of this post takes a step backwards and considers motivation, deployment, and survey design.

Motivation

According to David Wilson, director of strategic planning in LTS, a choice is being considered between Google Apps Education Edition and Microsoft Live@edu and should be made shortly (in Snowball 36 – November 2009). Students are definitely migrating. A decision is still being made about what to do with e-mail addresses for associate lecturers.

I thought it would be useful to survey interested parties about their preference if they had to choose between the two systems. I was especially interested in obtaining some indication of preference from students, who are guaranteed to be affected. The Business Steering Group, the group responsible for making the decision, will be meeting again soon and I will forward the findings of the survey to them for consideration.

Methodology

The survey itself was very simple, consisting of only three questions:

1. Which one of the following roles best describes your main role at the Open University? Your main role will be where you spend the majority of your time or where moving your existing FirstClass e-mail to the cloud will have the most impact.
2. Which cloud-based system would you prefer, if you had to choose one or the other? Choices are randomised.
3. I confirm that I am associated with the Open University as a student, associate lecturer, permanent staff, or in some other capacity.

The first question was intended to categorize the different respondents by their role at the university. It was recognized that some people have more than one role. They were asked to choose the one where the change would have the most impact. The role was then used to organize the results of the second question.

The second question is the heart of the survey. Respondents were give four choices:

• Google Apps Education Edition
• Microsoft Live@edu
• Don’t care either way
• Don’t really know enough to make a choice

The choices were randomized to avoid any suggestion of bias on the part of the survey giver.

There was also an opportunity to add some brief free-form comments on their choice. From comments in this section and comments received by e-mail, I know many people wanted the ability to say “Neither”. That was not a realistic choice given that one of the two systems will be adopted. That is also why it is worded as “if you had to choose…”

The third question was where the respondent agrees that they are associated with The Open University in some way. The survey is not very useful if it is completed by parties not affected by the outcome.

The survey was prefaced with some brief information about the motivation for the survey and how the survey results would be used. Respondents were also given two links from Google and two links from Microsoft on their respective products. Respondents were also given links to two articles from independent bloggers or education organizations reviewing the two products.

Respondents were assured that the survey was unofficial and no personal details, including computer IP addresses, were being recorded or stored with the survey. They were also assured that I would only be using the data for providing indicative preferences to the Open University and I had not sought or received permission from the Open University to conduct the survey. Contact details by e-mail or Twitter were included.

Survey Deployment

The survey questions were presented and answered electronically via the cloud-based SurveyMonkey poll service. The survey was open between Sunday, November 22nd, and Sunday, November 29th (23:59). Respondents were initially directed to the survey by one of three methods:

• A microblog entry on Twitter with a shortened URL leading to a blog post with a bit more background information on the survey and slightly expanded commentary on the survey than in the actual survey itself. I made several postings throughout the survey period, each time asking others to also pass the information on, which several people did.
• Postings in several FirstClass conferences consisting of a little background information about why I was doing the survey, how it would be used, and how to contact me. The posting included the URL for the a blog post as well as a direct link to the SurveyMonkey survey. The message asked readers to pass the message along to other interested parties, which resulted in it being posted to an unknown number of OUSA and course conferences. I personally made postings in the following FirstClass conferences:
  • MCT AL Discussion Forum
  • AL Common Room
  • Technology Cafe
  • Science Chat
  • Social sciences Cafe
  • R01 Arts Cafe
  • R03 Arts Cafe
  • OUSA Mac General
  • OUSA Open Access
  • OUSA Office Applications
  • OUSA Linux
  • OUSA London
  • OUSA Chat
  • OUSA Moderators
• A posting was made in the “Lounge” section of Platform, the Open University Community site. The posting was made the 25th of November and Platform claims “0 views”, but that seems to be an error as all threads have 0 views even when they have responses.

Conclusions

Even considering the various caveats in place, I think it is clear there is a strong preference for Google Apps Education Edition if people have to choose between one or the other. Examining the free-form comments, I know there is a belief from many people that e-mail should be kept in-house or that a choice of “none of the above” would have been preferred. Many people are concerned about keeping .open.ac.uk addresses for academic hardware and software purchases. Many people also expressed concern about security and data privacy issues with their e-mail being managed by either Google or Microsoft. I’ll examine these in more detail in a follow-up report.

Thank you to all those who took the time to respond and comment. I would also like to thank those people who reposted or re-tweeted the survey information. As promised, I will be passing this information along shortly to the Business Steering Group who is making the decision.

If you have any comments or questions, please feel free to leave a comment here, message me as @Eingang on Twitter, or e-mail me as mah383 on FirstClass server 2 (tutor.open.ac.uk).

[tweetthis]

Anal1a, in terms of clumping, doesn’t look very good, at least not anymore. That was not previously the case, but I had revised my algorithm to make it symmetrical as per the insructions of a computing statistician here at the University of Sussex. He claimed that the Procrustes Rotation needed symmetric data and my nonsymmetric data, where Doc1 vs Doc2 didn’t have the same similarity as Doc2 vs Doc1, was not going to work. That change has, I believe, altered the efficacy of the algorithm and things are no longer clumped together as promisingly as they were previously. The clumps should be a two- or three-letter short code followed by a digit. Therefore, ac1 and ac2 belong together. Pl1, pl2, and pl3 belong together, and so on. The clumping is significantly better in the already symmetric cosine normalization algorithm (anal2a). The two speech processing documents are clumped together (sp1 and sp2), all of the Power PC and G4 documents are together (pp1, pp2, g4c), and the three Pine Lake tornado stories are clumped far away from everything else (which is all computer-related) and together on their own. Excellent clumping, in fact. So the hypernym hypothesis looks like, on these short documents, it is working well with cosine normalization.

Here’s the final bit of loveliness: comparing one MDS cluster diagram against another. MDS output is mapped to the vector space independently. That is, the same data will produce the same visualization or mapping, but different data is mapped to a different vector space, so you cannot just compare one MDS matrix to another directly. That is where Procrustes Rotation comes in. It applies a series of intelligent matrix transformations, trying to map the second vector matrix onto the source vector matrix. As a side benefit, essential in my case, it always provides a fitness measure to tell you how close the two were. on a scale of 0 to 1. So these two, as you can see (see above image), even after the transformations, were not that close together. As it happens, though, this is not particularly useful information to know. I am currently more interested in assessing how close the two algorithms are to human classifiers.

This recent success gave us plenty to discuss, particularly with respect to metric and non-metric data. The MDS community calls source data metric when the similarity or dissimilarity data is symmetric. That is, the value at row 2, column 1 is the same as the value at row 1, column 2. Classical multi-dimensional scaling (MDS) is designed to only work with metric data. SPSS includes the ALSCAL and PROXSCAL MDS algorithms which can work with non-metric data, but MATLAB’s classical MDS does not because it treats things as Eucledean distances–another reason why I had to alter the Anal1a algorithm. The primary reason I now had metric data for everything, however, was because the computing statistician had told me I needed it for the Procrustes. Hawever, as we were examining my output, it occurred to me that Procrustes did not really care if the data was symmetric, so long as the dimensions of the data were the same (the same number of rows and columns). Which leads us to question whether the application of the method is statistically sensible or not. To that end, I need to track down a new computing statistician and perhaps a mathematician and discuss the process with them. My original computing statistician has retired.

Earlier I said that comparing one machine to another, to see how they fit is not useful information, but what would be interesting is to prepare a matrix of all the possible combinations of human judgements, cosine normalization, and weird formula:

cosine   wrd form.   human
cosine (anal2a)		x
weird formula (anal1a)           x
human                                        x

So that is my task for my next meeting (on the 16th of June). Before then, I need to figure out how to get MATLAB to take multiple tables as data. In SPSS, I could paste in several tables (representing all of the people’s individual data, for example) and it would work with that. That is necessary in order to aggregate the peopel to do the comparison. Onward ho, then! Progress at last!

Share/Save

Doc1	Doc2	Doc3	Doc4
Doc1	100	76	18	91
Doc2	76	100	22	35
Doc3	18	22	100	65
Doc4	91	34	65	100

This is not too dissimilar from the labelled diagram, part of the MATLAB documentation on data importing. Except that, if you look at the table below it, which describes which functions to use, they don’t have a function with a similar example to their labelled diagram. Early on I thought I should be able to use dlmread, which allows you specify rows/columns for starting points or a range. My idea was just to have a range which excluded the non-numeric troublesome labels. No matter what I did, though, I could not get it to work. It was frustrating, because I could paste the data into the Import Wizard and that could handle the data fine. I wrote people, I researched on the web, and I tried all sorts of things.

Eventually, I came full-circle back to dlmread and experimented by making a small data file with unrelated data in it. That worked fine. So I then copied half of one of my data tables into the test file and tried that. That also worked fine. I copied the whole data table into the test file and used dlmread on it. It worked fine! What was the difference between the two identical data files other than their filenames? When I uncovered the answer to that, I kicked myself. My data files were generated years ago and stored on my Mac OS 9-based laptop. My laptop and the data have since migrated to Apple’s swoopy BSD-based UNIX goodness and that’s the environment that MATLAB runs under. So… Have you guessed the problem? Yes, it was linefeeds! The data files had original Mac linefeeds and MATLAB wanted UNIX linefeeds. D’oh! It just goes to reaffirm that the things you don’t see can really hurt you.

Once that was solved, work proceded rapidly apace as I was now able to finish automating the whole comparison process from start to finish.

function  [Anal1Raw, Anal2Raw, Anal1MDS, Anal2MDS, fit] =
processEinCiteData(firstFile, secondFile, runName, labels)
% Read in the similarity matrices from the two data files
Anal1Raw = dlmread(firstFile, '\t', 1, 1);
Anal2Raw = dlmread(secondFile, '\t', 1, 1);
% Set up default document name labels if we didn't get any
if nargin < 4
labels = {'g4c', 'pp1', 'pp2', 'msc', 'pl1', 'pl2', 'pl3', 'sp1', 'sp2', 'ac1', 'ac2', 'bws'};
if nargin < 3
runName = '';
end
end
% Set up labels for the filenames
fileName1 = regexprep(firstFile, '\..*$', '');
fileName2 = regexprep(secondFile, '\..*$', '');
% Convert the similarity data to numbers below 1 for use in MDS
Anal1Raw = abs(100 - Anal1Raw)
Anal2Raw = abs(100 - Anal2Raw)
% Calculate the MDS and prepare a diagram showing the
% clusterings for the first document
[Anal1MDS, eigvals] = cmdscale(Anal1Raw);
figure(1);
plot(1:length(eigvals),eigvals,'bo-');
graph2d.constantline(0,'LineStyle',':','Color',[.7 .7 .7]);
axis([1,length(eigvals),min(eigvals),max(eigvals)*1.1]);
xlabel('Eigenvalue number');
ylabel('Eigenvalue');
plot(Anal1MDS(:,1),Anal1MDS(:,2),'bo', 'MarkerFaceColor', 'b', 'MarkerSize', 10);
axis(max(max(abs(Anal1MDS))) * [-1.1,1.1,-1.1,1.1]); axis('square');
text(Anal1MDS(:,1)+1.5,Anal1MDS(:,2),labels,'HorizontalAlignment','left');
hx = graph2d.constantline(0,'LineStyle','-','Color',[.7 .7 .7]);
hx = changedependvar(hx,'x');
hy = graph2d.constantline(0,'LineStyle','-','Color',[.7 .7 .7]);
hy = changedependvar(hy,'y');
title(['\fontname{lucida}\fontsize{18}' fileName1 ' MDS']);
xlabel(['\fontname{lucida}\fontsize{14}' runName ' on ' date], 'FontWeight', 'bold');
% Calculate the MDS and prepare a diagram showing the
% clusterings for the second document
[Anal2MDS, eigvals] = cmdscale(Anal2Raw);
figure(2);
plot(1:length(eigvals),eigvals,'rd-');
graph2d.constantline(0,'LineStyle',':','Color',[.7 .7 .7]);
axis([1,length(eigvals),min(eigvals),max(eigvals)*1.1]);
xlabel('Eigenvalue number');
ylabel('Eigenvalue');
plot(Anal2MDS(:,1),Anal2MDS(:,2),'rd', 'MarkerFaceColor', 'r', 'MarkerSize', 10);
axis(max(max(abs(Anal2MDS))) * [-1.1,1.1,-1.1,1.1]); axis('square');
text(Anal2MDS(:,1)+1.5,Anal2MDS(:,2),labels,'HorizontalAlignment','left');
hx = graph2d.constantline(0,'LineStyle','-','Color',[.7 .7 .7]);
hx = changedependvar(hx,'x');
hy = graph2d.constantline(0,'LineStyle','-','Color',[.7 .7 .7]);
hy = changedependvar(hy,'y');
title(['\fontname{lucida}\fontsize{18}' fileName2 ' MDS']);
xlabel(['\fontname{lucida}\fontsize{14}' runName ' on ' date], 'FontWeight', 'bold');
% Apply Procrustes to the two MDS results to map them
% into the same vector space and prepare a plot of the
% result
[fit, Z, transform] = procrustes(Anal1MDS, Anal2MDS);
figure(3);
plot(Anal1MDS(:,1), Anal1MDS(:,2), 'bo','MarkerFaceColor', 'b', 'MarkerSize', 10);
hold on
plot(Z(:,1), Z(:,2), 'rd', 'MarkerFaceColor', 'r', 'MarkerSize', 10);
hold off
text(Anal1MDS(:,1)+1.5,Anal1MDS(:,2), labels, 'Color', 'b');
text(Z(:,1)+1.5,Z(:,2),labels, 'Color', 'r');
xlabel(['\fontname{lucida}\fontsize{14}' runName ' on ' date], 'FontWeight', 'bold');
ylabel(['\fontname{lucida}\fontsize{14}' 'fit = ' num2str(fit, '%2.4f')], 'FontWeight', 'bold');
titleStr = ['\fontname{lucida}\fontsize{18}' fileName1 ...
' compared to ' fileName2];
title(titleStr, 'HorizontalAlignment', 'center', ...
'VerticalAlignment', 'bottom');
legend({firstFile, secondFile}, 4);

At the end, I had a quantitative number, the degree of fit, between two diagrams after applying the Procrustes Rotation to them. Finally! On a whim, I fed in the same data table as both arguments to my comparison program. That is, I compared the same data file to itself. My hypothesis was that the resultant degree of fit should be either 0 or 1 (depending on which the fitness was measured). Much to my surprise, no matter which data file I used, the result was never 0 or 1. My previous Procrustes Analysis code was taken from some sample code in the MATLAB documentation and looked like: [D,Z] = procrustes(Anal1aMDS, Anal2aMDS(:,1:2)); That last bit in () is some kind of MATLAB scaling, which, being a novice to MATLAB, I didn’t realize. So, in fact, my two diagrams weren’t the same which is why I wasn’t getting a 100% degree of fit. I do not want to say how long it took me to narrow that down. Once I did, though, it looked like I was basically set and I was able to quickly produce some comparisons between my “weird” half-baked metric and the cosine normalization one. One small step for EinKind.

This is a delayed entry from May 12th, 2004.

Share/Save