Image: Marbles, many marbles. I think I have lost mine in a sample of many marbles.

I’ve been re-running analyses today on my population of survey responses. I decided to remove some more responses to eliminate some the scatteredness in the population. The majority of responses were from European PvE (player versus the environment) realm players, so I removed the four American realm players and then the five non-PvE players, leaving me with a sample of 30.

The more I read about sampling, the more confused I am.

When we read the Oliver and Carr (2009) WoW communities of practice and learning paper the other day, we were somewhat dismissive because it only had five couples. However, the paper also mentioned that it used “theoretical sampling”, which we had not heard of. Someone looked it up quickly on Wikipedia and it sounded like you chose your sample based on it generating the features you wanted to look at. Now the description in the Oliver and Carr paper sounds more like “purposive sampling”, because they described their sampling in a way that seemed to fit with Cohen et al’s description of “…qualitative researchers handpick the cases to be included in the sample on the basis of their judgement of their typicality or possession of the particular characteristics being sought” (2007, p. 114-115):

Players were recruited through online guilds and real-world social networks. The first two sets of participants were sampled for convenience (two heterosexual couples); the rest were invited to participate in order to broaden this sample (one couple was chosen because they shared a single account, one where a partner had chosen to stop playing and one mother–son pairing).
Oliver and Carr (2009, p. 446).

I was browsing through Research Methods in Education today and it specifically mentions theoretical sampling as a feature of grounded theory and the sample size is immaterial. The important part is that you have enough data to saturate the categories in your theory. You collect more and more data until the acquisition of more data does not advance or modify the theory developed. It suggests that the size of the data set may be fixed by the number of people to whom one has access but you have to consider that it may be necessary to seek further data (Cohen et al. 2007, 116-117). A sample of five couples would then possibly be acceptable. Although I am taking a grounded theory approach, this does not feel quite like what I am doing, although I do have the intention of generating the theory from the data I have and then pursuing a larger-scale study later.

Another possibility is volunteer sampling. This is apparently different than convenience sampling. I suppose in a convenience sample, you have more control over how many people respond, e.g. a class of students, and you are directly asking them. In volunteer sampling, you rely on volunteers, like personal friends or friends of friends, although it can also be via, for example, a newspaper advertisement (Cohen et al. 2007, p. 116). This sounds similar to the approach that I took. I already knew I had to be careful about making generalizations and certainly the representativeness of the sample is lacking. This is probably acceptable, provided the lack of typicality is made clear.

Real World Research describes a convenience sample as one of the most widely used (Robson 2002, p. 265). Sensible uses of convenience samples, Robson suggests, are for piloting a proper sample survey or getting a feeling for the issues involved. This too feels like what I was doing, since I designed the study originally to be the starting point for a future, larger study. Providing a springboard for future research is also described as being acceptable by Bryman (2008, p.183) in Social Research Methods.

My section describing the survey distribution currently reads as follows:

A blog site was created for the overall project and readers invited to participate (Hoyle 2010) through an initial posting. Readers were given a brief explanation of the survey’s purpose, contact details for the author, and an explanation of the rules and time and effort expected. The page explained that there would be an opportunity to enter an optional draw to win a virtual in-game pet as a reward. This page also contained a link to the survey, hosted on SurveyMonkey, a third-party commercial web survey site.

At a minimum, 25 to 30 participants fully completing the survey were required and more than 50 to 75 would be burdensome. Advertising was therefore not  ambitious or comprehensive. Short messages were broadcast periodically on a European (player versus environment) game realm to a text communication channel shared by members of five allied guilds. A month before the survey, allied guild leaders were questioned about their current membership numbers. This information is available in the game and reflects the number of individual accounts that belong to a given guild. Total number of player accounts was 437. That count includes inactive players and players belonging to more than one allied guild. It is also possible for players to have more than one account, if they are willing to pay for it, resulting in the same person being counted more than once. However, after discussion with the guild leaders, the number of people with multiple accounts or multi-guild membership was believed to be small; the number of people reported is therefore probably fairly close. However, it is difficult to estimate what proportion would be active players or would have seen the periodic messages.

In addition to the in-game messages, the study was also advertised numerous times via the author’s main Twitter account and an account dedicated to news for the allied guilds. This resulted in a number of rebroadcasts as other researchers and followers tried to assist by passing along the message. Twitter messages, by their nature limited to 140 characters, were very brief, basically a tease along with the survey blog posting URL containing more information and the actual survey link. Finally, there was some promotion and requests for participation on guild forums belonging to the allied guild members, but not on the official Blizzard World of Warcraft forums, Elitist Jerks, Joystiq, or other large WoW community forums. Most participants would therefore be recruited from a community of people who knew of the author. This was intentional to benefit from social capital gained already by being a guild leader and co-leader of the allied guild group, especially as participants were expected to engage in a non-trivial task.

The study was designed as the first of a series investigating factors contributing to players persisting in learning and working in massively multiple online games, like World of Warcraft. Solicitation for participation was deliberately low-key to make the analysis of discursive responses manageable. Themes derived from the discursive responses could then be used to design a larger scale survey in the future. In this study, I particularly wanted to start collecting data on the following six research questions from a combination of qualitative and quantitative questions:

1. What motivates people to play World of Warcraft?
2. What motivates people to persist in playing?
3. Is there a relationship between gender and stated motivations?
4. Is there a relationship between age and stated motivations?
5. Is there a relationship between nationality and stated motivations?
6. Is there a relationship between character roles and classes and motivation?

In keeping with the overarching theme of learning, I hoped to see some evidence of learning behaviour or practices, prompting the most important research question:

7. What, if anything, are people learning in World of Warcraft?

The question therefore remains: convenience sample, volunteer sample, theoretical sample, or a mixture? I originally thought it was a convenience sample, but now I do not feel confident in that at all. Oliver and Carr describe two of the couples in their theoretical sample as being convenience samples. Are mixtures “acceptable”? I am leaning now strongly towards labelling it a volunteer sample. What have I done? Help!

Sincerely,
Confused in London

References

Bryman, A. (2008) Social Research Methods. 3rd edition. Oxford, United Kingdom, Oxford University Press.

Cohen, L., Manion, L. & Morrison, K. (2007) ‘Chapter 4: Sampling’, in Research Methods in Education, 6th edition. Milton Park, United Kingdom, Routledge UK.

Hoyle, M.A. (2010) WoW Learning: A Study of Learning in World of Warcraft by Michelle A. Hoyle, [online]. (Accessed June 24, 2010).

Oliver, M. & Carr, D. (2009) ‘Learning in Virtual Worlds: Using Communities of Practice to Explain How People Learn From Play’, British Journal of Educational Technology, 40 (3), pp:444-457. Also available from: http://doi.wiley.com/10.1111/j.1467-8535.2009.00948.x (Accessed June 14, 2011).

Robson, C. (2002) Real World Research: A Resource for Social Scientists and Practitioners-Researchers. 2nd edition. Oxford, United Kingdom, Blackwell Publishing.

Image: Photograph of street near Elephant and Castle on a rainy day in London through rain-streaked window

I’m about halfway through my initial coding of the motivation essays collected last April.  I should have been done this months ago, but I’ve somehow been scared to do it.  I think the big reason behind that is I’m afraid that I’m doing it or will do it incorrectly.  As I am going through and creating codes, I cannot help but feel that I am not always focussing on the motivation issue, which is the primary question. I am generally coding for content or themes I see appearing in the essays.  As an example, an essay may express that the author is more likely to assist someone else if they feel that other person has put some effort and thought into their character.  That is not their motivation for playing, but I have still created a code for it as “assist others”.  When I get to the end and review the list, I will not be able to tell which ones refer to motivation.  Some probably are where a participant has expressed it as a motivation, but other instances, even of the same code, might just be a theme that was raised.

At the moment, I have the following free nodes in NVivo:

Feeling a little insecure, I thought it might be time to consult a book I bought late last year but had yet to open: The Coding Manual for Qualitative Researchers by Johnny Saldaña (2009).  While I have many books now on research methods and specifically on qualitative analysis, I have found it difficult to get a grasp on the mechanics of coding.  I am somewhat reassured to read in the first chapter that “Rarely will anyone get coding right the first time” (p.10).

Saldaña differentiates between themes and codes, based on work of Rossman & Rallis: “think of a category as a word or phrase describing some segment of your data that is explicit, whereas a theme is a phrase or sentence describing some more subtle and tacit processes.” (Saldaña 2009, p. 13, his emphasis).  He goes on to say that “SECURITY can be a code, but A FALSE SENSE OF SECURITY can be a theme.”   He recommends avoiding coding thematically initially and to instead note potential themes down in an analytic memo.

In examining my list, aren’t most of my existing codes themes rather than categories, even if they’re a single word?  Maybe not necessarily.  If an essay’s author says they play World of Warcraft as stress relief, “stress relief” is an explicit thing.  That’s a category?   I am still unsure.  For the moment, I think I will continue on as I am.  This is only the first iteration and I can always improve it later.  However, I think I should start explicit coding some passages as “motivation” to delineate it from other points of interest that may also arise within a given essay and then go back and do the same for essays prior to case S1-028.

I suspected I was deviating from the main goals of the survey while doing my coding.  Saldaña addresses this by supporting the recommendation of Auerbach & Silverstein to make a one-page  summary of your research concerns, central research question, theoretical framework, goals of the study, and any other major issues (Saldaña 2009, p.18).  Then, keep that in front of you to aid you in maintaining your focus during coding.  Some questions were suggested as being applicable to coding field notes for all research by Emerson, Fretz, & Shaw (quoted in Saldaña 2009, p. 18):

1. What are people doing?
2. How, exactly, do they do this?  What specific means and/or strategies do they use?
3. How do members talk about, characterize, and understand what is going on?
4. What assumptions are they making?
5. What do I see going on here?  What did I learn from these notes?
6. What did I include them?

I have trouble seeing the applicability of those questions to my current task.  I do, however, agree with Saldaña’s addition of “What strikes you?”, suggested by Creswell (Saldaña, 2009, p.18).  I suspect it is that question that helps save all my existing work from having been useless, even if I did forget the purpose behind the study at times.

One thing I know I have not done is be rigorous about the codebook or code list.  MacQueen (quoted in Saldaña 2009, p. 21) recommends that a codebook entry should contain “the code, a brief definition, a full definition, guidelines for when to use the code, guidelines for when not to use the code, and examples.”  As I have created codes, I usually have not done any of that, although the odd one here or there has a brief description.  I have a plan to go back and “clean up” the codes.  For example, some codes need to be merged, like “exploration” and “exploring”.  Perhaps I can review how the codes have been used and write up descriptions for them at that point as well.

At the moment, I feel very much like the person looking through a rain-streaked window: everything is distorted and unclear. If I persevere, the hope is eventually the rain will stop and the streaks will fade away.

References:

Saldaña, J. (2009) The Coding Manual for Qualitative Researchers, London, United Kingdom, Sage Publications Ltd.

Doing Qualitative Research: The Book

I was reading Chapter 3 of David Silverberg’s Doing Qualitative Research: A Practical Handbook (Silverman, 2010 p.17-42) in September.  In it he gives three research diaries of Ph.D. students he had, detailing how they went from the start of their research projects, through methodology choice, and then through to data analysis.  While it was quite striking how coherent and “painless” the stories were, the more relevant realization I took away from it was the importance of having a framework around which to direct your research and to make sense of your data.  In two of the cases, the students used conversational analysis, a ethnomethodological approach.

Conversation analysis (commonly abbreviated as CA) is the study of talk in interaction (both verbal and non-verbal in situations of everyday life). CA generally attempts to describe the orderliness, structure and sequential patterns of interaction, whether institutional (in school, a doctor’s surgery, court or elsewhere) or in casual conversation.
Wikipedia (2010a)

In my case, conversational analysis wouldn’t be appropriate, because it requires audio or video data of conversations and then a detailed transcription is made.  The inductive data-driven analysis part to explain discovered patterns is a process I want to emulate, so I need a similar approach.  That reminded me that some of James Paul Gee’s work used discourse analysis (or what I think is discourse analysis) and that might be an appropriate framework for me to use to structure my own research.

Unfortunately, I think standard discourse analysis isn’t quite correct.  According to Wikipedia (2010b):

Discourse analysis (DA), or discourse studies, is a general term for a number of approaches to analyzing written, spoken, signed language use or any significant semiotic event.

The objects of discourse analysis—discourse, writing, talk, conversation, communicative event, etc.—are variously defined in terms of coherent sequences of sentences, propositions, speech acts or turns-at-talk. Contrary to much of traditional linguistics, discourse analysts not only study language use ‘beyond the sentence boundary’, but also prefer to analyze ‘naturally occurring’ language use, and not invented examples. This is known as corpus linguistics; text linguistics is related.

James Paul Gee’s use of discourse in social linguistics differentiates between “Discourse” (big Discourse) and “discourse” (little discourse).  Discourse (big) is a combination of the language and its contextual environment, such as behaviour, values, perspectives, etc., so it’s a form of situated language analysis.  He also looks at Discourse communities as part of that context.  A seminal work in discourse analysis is Gee’s An Introduction to Discourse Analysis: Theory and Method (1999).  I wonder if I should acquire this?  I note a second edition was released in 2005.  I’ve put in a request to the OU library to have their copy recalled and mailed to me.

References

Silverman, D. (2010) Doing Qualitative Research: A Practical Handbook, 3rd edition, London, United Kingdom, Sage Publications Ltd.

Wikipedia. (2010a) Conversation Analysis, [online] web page, Wikipedia. Available from: http://en.wikipedia.org/wiki/Conversation_analysis (Accessed September 15, 2010).

Wikipedia. (2010b) Discourse Analysis, [online] web page, Wikipedia. Available from: http://en.wikipedia.org/wiki/Discourse (Accessed September 15, 2010).

Share/Save

Hermeneutics (English pronunciation: /hɜrməˈnjuːtɨks/) is the study of interpretation theory, and can be either the art of interpretation, or the theory and practice of interpretation. Traditional hermeneutics — which includes Biblical hermeneutics — refers to the study of the interpretation of written texts, especially texts in the areas of literature, religion and law. Contemporary, or modern, hermeneutics encompasses not only issues involving the written text, but everything in the interpretative process. This includes verbal and nonverbal forms of communication as well as prior aspects that affect communication, such as presuppositions, preunderstandings, the meaning and philosophy of language, and semiotics.[1] Philosophical hermeneutics refers primarily to Hans-Georg Gadamer’s theory of knowledge as developed in Truth and Method, and sometimes to Paul Ricoeur.[2] Hermeneutic consistency refers to analysis of texts for coherent explanation. A hermeneutic (singular) refers to one particular method or strand of interpretation.
Wikipedia (2010)

It’s apparently related to computational semiotics or used in computational semiotics.  That reminds me of James Paul Gee again because he talks about the semiotics of things in his What Video Games Have To Teach Us about Learning and Literacy (2007).  Is it another sign that I need to be looking at Gee’s book on discourse analysis (Gee 2011)?

References

Gee, J.P. (2007) What Video Games Have To Teach Us About Learning and Literacy, 2nd edition, New York, NY, United States, Palgrave Macmillan.

Gee, J.P. (2011) An Introduction to Discourse Analysis Theory and Method, 3rd edition, Abingdon, United Kingdom, Routledge.

Silverman, D. (2010) Doing Qualitative Research: A Practical Handbook, 3rd edition, London, United Kingdom, Sage Publications Ltd.

Wikipedia. (2010) Hermeneutics, [online] web page, Wikipedia. Available from: http://en.wikipedia.org/wiki/Hermeneutics (Accessed September 21, 2010).

Doing Qualitative Research: The Book

Chapter 2 of David Silverman’s Doing Qualitative Research:  A Practical Handbook (2010, p.16) asks students to consider why they believe a qualitative approach is appropriate for their possible research topics.  In fact, I had not initially considered a qualitative approach at all.  With my background in artificial intelligence, software engineering, and information retrieval, I was tending towards quantitative methodologies.  Information retrieval is very much about calculations and measurement, so that was a natural fit. Wikipedia (2010) describes the qualitative method as one that “investigates the why and how of decision making, not just what, where, when.”

Much of my survey data, like population demographics, is very amenable to quantitative methods to usefully describe the types of people and characters who participated in the first survey.  However, the core questions I was interested in were more what some people would call “touchy-feely” or how and why questions:

1. How do people describe the guilds they belong to.
2. What motivated people to play World of Warcraft initially.

While the first of those questions could be approached in a quantitative way by coding each 140-character response into one of a number of categories, I found that approach unsatisfying.  Even in such short responses, there was more nuance than I could easily accommodate in a simple, quantitative coding scheme.  For the second question, which I had not yet even attempted to analyze, I knew the number of game players saying the same thing was not the important part; the variety was important because I was interested in the underlying themes being expressed and, because I gave survey participants the space to write an essay, one or two categories was definitely not going to capture the detail.  Traditional quantitative analysis tools would not easily allow me to explore and group themes dynamically either, which is why I started investigating NVivo, a qualitative analysis tool.

So for this study, I am looking at mixed methods research.  I will be using quantitative analysis for the demographic details and qualitative analysis for analyzing the content of free-form responses.  The moral of the story, and one which David Silverman tries to get across right at the start, is that you need to choose your methods based on your data and what you want to discover.  Don’t be wed to a methodology just because it is familiar to you or even necessarily just because it has always been done that way.

Silverman, D. (2010) Doing Qualitative Research: A Practical Handbook, 3rd edition, London, United Kingdom, Sage Publications Ltd.

Wikipedia. (2010) Qualitative Research, [online] web page, Wikipedia. Available from: http://en.wikipedia.org/wiki/Qualitative_research (Accessed September 14, 2010).

Share/Save

In the actual experiment, I gave Basil the following three essays:

Essay 1:
At first it was a way of keeping in touch with friends after I’d moved away. But I made more new friends throughthe online gaming community that occurs around the game. I’ve met a good number of my fellow guild members, including my guild leader and most of the other officers. To me, game has always been about exploring, storylines and the exotic locales presented therein. That’s all secondary to killing bosses, and trudging through raids really.

Essay 2:
I play WoW and other MMORPGS for the simple reason that I’m intrigued by the online community and game play aspects. WoW is my particular favourite that I return to again and again. I believe the reasoning behind this is the friendly community that has matured to quite a size over the number of years I’ve been playing. In addition to the community I find the story lines within the game interesting, challenging and sometimes, dare I say it, exciting. By exciting I mean, that like a good book, you want to see what is going to happen next!

Originally I started playing WoW for the simply reason it was an MMORPG. I was intrigued by the genre and WoW was really one of the first to be highlighted through the media, etc. As I progressed in the game, I discovered that it was a great way to relax after a busy day. As a form of escapism, it helped with relieving stress.

Now I rarely get to play WoW or any other MMORPG for that matter, however, for the same reasons of relaxation, online community, exciting stories, I still try to play as regularly as I can.

Essay 3:
Originally I moved to WoW simply because the majority of my guild had moved from DAoC, when WoW was released it was the next game that the existing guild members were collecting in. Ironically even though I followed my guild to the game I am actually motivated by the personal achievement.

I am the kind of player that likes to explore every location, complete every quest before moving on to the next zone and maximise trade skills. With each expansion, I spent most of the time solo’ing to the level cap, then exploring group content with my guild or raiding alliance.

With access to the raiding alliance I get to try challenging content which often require a level of skill and co-ordination. Currently I am motivated with the challenges of raiding with the aim to have completed as much as possible before the next content patch.

I know there is a sigma [sic] attached with gamers, but when you consider some people will return from work and just sit passively in front of a TV for 5hours. Similarly you see people sit all night on online chat channels. Given how some spend their time, how can spending your time problem solving and socialising with others with similar interests be so wrong.

Basil was asked to summarize the main ideas that occurred in each essay. Unfortunately, he was somewhat influenced by the question and noted down what people said their initial impetus for playing World of Warcraft was and then why they continue to play. I had to send him off to do it again. Table 1 illustrates our responses.

When I looked at essay 1, there was a question about things being “secondary to killing bosses , and trudging through raids…” Secondary implies that the other things were of lesser importance, but the negative tone implicit with words like “trudging” would seem to bely that, so I didn’t include the raiding. In talking to Basil, I know he had the same problem, because he asked me about it and I told him I would not give him an answer. As a result, he included raiding, whereas I did not.

On the whole, we don’t seem that different. If we had gone through the essays in advance together and agreed on some themes, I suspect the coding would have been similar. What do you think?