The Methodological Artist

Checklist questionnaire when revising a research assignment/project

7/27/2022

Designing research implies a lot of steps, and when comes the time to submit our project and we are stressed out and tired, one can easily find themselves overwhelmed with the number of things there is left to check before pressing the submission button.
Below I compiled a list of questions I encourage you to go through to self-assess the research design dimensions of your manuscript and rectify trajectory if necessary.
Obviously, no such checklist can be exhaustive nor can it match the specific requirements of the many types of format of research projects out there. This is a starting point with some common essentials and it is your responsibility to identify the more specific questions to cover all angles of your research design.
Good luck on pushing your project over the finishing line 😎 🏁 😎 🏁

Introduction

Is your research problem clearly articulated? (for more information about identifying a relevant research topic see how to identify a research topic)
Is your review of the literature structured and well organised?
Is your literature review concise or are there elements not directly supporting your demonstration and that you could therefore delete?
Is there a clear articulation of the gaps in/limits of the literature?
Are you contextualising your research problem so the reader has enough information to understand the stakes behind the issue and the rest of the project?

Research question

Do you have one strong research question directly connected to your introduction and that is answerable thanks to the data your will analyse?
Does your research question naturally follow the introduction for the reader? Would a reader be able to guess your research question just by reading your introduction?
If you have sub-questions, are they explicitly connected to the main research question?

For more information about research questions, read how to construct and formulate research questions.

Analytical framework

Are you using concepts/theories to go beyond describing the context and results? (check my blog post "what is analysis?" if you need more information regarding the difference between description and analysis)
Are the concepts/analytical framework you use clearly defined?
Are the concepts/analytical framework you use in clear alignment with the introduction as a whole?
Are the concepts/analytical framework you use only mentioned in the introduction or do you use them throughout your assignment including in your analysis?

Research Design

Do you justify your case selection? (thinking in terms of case goes beyond country-case, e.g. ableism can be thought as a case discriminatory practices and stereotypes)
If there are different dimensions within your case (e.g. discourses about public policy produced by young men working in the transport sector in London between 1990 and 1995) are all the elements of your case justified (or, for example, are you missing some dimensions by just justifying why "young men" and not why London, why this time period...)?
Do you justify the choice of your overall methodology/research design?
Do you discuss the steps of your analysis vis-à-vis evidence collected?
If your project is a multi-method research design, do you make explicit how you will use and assemble together the different sources/dimensions of the project?

Method of Data Collection

Do you justify why the method of data collection you use is the most adapted for your research project?
Do your justify the validity (and limits) of your data/sources?
Do you justify your sampling strategy?
Do you justify the criteria of inclusion and exclusion of your corpus (if you analyse text)?
Do you provide elements of contextualisation about your data/dataset/corpus so the reader can understand their value based on their role and context of production?
Is the type of material you analyse aligned with the different dimensions of your case, the literature review, the research question and the method?

Method of data Analysis

Did you check that you are not announcing you are doing a method but then using another method in the analysis? Did you check in some handbooks that you are not mixing up different methods without acknowledging it? (more common problem than you would think)
Is the method of analysis chosen aligned with your concepts/analytical framework?
Is the method of analysis chosen aligned with the type of sources/corpus you analyse?
Are you making explicit the trade-off of using this method (and not others)?
Are you using methodological literature to support your methodological choices?
Do you make explicit what type of evidence you will be looking for to answer your research question?
Do you make explicit the choices behind the construction of your analysis specific to the method you choose (for example the choice of the discourse analysis tools you are going to use, their definition and how they enable you to answer your research question)?

Results

Is your analysis structured (in paragraphs and using headings) or is it just a list of findings?
Is each argument you put forward supported by data?
Are you using concepts and literature to support your arguments?
Are you balancing the evidence between different sections of your argumentation?
Read the analysis as if you were a random reader, would you be convinced by the demonstration? Are there some missing elements?
Are all the elements of good analysis (check the blog about “the wheel of analysis”) present in your work or do you need to strengthen some dimensions?
At the end of your analysis, are you summarising in a clear, concise and straightforward way your results?
At the end of your result section (it could be in your conclusion), are you directly answering your research question in a clear straightforward way, like a one-sentence answer to a one-sentence question?
At the end of your result section (it could be in your conclusion), are you summarising in a clear, concise and straightforward paragraph your contributions to the literature you mentioned in the introduction?

Reflexivity, Ethics, and Limits of your project

Are you discussing the limits of your work and the steps that could be taken to address these limits?
Do you provide elements of reflexivity about how your position/trajectory/socialisation might have influence the construction/analysis in your project and the actions you took based on these reflexive insights?
Do you highlight the ethical dimensions of your project and how you address them?

0 Comments

Coding Qualitative Data - Am doing it the right way?

11/17/2020

0 Comments

asking questions for coding qualitative data

Whether you are a teacher teaching qualitative research methods or a student learning methodology, this blog post aims to help you answer the questions people learning how to code qualitative data always ask:

‘Am I coding the right way?’
‘I start by doing X and then doing Y, is that the right order?’
‘How do I know if I am coding right?’...

time to ask questions about coding qualitative data

When is it a good time to read this blog? (i.e. where does it fit in your learning curve?) = You have learned the basics of coding and you have started coding your material. You are in the process of reading your material over and over again. You have created some codes just to have to rename them the following day. Some of your themes start feeling like codes and some of your codes end up looking like themes. You wonder why you are so slow, what you must have missed, and expect others to be much more ‘efficient’, 'good' and 'quick' than you at doing all this. NOW is the time to read this post!

Let me start by telling you something useful right away: a lot of things you think you are doing wrong is just the way the method works! In this blog post, I will give you some keys to understand and explain how the coding process happens in practice. By the end of this post, I hope you will be able to substitute the questions above (that often fuel your frustration and take down our self-esteem) with more constructive questions that will boost the quality of your analysis and help you make the most out of your data.

But before going any further, let's define a few important words to make sure we understand each other:

Coding: “how you define what the data you are analysing are about” (Gibbs, 2007). Coding is often a key task of the nitty-gritty process of conducting qualitative text analysis. By coding, I mean the process of assigning codes that will represent stepping stones that you will eventually use as evidence to answer your research question. In this blog post, we will not expand on the basics of coding qualitative data (there are plenty of resources about this already).
Code: ‘is most often a word or a short phrase that symbolically assigns a summative, salient, essence capturing, and/or evocative attribute for a portion of language-based or visual data’ (Saldana, 2009). Codes are analytically useful terms or short sentences that we assign to certain segments of text to synthesise their meaning.
Theme: More abstract and synthetic than a code, a label or phrase which identifies an underlying meaning common between codes.
Analysis: A creative process that aims to produce a convincing demonstration based on empirical evidence describing, explaining and interpreting a social phenomenon (for more information check the blog post I wrote on this topic: What is analysis?).

What does coding look like in practice?

To set our expectations regarding coding as a research practice, we need to understand what happens when we code:

Coding is a pattern-finding activity;
Coding is an iterative process.

Coding as a pattern finding activity

coding qualitative data is a pattern finding activity

credit: Dr Eleanor Knott

Coding involves:
• An unconscious comparative process through which we identify elements of data that share similarities and that we distinguish from other elements of data;
• An unconscious relational process through which we analytically identify and name what connect these elements together.

You need to engage this double process on a certain amount of data to enable patterns to appear within your data. To put it differently, this means:

connecting what fits together and what doesn't;
expanding your imagination about what could have shown up in your data (and might somewhere) and what this interplay of absence/presence means in context.

Coding as an iterative process
Coding is not a linear process with clear cut steps that you take one after the other. Coding is an iterative process. By iterative, I mean a process for arriving at a decision or a desired result by repeating rounds of analysis or cycles of operations. The objective is to get closer to the desired results with each repetition, or rather, with each iteration.

To give you an example to illustrate what 'iteration' means, we can turn to how the word iteration is used in computer programming. In computer programming, iteration represents a process wherein a set of instructions or structures are repeated in a sequence a specified number of times or until a condition is met. When the first set of instructions is executed again, it is called an iteration.

In the case of coding qualitative data, this means that you keep repeating the process of coding until you meet the criteria of good analysis.

coding qualitative data is an iterative process between the research question, data, codes, themes, the analytical framework and the global theme

Exercise: How do university teachers experience funny cat videos on the Internet?

I have prepared an exercise to enable you to experience and feel first-hand what I mean by coding qualitative data being an iterative and pattern-finding process. If you want to get the full experience, follow my guidelines and do it seriously (no cheating!).

The exercise goes as follows. Let's say that you have conducted a series of interviews to answer the following research question: "How do university teachers experience funny cat videos on the Internet?" and that the first interview you have transcribed goes like this:

credit: Arria Belli

Excerpt from Interview 1:
"The thing I really like about kitties is that you can cuddle them. I don’t have kids so basically my cats… they are my best friends. So when I watch them online it just reminds me of that feeling. I love it."

The exercise goes as follows:
Task 1: Write a few codes for this excerpt (3-5). Work as if you are still in the process of constructing a research question and you are inductively exploring your data to figure out what you can make out of it. While you are coding, think about the kind of things you find relevant and interesting in the transcript. Don't look at the other excerpts yet (no cheating!).

Once you are done with Task 1, let's expand your material by adding excerpts from the transcripts of two other interviews, the interviews with Interviewer 2 and Interviewer 3. This is basically what happens when you start coding you material, you read a first interview and start coding it, not really knowing where it is going, and then you keep on by coding material from other sources.

Task 2: Code the excerpts from Interview 2 and Interview 3. As for the first excerpt, pick a few codes for each excerpt. At this stage, don't touch the codes you have created for the first excerpt (no cheating, again!).

Excerpt from Interview 2:
"I won’t lie, I love cats. I spend hours watching cats on the Internet doing silly things. My problem is that a lot of the time you see those very expensive cats, they are not normal cats. You see they are like supermodel cats and what I see is that people use them to show off. They use the cats. That I don’t like … like the cucumber thing you know, the cats are not ok, cats are not toys to play with… no, no, they just want to make internet money out of it."

Excerpt from Interview 3:
"These cats videos are genius, the cats are genius and the people posting them are genius. I read the comments all the time, it’s so good. People say Facebook is bad but this lol-cat culture, it’s giving me faith in humanity, always makes my day."

Ⓒ Audrey Alejandro – invented interviews

Task 3: Revisit/review/revise your coding of interview 1 based on your experience of reading and coding the excerpts from interview 2 and interview 3.

You might notice that what you find relevant and interesting in the first excerpt changes after reading the other two excerpts. By more or less consciously comparing the content of the different interviews, your interpretation of what the first excerpt means evolves. Expanding the material enables you to perceive patterns across the transcripts and differences between them.
For example, you might have created a code like "feelings" or "positive feelings" for the first excerpt, and then realise that there were feelings in all three interviews and you needed to change this code and be more precise about the kind of feelings that would best describe the experience of the interviewees in each excerpt. Also, interview 2 and 3 might have made you realise that the interviews were not only expressing different feelings, but also feelings about different things (cats, videos of cats, the humans producing such videos, the Internet...) and that these elements should also be accounted for in your coding etc.

To conclude, this exercise aimed to show you that it is normal for you not to find 'the right codes' the first time you read your material. Again, this is just the way the method works!

Don’t loose sight of your objectives!

when it comes to coding qualitative data, don't miss the forest for the trees

Original forest picture: Scott Wylie

Have you ever heard about the idiom 'Missing the forest for the trees'? This expression refers to a situation when someone is too involved in the details of a problem to look at the situation as a whole... And this is what happens when you start asking yourself questions like "is it ok if I turn my code into a theme?", "is it normal that I start changing the labels of my codes after having constructed my themes?" etc. Remember that coding is only a mean to an end, the end is for you to produce knowledge as interesting and rigorous as possible! Don't lose the big picture of what your objective is (=producing a good analysis) and the nature of the process that is taking you there (=a pattern-finding iterative process). Once you are clear about these points, I believe that all the questions you ask yourself can be better answered by asking yourself the two following questions:

Among the options available to me, what is the one that enables me to produce the best analysis?
Does what I am doing look like a pattern-finding iterative process?

If you are stuck or in case of doubt, try to substitute the questions you are asking yourself with these two to see what happens. In many cases, I bet it will help you move forward and take your analysis to the next level.

I hope this blog post has helped you unleash the coding force within you and that you are now ready to show the world your full coding potential 😉.

References
Gibbs, G. R. (2007) Qualitative Research kit: Analyzing qualitative data. London, England: SAGE Publications Ltd.
Saldana, J. (2009) The coding manual for qualitative researchers. Los Angeles, CA: SAGE.

You can download the pdf of the blog post here.

0 Comments