Member check

Summary

Member checks, also known as respondent validation, are used as a method of enhancing the validity or credibility of qualitative research findings. In quantitative research, the concepts of reliability and validity are used to judge and evaluate statistical findings. In member checks, descriptive summaries of a qualitative study are presented to the participants in the study. Member checks occur after the study is complete. Furthermore, the member checks can be done formally and informally. The researcher goes back to the participants at the end of the study and asks the participants if the information is accurate or needs correction or elaboration on subject matter. A researcher should document his or her qualifications, experiences, perspectives, and assumptions. Some researchers include those working together in the planning, conducting, and analysis of results. The participants either affirm that the summaries reflect their views, feelings, and experiences, or that they do not reflect these experiences. If the participants affirm the accuracy and completeness, then the study is said to have credibility. The most common source for standards regarding member checks is found in a book by Lincoln and Guba (1985). Since the most important issue in evaluating the rigor of qualitative research is trustworthiness, using the strategy of member checks, (along with other techniques such as prolonged engagement, triangulation, peer debriefing), is critical to minimizing distortion. It is critical to use member checking in qualitative research studies because these types of studies often involve interpretation. Thus, without allowing participants to validate the accuracy of their findings, one-sidedness will become a major concern. It is possible for researchers to be given the same results and form different inferences. Naturally, people tend to interpret findings based on their own motivations and thoughts even though a proper researcher should only report what they observe. Member checking is a means by which the risk of biased qualitative research can be diminished. The actual participants are able to validate the accuracy of their own results. Some qualitative researchers object to the use of member checks, usually because it can be difficult to complete a member check in which all of the participants verify that the information is correct. An explanation for this is that when the researcher presents the study, it is designed to portray the collective account of all of the participants so that it may be applied to a broader population. This portrayal may differ, slightly or considerably, from each single account causing participants to argue that the information is incorrect. However, a member check is a useful tool to try and ensure that personal bias is not included within a study as well as to make sure that the interpreter of the information is interpreting correctly. This process is done to warrant that the quantitative study remains both reliable and valid. Researchers should be aware of possible false information when reviewing responses to a member check, as well as, avoiding bias amongst themselves.. Member checking is a process of returning to the informants and asking them whether the data reported represent their experiences of the phenomenon under study. (Wood) where you go back to those researched, at the completion of the study, and ask them if you are accurate or need correction/elaboration on constructs, hypotheses, etc. Some take this to the point of the researcher and those researched working together in the planning, conducting, and analysis of results.

Interpreting Member Checking

Before you begin “member checking,” consider and answer each of the following questions:

• What is being checked? If it is simply a check that the participants have the demographic characteristics you recorded for them, or recall a discussion as you do, the task may be simple, but this is hardly validating your interpretation. Most commonly what is being “checked” is how you have “seen” the situation. What account of this interpretation are you to give out for rechecking? You are unlikely to ask the participants to read your whole book, report, or thesis. But to read a chapter out of context will not give them the broader picture.
• With whom are you checking? Who are the relevant “members” and what is their relation to you and to your research topic? By now, you know the diversity of the research area, of power, influence and interest, of perceptions and meanings people put on the topic. It is very unlikely that “members” are a homogenous group.
• Who is checking whom? It is never appropriate to think in terms of an “us and them” dichotomy, in which you are “doing research on” them. Would you expect some respondents to be closer to the research and researcher(s) than others, more likely to stand back and analyze? (In most ongoing field research projects, some participants become close advisers or informants.)
• How would you interpret agreement? This is a crucial question. If the participants agree with your analysis, given that they are probably not researchers, this would be surprising. (It may be a matter for concern. One of the goals that can be set for a good project is that it showed something the participants could not see.)
• What follows if “they” agree? This is a major ethical issue. If a participant agrees to your account and even to its publication, this does not clear you of the responsibility of considering the implications of publishing. Perhaps they are unable to see future fallout from your report? Perhaps those consulted have an interest in your publishing a critique of others?
• How to interpret disagreement? There are of course two different situations to consider: everyone disagrees with you, or (a different issue) there is disagreement between members on you interpretation. By now it should be clear that respondent checking does not establish truth of your report. Think through, before you start feedback processes, all the different reasons why disagreement might appear. It is very likely that if you have made a good job of your research, the report exposes or even hurts some participants. You may have probed motives and interests, revealed the ways high intentions disguise messy politics or fear. Perhaps your task included confronting the participants with this analysis, but do you expect them to agree and if they did, what would that tell you?

Whatever the response to the interpretation you present to your participants, what was it they responded to? How partial was the interpretation? A group meeting with a flip chart cannot possibly present a well-integrated theory. Examine with brutal honesty your own motives in presenting and what you presented.

Subtypes

Narrative accuracy checks

• subjects check accuracy of transcripts

Interpretive validity

• subjects check conclusions drawn from their narratives
• Interpretive validity is present when the researcher accurately portrays the meanings given by the participants to what is being studied. The goal is to accurately document the participants’ viewpoints and meanings

Member Check and Credibility

Researchers can complete a member check by submitting information to study participants that helps reassure that all the information is relevant. The term member check literally means that the members check the information presented. The main problem with having a member check is defining what exactly is to be checked, but it is required to determine the researcher's credibility. The member check is critical in making a researcher's work credible by supporting the study's results. Observers tend to be biased about what they see, and since qualitative research often involves interpretation, further problems occur when they misinterpret what is really happening. Observers in phenomenological terms, report their own reality rather than the reality they should be observing. The way to deal with this source of bias is to train researchers to be aware of and control their tendencies to be biased. An effective strategy to increase validity is to do a member check - that is, ask participants if they think the interpretation is valid. Member checks by research participants are increasingly recommended in qualitative research. There are differences however in qualitative research that suggests member checks may need to be approached with caution. Although member checking has been used for verification of the results, it is not actually a verification strategy. Many methodologists caution against using member check as verification by defining what participants say to be correct because it may actually pose a threat to the validity instead. This is due to results of studies being combined, made neutral, and abstracted from other participants as well as investigators wanting to be more responsive to their participants and restrain some of their results. Ultimately this causes the researchers study to be invalid. Credibility plays a crucial role in the dependability of a qualitative research, because it offers "accuracy, validity, or soundness of data".(Wood) In fact there are countless ways to establish credibility including: "prolonged engagement in the field, persistent observation, collection of sufficient data, triangulation )use of multiple raters, cases, theme interpreters), peer review, member checks (confirmation by participants), and a search for negative instances that challenge emerging hypotheses and demand their reformulation."

Qualitative studies are concerned with reliability and validity of instruments, as well as internal and external validity criteria. Credibility is the truth of findings as judged by participants and others within the discipline. (Wood). For Example, a researcher interviewed a women who is pregnant and will be giving birth in a month, the researcher will return to interview the women after she has given birth since she has now gone through the birthing process to validate accuracy of the experience.

Member Check Considerations

Member checks are one of the most important mechanisms of assuring reliability in qualitative inquiry, which involves seeking the person’s views on the honesty and consistency of the research (McBrien). The process of member checking often informs researchers of possible misinterpretations of answers to questions performed within interviews. The process of a member check also is important in revealing missing information that should be addressed before concluding the study. This is a step of reevaluation within a study that allows for researchers to implement changes and conduct further interviews in areas where a study is weak. However, the responses received from participants may not always be accurate and should be carefully reviewed by researchers. It is important for researchers to review responses in order to avoid altering sound data. Participants in a study may have knowledge of incomplete or incorrect information that leads to misinformed responses. Likewise, they may also respond falsely in order to avoid social judgment or societal views on the subject, despite anonymity. Participants’ responses may also stem from myth-based knowledge or delusional thinking (Douglas, 1976). When hypotheses are formed, researchers often have predictions on the outcome; therefore, it is vital that researchers avoid their own biases to data. Avoidance of bias can be aided by having separate researchers review the member check responses than those who conducted the interviews. There are no clear-cut means of clearly avoiding incorrect participant feedback or researcher bias from tainting gathered research. However, by bearing these points in mind, researchers can minimize these factors and strengthen the external validity of their research. Tip: If member checking is used as a validation strategy, participants should be encouraged to provide critical feedback about factual errors or interpretive deficiencies. In writing about the study, it is important to be explicit about how member checking was done and what role was played as a validation strategy. Reader’s cannot develop much confidence in the study simply by learning that member checking was done.

Based on their experiences of using member validation, Emerson and Pollner (1988) have pointed to several other difficulties with its use: problems arising from a failure to read all of a report and/or from misunderstanding what was being said, establishing whether certain comments were indicative of assent or dissent, members with whom the authors had been quite close being unwilling to criticize, inadvertently asking member validation questions with which it would be difficult to disagree, and fears among members about the wider political implications of agreement or disagreement. Bloor (1997) has similarly argued that it is sometimes difficult to get members to give the submitted materials the attention the researcher would like, so that indifference can sound like corroboration.

Types of Interviews

Kvale defines qualitative research interviews as "attempts to understand the world from the subjects' point of view, to unfold the meaning of peoples' experiences, to uncover their lived world prior to scientific explanations." Interviews for research or evaluation purposes differ in some important ways from other familiar kinds of interviews or conversations. Unlike conversations in daily life, which are usually reciprocal exchanges, professional interviews involve an interviewer who is in charge of structuring and directing the questioning. In some professional interview situations, such as job interviews or legal interrogations, the power of the questioner is much greater than the power of the one being questioned. Therapeutic or clinical interviews are another special kind of professional interview, in which the purpose is to increase understanding and produce change in the person being interviewed. While interviews for research or evaluation purposes may also promote understanding and change, the emphasis is on intellectual understanding rather than on producing personal change (Kvale, 1996).[1]

Unlike conventional, positivist research, there is no single accepted outline for a qualitative research proposal or report. Qualitative interviewing utilizes open-ended questions that allow for individual variations. The richness of the responses in both breadth and depth can add markedly to the understanding of the situation. Given that member checks originate from a form of data collection, it is important to note a few of the common ways in which data collection is obtained. In qualitative research, qualitative interview strategies can include: informal, conversational interviews; semi-structured interviews; or even standardized, open-ended interviews. In all types of qualitative interviews it is important for the researcher to have a list of questions or general ideas to help shape the interview. In having a reference the interviewer can insure that the content of the interviews will move along the same lines, although individual responses may vary. By keeping a guided focus the interviewer helps establish a similar system (strategy) of interviewing as well as utilizing time with the participant adequately.

Three different types of interviews that Britten wrote about are explained below:

Structured = consist of giving structured questionnaires and the interviewers are trained to ask questions in a standardized manner.

Semi Structured = the interviewer asks open-ended questions to get the person talking about a subject such as; What do you think good health is?

Depth = covers only one or two issues but in greater detail.

Relation Between Unstructured Interviews and Member Checks

It is best to ask permission to visit the participant again at the time of the unstructured interview. According to Cohen D. and Crabtree B. in "Qualitative Research Guidelines Project," unstructured interviews help the researcher gain insight about their participants through open-ended questions , but there is no way to verify that without the participant. Interviews are used as a way for the respondent to express their emotions and thoughts about their experiences and allow the interviewer to have a better understanding of a situation . There are numerous ways to perform unstructured interviews, but one of the easiest is to schedule a time with each participant and simply engage them in a formal interview. After the interview, it is important to gain permission for further interviews or for the member check. It is also important to explore inconsistencies within the client's answers. Inconsistencies may be cleared up by simply re-wording the question or expanding the depth of the client's answer.

Rapport and Member Checks

If rapport is not established during the [unstructured interview][2], the participant may not agree to participate in the member check. If most of the participants do not agree in member check participation, it becomes much more difficult to establish the validity or credibility of the findings. The interviewer should strive to build rapport with the interviewee in order to obtain honest and open responses. Conversing with the participant in a casual conversational manner, taking care to listen closely, and responding in a nonjudgemental manner are means to achieve this goal. Through casual, comfortable conversation the interviewer can create a more relaxed and inviting atmosphere in which to engage the participant. Another way to facilitate rapport is to use colloquial language. Furthermore, use of technical jargon and unfamiliar wording should be avoided, as these tend to distance the participant from the interviewer and create a non-inviting atmosphere. In any case, such interaction will prove to facilitate a common thread to build upon.

The Ladder of Analytical Abstraction and Member Check

What is the ladder of analytical abstraction? The first level is the process of summarizing the data that is collected. Then the second rung, or level, in the ladder focuses on themes and trends that appear in the information. Finally, the last level is finalizing the patterns and drawing conclusions or explanations. How do member checks, or member validation, play a significant role in the ladder of analytical abstraction? At the first level, member checks help verify the facts. The researchers can then correct any mistakes or throw out unverifiable data. At the second level, themes and trends, member checks help build the "case narrative". If a researcher cannot get key participants in the study to agree on the outcomes being drawn, then this shows a weakness in the research itself. The third or last rung is more tricky. While the researchers derive explanations based on their research of external sources, the participants would base their knowledge simply on their own experiences. If the participant pool is well-educated and understands the purpose of the research, then member validation at this stage could be progressive. However, if the participants don't understand the context of the research then it could be detrimental.

Advantages and Disadvantages of Member Checks

Advantages (Strengths)

The greatest advantage is that participant may verify the accuracy and completeness of the findings, which is a strong support for the correctness of the findings.

A second advantage is that it is not particularly expensive or time consuming to conduct member checks, unless the members are completely inaccessible for some reason.

Other Advantages:

• Provides an opportunity to understand and assess what the participant intended to do through his or her actions
• Gives participants opportunity to correct errors and challenge what are perceived as wrong interpretations
• Provides the opportunity to volunteer additional information which may be stimulated by the playing back process
• Gets respondent on the record with his or her reports
• Provides an opportunity to summarize preliminary findings
• Typically, member checking is viewed as a technique for establishing to the validity of an account.
• This is when data, analytic categories, interpretations and conclusions are tested with members of those groups from whom the data were originally obtained.
• This can be done both formally and informally as opportunities for member checks may arise during the normal course of observation and conversation.
• Member checks can be useful in action research projects, where researchers work with participants on a continuing basis to help with change.
• Completing a member check allows the researchers, at the completion of the study, to ask the participants if they are accurate or if they need to improve the constructs, hypothesis, etc. through elaboration or correction. Some researchers even go far enough to involve the participants in the planning, conducting, and analyisis of the results.

Disadvantages (Weaknesses)

In many cases, the researcher sees things in the findings that cannot be easily conveyed to the participant. This is most true in addiction and criminal research.

Sometimes, during member checks, researchers choose to disregard their own interpretations and to accept those of respondents at face value. This can be cosy but may lead to collusion: Atkinson has warned of the dangers of "romanticising" respondents' accounts.

Also, Member check exercises, such as reading of drafts, make considerable demands on participants' time and, depending on the research topic and content of transcripts, can even be exploitative or distressing.

Other Drawbacks and Problems of Member-checking

Although many researchers view member checking as the strongest available check on the credibility of a research project, it has its limitations.

• Member checking relies on the assumption that there is a fixed truth of reality that can be accounted for by a researcher and confirmed by a respondent
  • From an interpretive perspective, understanding is co-created and there is no objective truth or reality to which the results of a study can be compared
  • The process of member-checking may lead to confusion rather than confirmation because participants may change their mind about an issue, the interview itself may have an impact on their original assessment, and new experiences (since the time of contact) may have intervened
• Respondents may disagree with researcher's interpretations. Then the question of whose interpretation should stand becomes an issue. The researcher then must decide whether to exclude the data or record the dissenting opinion in some way

The researchers and members are stakeholders in the research process and have different stories to tell and agendas to promote. This can result in conflicting ways of seeing interpretations.

• Members struggle with abstract synthesis
• Members and researchers may have different views of what is a fair account
• Members strive to be perceived as good people; researchers strive to be seen as good scholars. These divergent goals may shape findings and result in different ways of seeing and reacting to data
• Members may tell stories during an interview that they later regret or see differently. Members may deny such stories and want them removed from the data
• Members may not be in the best position to check the data. They may forget what they said or the manner in which a story was told
• Members may participate in checking only to be 'good' respondents and agree with an account in order to please the researcher
• Different members may have different views of the same data
• On occasion, researchers opt to ignore their own interpretations and to believe those of respondents at face value.

According to member check experiences, Emerson and Pollner recognized several disadvantages with member check usage.

• Problems may arise from a members failure to read an entire report and/or misunderstanding of what is being asked by the researcher.
• Members may be unwilling to criticize or disagree to member validation questions due to their relationship with the author.
• Members may fear the political implications of agreement or disagreement.

Member Checks and Transferability

If the member check procedure is used in a sample of people who were not the original participants in the study, the procedure can be used to assess transferability (also known as applicability or external validity) of the original findings. If the people who were not the original participants do not agree with the information, then the findings can be biased or not transferable.

The transferability of a working hypothesis to other situations is contingent upon the degree of similarity between the original situation and the situation to which it is transferred. The researcher cannot specify the transferability of the findings. He or she can only present sufficient information that can then be used by the reader to ascertain whether the findings are applicable to the new situation.

The account produced by the researcher is designed for a wide audience and will, inevitably, be different from the account of an individual informant simply because of their different roles in the research process. As a result, it is better to think of respondent validation as part of a process of error reduction which also generates further original data, which in turn requires interpretation.

Another term for transferability is fittingness. Fittingness is being faithful to the everyday reality of the participants, by describing in enough detail to allow others to evaluate the importance for their own research, practice or theory development. Or does it “ring true” in your own experience. Member checks help confirm the fittingness of the information.

Although there are limitations to transferability, Marshall and Rossman (1999) argue, findings can still relate to other cases and contexts. This is especially important in case study research, where researchers use the lessons of one case to make recommendations that can apply to others. Good qualitative works tends to be "tied into "the big picture"... It is important to remember that transferability is of high importance in qualitative research.

Objections to Member Checks

Many writers establish internal validity – truthfulness and representation of the reality of the participants - by showing that they have carried out a ‘member check’ as Lincoln and Guba (1985) suggest. However, some researchers disagree with the use of a member check. Many researchers have noted that when the essence of the participants' experiences are similar then their stories represent social reality. Therefore, in their opinion, a member check can adversely transform the data through the process of analysis and writing. Phenomenologists believe each individual has their own unique perspective, inhabits a social world and with others recognize others’ reality to some extent. They believe that for an account to have validity, its readers will have grasped not only the essence of the phenomenon but also understood something of the human condition they have in common with the participants – intersubjective understanding.

References

• Barbour, R.S. (2001).Checklists for improving rigour in qualitative research: a case of the tail wagging the dog. BMJ. 322, 1115-1117.

• Britten, Nicky. BMJ 1995; 311: 251-253 (22 July).

• Bryman, Alan. (Ed.). Member Validation. In Addressing social problems through qualitative research Loughborough: Reference World.

• Bygstad, B., Munkvold B. (2007) "The Significance of Member Validation in Qualitative Analysis: Experiences From a Longitudinal Case Study." Proceedings of the 40th Hawaii International Conference on System Sciences.

• Byrne, M.M. (2001). Evaluating the findings of qualitative research. AORN Journal.

• Cohen, D., Crabtree, B. "Qualitative Research Guidelines Project." July 2006.

• Gordon, C., Prince, M.B., Benkendorf, J.L. (2002). Hamilton HE Journal of Genetic Counseling. 4, 245-63.

• Grinnel R., Unrau Y., (2008) Social Work Research and Evaluation: Foundations of Evidence-based Practice. Oxford University Press US.

• Holloway, I. (2005) Qualitative Research in Healthcare. McGraw-Hill.

• Kirk, J., & Miller, M. L. (1986). Reliability and validity in qualitative research. Beverly Hills, CA: Sage.

• Lincoln, Y., Guba, E. (1985) Naturalistic Inquiry. Sage Publications, Newbury Park, CA.

• Lofland, J., & Lofland, L. H. (1984). Analyzing social settings. Belmont, CA: Wadsworth Publishing Company, Inc.

• McBrien,Barry. British Journal of Nursing, 2008 Nov 7-20

• Morse, J.M., Barrett, M., Mayan, M., Olson, K., and Spiers, J. (2002). Verification Strategies for Establishing Reliability and Validity in Qualitative Research. International Journal of Qualitative Methods. 1, 2.

• Ratcliff, D. (1995). Notes for Five Part Seminar on Qualitative Research.

• Rubin, A., Babbie, E.R. (2008). Research methods for social work. Belmont, CA:Thomson Brooks/Cole.

See Also

• Analytic induction
• Content analysis
• Focus group
• Interview

• Online research communities
• Participatory action research
• Qualitative research
• Sensemaking