- About Us
- Statewide Resources
- Get Involved
- For Employees
- Find Us
Evaluation is an Everyday Activity
Program Evaluation Discussions
Updated: 12 hours 16 min ago
Having written about evaluation history previously, I identified those who contributed, not those who could be called evaluation pioneers; rather those who had influenced my thinking. I think it is noteworthy to mention those evaluation pioneers who set the field on the path we see today, those whom I didn’t mention and need to be. As a memorial (it is Memorial Day weekend , after all), Michael Patton (whom I’ve mentioned previously) is coordinating an AEA365 to identify and honor those evaluation pioneers who are no longer with us. (Thank you, Michael). The AEA365 link above will give you more details. I’ve also linked the mentioned evaluation pioneers that have been remembered. Some of these pioneers I’ve mentioned before; all are giants in the field; some are dearly loved as well. All those listed below have died. Patton talks about the recent-dead, the sasha, and the long-dead, the zamani. He cites the Historian James W. Loewen when he makes this distinction. Some of the listed are definitely the sasha (for me); some are zamani (for me). Perhaps photos will help (for whom photos could be found) and dates. There are other pioneers that are not mentioned here. Who was instrumental in your development? (They can still be alive. Or not.)
Patton starts by remembering Brenda Zimmerman (1956-2014), .
Marcia Guttentag (1933-1977) by Sara Miller McCune;
Donald Campbell (1916-1996) by Mel Mark;
Asa Hilliard III (1933-2007) by Rodney Hopson;
Egon Guba (1924-2008) by Jennifer Greene;
Robert Ingle (1926-1998) by Jean King;
Carol Weiss (1927-2013) by Sharon Rallis;
Will Shadish (1949-2016) by Laura Leviton;
Lee Sechrest (1929-2015) by Eleanor Chelimsky; and
Paul Lazarsfeld (1901-1976) , and
Alva (1902-1986) and Gunnar (1898-1987) Myrdal by Charmagne Campbell-Patton.
My sasha; my zamani.
I knew Bob Ingle, Will Shadish, Kathy Bolland, Lee Sechrest, and Egon Guba. Bob (and his wife, Maria), Will, and Kathy were friends of mine as well as colleagues. I miss them. A lot. Egon and Lee were colleagues; I miss them as well. They are sasha to me; they added much to the field. As did those who are zamani to me, even though they may have died within the last few years (Brenda Zimmerman, Don Campbell, Marcia Guttentag, Asa Hillard, Peter Rossi, Carol Weiss, Barry MacDonald, Paul Lazarsfeld, Alva Myrdal, and Gunnar Myrdal). I only knew Don Campbell, Carol Weiss, and Peter Rossi through their writings; there are books by them on my shelf. Although all these folks made their mark in other fields as well as evaluation, evaluation wouldn’t be where it is today without their conscientious application of what they knew (from their original fields) to what could be and made their contribution. They are remembered.
Mistakes are a great educator when one is honest enough to admit them and willing to learn from them.
– Alexander Solzhenitsyn
Even after 30+ years of evaluation, I make mistakes . It may be a mistake that occurs in the planning and modeling; it may be a mistake that occurs in the implementation, monitoring, and delivery; or a mistake in data management (qualitative or quantitative); or more than likely, a mistake in the use of the findings.
Assumptions are a form of bias that is pervasive in humon; we cannot escape the bias with which we grow up, the bias that come from the various systems we experience daily, the bias we read, see, and hear from the media. Some of those biases are so subtle that we do not know that we are being exposed to them, they just become a part of us as we go about our work, our lives. This is important to recognize. We assume that what worked will always work. We do not take into consideration the changes that have occurred–changes to systems, changes to locale, changes in you. We tend to think that what we see is all of what there is, like the ice berg when we know that the majority of the ice berg is below the water. I’ve been studying bias since the early 1980s. (I did my dissertation on personal and situational bias.) I work hard to NOT make assumptions. I fail miserably. I make assumptions without even knowing it. I wonder how much is a function of my privilege and how much is learned and did I learn it because of my privilege? I do not know. Are my assumptions always a reflection of that learning (and privilege)? Probably.
What was that mistake I made that has stuck with me? One that was based on assumptions I did not clarify? In the planning stage? I assumed that the docs who had agreed to “play” were in charge. They were not. This assumption put me (and the grant from NIDA) way behind. I learned (then at least) to clarify my assumptions. I still work on clarifying my assumptions.
Focus groups are a wonderful data gathering collection methodology. Not only are there different skills to learn for interviewing, analysis gives you the opportunity to explore qualitative data analysis. (It is all related after all.)
Now, I will confess that I’ve only ordered the 5th edition of the Krueger and Casey book (I don’t have it). I’m eager to see what is new. So I’ll settle for the 4th edition and try and regale you with information you may not know. (I will talk in a future post about the ways virtual focus groups are envisioned.)
Focus group describes (although sometimes incorrectly) a variety of group processes. Krueger and Casey give the reader a sense of to what to pay attention and to what is based on faulty data. So starting at the beginning, let’s look at an overview of what exactly is a focus group.
Groups are experiences that affect the individual throughout life and are used for planning, decision making, advising, learning, sharing, self-help, problem solving, among others. Yet group membership often leaves the individual unfulfilled, as time wasted. Krueger and Casey (abbreviated KC henceforth) as say that this failure of group activity is probably due to “unclear purpose and inappropriate process”.
Purpose is critical. Unless the purpose is clear at the outset, the group remains vague and unfocused, the participants confused or frustrated. The purpose is different for different types of groups, such as evaluative, marketing, political, etc. The other half of the above equation is process. If the leader doesn’t have the group process skills necessary to guide the group, the purpose may not be made clear because the direction is flawed. KC indicate the group process skills needed for one type of group may not work for another.
Focus groups provide the opportunity to gather information in the form of opinions, to listen to those opinions non-judgmentally, and to attempt to understand how people think about an issue, product, or service.
Participants are there because they all share similar characteristics, for example, they are all employed full time outside the home. They typically do not know each other, or know each other only in passing. The participants are not pressured to come to consensus. (Focus groups are not an example of nominal group technique as solutions are not sought nor ranked.) There is more than one group conducted so that trends and patterns can be collected. Then those comments (trends and patterns) are systematically analyzed.
Previously, I talked about Survey’s (even though I posted it April 27, 2016). Today, I’ll collect all the posts about focus groups and add a bit more.
2010/01/05 Talks about the type of questions to use in a Focus Group
2010/01/27 One of three topics mentioned
2010/09/09 Talks about focus groups in terms of analyzing a conversation
2011/05/31 Talks about focus groups in the context of sampling
2011/06/23 Mentions Krueger, my go to
2013/11/15 Mentions focus groups
2014/10/23 Mentions focus groups and an individual with information
2015/02/11 Mentions focus groups…
2015/05/08 Virtual focus groups
Although focus groups are a mentioned throughout many of my posts, there are few that are exclusively devoted to focus groups. That surprises me. I need to talk more about focus groups. I especially need to talk about what I found when I did the virtual focus groups, more than with the specific post. From the interest at AEA last year, there needs to be much discussion.
So OK. More about focus groups.
Although Dick Krueger is my go to reference for focus groups (I studied with him, after all), there are other books on focus groups. (I just discovered that Krueger and Casey have also revised and published a 5th edition.)
The others for example (in no particular order),
- Stewart, D. W. & Shamadasani, P. N. (1990). Focus groups: Theory and practice. Newbury Park, CA: Sage Publications. There is a 3rd edition of this book available
- Morgan, D. L. (ed.) (1993). Successful focus groups: Advancing the state of the art. Newbury Park, CA: Sage Publications.
- Greenbaum, T. L. (2000). Moderating focus groups. Thousand Oaks, CA: Sage Publications.
- Greenbaum, T. L. (2nd edition). (1998). The handbook for focus group research. Thousand Oaks, CA: Sage Publications.
- Carey, M. A. & Asbury, J-E. (2012). Focus group research. Walnut Creek, CA: Left Coast Press, Inc.
Mary Marczak and Meg Sewell have an introduction to focus groups here (it is shorter that reading the book by Krueger and Casey).
I think it is important to remember that focus groups:
- Yield qualitative data;
- Are used in evaluation (just not in a pre-post sense);
- Are a GROUP activity of people who are typically unfamiliar with each other.
Next time: More on focus groups.