- About Us
- Statewide Resources
- Get Involved
- For Employees
- Find Us
Evaluation is an Everyday Activity
Program Evaluation Discussions
Updated: 6 hours 17 min ago
Previously, I talked about Survey’s (even though I posted it April 27, 2016). Today, I’ll collect all the posts about focus groups and add a bit more.
2010/01/05 Talks about the type of questions to use in a Focus Group
2010/01/27 One of three topics mentioned
2010/09/09 Talks about focus groups in terms of analyzing a conversation
2011/05/31 Talks about focus groups in the context of sampling
2011/06/23 Mentions Krueger, my go to
2013/11/15 Mentions focus groups
2014/10/23 Mentions focus groups and an individual with information
2015/02/11 Mentions focus groups…
2015/05/08 Virtual focus groups
Although focus groups are a mentioned throughout many of my posts, there are few that are exclusively devoted to focus groups. That surprises me. I need to talk more about focus groups. I especially need to talk about what I found when I did the virtual focus groups, more than with the specific post. From the interest at AEA last year, there needs to be much discussion.
So OK. More about focus groups.
Although Dick Krueger is my go to reference for focus groups (I studied with him, after all), there are other books on focus groups. (I just discovered that Krueger and Casey have also revised and published a 5th edition.)
The others for example (in no particular order),
- Stewart, D. W. & Shamadasani, P. N. (1990). Focus groups: Theory and practice. Newbury Park, CA: Sage Publications. There is a 3rd edition of this book available
- Morgan, D. L. (ed.) (1993). Successful focus groups: Advancing the state of the art. Newbury Park, CA: Sage Publications.
- Greenbaum, T. L. (2000). Moderating focus groups. Thousand Oaks, CA: Sage Publications.
- Greenbaum, T. L. (2nd edition). (1998). The handbook for focus group research. Thousand Oaks, CA: Sage Publications.
- Carey, M. A. & Asbury, J-E. (2012). Focus group research. Walnut Creek, CA: Left Coast Press, Inc.
Mary Marczak and Meg Sewell have an introduction to focus groups here (it is shorter that reading the book by Krueger and Casey).
I think it is important to remember that focus groups:
- Yield qualitative data;
- Are used in evaluation (just not in a pre-post sense);
- Are a GROUP activity of people who are typically unfamiliar with each other.
Next time: More on focus groups.
NOTE: This was written last week. I didn’t have time to post. Enjoy.
Methodology, aka implementation, monitoring, and deliveryis important. What good is it if you just gather the first findings that come to mind. Being rigorous here is just as important as when you are planning and modeling the program. So I’ve searched the last six years of blogs posts and gathered some of them for you. They are all about Survey, a form of methodology. Survey is a methodology that is often used by Extension, as it is easy to use. However, organizing the survey, getting the survey’s back, and dealing with non-response are problematic (another post, another time).
The previous posts are organized by date from the oldest to the most recent:
2016/04/21 (today’s post isn’t hyperlinked)
Just a few words on surveys today: A colleague asked about an evaluation survey for a recent conference. It will be an online survey probably using the University system, Qualtrics. My colleague jotted down a few ideas. The thought occurred to me that this book (by Ellen Taylor-Powell and Marcus Renner) would be useful. On page ten of this book, it asks for the type of information that is needed and wanted. It lists five types of possible information:
- Participant reaction (some measure of satisfaction);
- Teaching and facilitation (strengths and weaknesses of the presenter, who may (or may not) change the next time);
- Outcomes (what difference/benefits/intentions did the participant experience);
- Future programming (other educational needs/desires); and
- Participant background (who is attending and who isn’t can be answered here).
Thinking through these five categories made all the difference for my colleague. (Evaluation was a new area.) I had forgotten about how useful this booklet is for people being exposed to evaluation for the first time and to surveys, as well. I recommend it.
The WECT program arbitrarily divided the WECT program into four parts. Those “modules” are:
- Program Planning and Logic Modeling;
- Program Implementation, Monitoring, and Delivery;
- Data Management and Analysis (divided into Qualitative data and Quantitative data); and
- Program Evaluation Utilization
I’ve talked about Program Planning and Logic Modeling. So now what? What does the evaluator do next? I think that the evaluator needs to think about and plan for program implementation, monitoring and delivery of the intervention. In other words, “HOW” are you going to do the program that you have planned and modeled?
When dividing the study of evaluation into parts, the next part is program implementation, monitoring and delivery. Mary Arnold, 4H Extension Specialist did a wonderful job of discussing this topic. Having developed a logic model that identifies who, what, and why, now is the time to see if the program works, or the “how”. This part, the “how” reminds me of the simple evaluation model I used when I first started in the field. That model had three parts: process, progress, and product. The “how” covers the process and progress or the implementation, the monitoring , and the delivery. It is the methods (methodology) of the program.
Before I give you the list of blogs that relate to methods (you know, survey, focus groups, demographics, etc), I thought I’d go back to my favorite thesaurus for clarity. (Thank you, Michael Scriven, for providing such a valuable resource.)
So what does the Evaluation Thesaurus say about these topics?
Implementation: The degree to which a program (or treatment) has been instantiated in a particular situation, typically in a field trial of the treatment or an evaluation of it. Ralph Tyler (1934) and Lou Smith (????) had important contributions to make to implementation. There is more. (page 190-191)
Monitoring: Usually a representative of the funding agency who watches for the proper use of funds, observes progress, provides information to the agency about the project and vice versa. There is more. (page 235)
Delivery (system): The link between a product or service and the immediate consumer (the recipient population)–which may or may not consist of those that need or want it. There is more. (page 120)
I’ve spent a lot of time thinking about implementation, monitoring, and delivery. I have realized that I do not have tags for these topics. And in talking with my supervisor, yesterday, I also realized that I conceptualize the “implementation, monitoring, and delivery” phase under the category of “methodology”. So I will add the tags and talk about methodology/methods in my next post.