- About Us
- Statewide Resources
- Get Involved
- For Employees
- Find Us
Evaluation is an Everyday Activity
Program Evaluation Discussions
Updated: 8 weeks 3 days ago
Recently I came across some old note of mine, from some meeting several years ago. I though it would be useful in my writing so I saved it; actually there were two notes that were similar in content. They both relate to blogging, although at the time I didn’t know I would be blogging.
I lump them all under the title of taking a stand, although stance would probably be more descriptive.
The notes are these:
- Know your audience.
- Be proactive to anticipate needs.
- Be reactive to meet needs.
- Be authentic.
- Be direct.
- Be unapologetic.
What you do with them affects you in your dealings, even your evaluation dealings.
If you do not know your audience , you cannot write to them; plan an evaluation with them; conduct an evaluation for them; teach them how to do the evaluation later. (That last sounds like you want to work yourself out of a job??? Maybe?) I have identified my audience as people who work for the Extension Service and need/want to know about evaluation (and sometimes other things… ) and other people who have an interest in evaluation in general–there are a lot of evaluators out there…
I listen to what folks are talking about and try to anticipate needs. Sometimes I’m not very good at anticipating needs; sometimes I am. I know that Fair Season is upon us and folks are probably not thinking EVALUATION right now. I think it is important to have evidence regardless of the season. Evaluation is one way to get evidence to support your contention.
When folks ask a question, I try to answer them (I see a question as a need–most of the time–and my knee jerk reaction is to find a solution). It may not be immediately. I look for answers and remember where those answers were. I send the answers (or at least where to find an answer) to whomever asked. No simple task. Fortunately, I’ve a bunch of good resources.
A long time ago, when I was first starting out in this business, I decided that being authentic (read: real) was the way to go. To me, that is the flip side of being direct. If you have to pussy foot around, you are not being real; you are not being direct. That doesn’t mean you have to be rude or insensitive. It does mean that you call a shovel a shovel, not that digging implement (unless you don’t know the name for something…).
At a certain point (probably after two, maybe after 18); there is no need to apologize for standing up for what you believe. You can only be a door mat if you lie down. So when it comes to taking a stand, no need to apologize. (I still find myself apologizing for things over which I have no control…I don’t need to do that). I do offer a caveat, however, letting the listener know this is my take on the issue.
I’m sure you can figure out how this is all evaluative.
How do you approach evaluation?
Are you the expert?
Do you work in partnership?
Are you one of the group?
To which question did you answer yes?
If you are the expert and know the most (not everything, no one know everything [although teenagers think they do]), you are probably “doing to”. Extension has been “doing to” for most of its existence.
If you work in partnership recognizing that the group with whom you are working has many cumulative years of knowledge and can give back to you, participants are co-equals, you are probably “doing with”.
If you are really one of the group, working daily to understand differences and biases, sharing that information and gathering information, you are probably “doing as”.
How does all this relate to evaluation? There are approaches to inquiry (of which evaluation is only one) that attempt to get the evaluator away from being the expert. David Fetterman has developed a model called empowerment evaluation ( and writes a blog about it here). His idea is basically to give the ability to evaluate away to the people who live the program/project…making them responsible, making them expert. The evaluator still needs to consult (obviously, or what would evaluators do?). Still it is an example of “doing with” that makes a world of difference. Community-based participatory research is another partnership form of inquiry often seen in public health and other outreach activities (read more about it here). Michael Quinn Patton talks about participatory evaluation in his book, Qualitative Research & Evaluation Methods . Participatory action research is another; I’m sure there are others…
The “doing as” concept comes from the diversity literature and includes information on cognitive bias. I heard it first from an evaluation colleague who is an indigenous person from NZ. And although I find this label compelling in its description, I find little or nothing on the concept in the literature. So let me see if I can describe it to you…when you evaluate from the perspective of “doing as” you evaluate as though you are a member of the community, owning the experience, and sharing what you know. It does include the “doing with” concept, to be sure, and goes further than that; the evaluator wears the hat, clothes, shoes of the group, the target group. It is being culturally aware, culturally competent; it is understanding, even if you cannot truly know, what it is like to be that person.
So, dear Readers. Are you doing to, doing with, or doing as when you evaluate?
We need to work diligently to do “doing as” when we evaluate.
The US has been a country for 238 years. A long time. Perhaps it is an opportunity to reflect on what are the rights, privileges, and obligations of citizenship. Perhaps it is just another holiday. Perhaps it is just a time for blueberry pie and peach ice cream. Perhaps it is a…fill in the blank.
I’m not feeling particularly patriotic. I am feeling very evaluative. Recently I viewed a map indicating that on a US passport an individual could travel to 172 different countries. The only country passports which were more powerful (i.e., able to visit more countries) were UK, Finland and Sweden. I wonder to where (what country) can’t I travel on my US passport? That question requires evidence. That is evaluative. I value my US passport. My girls and I travel with them even though driver’s license would be easier. (Being able to fly to Paris at a moment’s notice is important.. ) My passport is one of the privileges that comes with my citizenship. So is voting. So is freedom of speech and worship, and freedom from want and fear (FDR’s four freedoms).
What are you doing tomorrow…remembering?
Remember, evaluation is an everyday activity.
Enjoy the holiday.
What makes a blog engaging?
We know that blogs and blogging outreach to community members–those who have subscribed as well as those using various search engines to find a topical response.
Do the various forms of accessing the blog make a difference in whether the reader is engaged?
This is not a casual question, dear Readers. I will be presenting a poster at the Engagement Scholarship Consortium in October (which will be held in Edmonton, Alberta). I want to know. I want to be able to present to the various audiences at that meeting what my readers think. I realize that reading evaluation blogs may yield a response that is different from reading blogs related to food, or sustainability, or food sustainability, or climate chaos, or parenthood, or some other topic. There are enough evaluation blogs populating the internet that I think that there is some interest. I think my readers are engaged.
Only you, dear Readers, can tell me.
So are you engaged in reading my blog (even if you don’t comment).
Does the definition of engagement need to be broadened to be more inclusive? (see here for a definition used by the Consortium)
What exactly does collaboration mean in the context of blogs?
Chris Lysy in this week’s post talks about the why, what, who, how, and what next of blogging AND the post is peppered with cartoons. Using Chris as an example, I found his post engaging–I’m not sure it is collaborative. That is where I redefine collaborative…without his post, I doubt I would be writing this part of this blog…
So Readers–what DO you think? What makes a blog engaging?
A colleague asked, “How do you design an evaluation that can identify unintended consequences?” This was based on a statement about methodologies that “only measure the extent to which intended results have been achieved and are not able to capture unintended outcomes (see AEA365). (The cartoon is attributed to Rob Cottingham.)
Really good question. Unintended consequences are just that–outcomes which are not what you think will happen with the program you are implementing. This is where program theory comes into play. When you model the program, you think of what you want to happen. What you want to happen is usually supported by the literature, not your gut (intuition may be useful for unintended, however). A logic model lists as outcome the “intended” outcomes (consequences). So you run your program and you get something else, not necessarily bad, just not what you expected; the outcome is unintended.
Program theory can advise you that other outcomes could happen. How do you design your evaluation so that you can capture those. Mazmanian in his 1998 study on intention to change had an unintended outcome; one that has applications to any adult learning experience (1). So what method do you use to get at these? A general question, open ended? Perhaps. Many (most?) people won’t respond to open ended questions–takes too much time. OK. I can live with that. So what do you do instead? What does the literature say could happen? Even if you didn’t design the program for that outcome. Ask that question. Along with the questions about what you expect to happen.
How would you represent this in your logic model–by the ubiquitous “other”? Perhaps. Certainly easy that way. Again, look at program theory. What does it say? Then use what is said there. Or use “other”–then you are getting back to the open ended questions and run the risk of not getting a response. If you only model “other”–do you really know what that “other” is?
I know that I won’t be able to get to world peace, so I look for what I can evaluate and since I doubt I’ll have enough money to actually go and observe behaviors (certainly the ideal), I have to ask a question. In your question asking, you want a response right? Then ask the specific question. Ask it in a way that elicits program influence–how confident the respondent is that X happened? How confident the respondent is that they can do X? How confident is the respondent that this outcome could have happened? You could ask if X happened (yes/no) and then ask the confidence questions (confidence questions are also known as self-efficacy). Bandura will be proud. See OR OR (for discussions of self-efficacy and social learning).
1. Mazmanian, P. E., Daffron, S. R., Johnson, R. E., Davis, D. A., Kantrowitz, M. P. (1998). Information about barriers to planned change: A randomized controlled trial involving continuing medical education lectures and commitment to change. Academic Medicine 73(8), 882-886.