- About Us
- Statewide Resources
- Get Involved
- For Employees
- Find Us
Evaluation is an Everyday Activity
Program Evaluation Discussions
Updated: 7 hours 59 min ago
It all depends.
The classic evaluation response. In fact, it is the punch line for one of the few evaluation jokes I can remember (some-timers disease being what it is; if you want to know the joke, ask in your comment).
The response reminds me of something I heard (once again) while I was in Denver. One of the presenters at a session on competencies, certification, credentialing (an indirectly, about accreditation) talked about a criteria for evaluators that is not taught in preparatory programs–the tolerance for ambiguity. (What do you see in this image?)
What is this tolerance? What is ambiguity?
According to Webster’s Seventh, tolerance is the noun form of the verb “to tolerate” and means “…the relative capacity to endure or adapt physiologically to an unfavorable environmental factor…” also defined as “…sympathy or indulgence for beliefs or practices differing from or conflicting with one’s own; the act of allowing something; allowable deviation from a standard…”.
Using the same source, ambiguity (also a noun) means “…the quality or state of being ambiguous in meaning…” OK. Going on to ambiguous (the root of the word), it is an adjective meaning “…doubtful or uncertain especially from obscurity or indistinctness…capable of being understood in two or more possible senses…”. Personally, I find the “capable of being understood in two or more possible senses…” relevant to evaluation and to evaluators.
Yet, I have to ask, What does all that mean? It all depends.
Many evaluations are perfectly clear to the program designer(s) and not to the program participants (familiarity can be blinding). The process must be explained many times, in different phrasing; in different words before everyone involved understands, if then. And even then, do all participant understand the program the same way? Probably not because of cognitive biases that every person has and brings with them when they participate in anything. Every person has personal and situational biases which affect the understanding any individual has for what is currently occurring, even the program designer(s). If the program designer(s) then has someone else (say an external evaluator) conduct the evaluation, another layer of ambiguity may be added–often is.
Some folks will see ambiguity as uncertainty (in fact Webster’s Seventh uses uncertainty as a synonym). I don’t; for me not knowing (uncertainty) is different from being unclear (ambiguity);. Certainly, an argument can be made that they are the same. (I’ll leave that for another time.) I see it as incumbent on the evaluator to be clear. Tolerance for ambiguity is hard to teach because of the discomfort people experience when met with lack of clarity. Yet, to be a competent evaluator, tolerance for ambiguity is a competency that is needed.
Today is Veteran’s Day in the United States.
It is the day celebrated as a federal holiday by libraries, post offices, school districts; not the university. It originated as Armistice Day in celebration of the end of World War I, the war to end all wars, the Great War.
It wasn’t made a national holiday (celebrated by those institutions above) until 1938. The name was changed from Armistice Day to Veterans Day in 1954 after the Korean War to remember all veterans, not just those from WWI.
Yet these women and men often give the ultimate sacrifice and are often not recognized for their service. Metrics do not capture the value, merit, or worth of their service, yet it is usually metrics that is the focus of any evaluation done.
(This cartoon is the segue to the next US holiday.)
It has been about /years since I started this blog (more or less–my anniversary is actually in early December) .
Because I am an evaluator, I have asked several time is this blog making a difference. And those posts, the ones in which I ask “is this blog making a difference”, are the ones which get the most comments. Now, truly, most comments are often either about marketing some product, inviting me to view another blog, mirroring comments made previously, or comments in a language which I cannot read (even with an online translator). Yet, there must be something about “making a difference” that engages viewers and then engages them to make a comment.
Today, I read a comment that was directed to me specifically (most are not) which said:
Are you still up for some more updates?
I surely hope so, or perhaps someone else could maintain the blog-post. It was pretty cool to read about.”
So I keep blogging, and am learning to be satisfied with little differences and small, sometimes meaningful, comments from readers. This week’s post is about those little differences with small comments from readers.
First some statistics taken from my dashboard: Yesterday (November 4, 2014), there were 38 views (out of 101 views) all related to the posts that have “making a difference” in the title. Today, there are (at 11:00am PT) 17 views (out of 38 so far). The most views I’ve gotten in a day was 157 and that was this summer when you would think most people would be on vacation. Of those 157, 55 views were to the sites that had “making a difference” in the title (about 1/3 of all views).
What do I take away from those analytics? That this blog IS making a difference. Perhaps, not in the way I have traditionally viewed evaluation–changes in the “participant’s” life. Maybe using electronic programs (like a blog) results in small changes, changes that manifest in a return visit, a new idea, a formation of a community (in this case of readers). So I’m learning to look for small changes, not life changes. Maybe a lot of small changes result in life changes. Only you, the reader, can tell me that. So if you have read this far, let me know–do a lot of small changes result in life changes?
Blogging is a good way to express your views and defiantly it makes a difference when you post it with unique and (hopefully) new information. That difference is crafting the blog, thinking of what to write, and telling the relevant story. I continue to do that…that is my difference.
I strive to publish valuable information. I believe that this blog may make a difference to readers. After all, they found this blog, spent some time on this blog page, perhaps left a comment or two.
I read. A lot.
I also blog. Weekly, unless I’m not in the office.
This past week I read (again) Harold Jarche’s blog. He posts periodically on interesting social media finds. Some of these finds are relevant to evaluation (even if they are not framed that way). His post on October 17 included a post from Kate Pinner called Half-baked ideas (She is found on twitter @kmpinner ). She says, “Just because you know how to do something doesn’t mean you should: It’s rewarding to give other people a chance to shine.”
Pinner’s comment is related to a thought I’ve been mulling for some time now (a couple of years, actually). That is the whole idea of “doing as.”
David Fetterman talks about empowerment evaluation (a model) that allows evaluators to (in my understanding) give evaluation away, to work themselves out of a job. Wikipedia (a quick and easy resource) says that empowerment evaluation is “…designed to help communities monitor and evaluate their own performance. It is used in comprehensive community initiatives as well as small-scale settings and is designed to help groups accomplish their goals.” Many programs are done in small scale settings by (small) groups. Fetterman has a good idea; I don’t think he goes far enough, though certainly farther than the “doing to” one often sees with evaluation. (Evaluators are experts so listen to them.) Empowerment evaluation sounds a lot like “doing with”. Pinner says that maybe the other person has a good idea and that good idea needs to be used. Pinner’s statement sounds like it could fit in empowerment evaluation.
New and related topic.
I just finished watching a TedTalk by Susan Cain about the “Power of the Introvert”. She advocates that three activities to support introverts:
- Stop the madness for constant group work at schools and work;
- Go into the wilderness; and
- Take a good look at what is inside your “suitcase”–and take them/it out
I’m sure you are wondering how this introvert stuff ties in to the evaluation stuff I started with. Pinner says, “Creativity needs space: If you provide someone the solution they never really have a chance to think outside the box and innovate.” The someone may need to “go into the wilderness” of innovation. The someone may need to work alone and unpack their suitcase. (I know I do.) That is were “doing as” comes into play. If you as an evaluator work at being one of the group with whom you are doing the evaluation (although you may never actually BE one of the group because of the cultural divide), you can allow people to do the above three things. It is a “emptying the suitcase” opportunity. It allows creativity to exist.
For years my criteria for a “good” conference was the following
- See three long time friends and spend some time catching up;
- Meet three people I didn’t know before and would like to continue to know;
- Get three new ideas that I can use.
I think this year’s conference was a success (despite the difficulty in identifying who was doing what when because the management corporation minimized the program in an attempt to be ecological, if excluding). If I were to ask my daughters to rate the conference on a scale of one (1) to 10 (ten), one being not “good”, 10 being “good”, I think they would have said an 8 – 8.5. (They have their own following of friends and their own interests.)
I saw and talked to three long time friends, although I missed those who have chosen not to attend AEA any more (I must be getting old) and those with whom I didn’t spend time.
I met more than three people I didn’t know before and I must say, if they are any indication (and I think they are) of the evolution of the association, the association is in good hands (even though I miss the intimacy I “grew up with”).
Most importantly, I did get at least three new ideas.
- Competencies is a topic that evokes a lot of discussion both pro and con. One cannot talk about accreditation, certification, and credentialing without talking about competencies (the skills and knowledge that make evaluators distinct).
- Blogging is challenging. (I keep at it, even though.) Folks who blog about evaluation are a special lot. Blogging is probably no easy task; blogging on evaluation is challenging. It is one way to get ideas “out there”. Chris Lysy (one of those folks I finally actually met–he is the cartoon guy) says it so eloquently in his blog post on Freshspectrum. He says it helps him stay connected with colleagues all year. He uses the metaphor of analog vs. digital (read his post). Being in the digital world is definitely challenging for a digital immigrant like me. Still I blog.
- I thought I knew a lot about focus groups (and I do). Yet I learned new things from Michelle Revels in her session on Focus Group Research. Although she talked about using focus groups for collecting research data, focus groups are a wonderful tool for gathering qualitative data for evaluation questions, too.
- Certainly, my thinking and knowledge about needs assessments (needs as a noun, not verb) was increased. I think the fallacy is that too many people want to get it done quickly and don’t think of strengths of the target audience. Every time I do a professional development session on needs assessment with my long time friend and colleague, Jim Altschuld , I learn something about the process. This year was no different. (I really must finish his new book…)
There were other things which I have used in the last three days, for sure, and although they need to be mentioned, I’ve exceeded my self-imposed limit of 500 words.
If you went to AEA, let me know what you thought. Did you have a good conference? If so, what did you learn?