- About Us
- Statewide Resources
- Get Involved
- For Employees
- Find Us
Evaluation is an Everyday Activity
Program Evaluation Discussions
Updated: 6 hours 6 min ago
This will be short.
I showed a revised version of Alkin’s Evaluation Theory Tree in last week’s post. It had leaves. It looked like this:
It was taken from the second edition of Alkin’s book.
I have had two comments about this tree.
- There are few women represented in the tree. (True, especially in the draft version; in version above there are more.)
- I was reminded about the Fred Carden and Marvin C. Alkin’s article in the Journal of Multidisciplinary Evaluation, 8(17), January 2012. (There are still more leaves and the global south is represented.)
The person who reminded me about the Carden and Alkin (Thank you, Pablo Rodriguez-Bilella) article also gave me a reference that I am sharing (and since I cannot add a link I’ll just put the URL–hopefully, it will work in the post as it did in the comment–you may have to copy and paste):
This article is mentioned in the revised (yet again) version of the Evaluation Theory Tree in the article in number 2 above (see Table 1 and Figure 2).
So read the article in #2 above. and read the other article. The global south has a role here. It is important for evaluation.
In Chapter Two, he and Tina Christie talk about an evaluation theory tree and presents this idea graphically (all be it in draft form).
Think of your typical tree with three strong branches (no leaves) and two roots. Using this metaphor, the authors explain the development of evaluation theory as it appears in western (read global north) societies.
As you can see, the roots are “accountability and control” (positivist paradigm?) and social inquiry (post-positivist paradigm?).
The branches are labeled “use”, “methods”, and “valuing”. Scattered along those branches are various theorists who were/are significant in evaluation and its development. Some of these theorists have models that I have talked about in previous blogs (Lincoln/Guba, Stake, House, Eisner). Some are known to me and need to be shared here (Cousins, Stufflebeam, Greene, Rossi). Some are unknown to me (MacDonald, Wolf/Owens, Suchman, who have not been invited to contribute). In the first edition, Alkin has invited chapters by most of the folks listed in the tree.
The second edition lists more folks than the first. The metaphor of the tree has also been revised (see http://www.amazon.com/Evaluation-Roots-Perspective-Theorists-Influences/dp/1412995744#reader_1412995744). It now has leaves, some of which list the theorists,and a third root has been added: epistemology. Alkin says that the valuing branch stems from that root and is divided into objectivist and subjectivist views.
Although I don’t own the 2nd edition (yet), it takes a more global coverage but “…no chapter emerged on development theory in low and middle income countries (LMICs)”. This quote is taken from an article written by Alkin and Fred Carden in the January 2012 issue of Journal of MultiDisciplinary Evaluation (8, 17, 102-118). They suggest “…evaluation analysts…build a more comprehensive knowledge and documentation on development evaluation and more broadly on building the field of evaluation in LMICs.” It would be valuable for the global south to be represented.
The metaphor makes it easy to categorize the various views of evaluation; provides the reader with names to follow; and provides a history of sorts of evaluation. It would be interesting to see what is being done in the world they don’t address. Perhaps there is a novel approach that will be a newly leafed bud by someone who has yet to be named. Then it wouldn’t be a history…it would be contemporary.
(Pee Ess: I’ve been blogging for five years…)
Personal and situational bias are forms of cognitive bias and we all have cognitive bias.
When I did my dissertation on personal and situational biases, I was talking about cognitive bias (only I didn’t know it, then).
Then, I hypothesized that previous research experience (naive or sophisticated) and the effects of exposure to expected project outcomes (positive, mixed, negative) would affect the participant and make a difference in how the participant would code data. (It did.) The Sadler article which talked about intuitive data processing was the basis for this inquiry. Now many years later, I am encountering cognitive bias again. Sadler says that “…some biases can be traced to a particular background knowledge…”(or possibly–I think–lack of knowledge), “…prior experience, emotional makeup or world view”. (This, I think, falls under the category of, according to Tversky and Kahneman, human judgements and it will differ from rational choice theory (often given that label).
This is important for evaluators to remember…what you bring to the table does affect you; any assumptions you make because of your experience, world view, and/or perceptions affects you AND the evaluation. One way to help mitigate those assumptions is to make them explicitly clear–put them on the table.
Today, I was in a meeting about diversity. Although the term had been defined previously, there were many new players at the table for whom this term had not been clearly defined. Diversity is more than just the intersection of race and gender. Daryl Smith presents a model addressing this (she presented this model at a presentation at OSU in 2012). The discussion until that point had focused only on race; all the other forms of diversity including gender were not being addressed. Yet to talk about this topic all forms of diversity needed to be considered. Smith’s model included climate, access, success, education, scholarship, outreach, and capacity and they were all listed as “…overarching institutional goals for equity, inclusion, and diversity.” We had not clarified assumptions in this discussion. We were being influenced by personal and situational biases. If this had been an evaluation, there would have been a lot of cognitive dissonance; even not being an evaluation, there was a lot of cognitive dissonance. We will resolve this dissonance, even if it takes a while.
What I ask of evaluators is to remember that what you have experienced and what you know does affect the evaluation–any evaluation. Evaluations are not free of bias; evaluations can never be bias free. All we can do is try to mitigate the biases.
Sadler, D. R. (1981). Intuitive data processing as a potential source of bias in naturalistic evaluations. Educational Evaluation and Policy Analysis 3(4), 25-31.
The post Personal and Situational Bias-Cognitive Bias by another name? appeared first on Evaluation is an Everyday Activity.
A uniquely American holiday (although it is celebrated in other countries as well-Canada, Liberia, The Netherlands, Norfolk Islands),
For me it is an opportunity to to be grateful–and I am, more than words can express. I am especially grateful for my daughters, bright, articulate, and caring children (who are also adults).
What makes this holiday unique? That is an evaluative question.
What will make this holiday a good holiday for you? That, too, is an evaluative question.
This holiday will be good for me in many ways.
For me, it is an opportunity to think deeply about the various roles I fill: mother, sister, friend, evaluator, volunteer, among others.
It is an opportunity to think about what kind of guest I will be when I visit for the holiday.
It is an opportunity to think about the privilege that comes to me as an accident of my birth and those not so privileged.
It is an opportunity to count my blessings, of which there are many.