Evaluation is an Everyday Activity

Syndicate content
Program Evaluation Discussions
Updated: 5 hours 50 min ago

Evaluability assessment

Thu, 09/14/2017 - 2:25pm
Evaluability assessment.

I get my ideas to blog from a lot of places. One place I get ideas are from other blogs.

The content of this blog was Evaluability Assessment.

The blog author says that evaluability assessments tend to cover the following topics:

  • Clarity of the intervention and its objectives: Is there a logical and clear theory of change that articulates how and under what conditions intervention activities influence particular processes of change?
  • Availability of data: which data are available that can be used in assessing the merit and worth of the intervention (e.g. generated by the intervention, external data sets, policy and academic literature)
  • Stakeholder interest and intended use: to what extent is there a clear interest (and capacity) among stakeholders to use the evaluation’s findings and recommendations in strategic decision-making, program improvement, learning about what works, etc.?

What, you ask, is evaluability assessment?

You certainly can go to the blog and read what it says there. OR…You can go to Scriven’s book and read the history on page 138 .

Suffice it to say that evaluability is the extent to which they (projects and programs) can be evaluated. Scriven goes on to say: “It should be thought of as the first commandment of accountability or as the last refinement of Popper’s (Sir Karl Raimond Popper) requirement of falsifiability.

 

Sources.

I learned about evaluability assessment (EA) from Midge Smith (shown here with her husband Carl Wisler) in her book by the same name (published by Springer, search for it by title).  She says that EA is “…a diagnostic and prescriptive tool for improving programs and making evaluations more useful.” Like all tools used in evaluation, it is systematic and describes the structure of a program.

There is a newer volume  of that name. It is by Michael S. Trevisan and Tamara M. Walser (they do an AEA365 blog on that topic). It is not, unfortunately, on my shelf.  The blurb that accompanies the book (by the publisher, Sage) says: “Evaluability assessment (EA) can lead to development of sound program theory, increased stakeholder involvement and empowerment, better understanding of program culture and context, enhanced collaboration and communication, process and findings use, and organizational learning and evaluation capacity building.”

More detail than Midge offers, then her book is copyrighted in 1989.

EA is getting a lot of press lately (you may need to search for evaluability assessment when you go to AEA365).

I find it amazing how previously important things (EA) are now once again in vogue.

my .

 

 

The post Evaluability assessment appeared first on Evaluation is an Everyday Activity.

Categories: OSU Extension Blogs

Libraries.

Wed, 09/06/2017 - 3:58pm
LIBRARIES 

With them, we are amazing!

Without them, we humons are limited.

I can only speak for myself–I do not want to be limited. I am a library champion. (A library champion is someone who fosters public awareness on the extensive range of resources and services available at public, school, academic and special libraries nationwide.)

If you were to see my professional library, perhaps you would understand. (It looks like the photo above, only the shelves are white; I have four bookcases with extensions. The shelves are full.)

If you were to see my personal library, you would understand. Most of it is still in boxes.

Because I believe that literacy is important, When they were young and beginning to read, I gave four of my bookshelves to my daughters (who are now grown, although their bookshelves are still full, mostly, of books). One each is in their room; two are now in the quest room. Hence, my library is mostly still in boxes.

I’ve stopped buying books for personal use (I still get professional ones). I use the library to get hard copy. I have 13 books at home plus two book club books. I have 11 books on hold.

I have an iPad on which I have at least four books and an equal number on hold. (I read a lot.)

But libraries do so much more than provide us with books (still their most important function). They move information in new directions! And they have magic fingers on the keyboard. I would be lost without libraries.

The information to which they have access is astounding.

Libraries and evaluation.

I want to discuss my professional library. I have one whole bookcase (of seven plus shelves) which is filled with books relating to evaluation. Some books are in many editions. And that doesn’t include statistics books or measurement books or the hard-copy journals that have come over the years.

“Why?” you ask, do I have multiple versions of the same (well, almost) book, different editions? Ah. Perhaps one edition will provide the answer (to the puzzle) and the others do not. Does that mean that the answer is not relevant? No. Does that mean that the information is passe? Maybe. Maybe not. The book may be the seminal reference and needs to be sited. It may give a history that isn’t found any place else. It is important to see how the volume changes with each edition. Having multiple volumes adds value. (And the root of evaluation is value.)

Do I need all this? Probably not, especially in the age of the internet and access to all that it provides. Yet, there is something about hard copy; you know a book  (whether a paperback or not), with its binding, its smell, its feel, that cannot be duplicated on-line. Something that cannot be diminished. Something that definitely adds value, merit and worth.

 

“To live in the world without becoming aware of the meaning of the world is like wandering in a great library without touching the books.” ~~Manly P. Hall

My feeling exactly.

 

 

 

The post Libraries. appeared first on Evaluation is an Everyday Activity.

Categories: OSU Extension Blogs

Making things (life) better.

Fri, 09/01/2017 - 2:11pm
Get better.

Unless someone like you cares a whole awful lot, nothing is going to get better. It’s not.

~~Theodor Seuss Geisel

 

Dr. Seuss is a life-time favorite of mine. If he were still alive (unfortunately, he died in 1991), he would still be speaking out (however subtly) and making a difference (he was born in 1904). He opposed discrimination and isolationism and cartooned those issues between 1941-1943.

 

His comment above reminded me about why I’m an evaluator. I do care. A lot.

Yes, I like to solve puzzles.  I like to figure things out. Yes, I really like to make life better.

I use science. And scientific principles. And data. To get answers.

So I have to wonder, how can anyone go through this life without that support? (You know, the support of science and answers provided by data.)

Statistics

Yes, I know…statistics lie (this saying “statistics lie” is often attributed to Mark Twain) and liars use statistics, depending on what the statistic is representing and who is using it. Or figures don’t lie; liars do figure may be more accurate (used by Carroll D. Wright ). (Oh and while I’m talking about statistics, Hans Rosling  died in February [2017]; a major loss.)

There is a lot of press about “big data” these days. And artificial intelligence. Is this the future?  Given that I hear what sound like competing stories about the future I can only wonder.

Even though I care. A lot. Perhaps as one source said, “…is it just a bit of social theatre we perform to make ourselves feel virtuous, useful, and in the right?” Maybe I’m just feeling virtuous and useful.

Maybe I only think I can change the world.

Gloria Anzaldua , an American scholar of  Chicana cultural theory who died in 2004, is quoted as saying, “I change myself; I change the world.”

So because I care (Seuss) and I am changing myself (Anzaldua), perhaps life will get better. Perhaps.

I will stick around and find out. And remember.

The post Making things (life) better. appeared first on Evaluation is an Everyday Activity.

Categories: OSU Extension Blogs

Key Evaluation Questions

Fri, 08/25/2017 - 3:53pm
 Key Evaluation Questions.  My friend and colleague, Patricia Rogers shared Better Evaluation discussion of Key Evaluation Questions. The opening paragraphs were enough to get me reading. She says:

Key Evaluation Questions (KEQs) are the high-level questions that an evaluation is designed to answer. Having an agreed set of KEQs makes it easier to decide what data to collect, how to analyze it, and how to report it.

KEQs should be developed by considering the type of evaluation being done, its intended users, its intended uses (purposes), and the evaluative criteria being used.

Our BetterEvaluation task page ‘Specify Key Evaluation Questions’ runs through some of the main considerations to keep in mind when developing your KEQs, and gives some useful examples and resources to help you.

Choosing Key Evaluation Questions. How do we know we’re making a difference? Are the tactics we are using appropriate? What are the meaningful signs of progress? How does impact relate?

In the graphic above (for which I am grateful although I could not find a source) the closest these steps come to asking “key evaluation questions” is the step “define evaluation questions”.

I want to reiterate: They are NOT the “not specific questions that are asked in an interview or a questionnaire”.

So if you have done all the steps listed above, who defines the evaluation questions?

Or does it (as Rogers suggests) depend on the type of evaluation being done? (To be fair, she also suggests that the questions make it easier to decide what data to collect, how to analyze it, and how to report it.)

Now I’ve talked about the type of evaluation being done before.

 

Perhaps the type of evaluation is the key here.

 

There are seven types of evaluation (formative, process, outcome, economic, impact, goal-based, and summative). Each of those seven types do address a specific activity; each has a specific purpose. So it would make sense that the questions that are asked are different. Does each of those types of evaluation need to answer the above questions?

 

If we are looking to improve the program (formative, usually) do we really need to know if we are making a difference? If the question of “how” (process) the program was delivered, what do we really need to know? We do need to know if the tactics (approaches) we are using are appropriate. For any of the other purposes, we do need if a difference occurs. And the only way to know what difference we made is to ask the participants. Post Script Oh, since I last posted, I was able to secure a copy of the third edition of Levin’s book on economic evaluation.  Henry Levin wrote it with Patrick J. McEwan, Clive Belfield, A. Brooks Bowden, and Robert Shand.  It is worth a read (as much has changed since the second edition, specifically with “questions of efficiency and cost-effectiveness”.)

 

The post Key Evaluation Questions appeared first on Evaluation is an Everyday Activity.

Categories: OSU Extension Blogs