Evaluation is an Everyday Activity

Syndicate content
Program Evaluation Discussions
Updated: 15 hours 10 min ago

Assumptions

Tue, 05/16/2017 - 5:11pm
Assumptions.

Assumptions.

You know the old saying about when you assume.

I’ve talked about assumptions here and here. (AEA365 talks about them here.)

Each of those times I was talking about assumptions, though not necessarily from the perspective of today’s post.

I still find that making assumptions is a mistake as well as a cognitive bias. And it does… .

Today, though, I want to talk about assumptions that evaluators can make, and in today’s climate, that is dangerous.

So, let me start with an example.

Example.

Once upon a time, investigators received funds to provide women who used and abused cocaine safe, stable, and secure housing. The investigators planned to provide for women’s young children. By providing stable and secure housing, cocaine use would/could be reduced and could/would be measured by urine and blood samples (assumption #1). The investigators arranged for the children’s needs (assumption #2). And facilities were contracted to provide shelter for the women (assumption #3).

Would any of this work? Theoretically, it should (assumption #4).

As planning proceeded, the question was asked: “What do the women say?”

As it turned out, the women had not been asked. No one talked to the women about their needs.

The science was well documented.

Best practice was being employed.

A key stakeholder was, unfortunately, NOT “at the table”.

Planning stopped. The investigators invited the women. They came and offered much.

Was it another assumption to not include the children? Probably. The investigators made arrangements for the children.

What if those arrangements didn’t/wouldn’t/couldn’t work?

Clarification.

How often do we follow the assumptions?

What if (as in this case) those assumptions were faulty?

How do you over come those assumptions?

Where does culture fit into the discussion? Or does it?

Are the values we hold dear blinding us to the picture in front of us?

I think that clarifying, up front, when the planning begins, what assumptions are underlying the values you bring to the table. This is so important, this clarifying. Assumptions are only a place to start…not the answer.

my .

molly.

 

 

 

The post Assumptions appeared first on Evaluation is an Everyday Activity.

Categories: OSU Extension Blogs

Professional Development

Mon, 05/08/2017 - 3:35pm
Professional Development.

AEA365 shares some insights into its use in evaluating professional development.

The authors cite Thomas Guskey (1, 2). I didn’t know Thomas Guskey. I went looking.

Turns out, Donald Kirkpatrick (1924-2014) was the inspiration for the five level evaluation model of Thomas Guskey.

Kirkpatrick has four levels in his model (reaction, learning, behavior, results). I’ve talked about them before here and here. I won’t go into them again.

Guskey has added a fifth level. In the middle.

He talks about participant reaction (level 1) and participant learning  (level 2) (like Kirkpatrick).

His third level is different. Here he talks about organization support and change.

Then he adds two additional levels that are representative of Kirkpatrick’s model (level 3 and 4). He adds participant use of new knowledge and skills (Kirkpatrick’s behavior; Guskey’s level 4) and participant learning outcomes (Kirkpatrick’s results; Guskey’s level 5).

AEA365.

AEA365 (A Tip-a-Day by and for Evaluators) makes the point to “assess implementation readiness of participants and their organizations prior to the delivery of the PD”. The authors of AEA365 make the point to “obtain information about organizational readiness to support novel approaches” as the participants will return with novel approaches. They also suggest to gather information about participants’ prior content knowledge and classroom experience.  Participants perception of school or district buy-in is important to gather. As is participants’ attitudes about the training and future adoption of what they will be learning. The authors provide the following citations.

  1. Guskey, T. R. (2002). Does it make a difference? Evaluating professional development. Educational Leadership, 59(6), 45-51.
  2. Guskey, T. R. (2009). Closing the Knowledge Gap on Effective Professional Development. Educational Horizons, 87(4), 224-233 (http://files.eric.ed.gov/fulltext/EJ849021.pdf).

I have not used this model. That doesn’t mean much. The citations are 21st century. I was prepared in the 20th century. I draw heavily from citations offered then.

The Extension Service presents many professional development sessions each year. They may not call them professional development. Perhaps they are called seminars, lectures, training sessions. Whatever they are called, the professional is prepared to provide a service better.

That is professional development.

Why is evaluation important?

Professional development is learning. That learning can be formal, as in a classroom for continuing education or academic credits. Or it can be informal like free choice learning, learning experienced in a museum or a nature preserve. And you want to know if the participants learned. You might even want to know what they learned. That is evaluation.

Certainly, you want to know if behaviors will change. And what the participant plans to do with the information. Guskey makes a point, though. If the organization doesn’t support change (even though the have supported the professional development), what good is the professional development to the profession? (If you want to learn more about training and development see the website.)

my

molly.

The post Professional Development appeared first on Evaluation is an Everyday Activity.

Categories: OSU Extension Blogs

Differently and familiar

Tue, 05/02/2017 - 4:25pm
Differently.

People don’t want something truly new, they want the familiar done differently.

OK. I got this idea from a blog post on sushi, well, actually “California Roll”.

Made me think. Evaluation is a service; that service is familiar; over the years it is done differently.

That moves the profession along–like language drift, only evaluation drift.

It is valuable to know formative/summative. (Thank you, Michael Scriven.)

It is also valuable to know that evaluation wouldn’t be were it is today if you didn’t understand that concept and how it applies to what you are doing with your evaluation.

So evaluation is like sushi (California Roll). Evaluation takes what is familiar and repackages it into something that will advance the profession.

Change

If it didn’t take what is familiar it would result in unfamiliar interfaces that are more difficult to use and impede adoption.

We want adaption. Progress. Subtle change. People can accept adoption. They see it as progress (not necessarily change–even though it is).

Adaption is subtle change.

That subtle change takes place over time.

Evaluation is not the same profession it was when I entered it as a graduate student.

Evaluation changed.

And evaluators changed with the profession.

Although it makes much sense to know the history, doing evaluation today draws from many other disciplines.

Because evaluation draws from many disciplines, evaluation is a trans-discipline. One that includes many disciplines.

Knowledge is moved forward; evaluation adapts. It is not the same profession it was; it is still familiar.

 

 

The post Differently and familiar appeared first on Evaluation is an Everyday Activity.

Categories: OSU Extension Blogs