Evaluation is an Everyday Activity

Syndicate content
Program Evaluation Discussions
Updated: 5 weeks 2 days ago

Models

Fri, 03/07/2014 - 5:14pm

I’ve been reading about models lately; models that have been developed, models that are being used today, models that may be used tomorrow.

Webster (Seventh New Collegiate) Dictionary has almost two inches about models–I think my favorite definition is the fifth one: an example for imitation or emulation. It seems to be most relevant to evaluation. What do evaluators do if not imitate or emulate others?

To that end, I went looking for evaluation models. Jim Popham’s book has a chapter (2, Alternative approaches to educational evaluation) on models. Fitzpatrick, Sanders, and Worthen  has numerous chapters on “approaches”  (what Popham calls models). (I wonder if this is just semantics?)

Models have appeared in other blogs (not called models, though). In the case of Life in Perpetual Beta (Harold Jarche) provides this view of how organizations have evolved and calls them forms.(The below image is credited to David Ronfeldt.)

(Looks like a model to me. I wonder what evaluators could make of this.)

The reading is interesting because it is flexible. It approaches the “if it works, use it” paradigm; the one I use regularly.

I’ll just list the models Popham uses and discuss them over the next several weeks. (FYI-both Popham and Fitzpatrick, et. al., talk about the overlap of models.) Why is a discussion of models important, you may ask? I’ll quote Stufflebeam: “The study of alternative evaluation approaches is important for professionalizing program evaluation and for its scientific advancement and operation” (2001, p. 9).

Popham lists the following models:

  • Goal-Attainment models
  • Judgmental models emphasizing inputs
  • Judgmental models emphasizing outputs
  • Decision-Facilitation models
  • Naturalistic models

Popham does say that the model classification could have been done a different way. You will see that in the Fitzpatrick, Sanders, and Worthen volume  where they talk about the following approaches:

  • Expertise-oriented approaches
  • Consumer-oriented approaches
  • Program-oriented approaches
  • Decision-oriented approaches
  • Participant-oriented approaches

They have a nice table that does a comparative analysis of alternative approaches (Table 10.1, pp. 249-251)

Interesting reading.

References

Fitzpatrick, J. L., Sanders, J. R., & Worthen, B. R. (2011). Program evaluation: Alternative approaches and practical guidelines (4th ed.). Boston, MA: Pearson.

Popham, W. J. (1993). Educational Evaluation (3rd ed.). Boston, MA: Allyn and Bacon.

Stufflebeam, D. L. (2001). Evaluation models. New Directions for Evaluation (89). San Francisco, CA: Jossey-Bass.

 

 

 

Categories: OSU Extension Blogs

Do what I say not what I do

Mon, 02/24/2014 - 5:07pm

People often say one thing and do another.

This came home clearly to me with a nutrition project conducted with fifth and sixth grade students over the course of two consecutive semesters. We taught them nutrition and fitness and assorted various nutrition and fitness concepts (nutrient density, empty calories, food groups, energy requirements, etc.). We asked them at the beginning to identify which snack they would choose if they were with their friends (apple, carrots, peanut butter crackers, chocolate chip cookie, potato chips). We asked them at the end of the project the same question. They said they would choose an apple both pre and post. On the pretest, in descending order, the  students would choose carrots, potato chips, chocolate chip cookies, and peanut butter crackers. On the post test, in descending order, the students would choose chocolate chip cookies, carrots, potato chips, and peanut butter crackers. (Although the sample sizes were reasonable [i.e., greater than 30], I’m not sure that the difference between 13.0% [potato chips] and 12.7% [peanut butter crackers] was significant. I do not have those data.) Then, we also asked them to choose one real snack. What they said and what they did was not the same, even at the end of the project. Cookies won, hands down in both the treatment and control groups. Discouraging to say the least; disappointing to be sure. What they said they would do and what they actually did were different.

Although this program ran from September through April, and is much longer than the typical professional development conference of a half day (or even a day), what the students said was different from what the students did. We attempted to measure knowledge, attitude, and behavior. We did not measure intention to change.

That experience reminded me of a finding of Paul Mazmanian . (I know I’ve talked about him and his work before; his work bears repeating.) He did a randomized controlled trial involving continuing medical education and commitment to change. After all, any program worth its salt will result in behavior change, right? So Paul Mazmanian set up this experiment involving doctors, the world’s worst folks with whom to try to change behavior.

He found that “…physicians in both the study and the control groups were significantly more likely to change (47% vs 7%, p<0.001) IF they indicated an INTENT (emphasis added in both cases) to change immediately following the lecture ” (i.e., the continuing education program).  He did a further study and found that a signature stating that they would change didn’t increase the likelihood that they would change.

Bottom line, measure intention to change in evaluating your programs.

References:

Mazmanian, P. E., Daffron, S. R., Johnson, R. E., Davis, D. A., & Kantrowitz, M. P. (August 1998). Information about barriers to planned change: A randomized controlled trial involving continuing medical education lectures and commitment to change. Academic Medicine, 73(8), 882-886.

Mazmanian, P. E., Johnson, R. E., Zhang, A., Boothby, J. & Yeatts, E. J. (June, 2001). Effects of a signature on rates of change: A randomized controlled trial involving continuing education and the commitment-to-change model. Academic Medicine, 76(6), 642-646.

 

Categories: OSU Extension Blogs

Naturalistic models

Mon, 02/17/2014 - 4:53pm

When Elliot Eisner died in January, I wrote a post on his work as I understood it.

I may have mentioned naturalistic models; if not I needed to label them as such.

Today, I’ll talk some more about those models.

These models are often described as qualitative. Egon Guba (who died in 2008) and Yvonna Lincoln (distinguished professor of higher education at Texas A&M University) talk about qualitative inquiry in their 1981 book, Effective Evaluation (it has a long subtitle–here is the cover). They indicate that there are two factors on which constraints can be imposed: 1) antecedent variables and 2) possible outcomes, with the first impinging on the evaluation at its outset and the second referring to the possible consequences of the program. They propose a 2×2 figure to contrast between naturalistic inquiry and scientific inquiry depending on the constraints.

Besides Eisner’s model, Robert Stake and David Fetterman  have developed models that fit this model. Stake’s model is called responsive evaluation and Fetterman talks about ethnographic evaluation. Stake’s work is described in Standards-Based & Responsive Evaluation (2004) .  Fetterman has a volume called Ethnography: Step-by-Step (2010) .

Stake contended that evaluators needed to be more responsive to the issues associated with the program and in being responsive, measurement precision would be decreased. He argued that an evaluation (and he is talking about educational program evaluation) would be responsive if it “oreints more directly to program activities than to program intents; responds to audience requirements for information and if the different value perspectives present are referred to in reporting the success and failure of the program” (as cited in Popham, 1993, pg. 42). He indicates that human instruments (observers and judges) will be the data gathering approaches.  Stake views responsive evaluation to be “informal, flexible, subjective, and based on evolving audience concerns” (Popham, 1993, pg. 43).  He indicates that this approach is based on anthropology as opposed to psychology.

More on Fetterman’s ethnography model later.

References:

Fetterman, D. M. (2010). Ethnography step-by-step. Applied Social Research Methods Series, 17. Los Angeles, CA: Sage Publications.

Popham, W. J. (1993). Educational Evaluation (3rd ed.). Boston, MA: Allyn and Bacon.

Stake, R. E. (1975). Evaluating the arts in education: a responsive approach. Columbus, OH: Charles E. Merrill.

Stake, R. E. (2004). Standards-based & responsive evaluation. Thousand Oaks, CA: Sage Publications.

 

 

 

 

Categories: OSU Extension Blogs