- About Us
- Statewide Resources
- Get Involved
- For Employees
- Find Us
Evaluation is an Everyday Activity
Program Evaluation Discussions
Updated: 4 weeks 12 hours ago
I’ve just spent all of August and most of September editing chapters for a volume of New Directions in Evaluation (NDE) on Accreditation, Certification, and Credentialing. These topics all relate to competencies which all relate to building capacity. Now I can site a lot of references for competencies. (For example, Stevahn, King, Ghere, Minnema, 2005, AJE 26(1), pp. 43-59., among others see the work by King and cadre–that one cited just happens to be on my desk right now.) This group has been working on competencies for the last 15 or so years. This is important work–as well as problematic (hence the issue of NDE). I won’t go into details here because the NDE volume pretty much addresses these issues from a variety of perspectives. We (my co-editor, Jim Altschuld and I) have assembled (what I think is) a stellar collection of writers who have good ideas. Editing an issue of NDE (again) was a valuable experience for me: I learned again why I don’t write the definitive text on anything; I learned again how important Accreditation, Certification, and Credentialing are; I am reminded how complicated it is to assemble a list of competencies that adequately capture what is an evaluator; and I am once again humbled, recognizing that cynicism does not come with the territory–it is acquired.
Now, a bit on competencies and why they are important.
I think everyone will agree that there are certain knowledge (what a person can learn), skills (what a person can do), and dispositions/attitudes (what a person can think and/or feel–they are different BTW) necessary for an individual to function effectively as an evaluator. The question is what exactly are they? And can evaluation be a profession without an established list of competencies? The Worthen 1994 article is important here (Worthen, B. R. . Is evaluation a mature profession that warrants the preparation of evaluation professionals? In J. W. Altschuld & M. Engle [Eds.], New Directions for Program Evaluation: No. 62. The preparation of professional evaluators: Issues, perspectives and programs. [pp. 3–15]. San Francisco: Jossey-Bass). The Stevahn et al. article lists six different categories of competencies (professional practice, systemmatic inquiry, situational analysis, project management, reflective practice, and interpersonal competence). The CES list includes five categories (reflective practice, technical practice, situational practice, management practice, and interpersonal practice). They are similar, yet different.
The Canadian Evaluation Society has established a credentialing process that involves the a list of competencies that went through an extensive and exhaustive process research, consultation, and validation process. The AEA has yet to develop (or endorse) a similar list, and a similar list exists (see the Stevahn, King, Gere, & Minnema citation above).
How many of you who are practicing evaluators can honestly say you were taught in your preparation programs (even if you did a preparation program in a discipline other than evaluation) to analyze situations? To manage projects? To reflect on practice? About interpersonal communications? I’m guessing most people were exposed (even briefly) to professional practice (after all part of preparation is the socialization to the profession) and technical practice/systematic inquiry. With that disparity across preparation, how can evaluation be a profession?
This summer I spent a lot of time dealing with needs assessments and talking about needs and assets. It occurred to me that the difference between need and wants has a lot to do with evaluation (among other things). So what are needs? What are wants? How does all this relate to evaluation?
Maslow spoke eloquently about needs in his hierarchy, and although the hierarchy is often presented as a pyramid, Maslow didn’t present the needs this way. He did present this hierarchy as a set of building blocks with basic needs (physiological) as the foundation, followed by safety, loving/belonging, esteem, and self-actualization. He talks about this theory of motivation in his book, Motivation and Personality (a 3rd edition is available as well). This view of the individual ushered in the humanistic view of psychology (often called the third theory after behaviorism and psychoanalysis). He believed that human could not live without these needs and advocated that they are necessary for survival.
A “want” is often considered a desire based purely in economic, social, or psychological reality of human existence. It is something that an individual would like to have. (Chocolate, any one?) A want is not essential to human existence; it is only something an individual would like to have. Unfortunately, there are limited resources (as well as a large body of literature) talking about having enough. If you have enough, then wants are few and resources are available for everyone.
When does needing become wanting? Does wanting dominate even when there are needs? If you don’t have enough to eat, do you need food or want it? Or water? Same thing. Evaluation is like that. When do you have enough? When do you know enough? Are programs always about needs or are they about wants? If they are about wants, who is in the best position to determine if they are assets or needs? I’m sure it sounds like I’m going in circles; perhaps I am. I think (a caveat) that evaluation isn’t a want. I think (another caveat) that evaluation is a need, probably falling somewhere in the safety sphere (according to Maslow). Safety being security of various parts of an individuals life (body, employment, resources, morality, family, health, and property). I’m sure other arguments can be made as well.
Evaluation talks about the worth, merit, value of a program. Evaluation is one way to determine if something works, if the program has made a difference with the target audience. That sounds like security to me. Determining the worth, merit, value of a program, moves from wanting to needing. By determining the worth, merit, value of something (in this case a program), you help ensure security, you help ensure safety of the target audience.
I just finished a chapter on needs assessment in the public sector–you know that part of the work environment that provides a service to the community and receives part of its funding from the county/state/federal governments. Most of you know I’ve been an academic for at least 31 years, maybe more (depending on when you start the clock). In that time I’ve worked as an internal evaluator, a program planner, and a classroom teacher. Most of what I’ve done has an evaluative component to it. (I actually earned my doctorate in program evaluation when most people in evaluation came from other disciplines.) During that time I’ve worked on many programs/projects in a variety of situations (individual classroom, community, state, and country). I find it really puzzling that evaluators will take on evaluation without having a firm foundation on which to base those evaluations. (I know I have done this; I can offer all manner of excuses, only not here).
If I had been invited to participate in the evaluation at the beginning of the program, at the conceptualization stage, I would have asked if a needs assessment had been done and what was the outcome of that assessment. Was there really a lack (i.e., a need); or was this “need” contrived to do something else (bring in grant money, further a career, make a stakeholder happy, etc.)?
When I was writing the chapter, I revisited several “needs” assessments that actually provided valuable information that wasn’t about needs–that is, something lacking. Rather they provided support for the position taken by Altschuld (2014). in his current book on capacity building. He proposes a hybrid model that combines the two. Rather than looking at the glass half empty (needs), he looks at the glass half full (assets). Having stakeholders identify what they already have in terms of services that can be supported or augmented is a totally different approach to needs assessment.
Knowing what the stakeholders have can be very informative when developing programs. Recognizing that the stakeholders may have strengths that have been previously unrecognized is an important piece of information for an evaluator to have. It prevents duplication of services, it prevents inefficient programs, it promotes responsibility (of stakeholders, for sure; probably evaluators as well), and it promotes recognition of strengths currently held.
I was heartened to learn on retrospect that I supported asset/capacity building in the work I’ve done, long before the book was published. At the time, I’m sure we were looking for the gaps between what is and what could be; in reality, what could be was already present, we just had to recognize it.
“Humans are social animals for good reason. Without collaboration, there is no survival. It was not possible to defeat a Woolley Mammoth, build a secure structure, or care for children while hunting without a team effort. It’s more true now than then. Our reliance on each other grows as societies became more complex, interconnected, and specialized. Connection is a prerequisite for survival, physically and emotionally.”
This statement, which I found on Harold Jarche ‘s blog, applies to evaluation as much as it applies to the example provided by Psychology Today.
Evaluation is a collaborative effort; a team effort, a social effort. Without the collaboration, evaluation lacks much. I’m not sure that survival is dependent on evaluation and collaborative effort; perhaps. The evaluator may know all about evaluation and yet not be able to solve the problem presented by the client because the evaluator doesn’t know about the topic needing to be evaluated. The evaluator may know about something similar to and different from what the client needs and yet, not know about the specific problem. Let me give you an example.
I’ve done a lot of evaluation in the natural resources area and as a result, I’ve learned much about various natural resource topics, including horticulture, plant science, crop science, marine science. I do not know much about potatoes. A while back, a colleague called me and asked if I could/would serve as the evaluator on a ZEBRA CHIP project. Before I said, Sure, I asked about ZEBRA CHIP. Apparently, it is a potato disease transmitted by bacteria carrying psyllid that is causing much economic devastation among growers. It shows up best when the potatoes are made into chips (hence the name). It looks like this: . To me, it isn’t particularly stripped like the animal which offers its name, yet it doesn’t look like potato chips I’m used to seeing. I”m told that there is an unpleasant flavor to the chips as well. I knew a lot about growing things, not about potatoes, even though I’ve worked with potato growers before, just not about this disease.
So, I said sure, only to discover that I have 11 lines in which to write a cogent evaluation section for the work that Extension will be doing (if the grant is funded). If the grant is funded, it will be a five year effort. A continuation actually, which brings me full circle–a collaboration of multiple universities, multiple disciplines, multiple investigators. So what could I say cogently in 11 lines? I suggested that perhaps looking at intention and confidence would be appropriate because we (remember, I said, “Sure”) would not be able to measure actual behavior change. And to overcome the psyllids and eradicate this problem (not unlike the spotted wing drosophila which is affecting the soft fruits of the NW, specifically blueberries), we would need to get as close to behavior change as possible once the teaching has occurred. How can social media be used here? Good question–something to explore. At what level of Maslow’s hierarchy is this collaboration? Survival, sure. Somehow I don’t think Maslow was focused on economic survival.
The post Evaluation, Maslow’s hierarchy of needs, and social media appeared first on Evaluation is an Everyday Activity.