- About Us
- Statewide Resources
- Get Involved
- Find Us
Evaluation is an Everyday Activity
Program Evaluation Discussions
Updated: 3 days 15 hours ago
I was reminded about the age of this blog (see comment below). Then it occurred to me: I’ve been writing this blog since December 2009. That is 4 years of almost weekly posts. And even though evaluation is my primary focus, I occasionally get on my soap box and do something different (White Christmas Pie, anyone?). My other passion besides evaluation is food and cooking. I gave a Latke party on Saturday and the food was pretty–and it even tasted good. I was more impressed by the visual appeal of my table; my guests were more impressed by the array of tastes, flavors, and textures. I’d say the evening was a success. This blog is a metaphor for that table. Sometimes I’m impressed with the visual appeal; sometimes I’m impressed with the content. Today is an anniversary. Four years. I find that amazing (visual appeal). The quote below (a comment offered by a reader on the post “Is this blog making a difference?”, a post I made a long time ago) is about content.
“Judging just from the age of your blog I must speculate that you’ve done something right. If not then I doubt you’d still be writing regularly. Evaluation of your progress is important but pales in comparison to the importance of writing fresh new content on a regular basis. Content that can be found no place else is what makes a blog truly useful and indeed helps it make a difference.”
Audit or evaluation?
I’m an evaluator; I want to know what difference the “program” is making in the lives of the participants. The local school district where I live, work, and send my children to school has provided middle school children with iPads . They want to “audit” their use. I commend the school district for that initiative (both giving the iPads as well wanting to determine the effectiveness). I wonder if they really want to know what difference the electronics are making in the lives of the students. I guess I need to go re-read Tom Schwandt’s 1988 book, “Linking Auditing and Metaevaluation”, a book he wrote with Ed Halpern, as well as see what has happened in the last 25 years (and it is NOT that I do not have anything else to read…). I think it is important to note the sentence (taken from the forward), “Nontraditional studies are found not only in education, but also in…divers fields …(and the list they provide is a who’s who in social science). The problem of such studies is “establishing their merit”. That is always a problem with evaluation–establishing the merit, worth, value of a program (study).
We could spend a lot of time debating the merit, worth, value of using electronics in the pursuit of learning. (In fact, Jeffrey Selingo writes about the need to personalize instruction using electronics in his 2013 book “College (Un)bound”–very readable, recommended.) I do not think counting the number of apps or the number of page views is going to answer the question posed. I do not think counting the number of iPads returned in working condition will either. This is an interesting experiment. How , reader, would you evaluate the merit, worth, value of giving iPads to middle school children? All ideas are welcome–let me know because I do not have an answer, only an idea.
I keep getting comments about my post “Is this post making a difference” and the subsequent posts related to that. (The survey is closed–has been for a long time.) Although I was looking for some tangible measure of difference (i.e., what is the merit, worth, value of this blog), I find that this post elicits more positive comments than not. Although I still get the random advertising post on that blog, I mostly get thought provoking comments about how it is or can be making a difference. I also get the comment at least weekly that says my blog isn’t making any difference to the reader (perhaps the reader needs to follow the blog across time rather than just reading this post) …sigh…
One comment I received this week said, “I can confirm that the information that you share has: 1) made sense; 2) made a difference; 3) been worthwhile. Keep it up; don’t stop.” Nice. Not specific; still nice. That the blog is being read is important. That people take the time to comment is important. That this venue is valuable to many is important.
I get positive comments on my writing; I get positive comments on my content. I’m an evaluator. I want to know what difference this information is making in your life. This is a program and it needs to be evaluated. And number of page views isn’t the answer.
So I ask you to keep commenting, that way I know the blog is being read. That way I know I am making a difference.
For the first time in my lifetime the first day of Hanukkah is also Thanksgiving. The pundits are are sagely calling the event Thanksgivukkah. According to this referenced source, the first day of Hanukkah will not happen again for over 70,000 years. However, according to another source, this overlap could happen again in 2070 and 2165. Although I do not think I’ll be around in 2070, my children could be (they are 17 and 20 of this writing). I find this phenomenon really interesting–Thanksgiving usually starts the US holiday season and Hanukkah falls later, during Advent. Not so this year. I wonder how people combine latkes and Thanksgiving (even without the turkey). Loaded latkes? (My appreciation to Kia.)
So I’m sure you are wondering, HOW EXACTLY DOES THIS RELATE TO EVALUATION?
I decided that it was time to revisit my blog title, Evaluation is an Everyday Activity. Every day you evaluate something. Although you do not necessarily articulate out loud the criteria against which you are determining merit, worth, and value, you have those criteria. I have them for latkes AND Thanksgiving. Our latkes must be crispy; of winter vegetables including potatoes. This allows me to use a variety of winter vegetables I may have gotten in my CSA. (Beet latkes? Sweet potato latkes? Celeriac latkes? You bet!) Our Thanksgiving is to have foods for which we are truly thankful. That allows us to think about gratitude. Each year our menu is different because each year we are thankful for different things. (I must confess, however, we always have pie–pumpkin, which I make from home grown pumpkin/squash, and chocolate pecan, which is an original old family recipe.) One year when we put all the food on the table, all the food was green. We didn’t plan it that way; it just happened because they were foods for which we were thankful. This year, we will have mashed potatoes (by the Queen of mashed potatoes), Celebration Filo, both the gluten-free (made with rice wrappers and no onion, garlic, or dairy) and glutened versions (the version which we renamed and is in the link above), and something else that will probably be green. This year I’m thankful for my gluten-free; dairy-free friend who will join us for Thanksgiving and I’m working up alternatives to accommodate her and still satisfy the rest of us.
So you see, even when I’m thinking about Thanksgiving, latkes, and gratitude, I’m thinking about evaluation. What merit does the “program” have? What is its worth? What is its value? Those are all evaluative questions that apply to Thanksgiving (and latkes and gratitude).
So you see, Evaluation is an Everyday Activity.
We all know about independent variables, and dependent variables. Probably even learned about moderator variables, control variables and intervening variables. Have you heard of confounding variables? Variables over which you have no (or very little) control. They present as a positive or negative correlation with the dependent and independent variable. This spurious relationship plays havoc with analyses, program outcomes, and logic models. You see them often in social programs.
Ever encounter one? (Let me know). Need an example? Here is one a colleague provided. There was a program developed to assist children removed from their biologic mothers (even though the courts typically favor mothers) to improve the children’s choices and chances of success. The program had included training of key stakeholders (including judges, social service, potential foster parents). The confounding variable that wasn’t taken into account was the sudden appearance of the biological father. Judges assumed that he was no longer present (and most of the time he wasn’t); social service established fostering without taking into consideration the presence of the biological father; potential foster parents were not allerted in their training of the possibility. Needless to say, the program failed. When biologic fathers appeared (as often happened), the program had no control over the effect they had. Fathers had not been included in the program’s equation.
Recently, I was asked to review a grant proposal, the award would result in several hundred thousand dollars (and in today’s economy, no small change). The PI’s passion came through in the proposal’s text. However, the PI and the PI’s colleagues did some major lumping in the text that confounded the proposed outcomes. I didn’t see how what was being proposed would result in what was said to happen. This is an evaluative task. I was charged to with evaluating the proposal on technical merit, possibility of impact (certainly not world peace), and achievability. The proposal was lofty and meant well. The likelihood that it would accomplish what it proposed was unclear, despite the PI’s passion. When reviewing a proposal, it is important to think big picture as well as small picture. Most proposals will not be sustainable after the end of funding. Will the proposed project be able to really make an impact (and I’m not talking here about world peace).
I attended a meeting recently that focused on various aspects of diversity. (Now among the confounding here is what does one mean by diversity; is it only the intersection of gender and race/ethnicity? Or something bigger, more?) One of the presenters talked about how just by entering into the conversation, the participants would be changed. I wondered, how can that change be measured? How would you know that a change took place? Any ideas? Let me know.
A colleague asked whether a focus group could be conducted via email. I had never heard of such a thing (virtual, yes; email, no). Dick Krueger and Mary Ann Casey only talk about electronic reporting in their 4th edition of their Focus Group book. If I go to Wikipedia (keep in mind it is a wiki…), there is a discussion of online focus groups. Nothing offered about email focus groups. So I ask you, readers, is it a focus group if it is conducted by email?
I had a topic all ready to write about then I got sick. I’m sitting here typing this trying to remember what that topic was, to no avail. That topic went the way of much of my recent memory; another day, perhaps.
I do remember the conversation with my daughter about correlation. She had a correlation of .3 something with a probability of 0.011 and didn’t understand what that meant. We had a long discussion of causation and attribution and correlation.
We had another long conversation about practical v. statistical significance, something her statistics professor isn’t teaching. She isn’t learning about data management in her statistics class either. Having dealt with both qualitative and quantitative data for a long time, I have come to realize that data management needs to be understood long before you memorize the formulas for the various statistical tests you wish to perform. What if the flood happens????
Data dictionary. The first step in data management is a data dictionary. There are other names for this, which escape me right now…know that a hard copy of how and what you have coded is critical. Yes, make a back up copy on your hard drive…have a hard copy because the flood might happen. (It is raining right now and it is Oregon in November.)
Take a hard copy of your survey, evaluation form, qualitative data coding sheet and mark on it what every code notation you used means. I’d show you an example of what I do, only they are at the office and I am home sick without my files. So, I’ll show you a clip art instead… . No, I don’t use cards any more for my data (I did once…most of you won’t remember that time…), I do make a hard copy with clear notations. I find my self doing that with other things to make sure I code the response the same way. That is what a data dictionary allows you to do–check yourself.
Then I run a frequencies and percentages analysis. I use SPSS (because that is what I learned first). I look for outliers, variables that are miscoded, and system generated missing data that isn’t missing. I look for any anomaly in the data, any humon error (i. e. my error). Then I fix it. Then I run my analyses.
There are probably more steps than I’ve covered today. These are the first steps that absolutely must be done BEFORE you do any analyses. Then you have a good chance of keeping your data safe.