- About Us
- Statewide Resources
- Get Involved
- For Employees
- Find Us
Evaluation is an Everyday Activity
Program Evaluation Discussions
Updated: 12 hours 25 min ago
Not knowing may be “easier”; you know, less confusing, less intimidating, less fearful, less embarrassing.
I remember when I first asked the question, “Is it easier not knowing?” What I was asking was “By choosing to not know, did I really make a choice, or was it a default position?” Because if you consciously avoid knowing, do you really not know or are you just ignoring the obvious. Perhaps it goes back to the saying common on social media today: “Great people talk about ideas; average people talk about things; small people talk about other people” (which is a variation of what Elanor Roosevelt said).
Critical thinking (no, not negative thinking; reflective, thoughtful thinking) enables knowing. Talking about ideas allows people to think reflectively, to think thoughtfully; to know. Talking about people cuts off thinking about ideas; allows individuals to “not know”. And in that case, not knowing is seems easier; less effort. Perhaps, you say, that the people don’t know that they don’t know. That, too, is a choice. I am reminded of a thought that was stated with regard to “white privilege”: “…it is (emphasis original) your fault if you don’t maintain awareness of that fact” (referring to the fact that white people do benefit from white skin and therefore privilege). I’m not saying that not knowing (ignorance) is a form of white privilege; I am saying that there are similarities, that it is your fault to maintain ignorance, that there are choices.
And yes, this does relate to evaluation. What choices have you made recently that have kept you ignorant. Is it just too much trouble to…(fill in the blank)? I know there are days where I default to the not knowing (ignorance) position. (My philosophy teacher once told me that there are only three choices: to agree to do (know), to agree to not do (not knowing), and agree to not decide (default).) It takes too much work; I’ve too many other things that need to be done yesterday. OR, I am afraid of knowing. That is a choice.
I realized recently that I will encourage people to look at self-confidence/self-efficacy and to measure an individual’s intention to change as a way to identify outcomes. Yet there are other approaches to get to outcomes. I happen to believe that Mazmanian (1998*) did identify something important (intention to change) when evaluating programs (even though he was talking about continuing medical education). That is almost 20 years ago. Is it still relevant? (Don’t know.) It is still useful? (Yes.) Does it still help the evaluator get closer to condition change? (Yes.)
*Mazmanian, P. E., Daffron, S. R., Johnson, R. E., Davis, D. A., & Kantrowitz, M. P. (1998). Information about the barriers to planned change: A randomized controlled trial involving continuing medical education lectures and commitment to change. Academic Medicine 73(8), 882-886.
The use of a survey is a valuable evaluation tool, especially in the world of electronic media. The survey allows individuals to gather data (both qualitative and quantitative) easily and relatively inexpensively. When I want information about surveys, I turn to the 4th edition of the Dillman book (Dillman, Smyth, & Christian, 2014*). Dillman has advocated the “Tailored Design Method” for a long time. (I first became aware of his method, which he called “Total Design Method,” in his 1978 first edition, a thin, 320 page volume [as opposed to the 509 page fourth edition].)
Today I want to talk about the “Tailored Design” method (originally known as total design method).
In the 4th edition, Dillman et al. say that “…in order to minimize total survey error, surveyors have to customize or tailor their survey designs to their particular situations.” They are quick to point out (through various examples) that the same procedures won’t work for all surveys. The “Tailored Design Method” refers to the customizing survey procedures for each separate survey. It is based upon the topic of the survey and the audience being surveyed as well as the resources available and the time-line in use. In his first edition, Dillman indicated that the TDM (Tailored Design Method) would produce a response rate of 75% for mail surveys and an 80%-90% response rate is possible for telephone surveys. Although I cannot easily find the same numbers in the 4th edition, I can provide an example (from the 4th edition on page 21-22) where the response rate is 77% after a combined contact of mail and email over one month time. They used five contacts of both hard and electronic copy.
This is impressive. (Most surveys I and others I work with conduct have a response rate less than 50%.) Dillman et al. indicate that there are three fundamental considerations in using the TDM. They are:
- Reducing four sources of survey error–coverage, sampling, nonresponse, and measurement;
- Developing a set of survey procedures that interact and work together to encourage all sample members to respond; and
- Taking into consideration elements such as survey sponsorship, nature of survey population, and the content of the survey questions.
The use of a social exchange perspective suggests that respondent behavior is motivated by the return that behavior is expected, and usually does, bring. This perspective affects the decisions made regarding coverage and sampling, the way questions are written and questionnaires are constructed, and determines how contacts will produce the intended sample.
If you don’t have a copy of this book (yes, there are other survey books out there) on your desk, get one! It is well worth the cost ($95.00, Wiley; $79.42, Amazon).
* Dillman, D. A., Smyth, J. D. & Christian, L. M. (2014) Internet, phone, mail, and mixed-mode surveys: The tailored design method (4th ed.). Hoboken, N. J.: John Wiley & Sons, Inc.
Went looking for who said that originally so that I could give credit. Found this as the closest saying: “Destiny is no matter of chance. It is a matter of choice: It is not a thing to be waited for, it is a thing to be achieved.
William Jennings Bryan
Evaluation is like destiny. There are many choices to make. How do you choose? What do you choose?
Would you listen to the dictates of the Principal Investigator even if you know there are other, perhaps better, ways to evaluate the program?
What about collecting data? Are you collecting it because it would be “nice”? OR are you collecting it because you will use the data to answer a question?
What tools do you use to make your choices? What resources do you use?
I’m really curious. It is summer and although I have a list (long to be sure) of reading, I wonder what else is out there, specifically relating to making choices? (And yes, I could use my search engine; I’d rather hear from my readers!)
Let me know. PLEASE!
Erma Bombeck said “You have to love a nation that celebrates its independence every July 4th not with a parade of guns, tanks, and soldiers, who file by the White House in a show of strength and muscle, but with family picnics, where kids throw frisbees, potato salad gets iffy, and the flies die from happiness. You may think you’ve overeaten, but its patriotism.”
I heard this quote on my way back from Sunriver, OR on Splendid Table, an American Public Media show I don’t get to listen to very often and has wonderful tidbits of information, not necessarily evaluative. Since I had just celebrated July 4th, this quote was most apropos! I also heard snippets of a broadcast (probably on NPR) that talked about patriotism/being patriotic. For me, tradition is patriotic. You know blueberry pie on the 4th of July; potato salad; pasta; and of course, fireworks (unless the fire danger is extreme [like it was in Sunriver] and then all you can hope is that people will be VERY VERY careful!
So what do you think makes for patriotism? What do you do to be patriotic? Certainly, for me, it wouldn’t be 4th of July without blueberry pie and my “redwhiteblue” t-shirt. I don’t need fireworks or potato salad… What makes this celebratory for me is the fact that I am assured freedom from want, freedom of worship, freedom from fear, and freedom of speech and I realize that they are only as free as I make them.
Franklin Delano Roosevelt said it clearly in his speech to congress, January 6, 1941: “In the future days, which we seek to make secure, we look forward to a world founded upon four essential human freedoms.
The first is freedom of speech and expression — everywhere in the world.
The second is freedom of every person to worship God in his (sic) own way — everywhere in the world.
The third is freedom from want — which, translated into world terms, means economic understandings which will secure to every nation a healthy peacetime life for its inhabitants — everywhere in the world.
The fourth is freedom from fear — which, translated into world terms, means a world-wide reduction of armaments to such a point and in such a thorough fashion that no nation will be in a position to commit an act of physical aggression against any neighbor– anywhere in the world…”
This is an exercise in evaluative thinking. What do you think (about patriotism)? What criteria do you use to think this?