Applied, programmatic techniques for evaluating educational/training programs and projects. Units focus on planning, client relations, problem analysis, data collection and analysis, reporting, evaluation management, and ethics.
When I first reached out to the IDD&E staff about enrolling in the M.S. program, I mentioned that I had an interest in outcomes assessment as an eventual research goal and possible subject for doctoral work. I still have that interest, but the first thing one learns in IDE 641 is that evaluation is a different concept from assessment. Tessmer (1993) brings a useful definition to the concept of evaluation, specifically formative evaluation: “. …a data gathering process to determine the worth or value of the instruction, of its strengths and weaknesses.” (p. 11) As such, formative evaluation is a vital part of the instructional design and development process, the last step in the ADDIE paradigm. One cannot accurately gauge the quality of a designed instructional event without testing it out on both end users and content experts and incorporating their suggestions for improvement. Happily, in my consulting experiences formative evaluation was indeed part of the ADDIE process, just not labeled as such. It was often referred to as “peer review” and “subject matter expert review.”
Tessmer also describes summative evaluation as the process of determining the worth or merit of an instructional event that is fully formed—materials in their “adult” development stage (p. 12). This, perhaps bizarrely, makes me think of the metaphor of an insect developing. If summative applies to the adult, then formative applies to the larva. And as with an insect’s development, one usually knows how long one has to perform the formative evaluation and implement the suggested improvements, since educational projects are often if not always conducted on a finite schedule. In the projects I’ve been on, we know when system go-live is, we know when end user training will be conducted, and the remaining time is how long I have to conduct a successful formative evaluation.
A competent development process flow at the best projects I was involved with included at least one round of what I now know to be formative evaluation performed by a peer instructional designer, putting themselves in the shoes of an end learner; and at least one round performed by a subject matter expert, usually a super user on the client side. Sometimes we had two expert rounds. More than two can slow down the delivery process. On several projects, I performed the end-user evaluation role. I hope I was able to help my fellow ID consultants with my formative evaluations.
The weakest development processes at my past projects included no time for either user formative review or expert review. Happily, those don’t happen often.
This course, besides giving us the opportunity to conduct a formative evaluation with real users and real experts, and an opportunity to create but not execute a summative evaluation plan on the same learning, also gave us ample practice in the fine art of writing good evaluation questions. In some of my past projects, I’ve been tasked with writing summative evaluation questions on a Kirkpatrick (1954) level 1 basis. This class’s assignments gave me the chance to expand my skills in writing evaluation questions using criteria I had sometimes been using without knowing it, but also criteria I did not know about:
- Avoid asking yes/no questions if at all possible. Yes/no questions give the answerer an opportunity to just say “yes” or “no” without elaborating on why they feel that way.
- Leverage the phrase “To what extent…?” or “To what degree…?” to get the subject to truly ponder how the instruction affected them.
- Try to include an even number of rating choices for each question. This prevents a user from just giving “lukewarm” middle-of-the-road responses for any or all questions.
- Measure only one variable at a time with a question. For example, “To what extent will the tools and techniques you practiced in this class help you on the job?” should be broken into two separate questions, one about the tools and the other about the techniques. This will also aid when doing any statistical or quantitative analysis when following up with further research.
Below is a collaborative project I accomplished with fellow M.S. students Pooja Gupta and MAJ Michael Whitted to perform a formative evaluation for remote training materials created by the U.S. Department of Defense’s (DoD’s) Defense Information School (DINFOS). Even before the COVID-19 pandemic, DINFOS made the organizational decision to offer what would normally be an ILT training experience in a remote format, as a cost-cutting measure. We evaluated training materials for the US Air Force (USAF) Specialty Code (AFSC) 3N052 Radio and Television Broadcasting (USAF, 1998). AFSCs indicate what an airman’s job is in the USAF. The 5 in this AFSC code indicates that enlisted personnel at an E-5 level (Staff Sergeant) are the target audience for the materials. Being an active duty Army officer, Michael was invaluable to Pooja and me in providing military perspective on tasks and command structure within a service unit. All three of us conducted live Zoom interviews with end users of and an expert on the material. Those transcripts are included in the formative evaluation report.
Special thanks to fellow M.S. student Erin Smith, USAF Reserve, for conceiving of the project and providing us with all the training materials we needed to execute our projects.
Kirkpatrick, D. L. (1954). Evaluating Human Relations Programs for Industrial Foremen and Supervisors. (n.p.): University of Wisconsin–Madison.
Tessmer, Martin, (1993). Planning and Conducting Formative Evaluations (1st ed.). Routledge. https://doi.org/10.4324/9780203061978
United States Air Force. (1998, April). Occupational Survey Report. United States Air Force, United States Department of Defense. https://apps.dtic.mil/sti/pdfs/ADA348796.pdf
Final Grade: A
Beth's Journey Through the Instructional Design Universe 