Instructional Design for Improved Training Performance

Scenario: As your department training officer, you have noticed that performance at emergency scenes has not been up to standard. As a result, you put together a quick training program to address the performance gaps, but the solution hasn’t been as successful as you had hoped and now the Chief is questioning your capabilities and performance.

 

Introduction

Training firefighters, Emergency Medical Services, and other emergency services personnel is an expensive endeavor. Buric (2016) estimates that an organization, whether volunteer or career, will spend over $250,000 training a firefighter over the course of their career. Those who are outside the training and education discipline remain blissfully unaware of the process of instructional design and delivery. Instructional Systems Development (ISD) is a system designed to improve training and educational development and improve operational performance. Also, known as Training Systems Development, Systems Approach to Training, or Criterion Referenced Training, ISD is a “process based and following a series of steps or guidelines…shaped by the overriding idea that instruction in all formats must be consistent, reliable, and effective to facilitate learning” (Nichols-Hess and Greer, 2016). The ISD process is focused on leaner needs in it approach to instruction as it strives to support successful learning outcomes and is applicable to any discipline.  While there are several offshoots of ISD models, the ADDIE instructional design framework that is most frequently used. “The use of ADDIE can provide structure for developing instructional materials and student engagement, ensure that learning assessment is more accurate, and effectively guides the learning process” (Nichols-Hess and Greer, 2016). As Chuck Hodel points out, “ADDIE has become the gold standard in ISD models because at its very core, it reflects the five most common elements of the ISD process-analysis, design, development, implementation, and evaluation” (2016). 

The ADDIE model consists of five, interrelated and interconnected steps. These are:

Analysis, which includes gathering data about content, populations, delivery systems, methods, budgets, actual and desired performance, the impact and need of the training.

Design, during which the instructional designer will build a blueprint of the training course and develop project specifications, objectives, course content, and develop a design plan.

Development is the phase where educational materials are produced, pilot testing occurs, and draft design products transition to a deliverable educational product.

Implementation involves the delivery and evaluation of the training design product.

Evaluation occurs throughout the process, where each step is evaluated for success.

  

  ADDIE-Model.png

Figure 1: The ADDIE Model

 

The five phases of ADDIE may overlap or occur simultaneously and provide a dynamic and flexible template for developing effective and efficient instruction. By using this systems approach, training can be developed using a logical process that factors in constraints such as time and money; it ensures linkages between course materials, training activities, and performance benchmarks; it focuses training efforts and the actual knowledge and skills needed to successfully perform on the job; and products quality results that are supported by evaluations, revision schedules, and job performance improvement. 

Analysis Phase

The first step of the ADDDIE process is the analysis phase. Entry into this phase is triggered by recognizing an instructional issue that examines the target audience and their characteristics. In essence, asking and answering the question, “who can’t do what, and why?” According to Chuck Hodell (2016), “analysis is the singl3 most important building block in the practice of instructional design. Without it, y9ou have little in the way of information and data upon which to build a unit of instruction, or even to decide if training or designing a course is the best way to address a perceived training need.” 

The analysis phase consists of a series of steps designed to identify that there is a performance gap and that the gap is caused by training, describe the target audience and what the current level of performance is, and assess the resources, constraints, and goals of the organization with relation to the performance gap. The analysis phase should also describe what mastery of the performance should look like and how it will be evaluated. 

Analysis

Type

Tools to assist with conducting

Goal

Needs

Surveys, interviews, and observation of performance

Identify areas where training may be needed

Problem

Surveys, interviews, and observation of performance

Isolate and identify the cause of the problem

Goals

Surveys and interviews

Identify the goals of those interested in the training program

Population

Surveys and interviews

Identify characteristics such education and experience level

Resource

Surveys, observations, and interviews

Identify what is available and useful to the training program

Constraints

Surveys, observations, and interviews

Identify what limitation will impact the design, development, and delivery of the program, including money, time, and organizational support

Job

Surveys, observations, and interviews

Identify duties and responsibilities of performance

Task

Surveys, observations, and interviews

Identify and describe the distinct tasks and what is needed to perform each task

Delivery

Surveys and interviews

Describe how training will be delivered

Mastery

Surveys, observations, and interviews

Describes how mastery of performance will be evaluated

Revision

Surveys, observations, and interviews

Determines how often content and training need to be offered and revised

 

Table 1: Types of Analysis, Adapted from Old Dominion University (2001) and Hodell (2016)

Conducting a thorough analysis of the issue will allow decision makers in making informed decision with regards to supporting the training decision and the organization will ultimately save money by focusing its resources on a specific solution to a well-defined problem. By framing the training need into actionable items, the analysis helps guide the training strategies which helps to identify resource requirements, budget needs, and define expected outcomes for the organization. Remember that the first tasks are to identify that there is a need, determine the root cause of the problem, and determine that training is the most effective solution to remedy the issue.

 

Causes of Performance Gaps

Knowledge

Skills

Motivation

Habit

Environment

Communications

Task-Ability Misalignment

Staffing

Incentive/Disincentive

 

Table 2: Causes of Performance Failure

Design Phase

Once the performance issue has been described and the solution determined to be training, it is time to design the training program. Keeping in mind that the goal of training isn’t just the transfer of knowledge, but its application. As Hodell states, “having information doesn’t accomplish anything by itself. Something is accomplished when the learner uses that information to do things” (2016).  During the design phase, a design plan will be developed that includes:

  • Identifying the reason for training
  • Coordinating analysis, development, and evaluation processes
  • Identifying and describing the learning objectives
  • Defining the target audience
  • Describing any prerequisite knowledge or skills needed for successful performance by both participants and instructors
  • Describe the educational methodology to be used
  • Determine the sequence of content delivery and training strategies
  • Determine the training methods to be used
  • Describing how mastery of content will be evaluated
  • Describe the training media, materials, resources, and facilities needed to conduct successful training
  • Identifying what will be developed for the program, such as instructor manuals, student guides, etc.
  • Managing quality control of all processes and products

Ultimately, the design phase should lead to a design product that will result in knowledge and skill transfer. Two of the critical elements in this process are the development of learning objectives and learning strategies based that support the learning objectives. Learning objectives describe what performance should look like at the end of the training period. Objectives focus on key points, prepare the learner for what is expected, and describes the acceptable level of performance expected. The design document will develop four types of objectives: focusing objectives that draw attention to the most important parts of the learning module, performance objectives based on expected competency demonstration, instructional design objectives to guide the design and development of the lesson, and instructor evaluation objectives to assess the quality of instruction.

 

Types of Objectives

Type

Purpose

Focusing

These are provided to the learner before/with the learning material to guide their attention to the most important parts of the instructional content

Performance

Also given before/with the learning material to guide the learner’s attention towards the competencies they will be performing

Instructional Design

Helps instructional designers in the design and development process by guiding their focus to the objectives, evaluation, and design processes

Instructional Evaluation

Guides the evaluation of instructional performance

 

Table 3: Types of Objectives

Next, instructional strategies and the sequencing of information will be addressed in the design process. The instructional strategies selected should support the learning objectives (performance and focusing) and evaluation methods. The evaluation methods are directly tied to the learning objectives as well as the content. There are a variety of instructional methods that can be used, including:

  • Lecture
  • Role-play
  • Critique
  • Discussion
  • Case study
  • In-basket/out-basket
  • Simulations
  • On-the-job training
  • Gaming
  • Brainstorming
  • Critical incident review
  • Drill
  • Job aids

During the design phase, the methods of instruction will be selected and sequencing strategy to support the learning strategy identified.  Common sequencing strategies include providing step-by-step directions, moving from general knowledge to detailed information, or moving from the known to the unknown in order to bridge knowledge gaps and provide scaffolding for learning.  It is important to utilize the information developed from the population analysis as the level of knowledge or experience with the topic area will guide the level of support that is provided to the learner.

 

Level of Support to Provide based on Target Audience Background

Novice

Proficient

Lots of guidance

Introduction to new material

Provide high level organization (for example topic areas & outlines)

Provide visuals, examples, appropriate stories and metaphors

Structure

Achievable goals

Methods to improve confidence

Gradually increasing complexity or difficulty

Coaching &Feedback

Practice for new concepts

Build on known knowledge and skills

Advanced information or techniques

Coaching

Autonomy

 

Table 4: How to Support Learning Based on Learner Background & Experience

 

Finally, the evaluation strategy should be linked to the learning objectives and learning strategies. Activities and knowledge gain should be measurable and reflect actual job-related skills and knowledge. In addition, resources to support realistic evaluation should be available to support the process of ensuring content mastery. 

 

Development Phase

The development phase is focused on creating, purchasing, or leasing draft materials to support the lesson. When determining whether to produce rather than purchase course materials, the developer should consider:

  • Cost
  • Deadlines
  • Agreements
  • Samples of work
  • How final materials will be approved and by who
  • Impact on pilot testing
  • Adequacy of materials to meet objectives based on identified gaps

Facilities will need to be assessed to ensure that they meet the needs identified in the design document. This may include adequacy of training spaces such as classrooms, drafting facilities, and training props. Instructors will also be identified based on the criteria established in the design document. Once they have been identified, they will need to be trained to deliver the course content. All of the course materials will be developed, and facility, equipment, and resource needs will be addressed. Training materials may include:

  • Instructors Guide
  • Lesson Plans
  • Student Manual
  • Train the Trainer materials
  • Textbook
  • Guidebooks
  • Handouts
  • Non-print media such as software, mockups, models, and videos
  • Evaluation materials

Other materials produced may include:

  • Participant Recording System (attendance, evaluations, training record)
  • Instructor qualifications
  • Instructor lists
  • Course versioning
  • Material authorship/development team

Course developers may want to include a train-the-trainer element as part of the development plan. A train-the-trainer should be considered when

  • The course includes new or unfamiliar material
  • New or unfamiliar delivery systems
  • Instructor skill levels are in question
  • New or unfamiliar technology will be used
  • The course content requires a train-the-trainer component
  • License or certification requirements are tied to the training

An evaluation plan will be fully developed, based on the design document and when and how to evaluate learning, and may include scripting what is told to students about the evaluation process. Evaluation documents for learners, instructors, and support personnel. 

All Lesson Plans Should Include (Gagne’s Nine Levels of Learning)

 Gaining the attention of the learner

Direction (what to pay attention to and how to do tasks)

Recall of previously learned material

Lesson content

Application of materials and feedback #1

Application of materials and feedback #2

Application of materials and feedback #3

Evaluation

Closure

 

Table 5: Applying Gagne’s Nine Levels of Learning to Lesson Plan Development

Also, during the development phase, the materials will be delivered on a pilot basis, with modifications based on evaluation of all elements of the process and delivery. Pilot testing a learning module allows developers the opportunity to evaluate it before the project is fully implemented and is critical to successful course development. Pilot testing answers the following questions:

  • Does the lesson plan work?
  • Are the directions for the facilitator clear and concise?
  • Are the learner’s materials appropriate and thorough?
  • Are the facilitators materials appropriate and thorough?
  • Are the support materials what you expect?
  • Does the timing match estimates?
  • Are technology elements appropriate?
  • Are instructional methods appropriate?
  • What does/does not work?
  • What should be changed?

Hodell (2016)

Materials and delivery may be modified as needed based on the pilot evaluation process. Finally, there may be several pilot offerings, including post-revision pilots.

 

Implementation Phase

At the implementation phase, the course materials have been developed and tested and are ready for full implementation. At this time, the delivery and evaluation plans will be fully implemented with the training being delivered to the target audience as designed and evaluated against the criteria established in the evaluation plan. Evaluations of the course should include at least level 1 and 2 evaluations with level 3 and 4 evaluations preferred. Any further adjustments to the program will be based on the outcomes of the evaluations. 

Kirkpatrick’s Four Levels of Training Evaluation

Evaluation Level and Type

Description

Examples

Relevance

Type 1

Reaction

How participants felt about the training

Surveys, questionnaires, interviews to identify subjective feelings about course and environment during or immediately following the course.  Questions include:

Was the training relevant?

Did the training meeting your expectations?

Easy to do

Of limited value as participants may not be able to adequately judge their level of mastery.

Little relationship between student reaction and achieving mastery of content.

 

Type 2

Learning

Evaluation of learning during or immediately following training

Tests, observations, and similar end of course activities

Tends to be relatively simple to administer but may be difficult to evaluate complex topics.

Relevant for clear cut or technical topics.

Directly tied to the learning objectives.

Type 3

Behavior change

Assesses if the learning resulted in long-term behavior change on the job (was the material applied in the future)

Observation and supervisory feedback

Hard to quantify.

Requires participation and reinforcement of training concepts once participant leaves the training environment. Can be undone by “that may be what you learned in training, but in the real world, we do it like this”.

Type 4

Results

Assesses the impact of the training on the organization and community.  Return on training investment

Often seen in performance evaluation.

Ultimate measure of success is a positive return on investment for training delivered

Easier to evaluate as an individual v. larger training initiative with many participants.  Subject to external influences.

 

Table 6: Kirkpatrick’s Four Levels of Evaluation, North Dakota State University (2021), Hodell (2016)

Evaluation Phase

Evaluation is conducted during each phase of the ADDIE process, beginning during the analysis phase when determining if the performance issue is related to training and comparing performance against established benchmarks, through the design and development phases and finally during implementation of training. By evaluating continually throughout the process, quality training can be achieved and mastery of content, resulting in improved performance and elimination of the performance gap can be achieved. The feedback received during the ADDIE process allows for revision of materials linked with evaluation of content mastery, improving knowledge and skill transfer resulting in improved on-the-job performance, reducing extraneous costs, and improving return in investment for the organization.

Evaluation During the ADDIE Process

Evaluation during analysis

Is this a training issue?

Have the data been reviewed by

  • Stakeholders
  • SME’s
  • Target audience
  • Designers

Have findings been compared against benchmarks?

Evaluation during design

Has the design plan been reviewed by SME’s?

Has the design plan been reviewed by another designer?

Have the objectives been reviewed by SME’s and other designers?

Has the evaluation strategy and materials been reviewed by SME’s and designers?

Have the draft student and instructor materials been reviewed by SME’s and designers?

Have the media been reviewed by SME’s and designers?

Evaluation of objectives

ABCD have been addressed (audience, behavior, condition, & degree)?

Is the degree of difficulty appropriate?

Is there agreement between behavior and condition?

Evaluation during development

Review the evaluation plan

Review segment timing and compare estimates to pilot

Review all materials

Is there clarity of structure?

Evaluation during implementation

Learner

  • Reaction
  • Mastery
  • Value of training
  • Investment in change
  • Training reflects reality

Facilitator

  • Reaction
  • Usefulness of instructor materials
  • Content meets need
  • Course design

Organization

  • Learner mastery
  • Effectiveness
  • Training transitions to work performance

Instructional designer

  • Objectives are met
  • Content meets need
  • Materials of good quality
  • Participants, facilitators, and organizational needs met
  • Training reflects reality and need

Table 7: Evaluation during the ADDIE process

 

Objective Evaluation Job Adi

Objective

Write Objective Here

Behavior

Match / No Match

Condition

Match / No Match

Evaluation Task

Match / No Match

Learning Domain

Psychomotor / Cognitive / Affective

Job Aid 1: Objective Evaluation Job Aid

Conclusion

While instruction design may seem to be a cumbersome process, many departments currently apply training to solve all performance problems, even when training is not appropriate, resulting in lost money, time, and effort. By utilizing an instructional design process, such as ADDIE, training officers, company officers, and others who are responsible for the training of current and new responders can ensure that training is effective and efficient, resulting in reduced costs, improved performance, and better outcomes for both members of our organizations and the communities we serve.

 

 

References

Buric, A. (2016). Essentials of Great Instructional Design.  Retrieved from www.learnupon.com/blog/essentials-great-instructional-design.

Dirksen, J. (2016). Design for How People Learn, 2nd ed. New Riders. San Francisco: CA.

Hodell, C. (2016). ISD From the Ground Up.  A No-nonsense Approach to Instructional Design, 4th ed. ATD Press. Alexandria: VA.

McGriff, S. (2020). Instructional Systems Design (ISD): Using the ADDIE Model. College of Education: Penn State University.

Nichols-Hess, A. & Greer K. (2016). Design for Engagement: Using the ADDIE Model to Integrate High-Impact Practices Into An Online Information Literacy Course. Communications in Information Literacy, v 10#2, p. 264-282.

North Dakota State University. (2021). Kirkpatrick's Four Levels of Training Evaluation in Detail. Retrieved from https://www.ag.ndsu.edu/evaluation/documents/kirkparicks-four-levels-of-training-evaluation-in-detail

Northern Illinois University Center for Innovative Teaching and Learning. (2020). Gagné’s nine events of instruction. In Instructional guide for university faculty and teaching assistants. Retrieved from https://www.niu.edu/citl/resources/guides/instructional-guide

Old Dominion University. (2001). Instructional Systems Development (ISD). Occupational and Technical Studies. Old Dominion University. Norfolk: VA.

 

About the Author

Dave Donohue has over 40 years of fire, EMS, and emergency services experience, been involved in emergency services training since 1984, and has served as a responder at the local and federal levels in Florida, Maryland, Virginia, West Virginia, Pennsylvania, and Washington, DC.  He is the owner of Mid Atlantic Emergency and Safety Consulting, LLC.  He resides outside of Hagerstown MD, where he is a member of the Community Volunteer Fire Company of District 12 in Fairplay, serves as a member of the Maryland Wing of the Civil Air Patrol Hagerstown Squadron, is an instructor for the Maryland Fire Rescue Institute, works for a fire & EMS-based education center in Emmitsburg Maryland, and is looking forward to the return of minor league baseball to Hagerstown in 2022.  He can be reached at dkdonohue@aol.com

0 Comments
Recent Stories
Training with Anthony Avillo

Instructional Design for Improved Training Performance

Prevention vs. Suppression: Where do your priorities stand?