Evaluating the Effectiveness of Training & Development
Evaluating the Effectiveness of Training & Development
Guest writer Ron Wilson explores using the Kirkpatrick method as a way of evaluating the effectiveness of training & development.
Within the dealership the owner, CFO, Training Manager, Department Manager, and Supervisors all ask if the training and development are effective and worth the time and investment being provided. Many articles, lectures, and books have been written on this very topic. The following are some examples that have been applied utilizing the “Four Levels of Training Evaluation” utilizing the Kirkpatrick model.
The Kirkpatrick Model includes the evaluations into the following areas:
- Level 1 Reaction– How did the learning feel about the course, were they engaged, was the material relevant. This level can also be broken down to evaluate the course and the instructor.
- Level 2 Learning– This level measures the pre and post levels of knowledge utilizing a pre and post evaluation.
- Level 3 Behavior– Utilizing a pre-determined time to evaluate from the student’s and the supervisor’s view the level the behaviors are being utilized in the employee’s role. Are the tasks from the training class being utilized in the workplace?
- Level 4 Results– How has the training impacted the business? This can be a measure of customer satisfaction, amount of rework, reduced hours to rebuild a component.
Below are examples of how the information can be utilized and some comments on the implementation of the Kirkpatrick Model.
Reaction– This has also been called the “Smiley Face” survey. The survey is a recap of the student’s evaluation of the course material and the instructor. It is important to separate and review each individually.
The information below shows the responses from twenty-nine students relating to the course content and firsthand experience provided in the identified class. % E & VG combines the Excellent and Very Good score percentages. The comments provide valuable input to be collected and utilized for future improvements.
The information below shows a summary of the evaluation relating to the instructor leading the class and the comments provided. It is not uncommon to receive contradictory comments from a class. The class may have a wide variety of skill levels within the students attending the class.
Over time this information can be extremely helpful in evaluating the content and the individual instructors.
Learning- The level of learning obtained during the class provides a view of the knowledge level the student came into the class and a view of the knowledge the student is leaving the class. This is accomplished with a pre and post assessment. Below is an example:
Over time it can be determined that if a student came in with a low pre-test score the likelihood of not successfully completing the class. In this example 80% or higher is a passing score. The number of students passing, or not passing, the class can be a great tool for additional reviewing of content, instructor, and pre-requisites requirements.
Behavior– Level Three Evaluation reviews the application of the course material within the work environment based on input from the student and the supervisor. Utilizing a pre-determined time, the students and the supervisors evaluate if the behaviors are being utilized in the employee’s role. The evaluation is conducted by a survey of the students and the supervisors. These are specific tasks included in the class the student is expected to perform once returning to work.
Below is a list of tasks the student should be able to perform once returning to the shop area. The number of students and supervisors represents the ability of the students performing the tasks that was taught in the classroom. In this case there were more students that expressed their ability to perform the tasks as compared to the supervisors rating the employee’s performance of the task. Drilling down into the individual shops and supervisors, it can provide a little more detail of understanding the gap between the student’s score and their supervisor. Two items that may arise from the survey:
- A student may be less confident in their ability to perform the task as compared to the supervisor. We see this when the supervisor score is higher than the student’s score.
- The student may not have had the opportunity to perform the specific task since attending the class.
Results – The level of the results identified from attending the training class can be identified from expected business performance indicators such as:
- Measure of customer satisfaction
- Reduction of rework in the specific areas
- Reduced number of hours to rebuild a component.
All four of the levels require an elevated level of communication between the operating departments, the Training Department, and the students.
A couple of key points on the implementation of the Kirkpatrick Training Evaluation Model:
- There is a need for an LMS (Learning Management System) to record and analysis the information and data analysis skill set to establish the process, record, and report the results.
- The typical Training Department may not be able to evaluate “all” of the classes being provided (can be a data overload in levels 3 & 4). Start small and expand over time.
- This method can be utilized for all types of classes, soft skills as well as technical.
There are several resources to assist with understanding the model. James Kirkpatrick’s book “The Four Levels of Training Evaluation” is a great resource.