Level 3 Evaluation for eLearning: Behavior
In this series of articles about eLearning evaluation, I’m drawing upon Kirkpatrick’s Four-Levels Model to lay out the basics of what you should be accomplishing with your own efforts. Previous articles introduced the model (eLearning Evaluation: Kirkpatrick’s Four-Level Model), laid out more than 40 different sample questions across 7 different categories for Level 1 Evaluation (Level 1 Evaluation for eLearning: Reaction and More Questions for Level 1 eLearning Evaluation) to measure the audience reaction to your eLearning module or course, and gave you three best practices for Level 2 Evaluation of Learning (Level 2 Evaluation for eLearning: Learning). Now it’s time to tackle Level 3 Evaluation, which is all about behavior.
As I mentioned in the introductory article, the ultimate aim of most eLearning is to generate a change in people’s behavior, whether in terms of increasing the use of a particular skill or practice or what have you. Knowledge gained is useless unless it is applied in your people’s daily work lives. Oddly enough, it is at Level 3 that most eLearning evaluation falters, if it is engaged at all. This makes no sense if you want to really find out how effective your eLearning efforts are in making lasting changes that count.
Download the free white paper How You Can Assess The Effectiveness of Your Training- Kirkpatrick Model
Remember that the sequencing of the levels is important here. If you measure behavior and don’t find a change, that doesn’t mean your students didn’t learn anything or that the eLearning was a complete failure. You should already have designed and used your evaluations for the first and second levels, so you know if the audience liked the eLearning course or module, if they found it useful, and what they learned. Just keep in mind that there are things that can get in the way of the change in behavior you’d like to see. Sometimes conditions simply aren’t favorable, such as a reluctant supervisor, and that needs to be taken into account. You also need to allow sufficient time for a behavior change to take place before you try to measure it. Obviously, this level of evaluation is going to be very specific to the content of the eLearning course or module being evaluated. In general, however, here are some methods to employ:
Observation
In some instances, you may be able to gauge behavior changes by simply observing how course participants engage their work. This is a distinct possibility in smaller companies, where it’s easier to observe without much effort. In larger companies, it will be more of a challenge, but if you enlist managers and supervisors in the observations and recording of the data, you can conceivably find out a lot.
Self-Report Surveys
Another way to measure behavior change is to administer surveys from time to time to the people who took the eLearning course or module. You would have to keep in mind that such self-reporting can be problematic when people don’t want to admit they haven’t applied what they learned. However, you can also make this anonymous in order to encourage open and honest self-reporting, as well as give them ample opportunity to identify any organizational or managerial barriers that prevent them from applying what they learned.
Hybrid Approaches
It might even make the most sense to do both observation and self-reporting. It can be very enlightening to have these multiple perspectives and to ponder any differences that might emerge as well.
One of the problems that gets in the way of Level 3 eLearning evaluation is that whereas the first levels generate data that is typically held within the learning management system (LMS), the Level 3 evaluation data is typically generated by a performance management system, which is in turn typically “owned” and managed by a different department in the organization. That’s why Level 3 evaluation sometimes doesn’t happen at all. In larger organizations, you may need to get your learning and performance management departments to work together more closely to come up with a system that will work for Level 3 eLearning evaluation.
Once you tackle these substantial issues often encountered in Level 3 evaluation, then you’re ready for the final stage of the Kirkpatrick model, which is Level 4 evaluation focused on results, which is the topic of the final article in this series on eLearning evaluation.