How to determine the effect of learning
Our projects as examples
Does learning contribute to personal development? And to an organisation’s success and financial performance? One of the tools we use at TinQwise is the ‘Pyramid of Learning and Performing’ by Donald Kirkpatrick and Jack Phillips.
Why measure the ROI?
Because the organisation will be more engaged in and more supportive of the training programme. By calculating the return on investment, it will be easier to sell the learning intervention within your organisation. The ROI shows how much will be invested and what value will be created, either in terms of money or in results that can’t be expressed in money.
The Pyramid of Learning and Performing
How to use this pyramid?
The pyramid has five different levels. Determine your goals for each level. The ultimate goal is that you increase the learning ROI of your organisation.
Learning reaction and satisfaction (level 1)
Are you satisfied with the training course? Did you find it interesting?
This is often measured when the training programme or the online learning has been completed. You ask questions such as: Can you apply this new knowledge in your daily job? Would you recommend this training course to your colleagues? Which overall grade would you give this training course?
Learning evaluation and self-confidence (level 2)
Did you learn something?
This is where we measure whether employees feel confident in doing their job and whether they enjoy doing their job? For example, for an international retail client we developed a special report page for their onboarding process. We measure, for example, how many employees are on track with their online training, how many employees have completed the course within the time available and what their favourite topics are. What’s more, employees can give grades out of 10 for various aspects of the training course. They need to answer several questions at various stages in the learning process. For example: How likely is it that you will recommend this company as an employer to other people? Are you enjoying your onboarding process? How do you rate the guidance your buddy gives you?
Learning application and implementation (level 3)
Do you apply the training in real situations at work?
We developed a blended learning programme about the new planning system for one of our clients. The goals of the training was: make sure that jobs are planned more efficiently, that there is more time for ad hoc jobs and that employees are doing their work more efficiently.
Every team involved in the training started off with an intake and a baseline measurement. The following information was established: how many jobs are done per day; what is the average time needed to do a certain type of job; what is the average waiting time for ad hoc jobs, etc. etc. After that, the training programme started. The programme consisted of online learning, group classes and on-the-job coaching. By doing mid-term evaluations and on-the-job coaching, it soon became clear that the course participants were indeed applying the training in real situations at work.
During and after the training the same indicators were measured. What we saw was that people need time to get used to a new planning system at first. Some mistakes are made at the beginning. But soon there is a positive trend, which levels out after a while at a higher level than before the training. Employees are now doing more jobs in less time and they are more productive during their working hours. This means that the training has achieved a concrete goal.
Impact and result (level 4)
What impact does the training have on the organisation’s results?
Depending on the goal, you can measure job satisfaction of employees or customer satisfaction for a period of 6 months. Or you can measure absence due to sickness or brand awareness for your organisation, just to mention a few options.
At Sonos, for example, we organised a ‘train-the-trainer’. We trained the company’s own retail specialists. To make sure that they could train 1500 Sonos sales people at a major Sonos event. A few months after the Sonos Experience the company measured the brand recommendations for Sonos in shops. These recommendations had increased by 20% compared to the previous quarter.”
Return on Investment – ROI (level 5)
Does the value created for the company justify the investment made in the training?
This is about the balance between the investment and the value created.
ROI = value created / investment x 100
For example, we arranged an intensive blended learning programme for the sales staff of one of our clients. The goals that we wanted to measure were: NPS (Net Promoter Score: customer loyalty) and the conversion rate of new contracts signed.
Both the organisation’s customer loyalty and the number of new contracts had increased after the training. This means financial gains for the organisation. Using the formula above, we can calculate the value created for the company based on the investment made in training.
This client would like to take the next step and measure the duration of the new contracts. Do customers only say that they are more satisfied or do they actually stay loyal for longer periods of time? The company is now collecting information about this goal.
Measuring at all 5 levels is not always useful
Of course it is not always necessary to measure the effect of a training course at all five levels. For example, a brief one-day learning intervention does not need to be evaluated up to level 5. Obviously, the more extensive the measuring, the more time it takes and the more expensive it will be.
Taking the next step
Many of our clients are now measuring at levels 2 and 3. It is up to the developers of learning interventions to challenge their clients to measure one more level: level 4 at organisation level. By measuring at this level you can change training programmes that are not effective or discontinue them. Measuring will increase job satisfaction of your employees. And it will actively involve managers in the learning process and ensure better financial performance. Surely these are enough advantages to take measuring seriously?
Would you like to know more?
Would you like to create more value with your training programmes? We would love to talk to you about learning solutions, about the goals you would like to achieve and about measuring the effect of online learning.
Please contact: Oscar van der Horst, account manager. Email: firstname.lastname@example.org, Tel: +31 (0)6 86 810192.
Four levels of learning evaluation, Donald Kirkpatrick (Wikipedia)
The bottom line on ROI, how to measure the ROI in Learning and Development, Performance Improvement, and Human Resource, J. Philips
Jack Philips:The ROI Institute