Measuring much more than just learner satisfaction

How do you measure whether a training course is effective? At TinQwise we develop learning solutions with the best possible return on investment (ROI). A lot of time and effort goes into studying the target audience, the educational approaches, the experience of learning and technology. But it’s just as important to measure what a training course or learning campaign actually achieves. Only then are you able to really focus on the outcome of a learning intervention. Account manager Marguerite de Keijzer talks about the ROI of online learning.

Marguerite de Keijzer, TinQwise

Why is the return on investment so important?

“Our clients know what the advantages of online learning are: online learning saves time and money, people are learning in a flexible way tailored to their needs, the learning solution is scalable and it can enhance learning retention. However, the initial costs of an online learning campaign can seem relatively high. Which may be a reason for management not to invest in this option.

Decision-makers know what the advantages are but still they wonder: is it really worth investing in? By calculating the ROI of online learning, you will have a much clearer picture. And you will know whether it pays to invest time and money in online learning. Find out how to make the most of this investment!

What do companies measure when it comes to online training courses?

“Many companies only measure if course participants are happy about a training course. Or they measure how many employees were trained within a certain period of time. These things are measured afterwards. Occasionally, companies do a baseline measurement beforehand and a mid-term measurement during the training course. The advantage of this is that you can actually measure the progress that course participants are making and you can adjust the training course if necessary.”

How do you start measuring?

“First you need to be aware of the fact that you can measure various things. We call it measuring soft skills and hard skills. Soft skills are: branding, emotions, how can you do your job better? Hard skills are: do your sales increase? You can measure hard skills very accurately. Improving these skills often means making more money. Soft skills are part of the hard skills. They are harder to measure in a quantitative way. However, improving soft skills can often lead to financial gains too.”

How to decide what to measure?

“By figuring out what you want to measure. You can use KPIs (key performance indicators). There are many data in your company or organisation: things you can measure, figures you have access to. But it’s all about a handful of metrics that really matter. These numbers are of critical importance for your business. And you decide what they are. You need KPIs to compare your company’s performance with your goals. By using metrics you can determine whether you have achieved your KPIs.

The measurements that most companies carry out are too limited. Or they simply focus on the wrong things. For example, they only look at how many employees have completed a particular training course. It’s good to know, but it doesn’t give you any information about whether you are achieving other goals too. By determining in advance which KPIs you would like to influence, you can measure whether a training course has actually had the desired effect.”

Can you give us an example?

“Imagine you are implementing a training course about ‘Bribery & Corruption’ to raise awareness, to get employees to talk about it, and to let your staff know where they can report any suspicious behaviour. If your training course is successful, the chances are that the number of reported incidents will go up. Because people are paying attention, they start talking about it and they know where to report it. After a while, the number of reported incidents will go down again to a normal level. By monitoring the number of reported incidents every month for an entire year, you are indirectly measuring the effect of the training program.”

Very often, organisations only measure learner satisfaction levels, isn’t that right?

“Yes, that happens a lot. Of course it’s important to measure how happy a course participant is about a training program, because they can say how good the course was. That’s how you can collect information about the format and method of the training course. And whether the course content was complete.

But we’re also interested in what the course participants are going to do with the new information. How do you really see the effect of the training course when you look at staff behaviour? After a good quality training course, employees think that they will be better at their job. But is this actually the case? That’s something you also want to know. And you don’t measure that with a satisfaction survey.”

So how can we measure that?

“First of all by realising that there is a gap between: ‘I think I’m better at my job now’ and ‘I am better at my job.’ A training course is only the beginning. After a course, employees will start to implement what they have learned. And you would like to know what they retain after a course ends. And what the effect is on their work. Learning isn’t something for the short term. It’s a process that never stops.

That’s why you should measure at different moments in time, for example a baseline measurement, some mid-term measurements and a final measurement. The baseline measurement is done before the start of the training course. During the training course there are several mid-term measurements. And then after a certain period of time you do a final measurement.

You collect both quantitative and qualitative data when you measure. Based on this data you decide whether a training course is sufficient, whether you would like to make any changes to the programme and whether you would like to teach your employees more.”

Is measuring now becoming standard practice?

“We often tell our clients about the value of measuring. We recommend it. But it is often not done due to budget constraints. Even though it is very important. Evaluating and changing training course programmes can mean better training results and thus making more money.

However, once you start measuring you also need to be prepared to take action. You should be prepared to do something with the data you collect, otherwise there isn’t much point in measuring in the first place. What’s more, you can also use the information from measurements to manage expectations of your clients and stakeholders.”

“Of course you know your target audience, but sometimes the employees you thought you knew, do things a bit different. Sometimes people learn more when you do things in a slightly different way. It is worth finding out and creating the option to adjust and achieve more.”

Is it essential to measure and adjust?

“It is very useful. But you must be aware that the effects you are measuring are not only due to online learning. Our learning solutions are often part of a much larger project and there are many contributing factors (such as current affairs or an internal marketing campaign) that we as learning solution developers can’t influence.

These factors can have both a positive and a negative impact. It’s good to know that the effect of a training programme is hardly ever down to the training programme alone.

It is essential to determine your starting point. Prior to the training course we decide on the boundaries, objectives and conditions. We look at existing figures and our expectations are realistic.”

What is the biggest challenge when measuring the ROI of online learning?

“Wanting to measure everything you can measure. That’s a real pitfall. You can quite easily drown in that gigantic pool of data. Please be sure that you will only measure what is really relevant for your company’s goals. And please focus. Decide on your KPIs and metrics beforehand. What would you like to measure and which goals are realistic? Which results are you actually going to use and do something with? Show your employees that you take them seriously by really doing something with the outcome.”

Can online learning fail?

“No, online learning always has a positive effect. Training employees is always worthwhile. You have invested time, gathered valuable content and actively engaged employees. That’s why it’s important to take the next step: Measuring the effect of a training course on the actual job performance of participants.

There is always a positive outcome, whether they perform 6% better or sell 5% more or have more attention for each other which will improve a department’s atmosphere.

Learning always has a positive effect, also in terms of money. For example, you will need fewer people to do the same amount of work, your employees are making more money for the company, or your colleagues stay with the company for longer which saves a lot of money not having to recruit new staff.”