Reinforcement Metrics

How to Use Analytics to Optimize the Effectiveness of Your Training Reinforcement Plan


The fundamental purpose of corporate training is to empower employees to apply newly gained knowledge to their work. Without retention of what has been learned, reaching this goal is impossible. Think of retention as the gap that separates the learning moment from the doing moment. The bridge that closes this gap is the learning or training reinforcement plan. If you’re new to the concept, be sure to read “How to Overcome the Illusion of Knowing” and “How the Mobile Revolution can Overcome the Forgetting Curve”. But if you’re in the stage where you find yourself grappling with the question, “How can I measure and improve the effectiveness of my reinforcement plans?” read on.

Using analytics to improve the performance of your training reinforcement plans is easier than you think! With just a small investment, it’s relatively simple to measure and optimize employees’ retention. Implement the right metrics to gain insightful and actionable data, and then use that data to guide your efforts moving forward.

Why do training reinforcement metrics matter?

There are a countless number of reasons proving the value behind reinforcement metrics – you’d be hard pressed to find a valid reason not to measure your training reinforcement plans.

Picture this scenario: John is the Director of Learning and Development for a large retail chain. Over the last 5 years, he’s run several customer service training programs for the chain’s frontline employees. Realizing there was a problem with knowledge transfer, in the last 2 years, he implemented a 6-week reinforcement plan that followed the company’s in-store training event. John knows from the results of the exit test that the employees have acquired a good portion of what was taught. But John still doesn’t know the extent to which his training reinforcement plan is helping the employees retain that knowledge two or three weeks following the event. And to top it all off, Retail Operations is asking for assurance that the employees don’t have to be retrained a few months from now.

John needs metrics.

Insight into the learning journey

The learning journey of an employee doesn’t end when the initial training event is over. It continues during the critical few weeks following the event when the employee is back on the job and the “forgetting curve” is working hard to erode what’s been learned. Reinforcement metrics allow you to measure the effectiveness of regular retrieval practices and other activities in your training reinforcement plan. The insights you gain by expanding your radar from the learning event to the full learning journey are profound.

Maximizing ROI

The ROI for your training programs is intricately linked to your employees’ retention rate. The more they remember, the better they perform, and the better they perform, the greater your success and the higher the return on your training investment. Metrics are fundamental to true ROI maximization and yet so may training ROI projections don’t even include the retention phase. This is even more important than measuring the effectiveness of your training course. With reinforcement metrics, you have the ability to assess your learners’ journeys in real time months after the training event. ROI calculations that focus solely on test scores achieved at the end of the learning moment miss out entirely on the key measure of knowledge transfer.

Empowering learners

Picture how significant it would be for learners to be presented with insightful data about their own learning process. By sharing relevant and personalized reinforcement metrics with each employee, you allow them to become stakeholders in their individual learning journeys.

Accessibility and cost effectiveness

Using data to measure outcomes is nothing new but thanks to mobile, cloud and AI technologies, metrics have never been easier to collect and analyze. Generating, storing and analyzing large volumes of reinforcement measurements now cost next to nothing. The end result is the ability to leverage analytics in a way that achieves heightened retention effectiveness.

Getting the buy in

The objective and actionable data you can obtain from training reinforcement metrics helps you demonstrate unquestionable value to upper management. This makes it far easier to obtain approval for learning and development budgets as well as the expansion of your training reinforcement plan.

Fine tuning the content

A training reinforcement plan is a constant work in progress – and the right metrics play a key role in the continual tweaking and optimization of your plan. Metrics tell you what concepts require added reinforcement, allowing you to optimize both the initial training material, and fine-tune your training reinforcement schedules and retrieval practices.

Reinforcement metrics are different from traditional training measures

Were your learners paying attention during initial training? Do they understand what you taught them? Can they use the new knowledge effectively? Are they able to translate what they learned into what they do?

Training reinforcement plans are fundamentally different from traditional training programs. So it’s only logical that the way we measure them differs too.

Training programs are usually timebound (a 4-hour course, a 1-day course, a 3-day course), and as a result, there is very little opportunity to use metrics during the training event. Metrics are important but they’re typically used to make adjustments to the program for the next cohort of learners – not the current one.

On the other hand, training reinforcement plans last several weeks and there’s plenty of time to perfect the very plan you’re measuring. Subsequent retrieval practices can be altered dynamically to customize and improve the learner’s journey, both on an individual and collective level.

What to measure: reinforcement metrics

 Our experience has shown that the following three core metrics form the backbone of every training reinforcement plan:

  1. Learner Engagement
  2. Outcome
  3. Long-term Mastery

And in order to gain the greatest insight into your training reinforcement plans, you need to view these metrics along two dimensions:

  • Self-Dimension: how each employee’s metrics change over time
  • Cohort-Dimension: how each employee’s metrics compare to the average of the cohort of employees involved in the same training reinforcement plan

Engagement metrics

Learner engagement is defined as the extent to which the learner takes part in retrieval activities throughout the duration of the reinforcement plan. This measure is foundational in the sense that both the outcome and mastery metrics depend on it: no engagement means no outcome and no mastery. Frequency of use and the amount of time spent performing retrieval practices (ideally a blend of the two) are core indicators of a learner’s engagement. While measuring a learner’s engagement with the recommended learning resources of the reinforcement plan is secondary, it can still be useful.

A good reinforcement plan design assigns points to practice attempts and additional points to practice questions answered correctly. By doing this, you reward learners for their engagement.

Outcome metrics

Outcome is defined as the learner’s performance on tests and quizzes throughout the reinforcement plan. Test scores can be used to create gamified and competitive designs to encourage heightened engagement levels and learner interest (if you wish). For example, you might decide to use various team leaderboards to incentivize your learners’ levels of engagement during the reinforcement experience. The extent to which you decide to use gamification depends largely on your industry and company culture.

Other than using outcome metrics to promote user engagement, a good design avoids positioning outcome as a success criterion and rather focuses on measuring learners’ mastery levels, irrespective of quiz scores (see below for more).

Outcome metrics can include:

  • Number of questions answered correctly
  • Accumulated quiz scores
  • Number of awards earned
  • Position in a leaderboard

Mastery metrics

An employee achieves mastery in a given competency when he or she has comprehensive knowledge of that competency. To achieve long-term mastery, the employee needs to demonstrate proficiency in a balanced (across competencies) and sustained (over time) manner. Mastery is a binary measure. At any given time, an employee has either reached the mastery level for each competency or has not. The benchmark that separates the two situations is different for each industry and subject matter. For example, in one reinforcement plan, answering 2 questions in a row correctly may represent mastery, whereas in another reinforcement plan, mastery may not be achieved until 3 out of 5 questions are answered correctly over a 2-week period.

So how do we measure mastery? Start by mapping each retrieval practice question in your reinforcement plan to one or more competency. Then measure how frequently the learner correctly answers questions associated with a specific competency over a defined period of time. Use these measures to determine which employees have achieved mastery for each competency and how many have maintained that mastery.

Avoid the temptation to use quiz scores (an outcome metric) as a proxy for mastery. What does this mean? For example, if you use a leaderboard, you might assign a losing point to each wrong answer, but it would be inaccurate to factor this into the measure for mastery and here’s why: let’s say Mary engages in 3 retrieval practices each week over the course of 4 weeks. In the first two weeks, 80% of her answers are wrong, resulting in several losing points. But during the third week, only 40% of her answers are wrong and in the fourth week, only 5%! This, in fact, demonstrates long-term mastery! But your misdirected assignment of losing points incorrectly distorts this reality.

Mastery metrics are powerful. Use them to fine-tune your training reinforcement plan. And consider exposing your employees to their own mastery metrics or the average mastery metric of their peers. Show learners how their mastery of different competencies has changed over the duration of the reinforcement plan. And equip your frontline managers with the mastery metrics for their team members so they are better prepared for their coaching sessions.

When it comes to this metric, you have options. You can either choose to involve learners by showing them how their mastery metrics evolve throughout the training reinforcement plan or grant access to managers only. Achieving mastery doesn’t mean your learners should no longer be exposed to retrieval practices for that competency. Simply adjust your training reinforcement schedule to allow a little forgetting to set in before you recheck a learner on a mastered competency. As you’re deciding when to re-expose learners to particular practices, consider how often they use those concepts in any given day on the job.

Reinforcement metrics in action

Now that you know what metrics to use to measure your training reinforcement plan, how do you use these metrics to improve effectiveness?

While training reinforcement metrics help you optimize performance, you need solid A/B testing to perfect your plan. Create 2 groups: One group acts as a control and the other receives the “treatment” plan. However you choose to use A/B testing, be sure to use equivalent benchmark tests to assess your cohort’s knowledge levels at both the beginning and the end of the reinforcement plan.

In the early stages

When you first roll out your plan, use two groups, one that receives a reinforcement plan and one that receives no reinforcement whatsoever. This will give you an immediate indication of the training reinforcement plan’s positive effect on retention and mastery. This will go a long way to maintain executive buy in and dispel any possible skepticism within your company.

Fine-tuning stages

Once your reinforcement plans are better established, you’ll want to start testing different versions as you fine-tune the content (see “How to Create First-Rate Reinforcement Content” for more).

As you test and measure, be sure to focus on a single parameter at a time. A few of the most important parameters that your A/B tests can focus on optimizing are:

  • Spacing of retrieval practices
  • Types of questions used in retrieval practices (knowledge check, application questions, etc.)
  • Number of questions in each retrieval practice
  • Type of retrieval practices (multiple choice questions, open questions, flashcards, etc.)
  • Length of the entire reinforcement plan
  • Number of competencies covered
  • Type and timing of feedback for retrieval questions

For example, you set the spacing of the retrieval practices for Group A to every day and that of Group B to every other day. If the employees in Group A show a higher measure of the mastery metric, then serving daily retrieval practices is more effective. Now that you’ve determined an optimum value for the spacing variable, you’re ready to run the next A/B test, but this time to measure the impact of another parameter, let’s say, feedback timing.

The options are endless. But by testing, retesting, measuring and re-measuring, it won’t be long before the right changes become clear to see. And don’t forget! Of the three reinforcement metrics, the two you should focus on in your A/B tests are the engagement and mastery metrics – in that order.

Making your move

It’s never too early to start thinking about reinforcement metrics. A good reinforcement plan backed up by intelligent metrics is what puts you in a position to maximize your ROI, improve learning retention and boost company performance. When you take the time to effectively structure and measure your reinforcement plan, you end up with continuous, actionable, real time data that gives you a pulse on every learner. You’ll be able to understand what’s working and what’s not – and know how to fix it! In other words, as one of our clients said quite succinctly…

“Reinforcement metrics take the guesswork out of the successful transfer of learning.”