Instructional Triage: Getting the Right Help to the Right Learner...Fast!

One way to think about an education system it to think of it as a race against time for mastery.  There are only so many instructional hours and learners have so many different needs, some of the easy to address, some harder to address, and some that need a comprehensive intervention.  This way of framing an educational system made me think of a medical triage process during a large scale disaster.  The analogy goes like this:

You have a lot of people who need a lot of different kinds of help.  You can’t ignore any of them, because they all need your intervention, but you must to do two crucial things:

1.       You must reduce the average time to help for everyone in need to be as low as possible so no one dies simply because you haven’t gotten to them yet (due to the crowded, chaotic conditions).
2.       You must apply your most expert, most scarce, and therefore most expensive resources (expensive in time, salary, and sophistication) as precisely as you can to those who truly need them the most.

While not crucial in an emergency (no one is going to die immediately over this one), you have another goal:

3.       You should match the expertise of the other resources you have as precisely as you can to the degree of need.  You don’t want an RN treating someone who only needs a CNA or vice versa, because it is wasteful on the one hand, and on the other could turn a less severe need into a more severe one because you didn’t apply the right expertise/remedy soon enough.

To accomplish the above, you need to have a system for most of the people that provides an ever widening scope and degree of intervention (start with a first aid kit for self-treatment and work your way up) while quickly identifying those with severe needs that the other solutions aren’t going to be able to address and pulling them out of the queue and moving them directly to the comprehensive help they need.

Just about every organization that is responsible for helping others learn is in a similar race against time to get the right help to the right people in what can be an environment of scarce teaching/tutoring resources (making it "crowded") and with limited, sometimes conflicting information about the exact and diverse needs of each learner (making it "chaotic").  

So applying the principles of triage to a learning organization:

1.       You can reduce the average time it takes to get help in these ways:

a.       The more you know about the learner coming in, the quicker you are going to be able to identify needs and issues.  Every scrap of demographic information, every bit of data from their previous learning experiences, and any surveys or diagnostic tests they complete at sign up is all going to help you figure out their needs more quickly. 

IMPLICATION #1: It should be a priority to collect as much background information as possible on each learner and form predictive algorithms around it that are constantly refined and improved with the goal of anticipating their needs from day one.

b.      The smaller the chunk of meaningful learning you can assess and the quicker you can do so, the sooner you are going to be able to identify issues and needs.

  IMPLICATION #2: We should prioritize the creation and deployment of vastly more formative, granular assessments in our learning materials that are focused on assessing key concepts, the mastery of which are demonstrated to lead to essential learning outcomes.

c.       If you can identify major risk factors early on, you can save the inefficiencies of applying remedies that are not adequate and you can increase the amount of time the learner has exposure to the right remedies.

 IMPLICATION #3: We should become experts at identifying major risk factors very quickly and escalating the interventions immediately.

2.       You can apply expert resources precisely in these ways:

a.       You first must make sure that your instructors/mentors are undistracted so that their very expensive, expert time is not spent on paperwork or other administrative matters.

 IMPLICATION #4: It should be a priority to optimize the technology and administrative systems such that instructor time spent doing anything other than teaching/intervening with learners is as close to zero as possible and those tasks that are necessary but don’t lead directly to improved learning outcomes are delegated to others/automated.

b.      You must have experts that truly understand the proper, most effective interventions and how to apply them. 

 IMPLICATION #5: It should be a priority to hire staff with expertise in maximizing learning outcomes in a variety of challenging circumstances, to provide professional development to those who are expected to apply the expertise, and to run systematic pilots to figure out what to do when no one knows the answer and then operationalize and scale these interventions.

c.       You need to provide your instructors with the necessary diagnostic data so they can be efficient and accurate in their intervention. See 3.b. below.

3.       You can match expertise to degree of need in these ways:

a.       First, you must have an ecosystem that has a variety of interventions to apply.  If you only have a hammer, you are only going to pound things.  We need a diverse toolbox from which to choose.

 IMPLICATION #6: We need to develop diverse, calibrated, escalating, mutually aware, agile instructional interventions that are systematically applied if and when the previous one proves to be inadequate for timely mastery.

b.      You must have real-time, reliable data that lets you know which intervention to apply to whom. 

IMPLICATION #7: We need a highly sophisticated learning analytics system that measures learner progress velocity and mastery in real-time and can either automatically apply interventions reliably and intelligently or prompt a instructor/mentor to select an intervention from a vetted and targeted set of options to apply.

As an incredibly useful byproduct of the above, this system will generate an auditable trail of evidence of learning and mastery on a per learner basis that can be used to show all kinds of internal and external stakeholders exactly what the learner knows and how they learned it and we can use the same audit trail to teach ourselves how to teach the learners better.

The Power of Learning Analytics, Targeted Intervention, and Mastery Learning at Scale

This is the first time I have ever reblogged, but the posting by Bror Saxberg, CLO of Kaplan, regarding applying learning analytics at scale at the University of Wisconsin at Milwaukee is worth repeating.  It is a powerful combination of large scale analytics, careful experimental design, targeted intervention,  semi-automated student motivation, and mastery learning. Make sure you jump to the original via the link below to see some very compelling charts representing their success.

By the way, I ran in to Bror at the Educause Learning Initiative conference, and he is more than ready to lead the revolution for the application of learning sciences to higher education, and he has enough energy for a dozen mere mortals to pull it off!  I am interested in connecting with others who share this passion as well. 

Bror writes:

At the recent Educause Learning Initiative meeting in Austin Texas, I came across some very interesting recent randomized control trial results from the University of Wisconsin at Milwaukee.  They’re testing a combination of mastery learning plus “amplified assistance” (data-driven suggestions for faculty about who to intervene with and how) in an introductory Psychology course, and, with thousands of students (!!) having run through controlled trials, they’re showing significant improvements in pass rates and long term retention.

via brorsblog.typepad.com

The Chocolate and Peanut Butter of Learning Analytics

On October 23rd, 2009, Adobe Systems, Inc. competed its acquisition of Omniture, Inc. An Adobe press release describes the benefits of this merger as follows:

The combination of the two companies will increase the value Adobe delivers to customers. For designers, developers, and online marketers, an integrated workflow — with optimization capabilities embedded in the creation tools — will streamline the creation and delivery of relevant content and applications. This optimization will enable advertisers and advertising agencies, publishers, and e-tailers to realize greater ROI from their digital media investments and improve their end users' experiences.

Beneficiaries: advertisers, advertising agencies, publishers, etailers
Benefit: greater ROI, better end user experience

What this press release does not say is that on that fateful day the chocolate of Omniture's web analytics tools dropped smack dab into the middle of the peanut butter of Adobe' s eLearning suite. Or it could have/should have/will if someone stops trying to flip up the most enticing Flash banner ad possible for a minute and thinks about the eLearning world. If the red and black eLearning folks will wander over to their new 1.8 billion dollar, lime green roommates and make a modest proposal, we could have a match as classic and enticing as the Reece's peanut butter cup.

Does Adobe have any idea that they are sitting on the biggest revolution in eLearning since the browser? Do they realize that they now have at their finger tips all of the tools necessary to dominate the eLearning space with the hottest, most completely integrated, most elegantly implemented learning analytics suite on the market? Who else has the power to build learning analytics straight into the most popular tools for elearning design and development? Who else has the statistical and number crunching guns to process and display massive amounts of learner data in slick, easy to use dashboards? Could there be a more obvious fit? Does anyone there realize that they could be the engine that powers a massive emerging industry?

Probably not. Why would the people focused on ROI for advertisers start scribbling on the back of napkins with people who are focused on ROI for eLearning ? Nothing against Adobe; big corporations just don't innovate this way very often. If they did, the press release might read as follows:

The combination of the two companies will increase the value Adobe delivers to learners everywhere. For instructional designers, eLearning developers, and online colleges and universities, an integrated workflow — with learning optimization and tracking capabilities embedded in the creation tools — will streamline the creation and delivery of customized learning content and experiences. This optimization will enable teachers, trainers, instructional designers, and training organizations as well as online educators to realize greater ROI from their digital training and teaching investments and improve their end users' experiences and, ultimately, the overall appeal, effectiveness and efficiency of their learning.

It would be a crying shame if this didn't happen. Imagine a world without Reece's peanut butter cups. It would be that bad.

Web Analytics in Education

A few months ago, I saw a presentation by Clint Rogers on web analytics at BYU. Since then, I have been thinking a lot about the possibilities of web analytics for education. I even told my friends that I am pretty sure that whoever figures out how to do it right first is going to start an industry. I was pleased to see that Clint was teaching a seminar on that very topic this fall at BYU so I am sitting in on it even though I have almost no spare time right now. As a result, several upcoming posts will be related to this topic.

So what could we learn from web analytics that we don't already know and what could we do with that knowledge?

My brainstorm:

• We could know exactly who looked at what and for how long. We could know which of the 10 things we thought they absolutely had to read they actually did read (or at least left open on their browser) and for how long and then correlate that to their scores to see if they really did need to read those ten things or not.
  
• We could find out if the $5000 simulation we built gets more actual student face time than the $500 game.
  
• We could provide approach A 50% of the time and approach B 50% of the time and correlate to outcomes to see if one has better results.
  
• We could identify learners who are not logging in, or clicking randomly, or only doing the quizzes and intervene by notifying them automatically (but as if we are human) that we have noticed this pattern and we are concerned (a human would read the reply, of course).
  
• We could possbily identify profiles of people who are cheating.
  
• We could find out if online students really do cram the entire course in to the last three weeks of the semester and still get an A- on the final and reflect on how we feel about that.
• We could discover that you only need to skim this particular course to get a B-.
• We could discover that if you only read the intro and the summaries of each lesson you get a passing C.
• We could discover that those who do all the optional quizzes and pace themselves so that they complete three lessons a week get an A and then tell new students at the beginning of the course of this pattern for success in this particular course to help them invest in good study practices. And, if they fall off the wagon, we could remind them that their current, not so hot learning patterns correlate with a D for 90% of the students last semester that fell into this pattern and didn't change by October 1st. In fact, profiling the behavior of high performing students or of those who get off to a rough start and recover or of those who spend the least amount of time in the course but get the highest grades or, or, or..., I think, is one of the most interesting areas that could be investigated and could lead to a lot of good advice for others taking the course and entire course redesigns to make them more lean and mean and precisely helpful. Especially if we can profile the students entry characteristics and then correlate them to success patters for those specific characteristics.

  "Dear student, According to the survey and your past grades, you are very similar to 86 students who took this course in the last 2 years. These students also 'enjoyed working on their own' but 'felt that they learned slower than most' and had similar grades to you on the pre-requisite courses. Students with this profile were most successful in this course when they followed these study habits: yada yada However, most of these students were more inclined to follow these less effective patterns: yida yida. We have sophisticated tools that can produce a weekly report showing how close your study habits are to those of students with your profile who were sucessful in the past and warning if you fall into the less effective learning patterns common to students with your profile. Would you like us to send this report to you?"

None of this feels like TLC for the student, but I believe that the hard numbers and statistical patterns can be presented in a very human, non-threatening, helpful way that really will help students feel like the course designers/instructors know them and are there to help them and have this almost magical insight into how they can improve their performance in the course. Maybe not. But it is very much worth a try.