Instructional Triage: Getting the Right Help to the Right Learner...Fast!

One way to think about an education system it to think of it as a race against time for mastery.  There are only so many instructional hours and learners have so many different needs, some of the easy to address, some harder to address, and some that need a comprehensive intervention.  This way of framing an educational system made me think of a medical triage process during a large scale disaster.  The analogy goes like this:

You have a lot of people who need a lot of different kinds of help.  You can’t ignore any of them, because they all need your intervention, but you must to do two crucial things:

1.       You must reduce the average time to help for everyone in need to be as low as possible so no one dies simply because you haven’t gotten to them yet (due to the crowded, chaotic conditions).
2.       You must apply your most expert, most scarce, and therefore most expensive resources (expensive in time, salary, and sophistication) as precisely as you can to those who truly need them the most.

While not crucial in an emergency (no one is going to die immediately over this one), you have another goal:

3.       You should match the expertise of the other resources you have as precisely as you can to the degree of need.  You don’t want an RN treating someone who only needs a CNA or vice versa, because it is wasteful on the one hand, and on the other could turn a less severe need into a more severe one because you didn’t apply the right expertise/remedy soon enough.

To accomplish the above, you need to have a system for most of the people that provides an ever widening scope and degree of intervention (start with a first aid kit for self-treatment and work your way up) while quickly identifying those with severe needs that the other solutions aren’t going to be able to address and pulling them out of the queue and moving them directly to the comprehensive help they need.

Just about every organization that is responsible for helping others learn is in a similar race against time to get the right help to the right people in what can be an environment of scarce teaching/tutoring resources (making it "crowded") and with limited, sometimes conflicting information about the exact and diverse needs of each learner (making it "chaotic").  

So applying the principles of triage to a learning organization:

1.       You can reduce the average time it takes to get help in these ways:

a.       The more you know about the learner coming in, the quicker you are going to be able to identify needs and issues.  Every scrap of demographic information, every bit of data from their previous learning experiences, and any surveys or diagnostic tests they complete at sign up is all going to help you figure out their needs more quickly. 

IMPLICATION #1: It should be a priority to collect as much background information as possible on each learner and form predictive algorithms around it that are constantly refined and improved with the goal of anticipating their needs from day one.

b.      The smaller the chunk of meaningful learning you can assess and the quicker you can do so, the sooner you are going to be able to identify issues and needs.

  IMPLICATION #2: We should prioritize the creation and deployment of vastly more formative, granular assessments in our learning materials that are focused on assessing key concepts, the mastery of which are demonstrated to lead to essential learning outcomes.

c.       If you can identify major risk factors early on, you can save the inefficiencies of applying remedies that are not adequate and you can increase the amount of time the learner has exposure to the right remedies.

 IMPLICATION #3: We should become experts at identifying major risk factors very quickly and escalating the interventions immediately.

2.       You can apply expert resources precisely in these ways:

a.       You first must make sure that your instructors/mentors are undistracted so that their very expensive, expert time is not spent on paperwork or other administrative matters.

 IMPLICATION #4: It should be a priority to optimize the technology and administrative systems such that instructor time spent doing anything other than teaching/intervening with learners is as close to zero as possible and those tasks that are necessary but don’t lead directly to improved learning outcomes are delegated to others/automated.

b.      You must have experts that truly understand the proper, most effective interventions and how to apply them. 

 IMPLICATION #5: It should be a priority to hire staff with expertise in maximizing learning outcomes in a variety of challenging circumstances, to provide professional development to those who are expected to apply the expertise, and to run systematic pilots to figure out what to do when no one knows the answer and then operationalize and scale these interventions.

c.       You need to provide your instructors with the necessary diagnostic data so they can be efficient and accurate in their intervention. See 3.b. below.

3.       You can match expertise to degree of need in these ways:

a.       First, you must have an ecosystem that has a variety of interventions to apply.  If you only have a hammer, you are only going to pound things.  We need a diverse toolbox from which to choose.

 IMPLICATION #6: We need to develop diverse, calibrated, escalating, mutually aware, agile instructional interventions that are systematically applied if and when the previous one proves to be inadequate for timely mastery.

b.      You must have real-time, reliable data that lets you know which intervention to apply to whom. 

IMPLICATION #7: We need a highly sophisticated learning analytics system that measures learner progress velocity and mastery in real-time and can either automatically apply interventions reliably and intelligently or prompt a instructor/mentor to select an intervention from a vetted and targeted set of options to apply.

As an incredibly useful byproduct of the above, this system will generate an auditable trail of evidence of learning and mastery on a per learner basis that can be used to show all kinds of internal and external stakeholders exactly what the learner knows and how they learned it and we can use the same audit trail to teach ourselves how to teach the learners better.

Real-Time Business Case Method: Study What a Real Company is Doing. Right now. This Minute.

Since 1920, the basic case study method used first at Harvard Business School (HBS) and then by practically every business school on earth has changed little. Sure, they are putting them on tablets now, but otherwise the simple formula is 1. research something that happened, 2. write it up, 3. leave the key issues open for the students to analyze and interpret (even though the outcome is known to the case designer), and 4. hold a highly accountable class discussion about it. Often at the end, the outcome is revealed for comparison to the students prescriptions or predictions.  Nice. Tidy.  Print, repeat.

And print they did.  HBS alone has 17,000 case studies for sale!

But what if you took the whole concept up a notch?  What if the outcome was unknown because it simply hadn't happened yet?  Because the company was making the crucial decisions right now?  Today and tomorrow and throughout the course and, if the company survived, long after the students left the classroom.

And what if you had a front seat to the action? A reporter inside the company documenting what was happening week by week, interviewing executives and employees, and sending out updates once a week, including company documents, strategy, and financial forecasts?

And what if you could actually speak to the executives via video conference and ask them questions once a week?  And what if you were allowed to critique their business strategy using the same information they have available to them and provide recommendations for their consideration?

This is precisely the audacious experiment conceptualized and implemented by James Theroux at the Isenberg School of Management.  The results were phenomenal.  In the first implementation of this approach, 45% of  the students said they felt it was the most memorable or valuable course they had taken.

And was it good for the company?  Why would they air their dirty laundry in front of and answer questions from a bunch of students when they have urgent work to do and critical decisions to make?  Well, consider that the students identification of core weaknesses in the business model turned out to be accurate and their recommendations for action prophetic.

According to Theroux:

“Over time, and including [a] merger, the result has been to a large degree that [the students'] thoughts on positioning the company have come to fruition.”

Repeating this kind of success is difficult.  In a later iteration, a mere 25% of the students found it to be the most memorable or valuable course they had taken (I hate it when only a quarter of my students think my class was the best one ever!). The company in this case was slower moving, the technology field it was in was harder to understand, and the access to the leadership was, unexpectedly, more limited.

But the best things to do are way too often also the hardest things to do.  It is worth trying again and again to get a model like this right; to refine the selection criteria; to get the costs of engaging in the approach manageable (either by lowering execution costs or by scaling participation fees); to demonstrate benefits so compelling to both the company and the learning institution that the approach becomes accepted and routine and takes it place along side that other, once radical, approach that has been working since 1920.

Hats off to Dr. Theroux!

For more information on Real-Time Case Method, see:
Real-Time Case Method: Analysis of a Second Implementation. Theroux, James M.
Journal of Education for Business Jul/Aug 2009, Vol. 84 Issue 6, p367-373 7p; 2 Charts, 12 Graphs, 2009 

The Chocolate and Peanut Butter of Learning Analytics

On October 23rd, 2009, Adobe Systems, Inc. competed its acquisition of Omniture, Inc. An Adobe press release describes the benefits of this merger as follows:

The combination of the two companies will increase the value Adobe delivers to customers. For designers, developers, and online marketers, an integrated workflow — with optimization capabilities embedded in the creation tools — will streamline the creation and delivery of relevant content and applications. This optimization will enable advertisers and advertising agencies, publishers, and e-tailers to realize greater ROI from their digital media investments and improve their end users' experiences.

Beneficiaries: advertisers, advertising agencies, publishers, etailers
Benefit: greater ROI, better end user experience

What this press release does not say is that on that fateful day the chocolate of Omniture's web analytics tools dropped smack dab into the middle of the peanut butter of Adobe' s eLearning suite. Or it could have/should have/will if someone stops trying to flip up the most enticing Flash banner ad possible for a minute and thinks about the eLearning world. If the red and black eLearning folks will wander over to their new 1.8 billion dollar, lime green roommates and make a modest proposal, we could have a match as classic and enticing as the Reece's peanut butter cup.

Does Adobe have any idea that they are sitting on the biggest revolution in eLearning since the browser? Do they realize that they now have at their finger tips all of the tools necessary to dominate the eLearning space with the hottest, most completely integrated, most elegantly implemented learning analytics suite on the market? Who else has the power to build learning analytics straight into the most popular tools for elearning design and development? Who else has the statistical and number crunching guns to process and display massive amounts of learner data in slick, easy to use dashboards? Could there be a more obvious fit? Does anyone there realize that they could be the engine that powers a massive emerging industry?

Probably not. Why would the people focused on ROI for advertisers start scribbling on the back of napkins with people who are focused on ROI for eLearning ? Nothing against Adobe; big corporations just don't innovate this way very often. If they did, the press release might read as follows:

The combination of the two companies will increase the value Adobe delivers to learners everywhere. For instructional designers, eLearning developers, and online colleges and universities, an integrated workflow — with learning optimization and tracking capabilities embedded in the creation tools — will streamline the creation and delivery of customized learning content and experiences. This optimization will enable teachers, trainers, instructional designers, and training organizations as well as online educators to realize greater ROI from their digital training and teaching investments and improve their end users' experiences and, ultimately, the overall appeal, effectiveness and efficiency of their learning.

It would be a crying shame if this didn't happen. Imagine a world without Reece's peanut butter cups. It would be that bad.

Web Analytics in Education

A few months ago, I saw a presentation by Clint Rogers on web analytics at BYU. Since then, I have been thinking a lot about the possibilities of web analytics for education. I even told my friends that I am pretty sure that whoever figures out how to do it right first is going to start an industry. I was pleased to see that Clint was teaching a seminar on that very topic this fall at BYU so I am sitting in on it even though I have almost no spare time right now. As a result, several upcoming posts will be related to this topic.

So what could we learn from web analytics that we don't already know and what could we do with that knowledge?

My brainstorm:

• We could know exactly who looked at what and for how long. We could know which of the 10 things we thought they absolutely had to read they actually did read (or at least left open on their browser) and for how long and then correlate that to their scores to see if they really did need to read those ten things or not.
  
• We could find out if the $5000 simulation we built gets more actual student face time than the $500 game.
  
• We could provide approach A 50% of the time and approach B 50% of the time and correlate to outcomes to see if one has better results.
  
• We could identify learners who are not logging in, or clicking randomly, or only doing the quizzes and intervene by notifying them automatically (but as if we are human) that we have noticed this pattern and we are concerned (a human would read the reply, of course).
  
• We could possbily identify profiles of people who are cheating.
  
• We could find out if online students really do cram the entire course in to the last three weeks of the semester and still get an A- on the final and reflect on how we feel about that.
• We could discover that you only need to skim this particular course to get a B-.
• We could discover that if you only read the intro and the summaries of each lesson you get a passing C.
• We could discover that those who do all the optional quizzes and pace themselves so that they complete three lessons a week get an A and then tell new students at the beginning of the course of this pattern for success in this particular course to help them invest in good study practices. And, if they fall off the wagon, we could remind them that their current, not so hot learning patterns correlate with a D for 90% of the students last semester that fell into this pattern and didn't change by October 1st. In fact, profiling the behavior of high performing students or of those who get off to a rough start and recover or of those who spend the least amount of time in the course but get the highest grades or, or, or..., I think, is one of the most interesting areas that could be investigated and could lead to a lot of good advice for others taking the course and entire course redesigns to make them more lean and mean and precisely helpful. Especially if we can profile the students entry characteristics and then correlate them to success patters for those specific characteristics.

  "Dear student, According to the survey and your past grades, you are very similar to 86 students who took this course in the last 2 years. These students also 'enjoyed working on their own' but 'felt that they learned slower than most' and had similar grades to you on the pre-requisite courses. Students with this profile were most successful in this course when they followed these study habits: yada yada However, most of these students were more inclined to follow these less effective patterns: yida yida. We have sophisticated tools that can produce a weekly report showing how close your study habits are to those of students with your profile who were sucessful in the past and warning if you fall into the less effective learning patterns common to students with your profile. Would you like us to send this report to you?"

None of this feels like TLC for the student, but I believe that the hard numbers and statistical patterns can be presented in a very human, non-threatening, helpful way that really will help students feel like the course designers/instructors know them and are there to help them and have this almost magical insight into how they can improve their performance in the course. Maybe not. But it is very much worth a try.

Real-time Measure of Learner Engagement

I want to conduct this study:

Twenty college students (or learners of your choice) are given a means and a prompt (and a reward) to answer the following two questions every five minutes for all of their waking hours over the course of a day/week:

1. What are you doing right now (if different than your previous answer to this question)?
2. How engaged do you feel right now on a scale from 1 (I am bored to tears! Save me!) to 5 (Shhh! Go away, I am busy!)?

This would allow us to formulate some baseline data to see where learning experiences fall in the overall spectrum of a learner's environment in terms of engagement. I am guessing that most of them fall into the bottom twentieth percentile, most of the time. If so, that can't be good. This kind of study could also be instrumental in identifying those exceptional learning experiences that are maxing out the scales. Think we might want to look at those particular experiences a little more closely?

Or imagine this variation: You are the instructor of a course. Each student has a little engagement meter, asking to rate engagement on a scale from one to five every five minutes during your class time. You videotape the class. You synchronize the video with the data. You chart engagement across time. Where you see peaks (hopefully) and valleys (inevitably) you jump to that part of the tape to see what was/wasn't going on. How much do you think you could improve your course after just one session of this? After three? Five? Do you have the courage to have the data reported to you in real-time while teaching the class? (Would that even be a good idea?)

If you know of anyone anywhere who is doing anything along these lines, please let me know. I am aware of classroom clickers, but not aware of anyone using them to measure engagement throughout class time to create an engagement graph. And I have never heard of anyone trying to establish an "engagement baseline" for learners that compares their learning experiences to the rest of the daily experiences in their life.

In closing, could you please rate your level of engagement with this blog post on a scale from one to five? ;)