Tuesday, September 9, 2008

Web Analytics in Education

A few months ago, I saw a presentation by Clint Rogers on web analytics at BYU. Since then, I have been thinking a lot about the possibilities of web analytics for education. I even told my friends that I am pretty sure that whoever figures out how to do it right first is going to start an industry. I was pleased to see that Clint was teaching a seminar on that very topic this fall at BYU so I am sitting in on it even though I have almost no spare time right now. As a result, several upcoming posts will be related to this topic.

So what could we learn from web analytics that we don't already know and what could we do with that knowledge?

My brainstorm:

  • We could know exactly who looked at what and for how long. We could know which of the 10 things we thought they absolutely had to read they actually did read (or at least left open on their browser) and for how long and then correlate that to their scores to see if they really did need to read those ten things or not.
  • We could find out if the $5000 simulation we built gets more actual student face time than the $500 game.
  • We could provide approach A 50% of the time and approach B 50% of the time and correlate to outcomes to see if one has better results.
  • We could identify learners who are not logging in, or clicking randomly, or only doing the quizzes and intervene by notifying them automatically (but as if we are human) that we have noticed this pattern and we are concerned (a human would read the reply, of course).
  • We could possbily identify profiles of people who are cheating.
  • We could find out if online students really do cram the entire course in to the last three weeks of the semester and still get an A- on the final and reflect on how we feel about that.
  • We could discover that you only need to skim this particular course to get a B-.
  • We could discover that if you only read the intro and the summaries of each lesson you get a passing C.
  • We could discover that those who do all the optional quizzes and pace themselves so that they complete three lessons a week get an A and then tell new students at the beginning of the course of this pattern for success in this particular course to help them invest in good study practices. And, if they fall off the wagon, we could remind them that their current, not so hot learning patterns correlate with a D for 90% of the students last semester that fell into this pattern and didn't change by October 1st. In fact, profiling the behavior of high performing students or of those who get off to a rough start and recover or of those who spend the least amount of time in the course but get the highest grades or, or, or..., I think, is one of the most interesting areas that could be investigated and could lead to a lot of good advice for others taking the course and entire course redesigns to make them more lean and mean and precisely helpful. Especially if we can profile the students entry characteristics and then correlate them to success patters for those specific characteristics.
    "Dear student, According to the survey and your past grades, you are very similar to 86 students who took this course in the last 2 years. These students also 'enjoyed working on their own' but 'felt that they learned slower than most' and had similar grades to you on the pre-requisite courses. Students with this profile were most successful in this course when they followed these study habits: yada yada However, most of these students were more inclined to follow these less effective patterns: yida yida. We have sophisticated tools that can produce a weekly report showing how close your study habits are to those of students with your profile who were sucessful in the past and warning if you fall into the less effective learning patterns common to students with your profile. Would you like us to send this report to you?"
None of this feels like TLC for the student, but I believe that the hard numbers and statistical patterns can be presented in a very human, non-threatening, helpful way that really will help students feel like the course designers/instructors know them and are there to help them and have this almost magical insight into how they can improve their performance in the course. Maybe not. But it is very much worth a try.

5 comments:

dr. clint said...

I loved reading your brainstorm, hopefully we will get to talk about almost all of those ideas at some point during the semester. I think you are so right that there will be huge value in this, and soon.

John Hilton III said...

thanks for a great post...i'm not in the web analytics class, but after reading your insights i wish i were!

jasonpriem said...

Great points. The ability to gather rich, deep data about students is one of the most transformative--and to date, neglected--potentials of online teaching. You point out how this data can be a great resource for study after-the-fact, and I completely agree. But I think there is tremendous potential in using analytics during instruction, as well. Many folks have pointed out the problem of on-the-fly assessment in distance learning; it's a lot harder to gauge students' progress without all the facial and vocal cues we get in person. But detailed web analytics, combined with intelligent processing and well-designed visualizations, may actually turn this deficiency upside-down, offering the online educator a clearer, richer picture of her classes' progress than she'd get in a crowded lecture hall.

I'd encourage you to check out the work of Moodie and Kunz, Kosba, Romero, and Mazza, all of whom are doing really interesting things involving using analytics data in real time. My colleagues and I recently published a piece on this in the journal Internet and Higher Education that lists some more sources, too.

Joseph said...

Thanks, Jason. I couldn't agree more with your observation that real time analytics has huge potential and can hopefully steer a learning experience into more productive directions as it unfolds. I have already downloaded your article and am looking forward to a good read.

Thanks for the leads.

Joseph

Michael said...

I loved the insight that you put into the post of the possible benefits of educational web analytics. I think it would be very interesting for someone like Independent Study to create their own web analytics program to be able to better assess things that you are talking about. I really like how Mike Griffiths is using webcam assignments to get in touch with his online students more. I'm just wondering how cost effective that would be to work directly with the students and get to know them personally, instead of relying on web analytics.