Very few colleges and universities are revealing their assessment practices for MOOCs; Duke is one exception. This report explains that selective university's experiences with one course, and demonstrates how their administrators and faculty thought about measuring success. In my view, some of it is good, and some of it is exactly what not to do.
So here is my overview of an initial evaluative framework for thinking about whether or not a MOOC was a "success." I'm posting this with the explicit intention of generating discussion to refine this framework and build it out, so please add your two cents.
The framework has three components: an assessment of costs expended to generate and implement the MOOC, an assessment of its intended impacts in multiple domains, and as assessment of unintended consequences. Only by considering all three can one make an informed decision about whether a MOOC was worthwhile, yet reporting on MOOCs suggests that many institutions are focusing on only one or two of the three.
- COSTS
- How many total hours of effort were required to build the MOOC?
- How many of these were on the part of the faculty member?
- How many involved TA time?
- How many involved staff time?
- What technological costs were incurred?
- How much was spent to get the MOOC off the ground?
- What are the regular maintenance costs?
- What are the costs of upgrading the MOOC?
- How many hours of administrative time were devoted to meetings about the MOOC?
- How much time and money is spent advertising the MOOC?
- How much time will be required to regulate it?
- How much time will be required to assess its outcomes?
- IMPACTS
- On Students.
- How many students initially enrolled in the MOOC?
- How many of these were students were not already enrolled in the university?
- How many countries are represented among MOOC users?
- How many of these countries were previously not served by the university, or represent new relationships?
- How many of these countries are home to under-educated populations?
- How many of these students had no college experience prior to the MOOC?
- How many of these students already held a college degree of any kind?
- What percent of the total initial enrollment completed the first week of the course?
- What percent of the total initial enrollment completed the first assessment?
- What percent of the total initial enrollment completed the entire course?
- On Faculty.
- What was the professor's level of satisfaction with the course, as compared to satisfaction of teaching a usual university course?
- What new contacts for the professor resulted from the MOOC, related to his/her research agenda?
- How did teaching the MOOC inform the professor's research agenda?
- How did teaching the MOOC change how the professor approaches his/her regular university courses?
- Is there any evidence that the professor and/or his/her research gained more recognition in the community where MOOC students came from? (e.g. media mentions, etc)
- SIDE EFFECTS
- How many course releases (from campus teaching) were required for the professor to teach the MOOC, and how many students on average are taught per course (typically)?
- How many press mentions occurred regarding the MOOCs?
- How many were positive?
- How many were negative?
- What did state legislators say about the MOOCs?
- How many felt the MOOC was a welcome activity?
- How many felt the MOOC was an unwelcome activity?
- What is the dollar value of the activities incurred by the MOOC contracted out to the provider rather than performed "in-house"?
As I said, this is just a rough start. Write in, and help me flesh this out!
