We have conducted three studies to evaluate WebIDE at Cal Poly - San Luis Obispo.  In all three studies, two sections of the same course were taught by the same instructor.  Each section had approximately 30 to 35 students.  One section was randomly selected as the control group, and the other as the experimental group.  Both sections were given weekly labs that contained identical content.  The only difference was that the experimental group completed the labs in WebIDE, using the lock-step framework and incremental feedback.  The control group used traditional development environments (e.g. Eclipse, vim/gcc).

The first two studies were pilot studies conducted in Fall 2010 and Spring 2011 while WebIDE was still being developed.  WebIDE was still a bit unstable during these times, but we did report preliminary results at CSEE&T in May 2011.  The WebIDE group showed a significant improvement in performance when writing a simple Android application. Additionally, among students with some programming experience, the WebIDE group was more proficient in writing unit tests.

The most complete evaluation of WebIDE was conducted in Fall 2011 in an Introduction to Computing (CS0) course that taught beginning programmers how to build Android apps.  Results are currently being prepared for publication.  A brief overview of the course and WebIDE evaluation are provided below.  

The course contained nine labs.  The first three labs used visual drag-and-drop programming environments (Scratch and App Inventor).  The fourth, fifth, sixth, and eighth labs introduced Java programming concepts (data types, variables, operations, methods, classes, selection, loops, arrays).  The seventh and ninth labs introduced Android with Java.  

Students were given a pre-study quiz to assess their programming skills coming into the course.  As the chart below indicates, the WebIDE section had weaker programming skills coming into the course, and they performed worse on the initial Scratch and App Inventor labs (prior to WebIDE).  Just prior to lab four, a coin was flipped to determine which section would be control/experimental, without having analyzed results on the Java pre-quiz or the first three labs.  Labs four through eight included online quizzes conducted at the end of closed lab sessions.  The chart below shows the percentage differences on these quiz assessments.  Students who used WebIDE scored between 2.51% and 12.19% better on all assessments after WebIDE was introduced.  A more complete analysis of student scores and qualitative survey results will be added soon.

WebIDE Publications:
Related Publications: