Andrew's Masters Project

I’ve completed what I’m going to call the “primary” literature review for my project. I printed all the relevant papers I found to date and in the last two days I read and annotated the most immediately relevant of those – authored by Beryl Plimmer et al. I mentioned them in previous posts. I’ll post some of my notes here also.

“A Pen-based Paperless Environment for Annotating and Marking Student Assignments” (2006):

• An advantage of doing reviews electronically is that it can be done remotely. Paper allows free-form, unstructured, unrestricted, expression and that’s difficult with a simple textbox on a webpage, but a tablet allows for the best of both worlds.
• An interesting point was mentioned that got me thinking: using different colours of ink for different types of statements (positive/negative, low/high severity) turned out to be unusable – very error prone. I will probably not offer a way to categorise freehand comments, but this is something worth keeping in mind if I get ideas in the future.
• Think-aloud was also a failure, authors attributed it to that code review takes a lot of concentration and saying what you’re thinking breaks that concentration. As a result – no any useful information was gathered from the talk-while-you-do-the-review process.
• The participants didn’t have previous experience with using a tablet PC but figured it out quickly. I have to make sure I come up with some exercise for my participants.
• The lack of a horizontal scrollbar was mentioned as a problem brought up by the participants. I have not even considered how scrolling works when you’re using a stylus. Something to keep in mind.
• A couple of participants suggested that colour syntax highlighting would have made the code easier to read. This will probably not work for me since I’ll be working with patches, but maybe.
• “Text-entry to the assignments via the on-screen keyboard proved to be tedious, so an external keyboard was added during the first participant’s session […] We noted that markers used the pen and the keyboard simultaneously”. Hm. This is likely something I will run into also. Typing is much quicker than handwriting, quicker than talking even. So when the reviewer wants to provide a paragraph of comments they will certainly want a keyboard.
• “they had initially tried erasing annotations with the back of the pen”. I’m not sure I would have, but erasing annotations is important, and whether it works with the back of the pen depends on the hardware. Have to keep this in mind too.
• Here and in one other paper the Word review functionality was mentioned. Another paper (can’t remember which one) lists the advantages and disadvantages of that. I should probably have a similar list.
• Because of implementation troubles one computer ran the review software and another had the IDE (the second probably to build/run the code). Probably won’t apply to me.

“RCA: Experiences with an IDE Annotation Tool” (2006):

• When the underlying layout changes (e.g. font size, zoom) the review comments need to stay in place. This was mentioned in a few other places as “reflowing annotations”, and I thought it didn’t apply to me because the code I’m concerned with will not change. It will probably be a diff that will stay in the system forever whether it’s checked in or altered or not. But if I’m to integrate with ReviewBoard – the user will certainly be able to change the font size. Maybe for my study I can not care about this.
• Another vote for the ability to modify the review comments, here including not only erasing but selecting, moving, and recolouring. Eeee.. too fancy for my project, probably.
• It’s not clear what the final implementation was, but I think their ideal would have been a transparent canvas for the ink overlaid on the code editor. I’m not sure this is possible to do with the HTML canvas, but I expect I can just render whatever used to be rendered into the browser window into the canvas instead. Thankfully I don’t need to support editing of the code. Fingers crossed.
• A system similar to the Visual Studio breakpoints was used in RCA to choose a severity for the comment. They didn’t mention here whether it was used/useful or not. I’m not sure whether this paper is newer or older than the one I mentioned above.

“CodeAnnotator: Digital Ink Annotation within Eclipse” (2007):

• Nothing new here, but I’ve been kindly offered a chance to review the source code, might learn some useful things from it.

“Issues of Extending the User Interface of Integrated Developmnet Environments”:

• “size and average complexity of projects has grown, the traditional way of reviewing code is no longer feasible”. I wonder if this is in reference to reviewing entire codebases, rather than specific changes as listed in a patch file. I think I will ignore this problem, assume that a diff is still the primary method of reviewing changes because it works well.

“A comparative Evaluation of Annotation Software for Grading Programming Assignments” (2010):

• There’s a mention of prior work that found “Ink annotations are rich and expressive due to their free-format”, and “Ink annotations also have a part to play in supporting active reading”. I have to read these two papers to see if any of it has data useful for my study.
• An adapted means of categorising comments as  “either a tick or cross, comment, grade, or other” was used. Given that this paper is a few years newer, perhaps this was found to be a good solution to the difficulties of categorising mentioned in the papers quoted above.
• “Latin squares arrangement” – something I might want to use.
• This study was run on 600 assignments from 200 students, with real TAs giving real comments and marks. The data resulting from that is very useful and interesting:
  • Paper comments are much more useful than database (form field) comments, but grading on paper was universally disliked because it’s so cumbersome. 70% of the students never collected the marked papers.
  • In the TAs feedback on the comparison of paper with the tablet the digital eraser was voted one of the biggest advantages.
  • It did not take any longer to mark using the tablet than using either paper or the database method. This is encouraging, and something I hope to confirm in my study.
  • Overall the tablet method was found to be the best of the three.

There is so much interesting stuff in these papers I almost feel like starting to add that stuff as notes to my own, but I think it will be more prudent to just make notes on the side and in the blog for now, until I make some final decisions about what my study is going to look like.