{"id":73,"date":"2010-04-04T09:02:18","date_gmt":"2010-04-04T13:02:18","guid":{"rendered":"http:\/\/littlesvr.ca\/masters\/?p=73"},"modified":"2010-05-30T16:31:06","modified_gmt":"2010-05-30T20:31:06","slug":"code-review-using-tablets-literature-review-3","status":"publish","type":"post","link":"http:\/\/littlesvr.ca\/masters\/2010\/04\/04\/code-review-using-tablets-literature-review-3\/","title":{"rendered":"Code review using tablets \u2013 literature review (3)"},"content":{"rendered":"<p>I&#8217;ve completed what I&#8217;m going to call the &#8220;primary&#8221; literature review for my project. I printed all the relevant papers I found to date and in the last two days I read and annotated the most immediately relevant of those &#8211; authored by Beryl Plimmer et al. I mentioned them in previous posts. I&#8217;ll post some of my notes here also.<\/p>\n<p>\u201c<a href=\"http:\/\/crpit.com\/confpapers\/CRPITV50Plimmer.pdf\">A Pen-based Paperless Environment for Annotating and Marking Student Assignments<\/a>\u201d (2006):<\/p>\n<ul>\n<li>An advantage of doing reviews electronically is that it can be done remotely. Paper allows free-form, unstructured, unrestricted, expression and that&#8217;s difficult with a simple textbox on a webpage, but a tablet allows for the best of both worlds.<\/li>\n<li>An interesting point was mentioned that got me thinking: using different colours of ink for different types of statements (positive\/negative, low\/high severity) turned out to be unusable &#8211; very error prone. I will probably not offer a way to categorise freehand comments, but this is something worth keeping in mind if I get ideas in the future.<\/li>\n<li>Think-aloud was also a failure, authors attributed it to that code review takes a lot of concentration and saying what you&#8217;re thinking breaks that concentration. As a result &#8211; no any useful information was gathered from the talk-while-you-do-the-review process.<\/li>\n<li>The participants didn&#8217;t have previous experience with using a tablet PC but figured it out quickly. I have to make sure I come up with some exercise for my participants.<\/li>\n<li>The lack of a horizontal scrollbar was mentioned as a problem brought up by the participants. I have not even considered how scrolling works when you&#8217;re using a stylus. Something to keep in mind.<\/li>\n<li>A couple of participants suggested that colour syntax highlighting would have made the code easier to read. This will probably not work for me since I&#8217;ll be working with patches, but maybe.<\/li>\n<li>&#8220;Text-entry to the assignments via the on-screen keyboard proved to be tedious, so an external keyboard was added during the first participant&#8217;s session [&#8230;] We noted that markers used the pen and the keyboard simultaneously&#8221;. Hm. This is likely something I will run into also. Typing is much quicker than handwriting, quicker than talking even. So when the reviewer wants to provide a paragraph of comments they will certainly want a keyboard.<\/li>\n<li>&#8220;they had initially tried erasing annotations with the back of the pen&#8221;. I&#8217;m not sure I would have, but erasing annotations is important, and whether it works with the back of the pen depends on the hardware. Have to keep this in mind too.<\/li>\n<li>Here and in one other paper the Word review functionality was mentioned. Another paper (can&#8217;t remember which one) lists the advantages and disadvantages of that. I should probably have a similar list.<\/li>\n<li>Because of implementation troubles one computer ran the review software and another had the IDE (the second probably to build\/run the code). Probably won&#8217;t apply to me.<\/li>\n<\/ul>\n<p>\u201cRCA: Experiences with an IDE Annotation Tool\u201d (2006):<\/p>\n<ul>\n<li>When the underlying layout changes (e.g. font size, zoom) the review comments need to stay in place. This was mentioned in a few other places as &#8220;reflowing annotations&#8221;, and I thought it didn&#8217;t apply to me because the code I&#8217;m concerned with will not change. It will probably be a diff that will stay in the system forever whether it&#8217;s checked in or altered or not. But if I&#8217;m to integrate with ReviewBoard &#8211; the user will certainly be able to change the font size. Maybe for my study I can not care about this.<\/li>\n<li>Another vote for the ability to modify the review comments, here including not only erasing but selecting, moving, and recolouring. Eeee.. too fancy for my project, probably.<\/li>\n<li>It&#8217;s not clear what the final implementation was, but I think their ideal would have been a transparent canvas for the ink overlaid on the code editor. I&#8217;m not sure this is possible to do with the HTML canvas, but I expect I can just render whatever used to be rendered into the browser window into the canvas instead. Thankfully I don&#8217;t need to support editing of the code. Fingers crossed.<\/li>\n<li>A system similar to the Visual Studio breakpoints was used in RCA to choose a severity for the comment. They didn&#8217;t mention here whether it was used\/useful or not. I&#8217;m not sure whether this paper is newer or older than the one I mentioned above.<\/li>\n<\/ul>\n<p>\u201cCodeAnnotator: Digital Ink Annotation within Eclipse\u201d (2007):<\/p>\n<ul>\n<li>Nothing new here, but I&#8217;ve been kindly offered a chance to review the source code, might learn some useful things from it.<\/li>\n<\/ul>\n<p>&#8220;Issues of Extending the User Interface of Integrated Developmnet Environments&#8221;:<\/p>\n<ul>\n<li>&#8220;size and average complexity of projects has grown, the traditional way of reviewing code is no longer feasible&#8221;. I wonder if this is in reference to reviewing entire codebases, rather than specific changes as listed in a patch file. I think I will ignore this problem, assume that a diff is still the primary method of reviewing changes because it works well.<\/li>\n<\/ul>\n<p>&#8220;A comparative Evaluation of Annotation Software for Grading Programming Assignments&#8221; (2010):<\/p>\n<ul>\n<li>There&#8217;s a mention of prior work that found &#8220;Ink annotations are rich and expressive due to their free-format&#8221;, and &#8220;Ink annotations also have a part to play in supporting active reading&#8221;. I have to read these two papers to see if any of it has data useful for my study.<\/li>\n<li>An adapted means of categorising comments as\u00a0 &#8220;either a tick or cross, comment, grade, or other&#8221; was used. Given that this paper is a few years newer, perhaps this was found to be a good solution to the difficulties of categorising mentioned in the papers quoted above.<\/li>\n<li>&#8220;<a href=\"http:\/\/en.wikipedia.org\/wiki\/Latin_squares\">Latin squares arrangement<\/a>&#8221; &#8211; something I might want to use.<\/li>\n<li>This study was run on 600 assignments from 200 students, with real TAs giving real comments and marks. The data resulting from that is very useful and interesting:\n<ul>\n<li>Paper comments are much more useful than database (form field) comments, but grading on paper was universally disliked because it&#8217;s so cumbersome. 70% of the students never collected the marked papers.<\/li>\n<li>In the TAs feedback on the comparison of paper with the tablet the digital eraser was voted one of the biggest advantages.<\/li>\n<li>It did not take any longer to mark using the tablet than using either paper or the database method. This is encouraging, and something I hope to confirm in my study.<\/li>\n<li>Overall the tablet method was found to be the best of the three.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<p>There is so much interesting stuff in these papers I almost feel like starting to add that stuff as notes to my own, but I think it will be more prudent to just make notes on the side and in the blog for now, until I make some final decisions about what my study is going to look like.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>I&#8217;ve completed what I&#8217;m going to call the &#8220;primary&#8221; literature review for my project. I printed all the relevant papers I found to date and in the last two days I read and annotated the most immediately relevant of those &#8211; authored by Beryl Plimmer et al. I mentioned them in previous posts. I&#8217;ll post [&hellip;]<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"_links":{"self":[{"href":"http:\/\/littlesvr.ca\/masters\/wp-json\/wp\/v2\/posts\/73"}],"collection":[{"href":"http:\/\/littlesvr.ca\/masters\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/littlesvr.ca\/masters\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/littlesvr.ca\/masters\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"http:\/\/littlesvr.ca\/masters\/wp-json\/wp\/v2\/comments?post=73"}],"version-history":[{"count":4,"href":"http:\/\/littlesvr.ca\/masters\/wp-json\/wp\/v2\/posts\/73\/revisions"}],"predecessor-version":[{"id":76,"href":"http:\/\/littlesvr.ca\/masters\/wp-json\/wp\/v2\/posts\/73\/revisions\/76"}],"wp:attachment":[{"href":"http:\/\/littlesvr.ca\/masters\/wp-json\/wp\/v2\/media?parent=73"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/littlesvr.ca\/masters\/wp-json\/wp\/v2\/categories?post=73"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/littlesvr.ca\/masters\/wp-json\/wp\/v2\/tags?post=73"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}