{"id":65,"date":"2010-03-21T21:07:04","date_gmt":"2010-03-22T01:07:04","guid":{"rendered":"http:\/\/littlesvr.ca\/masters\/?p=65"},"modified":"2010-03-21T21:07:04","modified_gmt":"2010-03-22T01:07:04","slug":"potential-experiment-attempt-1","status":"publish","type":"post","link":"http:\/\/littlesvr.ca\/masters\/2010\/03\/21\/potential-experiment-attempt-1\/","title":{"rendered":"Potential experiment &#8211; attempt #1"},"content":{"rendered":"<p>One of the things I can do now is try and define what my experiments with people will look like. To perform these experiments I would need to get approval from the UofT ethics board, and to get that approval I need to fill out an application describing what sort of information I collect from whom under what circumstances.<\/p>\n<p>This is the first attempt. Let&#8217;s assume for the purpose of this excercise that the software will be an extension of Review Board.<\/p>\n<p>This will give me potential access to the Basie project (my fellows <a href=\"http:\/\/zuzelvp47uoft.wordpress.com\">Zuzel<\/a> and <a href=\"http:\/\/mikeconley.ca\/blog\">Mike Conley<\/a> are working with it), a bunch of open source projects and perhaps even a business or two of those listed on the <a href=\"http:\/\/www.reviewboard.org\/users\/\">Review Board site<\/a>, and whomever else I can find.<\/p>\n<p><a href=\"http:\/\/pyre.third-bit.com\/blog\">Greg<\/a> mentioned I need to be in physical proximity to the subjects, I forgot to ask why &#8211; perhaps it&#8217;s not necessary for the experiment the way I define it. The most likely candidate is Basie.<\/p>\n<p>I would very much prefer to have a controlled experiment, but in a real environment. Which means that review board has to be already actively used in the project I&#8217;m looking at, and at least several reviews per week are done. This may be a challenge, but I&#8217;ll work with the assumption that I can find such a project.<\/p>\n<p>Oh, the bigger problem is that the reviewers need tablets (using a mouse will be useless). This is probably what Greg meant.<\/p>\n<p>I will have N reviewers participationg. For this description I will only mention one, and I&#8217;ll call this person Roy the reviewer. I will give Roy a tablet and a URL to 4 code submissions:<\/p>\n<ul>\n<li>Two simple changes (something like a new 2nd year undergrad hello world function)<\/li>\n<li>Two complex changes (maybe database access, or more than one file affected, or changes spread wide across one file)<\/li>\n<\/ul>\n<p>Each of these will have the same number of problems, but they will be different so that one review doesn&#8217;t affect the other.<\/p>\n<p>I will ask Roy to review one of the first and one of the second normally, by typing in comments. This would tell me:<\/p>\n<ol>\n<li>What number of substantively different comments they provided<\/li>\n<li>How many of those are of what type (design flaw, bug, usability problem, style issue)<\/li>\n<li>How much time was spent on the review<\/li>\n<\/ol>\n<p>Then I will ask Roy to try the stylus, to have him get used to using it as a pen, then to use the stylus to add review comments either inline or on the side of the code they&#8217;re talking about, treating the monitor as a printout.<\/p>\n<p>Then I would compare the results.<\/p>\n<p>Using a stylus should not be a problem. But experienced reviewers have a system developed for providing good feedback using the oldschool method, so I have to account for that somehow. Perhaps a few days later I can repeat the experiment and see if their use of the new system is any different.<\/p>\n<p>Do this with N people, and perhaps some statistically significant results can be deduced.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>One of the things I can do now is try and define what my experiments with people will look like. To perform these experiments I would need to get approval from the UofT ethics board, and to get that approval I need to fill out an application describing what sort of information I collect from [&hellip;]<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"_links":{"self":[{"href":"http:\/\/littlesvr.ca\/masters\/wp-json\/wp\/v2\/posts\/65"}],"collection":[{"href":"http:\/\/littlesvr.ca\/masters\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/littlesvr.ca\/masters\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/littlesvr.ca\/masters\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"http:\/\/littlesvr.ca\/masters\/wp-json\/wp\/v2\/comments?post=65"}],"version-history":[{"count":3,"href":"http:\/\/littlesvr.ca\/masters\/wp-json\/wp\/v2\/posts\/65\/revisions"}],"predecessor-version":[{"id":68,"href":"http:\/\/littlesvr.ca\/masters\/wp-json\/wp\/v2\/posts\/65\/revisions\/68"}],"wp:attachment":[{"href":"http:\/\/littlesvr.ca\/masters\/wp-json\/wp\/v2\/media?parent=65"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/littlesvr.ca\/masters\/wp-json\/wp\/v2\/categories?post=65"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/littlesvr.ca\/masters\/wp-json\/wp\/v2\/tags?post=65"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}