P.S. This was written on the 18th of july, before I started the experiments. I’ve decided not to publish it at that time but to wait until the experiments are complete, which is now.
Last wednesday two of my peers Mike and Zuzel have kindly helped me by reviewing some code using my new tablet, a customised Slice, and a Java assignment I wrote in my 2nd year undergraduate degree. The point in doing this was to find problems with the experiment, and get other insights that might help do it right with actual participants (because of their close involvement with my project they cannot count as participants in the experiment). It’s turned out to be very valuable, I’ve learned quite a bit – thanks guys!
Here I will relate things that I feel should not contaminate participants if they happen to stumble across this post.
One of the first things that became obvious was that the stylus needs to be calibrated for each participant. I watched Mike calibrate it and he’s done the same I would have, but Zuzel’s settings have been quite different. This will not be a problem, just put the control panel icon on the desktop and ask each participant to run it, it only takes 6 clicks for the entire process.
I failed to time either of them. As I was thinking about it I realised this is a much more complicated problem than I thought. Both Mike and Zuzel stopped during their review – either to ask something or to join a conversation. Do those pauses count as part of the 20 minutes I’m planning, or not really? Should I let them at it for as long as they are willing? And should I instrument some breaks/distractions so the participants don’t get bored?
Which reminded me – I have not given much thought to something Greg mentioned a while ago: what order do the participants do the reviews in? Always the same (such as tablet then form then paper), or a random order for each person? Given the expected small number of participants – random order would probably still need analysis, and I would rather not have the extra variable. So I’ll probably have them all done in the same order, just have to decide on it.
A keyboard was not available, even though it’s in my project plan to add keyboard input functionality for Slice. I asked both Mike and Zuzel whether they missed it, and whether they would have used it if it were available. Mike thought so, but wasn’t sure how its use would balance with that of the stylus, Zuzel was sure she wouldn’t have.
Neither used the line numbers, which is good I guess (I can just get rid of them), and not unexpected given the simplicity of the code.
It’s become apparent that a problem with the software I’ve been aware of is a major one. The eraser can erase not only the stuff penned in, but also the line numbers and the code itself. Zuzel had this happen 3 times, Greg had this happen the one time he used the software, and it happens to me periodically. That means 75% of the participants will run into this problem, which has the following effect:
I can fix it post-mortem by overlaying the comments over the code, but such an occurrence pretty much invalidates the data gathered from the participant it’s happened to. It definitely needs to be fixed.
Also I thought the code was too complicated. It’s not really complicated code, but when you have 20 minutes to become familiar enough with it and review it – anything other than a single function is too complicated. I’ve decided to leave it as it is though, it seems to be more a realistic review this way.