For my study I need 3 code samples of equal complexity and with the same number of mistakes in Java, plus 3 in C/C++.
I dug through my entire undergraduate homework tree (which I kept for sentimental value) and only found one assignment that was appropriate – an introductory Java assignment.
I forgot about this problem for some weeks, but now that I’m getting close to running the experiments I need to get it ready. My search for other people’s assignments failed, so I decided to write my own.
Immediately it became obvious that I haven’t seriously considered this problem. Each participant gets 3 pieces of code to review, one for each review method, and I haven’t thought the following through:
- Should they be the same code? Apparently obviously not, the second and third reviews would be tainted by the participant’s comments on the first.
- Should it be completely different code? The problem above goes away but then this becomes a huge variable, since a student may be well familiar with one concept (e.g. string manipulation) but completely unfamiliar with another (e.g. recursion).
- Should it be code that uses similar techniques, but is solving a different problem?
I picked (3). I don’t think it entirely solves the problems in (1), since the same feedback may apply to the same mistakes in all 3 sets of code, but that problem would exist regardless. Actually I know someone who may have an answer to that before I get to the analysis stage of my dissertation, that will be nice to mention.
Having made this decision I dug up another introductory Java assignment specification and implemented it. The results were astounding. One would think that given several years of work experience it should be easy, but if I were still an undergrad student and I submitted this, I would have received a failing grade. The specifications are completely confusing and the requirements make no sense. Not having a prof to ask what he meant makes it nearly impossible to implement properly.
Which I later thought is ok. My goal was to make sure that the second set of code is of the same quality as the first, and adherence to assignment specifications is irrelevant. I made sure the code compiles and did my best to make sure that fundamentally it makes sense.
Then I introduced some errors of different types, in both assignments. Hopefully they are roughly the same now.
This exercise took 4 solid hours of hard work, and I have to do 4 more (1 java and 3 C/C++). I’ll try to find time somewhere.