First, here’s a spreadsheet that shows all the comments. Nice comments are in blue, problems are in red:
To aggregate, I’ve added a column on the right that has a “meta” description of the issue. After sorting, it can be seen that there are four problem issues that relate to “help”, and then three issues regarding layout and formatting. There is one “Transition” issue.
Clearly,the main issue here is that there is no sitewide help. The other two issues are primarily appearance-related. This isn’t surprising. Content-management systems are pretty good about providing functional websites. Making them uncluttered and is another matter – particularly in volunteer organizations where everything is done by committee These sites tend to have lots of content, but no aesthetic theme.
Reflecting on this exercise, I’m of two minds. First I’m glad I don’t do this for a living. I would go mad. Which leads me to my next thought. Does anyone like doing this? Or is this still done mostly because we have yet to figure out how to do this using automation? I’m not talking about bug-tracking here, because that is more tied into the process of development (for example, some incorrect calculation feature in a spreadsheet). And in fact, bug-tracking and fixing is reasonably mature. We have white box and back box tests that are run under continuous integration systems. And yet there seems to be nothing of the kind for GUIs.
Recently, there has been some research in the use of Machine vision for this kind of testing: [http://groups.csail.mit.edu/uid/projects/sikuli/sikuli-chi2010.pdf], [http://link.springer.com/article/10.1007%2Fs11219-011-9135-x?LI=true]. This work seems to point to the possibility that the era of tedious, manual examination of GUIs may be coming to an end. Personally, I can’t wait.