Saturday 16 October 2010
Distrusting Research
Here's an interesting article about bias in medical research. Almost everything they say about medical research applies to computer science too.
So you have to be very skeptical of empirical results in computer science, especially "we tried this and it worked" results. Fortunately --- unlike biology, I suspect --- at the heart of many computer science papers is just a clever idea or observation someone came up with. You can take that idea and try it for yourself. But don't bet the company on it just because some researcher said it worked great.
Of course, the situation is still pretty bad overall. The sick research culture that only rewards positive results doesn't just create selection bias in reported results; it also deprives us of interesting data about ideas that don't work. After all, people only try things they think will work, so failures on average should be more surprising --- and therefore more interesting --- than successes.
At PLDI this year not a single paper reported a negative result. However, during the short talks session one group from Microsoft presented a cute piece of work applying an automated reasoning tool to perform compilation; nice idea, but it didn't really work. I was so excited I ran up to the microphone just to say "thank you for presenting a negative result!". Then I added "We need more results like this", and everyone laughed...
Comments
http://www.bluebytesoftware.com/blog/2010/01/03/ABriefRetrospectiveOnTransactionalMemory.aspx
And Simon Peyton-Jones' mailing-list followup on why Haskell's STM avoids many of the difficulties the Microsoft people ran into:
http://article.gmane.org/gmane.comp.lang.haskell.cafe/78833
Of course, neither of those were academic papers. I'm not holding my breath for academic incentive systems to reorganize themselves to encourage the sharing of negative results, but I do have some hope for public discussions about negative results in other (less formal) contexts. Maybe those other contexts, like blogs and mailing lists, are actually better suited to discussing negative results? They facilitate a rapid back-and-forth discussion that may make it easier to suss out the specific reasons for a particular negative result. And those discussions might also shed some light on the remaining opportunities to move forward that skirt the negative result.
Have any negative results of your own that you'd like to share?
And of course, this is another evidence of ASCII rule ... whatever is interesting to know, is usually in text/plain :)
Yes mistakes are made, but somehow, great progress is also made. Just to give you one example, my wife is an oncology nurse. In the past 25 years or so the practice has improved tremendously. Patients tolerate the new drugs much better, and they live longer. It's a good bet that without medical science, most of us would not be alive now. Astonishing discoveries continue to be made in all fields, and I can't think of a single area of science that hasn't made great strides in recent years.
Biomedical science has its own unique problems, but remember, it's also unusually difficult. In most cases it's difficult or impossible to do experiments on human subjects. There are also obvious problems with the system, as in all human endeavors, but here too I am confident that progress will be made.
Nutrition is one example of the latter. Computer security is another.
I agree with you. The CHI community has a special place for papers that report negative / null / controversial results, called alt.CHI. The community get to read the papers and make comments about them and then a final jury of experts make the decision about which papers should be presented at the conference.
http://chi2011.org/authors/altchi/index.html
Cheers Craig