Sunday, 17 October 2010

Distrusting Research

Here's an interesting article about bias in medical research. Almost everything they say about medical research applies to computer science too.

So you have to be very skeptical of empirical results in computer science, especially "we tried this and it worked" results. Fortunately --- unlike biology, I suspect --- at the heart of many computer science papers is just a clever idea or observation someone came up with. You can take that idea and try it for yourself. But don't bet the company on it just because some researcher said it worked great.

Of course, the situation is still pretty bad overall. The sick research culture that only rewards positive results doesn't just create selection bias in reported results; it also deprives us of interesting data about ideas that don't work. After all, people only try things they think will work, so failures on average should be more surprising --- and therefore more interesting --- than successes.

At PLDI this year not a single paper reported a negative result. However, during the short talks session one group from Microsoft presented a cute piece of work applying an automated reasoning tool to perform compilation; nice idea, but it didn't really work. I was so excited I ran up to the microphone just to say "thank you for presenting a negative result!". Then I added "We need more results like this", and everyone laughed...



6 comments:

  1. I liked Joe Duffy's blog retrospective on Microsoft's STM experiments:
    http://www.bluebytesoftware.com/blog/2010/01/03/ABriefRetrospectiveOnTransactionalMemory.aspx
    And Simon Peyton-Jones' mailing-list followup on why Haskell's STM avoids many of the difficulties the Microsoft people ran into:
    http://article.gmane.org/gmane.comp.lang.haskell.cafe/78833
    Of course, neither of those were academic papers. I'm not holding my breath for academic incentive systems to reorganize themselves to encourage the sharing of negative results, but I do have some hope for public discussions about negative results in other (less formal) contexts. Maybe those other contexts, like blogs and mailing lists, are actually better suited to discussing negative results? They facilitate a rapid back-and-forth discussion that may make it easier to suss out the specific reasons for a particular negative result. And those discussions might also shed some light on the remaining opportunities to move forward that skirt the negative result.
    Have any negative results of your own that you'd like to share?

    ReplyDelete
  2. Robert O'Callahan17 October 2010 22:14

    My thesis. Using Hindley-Milner type inference (even plus extensions) for alias analysis in Java didn't work.

    ReplyDelete
  3. The article is really excellent, thank you for the link. I have to add in this context old good http://www.lhup.edu/~DSIMANEK/cargocul.htm Feynman is just briliant, but still when looking around myself I am afraid most people (by far not only in computer so called science) act as this was never spoken.
    And of course, this is another evidence of ASCII rule ... whatever is interesting to know, is usually in text/plain :)

    ReplyDelete
  4. VanillaMozilla21 October 2010 17:26

    There is something very troubling about that article. It must be tempting to read it and ignore all research, or worse yet, oppose its funding.
    Yes mistakes are made, but somehow, great progress is also made. Just to give you one example, my wife is an oncology nurse. In the past 25 years or so the practice has improved tremendously. Patients tolerate the new drugs much better, and they live longer. It's a good bet that without medical science, most of us would not be alive now. Astonishing discoveries continue to be made in all fields, and I can't think of a single area of science that hasn't made great strides in recent years.
    Biomedical science has its own unique problems, but remember, it's also unusually difficult. In most cases it's difficult or impossible to do experiments on human subjects. There are also obvious problems with the system, as in all human endeavors, but here too I am confident that progress will be made.

    ReplyDelete
  5. Robert O'Callahan21 October 2010 22:24

    I agree that great progress has been made in some areas, but in other areas vast resources have been expended and apparently not much real progress has been made.
    Nutrition is one example of the latter. Computer security is another.

    ReplyDelete
  6. Hi Rob,
    I agree with you. The CHI community has a special place for papers that report negative / null / controversial results, called alt.CHI. The community get to read the papers and make comments about them and then a final jury of experts make the decision about which papers should be presented at the conference.
    http://chi2011.org/authors/altchi/index.html
    Cheers Craig

    ReplyDelete