Twenty tips for interpreting scientific claims

  • Posted by a hidden member.
    Log in to view his profile

    Dec 01, 2013 6:37 AM GMT
    A friend of mine posted this on Facebook and thought it was an interesting read.

    Policy: Twenty tips for interpreting scientific claims
    http://www.nature.com/news/policy-twenty-tips-for-interpreting-scientific-claims-1.14183

    "This list will help non-scientists to interrogate advisers and to grasp the limitations of evidence, say William J. Sutherland, David Spiegelhalter and Mark A. Burgman."

    Differences and chance cause variation.
    No measurement is exact.
    Bias is rife.
    Bigger is usually better for sample size.
    Correlation does not imply causation.
    Regression to the mean can mislead.
    Extrapolating beyond the data is risky.
    Beware the base-rate fallacy.
    Controls are important.
    Randomization avoids bias.
    Seek replication, not pseudoreplication.
    Scientists are human.
    Significance is significant.
    Separate no effect from non-significance.
    Effect size matters.
    Study relevance limits generalizations.
    Feelings influence risk perception.
    Dependencies change the risks.
    Data can be dredged or cherry picked.
    Extreme measurements may mislead.
  • Posted by a hidden member.
    Log in to view his profile

    Dec 01, 2013 9:04 AM GMT
    Cool. icon_smile.gif
  • Posted by a hidden member.
    Log in to view his profile

    Dec 01, 2013 1:09 PM GMT
    Interesting list.
  • Posted by a hidden member.
    Log in to view his profile

    Dec 01, 2013 1:15 PM GMT
    Did I miss seeing:

    Has the claim gone through peer review
  • Posted by a hidden member.
    Log in to view his profile

    Dec 01, 2013 1:34 PM GMT
    I've been watching the Science Channel all weekend and, I swear, I'm getting spaced out (no pun intended) over having all this information poured into my brain and trying to comprehend over and over again what a "trillion trillion trillion trillion trillion" of anything means.
  • Posted by a hidden member.
    Log in to view his profile

    Dec 01, 2013 1:37 PM GMT
    ART_DECO saidDid I miss seeing:

    Has the claim gone through peer review


    Exactly what I was gonna say. This seems more like "20 reasons to be skeptical," which is also good! Proper use of skepticism and understanding statistics helps in identifying truths, or at least that which is as close to factual as anything can be in our universe. But the source and spread of claims are equally important these days, and usually the first indicator of whether something is worthy to even consider being skeptical about.
  • Posted by a hidden member.
    Log in to view his profile

    Dec 01, 2013 1:38 PM GMT
    Brian Cox is annoying though. I wish he'd stop smiling so much.
    Professor-Brian-Cox-holds-010.jpg
  • Posted by a hidden member.
    Log in to view his profile

    Dec 01, 2013 1:40 PM GMT
    I prefer to listen to Morgan Freeman. And look everybody. He's BLACK.
    gty_morgan_freeman_dm_120719_wblog.jpg
  • aguyinbrist

    Posts: 98

    Dec 01, 2013 2:25 PM GMT
    I have been PUBLISHED IN NEWSCIENTIST WHOOP WHOOP haha
  • aguyinbrist

    Posts: 98

    Dec 01, 2013 2:29 PM GMT
    Also a lot of what the OP posted is rubbish and only applies to a lot pseudo science out there.

    As a mathematician, I can tell you that mathematical rigour is so stringent that you simply cannot make unsustainable claims in mathematics or you'll be found out very very very quickly. There's a reason why very very very few people undertake mathematics at the research level like I did, I guarantee you it's not easy!
    But at least you have no fallacies such as bias and what not (statistics is not mathematics, probability on the other hand is mathematics). In fact, mathematicians can even compute distributions for correlations of variables which tell you how they move together on avaerage, formidable and EXTREMELY DIFFICULT results indeed.
  • GWriter

    Posts: 1446

    Dec 01, 2013 2:46 PM GMT
    aguyinbrist saidAlso a lot of what the OP posted is rubbish and only applies to a lot pseudo science out there.

    As a mathematician, I can tell you that mathematical rigour is so stringent that you simply cannot make unsustainable claims in mathematics or you'll be found out very very very quickly. There's a reason why very very very few people undertake mathematics at the research level like I did, I guarantee you it's not easy!
    But at least you have no fallacies such as bias and what not (statistics is not mathematics, probability on the other hand is mathematics). In fact, mathematicians can even compute distributions for correlations of variables which tell you how they move together on avaerage, formidable and EXTREMELY DIFFICULT results indeed.


    Why is it rubbish? There are a lot of fields, especially related to health and fitness (e.g. nutrition, biochemistry, physiology) that are legitimately scientific, but highly susceptible to error and misinterpretation. Not everything is as exact as mathematics. That doesn't make it "pseudo science." It just means a healthy skepticism is warranted, which was the OP's point. I thought the list was very useful.
  • Posted by a hidden member.
    Log in to view his profile

    Dec 01, 2013 3:02 PM GMT
    aguyinbrist saidAlso a lot of what the OP posted is rubbish and only applies to a lot pseudo science out there.

    As a mathematician, I can tell you that mathematical rigour is so stringent that you simply cannot make unsustainable claims in mathematics or you'll be found out very very very quickly. There's a reason why very very very few people undertake mathematics at the research level like I did, I guarantee you it's not easy!
    But at least you have no fallacies such as bias and what not (statistics is not mathematics, probability on the other hand is mathematics). In fact, mathematicians can even compute distributions for correlations of variables which tell you how they move together on avaerage, formidable and EXTREMELY DIFFICULT results indeed.


    If you would single out which of those points were 'rubbish and only applies to pseudo science out there' that might be kind to the non-initiated.
  • aguyinbrist

    Posts: 98

    Dec 01, 2013 3:15 PM GMT
    rodo said
    aguyinbrist saidAlso a lot of what the OP posted is rubbish and only applies to a lot pseudo science out there.

    As a mathematician, I can tell you that mathematical rigour is so stringent that you simply cannot make unsustainable claims in mathematics or you'll be found out very very very quickly. There's a reason why very very very few people undertake mathematics at the research level like I did, I guarantee you it's not easy!
    But at least you have no fallacies such as bias and what not (statistics is not mathematics, probability on the other hand is mathematics). In fact, mathematicians can even compute distributions for correlations of variables which tell you how they move together on avaerage, formidable and EXTREMELY DIFFICULT results indeed.


    If you would single out which of those points were 'rubbish and only applies to pseudo science out there' that might be kind to the non-initiated.


    It was with regards to mathematical sciences. In mathematics there are not MANY interpretations, there is usually ONE interpretation and it is the CORRECT one.
  • Apparition

    Posts: 3529

    Dec 01, 2013 3:42 PM GMT
    another rule...follow the money.

    if it seems like a situation is intuitively harmful or bad for the environment, and someone will profit from the opposite answer (that it is not harmful) and they paid for the study that declared something unharmful...it will generally be wrong.

    ie. we can burn all the coal we want and nothing bad will happen...all the studies by the coal industry say so.

    or

    fracking is completely neutral as to the influence on the local water supply...just like the fracking industry says. (just ignore those flames shooting out of your shower)
  • HottJoe

    Posts: 21366

    Dec 01, 2013 5:11 PM GMT
    Apparition saidanother rule...follow the money.

    if it seems like a situation is intuitively harmful or bad for the environment, and someone will profit from the opposite answer (that it is not harmful) and they paid for the study that declared something unharmful...it will generally be wrong.

    ie. we can burn all the coal we want and nothing bad will happen...all the studies by the coal industry say so.

    or

    fracking is completely neutral as to the influence on the local water supply...just like the fracking industry says. (just ignore those flames shooting out of your shower)

    Qft

    Energy from fossil fuels is bad for our environment, our economy, and our societies. We need renewable energy!!!
  • Posted by a hidden member.
    Log in to view his profile

    Dec 01, 2013 5:21 PM GMT
    Stop this nonsense.
    Only *my* science is valid!
    SouthPark2.jpg
  • Posted by a hidden member.
    Log in to view his profile

    Dec 01, 2013 6:58 PM GMT
    aguyinbrist saidI have been PUBLISHED IN NEWSCIENTIST WHOOP WHOOP haha


    Congratulations! Which issue is it in?
  • Paperless_Pen

    Posts: 573

    Dec 01, 2013 7:09 PM GMT
    aguyinbrist saidAlso a lot of what the OP posted is rubbish and only applies to a lot pseudo science out there.

    As a mathematician, I can tell you that mathematical rigour is so stringent that you simply cannot make unsustainable claims in mathematics or you'll be found out very very very quickly. There's a reason why very very very few people undertake mathematics at the research level like I did, I guarantee you it's not easy!
    But at least you have no fallacies such as bias and what not (statistics is not mathematics, probability on the other hand is mathematics). In fact, mathematicians can even compute distributions for correlations of variables which tell you how they move together on avaerage, formidable and EXTREMELY DIFFICULT results indeed.


    But this isn't about mathematics. Science is an empirical pursuit.
  • Posted by a hidden member.
    Log in to view his profile

    Dec 01, 2013 8:02 PM GMT
    Paperless_Pen said
    aguyinbrist saidAlso a lot of what the OP posted is rubbish and only applies to a lot pseudo science out there.

    As a mathematician, I can tell you that mathematical rigour is so stringent that you simply cannot make unsustainable claims in mathematics or you'll be found out very very very quickly. There's a reason why very very very few people undertake mathematics at the research level like I did, I guarantee you it's not easy!
    But at least you have no fallacies such as bias and what not (statistics is not mathematics, probability on the other hand is mathematics). In fact, mathematicians can even compute distributions for correlations of variables which tell you how they move together on avaerage, formidable and EXTREMELY DIFFICULT results indeed.


    But this isn't about mathematics. Science is an empirical pursuit.


    Yes. It's about scientific skepticism and ignorance among politicians. I'm a cynic though. If a politician has a vested interest in the outcome of a scientific study, no amount of scientific training or empirical evidence is going to stop them from spinning the result to their advantage. Science is a human endeavour, but politics is an unashamedly human endeavour!
  • Posted by a hidden member.
    Log in to view his profile

    Dec 01, 2013 8:14 PM GMT
    Brian Cox has just informed me that in all the uncountable years in the life of the universe, it's only the middle 1 trillion trillion trillion (I lost count of how many times he said "trillion") trillionth that can support life. Interesting.
    AgeUniverseIcon.gif
  • GWriter

    Posts: 1446

    Dec 01, 2013 9:05 PM GMT
    aguyinbrist said
    rodo said
    aguyinbrist saidAlso a lot of what the OP posted is rubbish and only applies to a lot pseudo science out there.

    As a mathematician, I can tell you that mathematical rigour is so stringent that you simply cannot make unsustainable claims in mathematics or you'll be found out very very very quickly. There's a reason why very very very few people undertake mathematics at the research level like I did, I guarantee you it's not easy!
    But at least you have no fallacies such as bias and what not (statistics is not mathematics, probability on the other hand is mathematics). In fact, mathematicians can even compute distributions for correlations of variables which tell you how they move together on avaerage, formidable and EXTREMELY DIFFICULT results indeed.


    If you would single out which of those points were 'rubbish and only applies to pseudo science out there' that might be kind to the non-initiated.


    It was with regards to mathematical sciences. In mathematics there are not MANY interpretations, there is usually ONE interpretation and it is the CORRECT one.

    Based on this thread, I'm not impressed with the social and conversational skills of mathematicians.
  • Posted by a hidden member.
    Log in to view his profile

    Dec 01, 2013 9:24 PM GMT
    aguyinbrist said
    rodo said
    aguyinbrist saidAlso a lot of what the OP posted is rubbish and only applies to a lot pseudo science out there.

    As a mathematician, I can tell you that mathematical rigour is so stringent that you simply cannot make unsustainable claims in mathematics or you'll be found out very very very quickly. There's a reason why very very very few people undertake mathematics at the research level like I did, I guarantee you it's not easy!
    But at least you have no fallacies such as bias and what not (statistics is not mathematics, probability on the other hand is mathematics). In fact, mathematicians can even compute distributions for correlations of variables which tell you how they move together on avaerage, formidable and EXTREMELY DIFFICULT results indeed.


    If you would single out which of those points were 'rubbish and only applies to pseudo science out there' that might be kind to the non-initiated.


    It was with regards to mathematical sciences. In mathematics there are not MANY interpretations, tere is usually ONE interpretation and it is the CORRECT one.


    I trust the original points were formulated with regards to empirical science. As you know, it is ill-advised to refer to that as "pseudo science" given its inability to provide with 'definitive' conclusions (or given anything else in general).
  • Posted by a hidden member.
    Log in to view his profile

    Dec 01, 2013 9:33 PM GMT
    Can you design an experiment in the field of mathematics?
  • Posted by a hidden member.
    Log in to view his profile

    Dec 01, 2013 10:52 PM GMT
    "This list will help non-scientists to interrogate advisers and to grasp the limitations of evidence, say William J. Sutherland, David Spiegelhalter and Mark A. Burgman."

    Not sure about that. The list is good, but I doubt the vocabulary speaks to non scientific people. Regression, extrapolation, significance etc... refer to precise science notions.

    As for mathematics vs experimental science, there is no competition or hierarchy. Experimental science use math rather rigorously, the same way biology use chemistry.

    aguyinbrist saidthat you simply cannot make unsustainable claims in mathematics


    Sound to me that Godel demonstrated you can build a true proposition that you can't demonstrate, sound close to 'unsustainable claim', no ? ;)
  • SargeKO

    Posts: 42

    Dec 02, 2013 1:10 AM GMT
    POWER! You need always consider the power (statistical definition) of your experimental design!