CTT introduce compulsory helmets and front lights

[Jdsk]

One thing I'm reasonably confident about is that different people respond very differently to risk, and that what one is looking for in policy is a very broad common denominator, i.e. this intervention will do significantly more good than harm. This makes psychological effects potentially very difficult to factor in to policy decisions beyond understanding that they will make the error bars bigger. But as anyone who's done physics knows, you do have to account for the size of the error bars when stating something with confidence. It's still science if we say "we don't know enough in here, and here are the wildly varying results we got that show we don't know enough, but there are effects in there we haven't learned to measure usefully yet".

Yes. I'd start by expecting big differences between individuals and between settings. And good studies will measure the different sources of variation as well as any average effect.

Policy should include assessment of evidence along with other factors. And often that should include stating that there are things that we don't know yet.

This discussion would be much more constructive if it separated evidence from policy, and especially from fear of possible future policy.

Jonathan

[Stevek76]

And even that brings you awkward confounders.

Absolutely. The best that could be done is ensuring a suitably random sample (and of sufficient size) and randomizing the order in which the helmet and non-helmet run were undertaken. Which is another issue beyond the ethics one. Good quality samples are often a prohibitively time consuming or expensive.

Rather too many decisions on public policy, whether that's informed by academic research where the sample was whoever responded to a campus email and/or social media or a council producing quantitative analysis from a public consultation, are based on statistically worthless data.

And it's going to be difficult to run a study of adequate power because the rate of injury is so low.

There are proxies that could be used, e.g. in the mtb example the descent time is a perfectly adequate proxy. General road traffic is much trickier as time will be much more variable on matters beyond the study control (traffic & signals variations) but it would probably be possible to concoct something with from speed, acceleration and proximity data, sample requirements would be much larger though.

[thirdcrank]

One thing that struck me is that it seems some posters appear to have segued imperceptibly from the Helmets in the HC thread, to one concerned with the rules (and rulemaking) of a sport's governing body and that's one which has probably passed its heyday.

Here's something from an earlier age.

Time Trials 1940's and 1950's

https://www.classiclightweights.co.uk/r ... 40s-1950s/

[Jdsk]

There are proxies that could be used, e.g. in the mtb example the descent time is a perfectly adequate proxy. General road traffic is much trickier as time will be much more variable on matters beyond the study control (traffic & signals variations) but it would probably be possible to concoct something with from speed, acceleration and proximity data, sample requirements would be much larger though.

Yes, proxies can make it easier.

But in this forum I'd expect them to be immediately criticised because of the different outcome or different setting. And then discounted because they aren't perfect.

Those studies should be done and submitted for publication. And they should include their own statement of limitations, as in all of the recommended frameworks for evidence-based methods.

Jonathan

[Jdsk]

One thing that struck me is that it seems some posters appear to have segued imperceptibly from the Helmets in the HC thread, to one concerned with the rules (and rulemaking) of a sport's governing body and that's one which has probably passed its heyday.

And I expect this to keep happening until we start addressing different questions one at time and separating evidence from policy, whether it's individual, in an organised event, or national.

Jonathan

[thirdcrank]

I'm not going back to look but IIRC, the "Helmets in the HC" thread lost impetus when I suggested an answer to the unasked question What is to be done?

It would be easy to believe there's a greater appetite for discussion than action. Perhaps we might crowd-fund a university chair of cycle helmet science. I could nominate some worthy recipients of an honorary PhD.

Что дѣлать?

[Jdsk]

I'm not going back to look but IIRC, the "Helmets in the HC" thread lost impetus when I suggested an answer to the unasked question What is to be done?

That's a specific question. Thankyou. And of course it's about policy.

It would be easy to believe there's a greater appetite for discussion than action. Perhaps we might crowd-fund a university chair of cycle helmet science. I could nominate some worthy recipients of an honorary PhD.

Nobutseriously I'd encourage people to contribute to the development of the science as well as posting here. Could be in carrying out better systematic reviews. Could be in designing and performing better primary studies. Probably easiest done by working with a University department.

Jonathan

PS: Did Lenin cycle?

[thirdcrank]

More to the point, I don't know if he wore a helmet, of whatever type.

[pjclinch]

An application would be much more likely to succeed if it didn't rely on ad hominem criticisms and didn't suggest that a 20y old monograph based on a 30y old report outweighs all subsequent studies.

Risk's main point is outlining, with plenty of evidence, that anything involving risk is far more complicated to sort out than most people had previously thought. With the intervening experience I don't think there's anything contentious about that, despite the number of people that still think it's a case of engineering.

Until subsequent studies address that point and stop assuming that risk analysis of a complex real-world issue with significant psychological components is just like a blind random drug trial it's not so much that Risk outweighs them as it simply illustrates that they haven't had enough thought put in to their design to come up with decent results that mean anything.

What is it about the subsequent studies that you feel properly addresses the psychological complexities of helmet use that Adams suggests will confound them?

Much like the infamous BMJ editorial, the real conclusion of Risk is something along the lines of "actually it's all rather complicated". That's a very different kind of conclusion to "this device makes you x% more protected" or "this device does not engender any behavioural change to increase the risk of a crash". One is a clear finding, the other is a warning that clear findings are hard to come by. Thus, to produce clear findings you either need to do significantly better work than has been done, or demonstrate that the warnings don't apply. I don't see work on helmet efficacy that has managed either.

Pete.

[mattheus]

An application would be much more likely to succeed if it didn't rely on ad hominem criticisms and didn't suggest that a 20y old monograph based on a 30y old report outweighs all subsequent studies.

Risk's main point is outlining, with plenty of evidence, that anything involving risk is far more complicated to sort out than most people had previously thought. With the intervening experience I don't think there's anything contentious about that, despite the number of people that still think it's a case of engineering.

Until subsequent studies address that point and stop assuming that risk analysis of a complex real-world issue with significant psychological components is just like a blind random drug trial it's not so much that Risk outweighs them as it simply illustrates that they haven't had enough thought put in to their design to come up with decent results that mean anything.

What is it about the subsequent studies that you feel properly addresses the psychological complexities of helmet use that Adams suggests will confound them?

Much like the infamous BMJ editorial, the real conclusion of Risk is something along the lines of "actually it's all rather complicated". That's a very different kind of conclusion to "this device makes you x% more protected" or "this device does not engender any behavioural change to increase the risk of a crash". One is a clear finding, the other is a warning that clear findings are hard to come by. Thus, to produce clear findings you either need to do significantly better work than has been done, or demonstrate that the warnings don't apply. I don't see work on helmet efficacy that has managed either.

Pete.

Well put.

All the evidence so far says - to ME - that any benefits from helmets are too small to detect reliably.

If you run the "toss a coin 100 times" study often enough in different towns, you will find a town that gets more heads than normal. That doesn't mean it was a sensible study.

[drossall]

Would I trust a systematic review of risk compensation?

My BSc is in physics, so I'm in a different part of your joke. But I'm trying to understand systematic reviews at all, to be honest - which may just show my ignorance.

As I understand it, the point of the scientific method is that published papers are subject to comment and testing by others, who may well set out to do related studies that will support, or contradict, the conclusions of the original. So, no scientific paper is an island, but rather they are an interconnected network. And the conclusions are drawn (in part) from the relationships - a paper plus the replies and further studies that it engendered.

So, just looking at each paper in isolation is likely to fail to identify both those that have been central in forming the current consensus, and those which have been so contradicted by later results that their conclusions are of little value. I've not seen much in the reviews of, say, helmet studies to persuade me that they are taking account of these factors. But again, I'm not sure whether I'm qualified to comment.

[Jdsk]

Would I trust a systematic review of risk compensation?

My BSc is in physics, so I'm in a different part of your joke. But I'm trying to understand systematic reviews at all, to be honest - which may just show my ignorance.

As I understand it, the point of the scientific method is that published papers are subject to comment and testing by others, who may well set out to do related studies that will support, or contradict, the conclusions of the original. So, no scientific paper is an island, but rather they are an interconnected network. And the conclusions are drawn (in part) from the relationships - a paper plus the replies and further studies that it engendered.

So, just looking at each paper in isolation is likely to fail to identify both those that have been central in forming the current consensus, and those which have been so contradicted by later results that their conclusions are of little value. I've not seen much in the reviews of, say, helmet studies to persuade me that they are taking account of these factors. But again, I'm not sure whether I'm qualified to comment.

Systematic reviews are the current best tool for gathering information from multiple primary studies. Compared to previous review methods they're especially better at reducing selection bias.

They should make the methods of both selection and (any) synthesis transparent. And ideally they'd provide the data for anyone else who wants to carry out their own further processing.

There are quite a lot of different templates for carrying them out.

And some journals are now saying that they'll only publish reviews that meet some criteria of being done systematically.

Just a tool. Not perfect.

https://en.wikipedia.org/wiki/Systematic_review

And a quick guide to support critical appraisal of systematic reviews:
https://www.cebm.ox.ac.uk/resources/ebm ... isal-tools
It's definitely worth working through this on a real systematic review to get a flavour of how they work.

Jonathan

PS: Always happy to discuss SRs in general. But probably best to pick another domain. : - )

[drossall]

Thank you. That's really helpful. I'm a bit out of date (and I've never published any research in my life, nor been in danger of doing so).

Presumably they would only really work for eliminating selection bias when the primary studies were taking similar approaches?

[pjclinch]

SR is a way to concentrate signal to improve the S/N ratio over a body of papers. But you can only usefully do that if there's an underlying consistent signal.

I find it telling that Goldacre is very much a proponent of SR and in much of his writing goes out of his way to promote and encourage their use, but that Cycle helmets and the law says the current research body isn't really conclusive and there is a suggestion that more work along the same lines won't be either. He doesn't say all someone needs to do is systematically review it and we'll have progress. The drift (especially if read with the background on the badscience website or in I think you'll find it's a bit more complicated than that) is that it's a field where the current tools don't appear to be up to this particular job. Those current tools include SR.

Pete.

[Jdsk]

SR is a way to concentrate signal to improve the S/N ratio over a body of papers. But you can only usefully do that if there's an underlying consistent signal.

Systematic reviews can do that. But they can also be useful in establishing that there isn't a consistent signal there at all. Or that nothing at any appropriate level of the evidence hierarchy has been published to discover if there is a signal.

They're just a better tool for reviewing the literature than nonsystematic reviews or cherrypicking individual studies.

Jonathan