In the summer, men's health campaigners in the UK were amazed to see headlines on the BBC claiming that the 'vast majority of patients' were satisfied with their doctors' surgery hours. This is NOT what most men tell us. We rely on statistics but EMHF website editor Jim Pollard, a self-confessed stats cynic, argues that campaigners across Europe need to be aware of their uses and abuses.
The headlines on the BBC (right) and elsewhere reporting widespread satisfaction with family doctors (GPs) opening hours and other aspects of the service followed publication of the GP Patient Survey (GPPS) conducted by the British Department of Health. Some interpreted the results as proof that GP practices are open at times to suit the vast majority of patients.
If you find this claim hard to credit, you're not alone. Most of us have heard men complain about doctors' surgery hours.
Even the DH’s own sources suggest that this supposed satisfaction is a statistical anomaly rather than a true picture. Certainly the Healthcare Commission (HC) survey from January, also commissiond by the Department, gives a very different impression.
In the HC survey, based on an admittedly smaller but still large sample, 25% of people said specifically that inconvenient opening hours had put them off going to the GP within the preceding 12 months. That's one in four. Furthermore, only 28% of people said that additional opening hours were not necessary.
So why the difference?
These two surveys make an interesting case-study in the sort of scrutiny to which we should be subjecting survey findings. The resulting headline may be less interesting but the picture will be more complete.
The GPPS carried out by Ipsos MORI survey appears to have actually been two simultaneous surveys. One of these surveys was not postal or online but completed by the GP with their patients. Moreover, both surveys within the GPPS chose, or were weighted to cover, only patients who had made an appointment in the previous six months.
People sitting with their GP are less likely to be critical while those who have had a recent appointment are demonstrably less inconvenienced by opening hours than the many thousands who didn’t make an appointment.
Don Redding, head of communications at the Picker Institute which carried out the HC survey, says: ‘In general the results differ because the surveys themselves differ. The questions are worded differently, there is a big difference in the number of response options, and the base for analysis is different. The access questions in the GPPS survey are based on those who made an appointment with a GP within the 6 months prior to the survey, whereas our sample was based on anyone registered with the practice. Not surprisingly this gives a massive disparity.’
Specifically, the two questions on opening hours could hardly have been more different and still be about the same subject.
The GPPS asked ‘Over the last 6 months or so, were you satisfied with the hours your GP surgery was open?’ There were just two options: 'Yes I was satisfied with the opening hours' or 'No, I was dissatisfied with the opening hours'. In this context, it is less surprising that 84% of people said they were satisfied.
By contrast, the HC survey asked: 'In the last 12 months, have you ever been put off going to your GP practice/ health centre because the opening times are inconvenient for you?' There were three possible responses: Yes, often; Yes, sometimes; and No. As we have seen, in response, 25% stated that in the last 12 months they had been put off because they found opening hours inconvenient.
Don Redding explains the differences: ‘GPPS asks respondents to generalise their satisfaction with the opening hours over the last 6 months. The HC question asks more specifically about experience – have they EVER been put off from going. The HC response options discriminate between people more. Those 18% who said "yes, sometimes" could have said they were satisfied with the opening hours in the GPPS survey.’
There was a similar gulf with other questions. Take, for example, the GPPS finding: 86% of people reported that they were satisfied with their ability to get through to their doctor’s surgery on the phone. This contrasts starkly with the HC survey in which 58% reported difficulty contacting the local practice/health centre by phone.
The GPPS question was 'In general, are you satisfied with how easy it is to get through to someone on the phone at your doctor’s surgery?' The HC question was 'Have you had a problem getting through to your GP practice/ health centre on the phone?'
Do Redding says: ‘The questions are asking two conceptually different things – the GPPS question is asking for patients to generalise and to rate their satisfaction with getting through on the phone, where as our question asks more specifically about the patient’s experience of having a problem getting through. Our methodology at Picker is to always use reports of actual experience. We consider satisfaction questions as less precise and less useful.’
As Men's Health Forum CEO Peter Baker said in his letter to the BMJ on this subject: ‘Relying just on the last set of statistics is always an unwise path to tread’. Arguably, relying on any set of statistics is unwise if their methodology has not been scrutinised.
