Yesterday OfCom released a report [pdf] on the current state of UK mobile networks – a report similar to ours on the subject from last week, which found that 4G speeds had halved over the past year in the UK. OfCom focussed their report on network performance, rather than coverage, and cited our report to show that independent analysis is being carried out on the comparative coverage provided by MNO’s. Importantly, OfCom carried out their study by comparing performance in sites around the UK, both indoor and outdoor, where all four networks were present – and their findings were different to ours, with higher reported speeds on both 3G and 4G. In the OfCom report EE ranked as the fastest 4G LTE network while Vodafone were the fastest 4G network based on OpenSignal user data.
We feel very strongly that these kind of reports are exactly what OfCom should be doing, and that any and all independent data made available to consumers is a positive step leading to a more efficient market. We have made clear in the past that we want OfCom to do more independent testing, and make use of different datasets (such as OpenSignal) – so that better consumer information on network performance will force the operators to compete more in terms of the actual cellular service they provide. We are supporters of all independent information that helps to achieve this end, and we are delighted that OfCom chose to cite our report, as it shows they are open to innovation and aware of the limitations of traditional testing methods. That being said, we retain a number of reservations about the methodology used in the OfCom study.
1) The data is up to eight months old: With the current pace of network upgrades, using data up to eight months old is not likely to reflect the current state of network performance. OfCom’s data collection ran from March to June, while our OpenSignal report covers the three months leading to October 1st. This is especially a concern when it comes to 4G measurements, as our UK report showed that average 4G LTE speeds across all networks had fallen 15% from March to October, meaning that OfCom may be overstating the speeds currently experienced by real consumers.
2) The type of device used: OfCom tested using a Galaxy Note III, a mobile phone which is available across all networks. They justify this by saying they want to test the network without it being affected by the user’s device, as different devices experience different speeds. While this is a legitimate way to judge comparative network performance, we do not feel that it is the best way to show the real user experience of the network. Testing using a consumer device in this way sits at an uncomfortable halfway point between using specialised testing equipment and data crowdsourced from real user devices, and is unable to fully capture the full spread of user experience. Many users do not have high-end devices, so testing using a device such as the Galaxy Note III is likely to overstate speeds. Our network averages represent the fact that users have many different devices, and our data is therefore naturally weighted by device market share – as every user counts the same – giving a more accurate picture of the diversity of the UK’s mobile device usage. We feel our methodology complements the OfCom data, as it would be too time consuming to road-test using a full spread of devices and OfCom do not claim to be directly measuring the user experience, despite using a consumer device for their testing.
3) The location of tests: OfCom test both indoors and outdoors (a positive development from the days when network testing included almost no indoor tests) but they test 50% on each, which is not necessarily representative of typical use. Above all, by testing in areas of good connectivity (where all four networks are present) they are potentially skewing their results to the faster end.
4) Testing on EE ‘double speed’: OfCom ran their tests using the EE 4G ‘double speed’ sim (without fully explaining why, as they stated they intended to test both the ‘single’ and ‘double’ speed tariffs, but ended up only testing the faster one). Many EE 4G users are not on the faster speed tariff and so reported speeds for EE are only representative of the experience available to users who are paying for the faster data, and therefore possibly not representative of those who are paying for 4G LTE on EE on the ‘single speed’ network.
5) What OfCom are actually measuring: OfCom are not measuring the typical speeds users actually experience (and this is by design), and therefore overstate the speeds consumers are likely to get. This explains why we report the mobile networks to be slower on both 3G and 4G than OfCom, as our testing is directly measuring performance as experienced by users rather than modelling it based on controlled tests.
6) Lack of Coverage data: OfCom rely on coverage data self-reported by the operators themselves, looking at the proportion of premises covered in the UK. This metric is an attempt to combine raw geographic coverage with is impact on users, but we feel that, in isolation, it is some way divorced from the actual availability of networks for consumers. OfCom cite our ‘time on’ metric, which looks at the proportion of time users have a connection, as an alternative metric for coverage and we feel it is entirely complementary to the more traditional geographic metrics used by OfCom, as it helps put the ‘premises’ figure into perspective. OfCom’s testing methodology is not able to gather accurate data on coverage, and this means that their report on network performance cannot be entirely complete, as it only records data from where all networks are present. While differences in performance are right to be noted, and can be significant, what is more significant is the actual availability of the network itself.
The rise of independent reporting is vital for the on-going success of a competitive mobile market in the UK, and we feel that OfCom’s report is important for bringing questions of network performance to the forefront of consumers’ minds. We do, however, believe that additional useful and up-to-date information can be made available to consumers through additional techniques, such as crowdsourcing, and would encourage the OfCom [pdf] and OpenSignal reports to be read side-by-side to paint a more complete picture of the current state of network provision in the United Kingdom.