In this blog post, we’re taking a deep dive into the very core of OpenSignal: explaining the metrics we use, what they all mean and what their roles are in measuring the real-world mobile network experience as users see it.
How do we collect the data in the first place?
We collect and analyze more than 3 billion measurements every day, from more than 100 million smartphones across the world. We collect data every day of the week, at all hours and in all the places people live, work and travel: no simulations, no predictions, no idealized testing conditions. Our data comes from actual smartphone users and we report users’ actual network experience, whether they are indoors or out, bustling in a busy city or trekking in the countryside.
We collect the vast majority of our data via automated tests that run in the background, enabling us to report on users’ real-world mobile experience at the largest scale and frequency in the industry. These automated tests are run at random points in time and therefore represent the typical experience available to a user at any given moment.
Availability – Adding time to the equation
Our 4G availability metric shows the proportion of time users with a 4G device and subscription, have an LTE connection. When we report an average 4G availability of 75%, that means our LTE users were, on average, connected to LTE services on their network 75% of the time.
Availability contrasts with coverage metrics such as population coverage which considers an individual 100% ‘covered’ just by looking at if they have coverage at their home. The reality is that every user experiences some degree of both being covered and not being covered in different places, but this nuance is lost with the population coverage metric. What really matters is that mobile users get coverage in the places they actually use their phones, whether inside their homes, at their offices, in the supermarket or on the go. Looking at the proportion of time that users can get an LTE connection provides us with a more precise reflection of the truth.
We also keep track of the instances that leave mobile users most frustrated: when there is no signal to connect to at all. The most common dead zones users struggle with occur indoors. As most of our availability data is collected indoors (not surprisingly, since that’s where users spend most of their time), we’re particularly astute at detecting those areas of zero coverage. (You can find poor connectivity areas and compare operators’ real-world coverage with our online coverage maps).
Download Speed – The real speeds users get
When it comes to average download connection speeds we report publicly on three variations: 3G, 4G and overall speed. Measured in Mbps, our 3G and 4G speed metrics reflect the average download speeds our users’ experience on each type of network over the course of the data collection period.
Overall download speed is calculated as a weighted average of the individual 3G and 4G speeds based on the proportion of time typical users spend connected to each. If users cannot connect to an LTE network due to low availability, their overall speeds will also drop as they will spend more time on 3G connections (which can be up to 4 to 5 times slower than an LTE connection).
Latency – Quantifying the lag
Latency refers to the delay users experience as data makes a round trip through the network. If the latency of your network is high, you’ll experience a lot more lag time. We measure our latency metrics in milliseconds for both 3G and 4G connections – the lower the latency value, the more responsive the network.
Both our download and latency speed tests are run on content delivery networks (CDN) that host the most popular internet destinations (not on dedicated testing servers where requests might follow a different route to normal traffic or indeed be prioritised in a way that isn’t representative), helping us to accurately understand typical user experience.
Metrics in the pipeline
We are constantly evaluating new ways to measure users’ true mobile experience and frequently report on our findings as these metrics progress through the development stage. In time, we aim to include these in our reports as standard. Here are a few:
Peak Speed – The fastest experienced speed
Peak speed measures the fastest experienced speeds OpenSignal users experience on real-world networks. For this metric, we disregard technical or congestion limitations, focus on the most optimized connections in our measurements and simply report on what the peak speeds in real-world live networks are for real users. This is different to the best-case speeds measured in idealized conditions that users themselves can never achieve.
To calculate these peak speeds, we extract data on the speed tests our users experience and analyse the fastest ones. For a more practical grasp, read our case studies on peak speeds in the UK and India.
Place Coverage – The new way to look at coverage
While 4G availability takes into account when our users are able to latch onto an LTE signal, place coverage focuses on where. Place coverage measures the percentage of locations where people have LTE connections available to them. Together these two metrics are a powerful way to understand the real coverage that our users experience, uniquely combining time on the network and the places that matter to people – where they live, work and travel.
Rather than using mathematical models and estimating where operators have coverage, OpenSignal’s place coverage metric is grounded in a person-centric approach and uses real-world measurements wherever people go, in urban and rural areas, indoors and outdoors.
Our upload speed metric measures the average upload speeds for each operator on LTE connections. Just as with our download speed metrics, 4G upload speed is measured in Mbps and reflects the average uplink speeds our users experience over the course of the data collection period.
Typically upload speeds are slower than download speeds, as operators concentrate their resources on providing the best possible download speed for users consuming content on their devices. As mobile internet trends move away from downloading content to creating content and real-time communications services, upload has become vital.
This metric measures the quality of experience for video streaming via a mobile network for 720p HD video based on an International Telecommunication Union (ITU)-vetted algorithm that has widespread acceptance throughout the industry. Taking into account different factors including: picture quality, video loading time and stall rate, the video experience metric scores each operator on a scale from 0 to 100. Here’s how to interpret scores within that scale:
- 83.3 – 100 Excellent: Generally video starts with almost no delay, plays continuously without interruption and renders at 720p HD resolution with optimal bitrate and high quality.
- 66.7 – 83.3 Very Good: Video loads with only a small delay and typically plays without interruptions.
- 50 – 66.7 Good: Typically video loads with a noticeable but not too long delay and is occasionally subject to short playback interruptions.
- 33.3 – 50 Poor: Video loads slowly and prolonged playback interruptions are common.
- 0 – 33.3 Bad: Video loads extremely slowly or stalls frequently for long periods of time.
With video being the single largest category of traffic carried on mobile networks and consumption expected to grow to keep pace with consumer demand, this is an extremely relevant metric. It is another key element in OpenSignal’s ongoing mission to measure the actual consumer experience on mobile networks.
We’re constantly working on new, better and more meaningful ways to measure users’ real-world mobile experience. Stay tuned to our blog for the latest updates and or sign up to receive our newsletter.
Have a question or a comment? Leave a comment below.
Feel like checking the numbers for yourself? Download our OpenSignal or Meteor app. Both are, available on Android and iOS!