OpenSignal data is collected from regular consumer smartphones and recorded under conditions of normal usage. As opposed to drive-test data, which simulates the typical user experience by employing a limited set of devices in a small number of locations, we take our measurements from millions of smartphones owned by normal people who have downloaded the OpenSignal app. Those measurements are taken wherever users happen to be, whether indoors or out, in a city or in the countryside, representing performance the way users experience it.
The OpenSignal Android app runs constantly in the background at low power, noting changes in network conditions and performing over one hundred individual tests. This gives us an incredibly rich dataset that allows us to build up a complete picture of network experience.
Instead of measuring experience at a subset of random locations and times, we’re collecting our data at the precise places people spend their time and the specific times they are actually using their devices. We run tests when people are indoors, when they are outdoors and when they are in vehicles. This allows us to measure the full range of consumer behavior in a way that is not possible through any other testing methods.
We also test across the full range of Android and iOS smartphones in the market, rather than limit ourselves to a handful of specific test devices. As new technologies like LTE-Advanced emerge or operators bring new 4G networks online in new spectrum, we see those capabilities emerge as soon as the smartphones that support them make it into the market.
We’ve collected terabytes of signal data from over 200 countries and we utilize the latest big-data technologies to analyze and process it. We have a team of data scientists dedicated to investigating our data as well as a team of analysts generating insights we can share in an easily digestible format.
Before any data is included in an analysis it is subject to a detailed quality assurance procedure. For example, we use a series of tests and filters to determine if a device is operating correctly and reporting accurate values. In addition, we have established thresholds on the amount of data we must collect to ensure our metrics meet conditions of statistical significance. Finally any statistical analysis contains a certain margin of error, which we factor into all of our country level reports. In some cases, that results in statistical draws for categories such as speed or availability and we make this clear in our reports.
Our high standards have led to our data being used by dozens of academics in over 100 published academic papers, including several authored by OpenSignal directly. For example, our data has been used to correlate telecom infrastructure against economic development in Boston neighborhoods, to show the variations in performance for different device models and to explore the links between rain and cellular availability.
OpenSignal testing is based on both user-initiated tests and background automated testing. While both types of tests can be useful, user-initiated measurements can be impacted when users choose to run them whereas background tests can be run at regular intervals throughout the day, allowing the app to capture a much broader range of network measurements throughout any given time period. By combining user-initiated and background testing we are able to accurately reproduce the complete user experience and remove the impact of relying solely on user-initiated speed tests.
We record data on speed, reliability, and our own metric, availability, which looks at the proportion of time users have a network connection. As we measure only where and when the user is using their phone, the availability metric determines whether the network is ‘doing its job’ by providing a connection where it’s needed most. Looking at availability adds a key element of network assessment, time, that is not covered by more traditional metrics such as population and geographic coverage. Also, measuring every aspect of network performance gives us much more insight into consumer experience including, for example, the proportion of time people use Wifi and how that impacts their experience.
With over 20 million downloads and a high app store rating, the OpenSignal app is the world’s most popular network measuring app. We’ve also launched a companion app called Meteor that shows users what their speeds mean in simple terms.
We have achieved this by putting our users first, making our apps fun and useful while taking great care to protect user privacy. As a custodian of data donated by our users we take privacy very seriously, making sure we only record data related to network performance, and not sharing any personally identifying information with 3rd parties. We also invest significant resources to ensure the app minimises any battery and data consumption.