At OpenSignal we love sensors. We’ve written several blogposts on the different types to be found on Android and iPhone devices and have recently published a comprehensive report on the topic.
But when thinking about sensors, one of the most basic and widespread is often overlooked: the microphone, present in every single cell phone – be it a super slick smartphone or the Samsung X840 I had for many years.
Even my old Samsung X480 had a couple of sensors…
Some well known applications take advantage of this ubiquitous sensing device. On chat and social apps such as WhatsApp, WeChat or iMessage, the mic is put to work to record and share voice messages. The microphone is also used by Apple’s Siri and Google voice search – my husband “OK Google”s his phone at least a couple of times a day, often with questions (“How high is Cusco?”), sometimes with requests (“Show me a picture of a polar bear”). It is likewise thanks to the microphone that Shazam, SoundHound or TrackID™ can take a digital fingerprint of a song you’re listening to, which is then matched against large databases of tracks to give you the title of your tune of choice.
The potential uses of the microphone as a repurposed sensor are in fact multiple, diverse and very promising, as I discovered diving into a bunch of extremely interesting articles and papers. Here are some of the projects, apps and proof of concept software that reinvent the functions of the mobile’s mic.
Mapping noise pollution
I recently came across a wonderful website that any French-speaker map lover should check out: http://veillecarto2-0.fr. Among their many cool blogposts features an entry on crowdsourced maps to measure noise pollution. Some of these are generated through Android and iPhone apps that employ the cell phone’s microphone to take readings of urban ambient noise. Examples include NoiseWatch, Noisemap, NoiseTube and WideNoise. The literature on the subject is quite extensive as these applications are mostly the work of academics and environmental agencies – have a look at the table at the end if you want to read further!
Into the wild
But it’s not all about urban noise: cell phones also lend an ear to nature’s voice. And if you happen to be planning a visit to the New Forest National Park, you might consider downloading beforehand the Cicada Hunt app, an original piece of software that uses your phone’s mic to capture the sounds of the surrounding environment and let’s you know if there’s a cicada anywhere near you. More info about it can be found in one of our previous blogposts. Another wonderful project that plunges into the woods is Rainforest Connection. Their aim is to fight illegal logging by camouflaging solar-powered cell phones in the tree canopy; when the microphones in these devices catch the sound of chainsaw, the responsible authorities are instantly alerted.
Phone’s mics can help save trees
Health comes first
Smartphones’ microphones could open up an array of possibilities in the field of mHealth. They have been tested as a mean to monitor sleep patterns by listening to your movements – HappyWakeUp does exactly that to rouse you from your dreams at the most convenient stage of the sleep cycle. The microphone can be equally employed to measure social isolation based on the duration of ambient conversations: this is one of the functions of BeWell, an app that keeps track of your general wellbeing focusing on your sleep habits, physical activities and social interactions. The developers of BeWell have also been working on StressSense, a software that uses the microphone to identify stress in the user’s voice. As a matter of fact, cell phones’ mics can listen up not only for stress signs but also for pathological lung sounds, as shown by the creators of SpiroSmart, a smartphone-based spirometer. Last but not least, apps such as BioAid or soundAMP Lite can transform your phone into a hearing aid by amplifying the audio feed from the microphone and playing it back through your headphones.
What was that noise?
Smartphones have the potential to become extremely precise sound sensing devices, as proved by several apps and projects. In 2009, a research team from Dartmouth College published a paper describing SoundSense, an iPhone application designed to recognise and classify sound events detected by the cell phones’s microphone. In a similar fashion, the Batphone app works as a novel method of indoor localisation by recording room ambiance – the little snippets of noise are used to identify rooms previously tagged by the user. What’s more, a study conducted by researchers at Queen Mary University of London has shown that acoustic scene classification algorithms, exploited by applications of this kind, can achieve a mean accuracy matching the median performance of humans. That’s right: phones can be as good as humans at understanding where they are based on noises heard.
Cell phones’ mics will not only process and analyse different kinds of sound: they can also be programmed to produce particular noises for very specific purposes. Sonar, for example, works exactly as its name suggests – emitting sound to calculate the distance of objects from the echo. Another innovative creation is Chirp: this application produces a short bird-like song that triggers a download when “heard” by another phone running the app.
Chirp away your data!
The starting point for new developments
Several of the apps mentioned here are the work of academic research, and as such, not always available on Google Play or Apple Store; but the prospective uses of the microphone as a repurposed sensor are no less exhilarating for this. Even more is to be expected if new hardware and software are combined. We have smartwatches in mind, but also projects such as iBats – an app that uses an ultrasonic detector plugged into the phone to collect bat sounds and map their distribution. Similarly, a team at Cornell University are working on BodyBeat, a sensing system composed of a custom-built microphone and an Android app that studies non-speech body sounds – those of food intake, laugh, breathing, cough. We will certainly keep an ear out for promising developments such as these ones.
|"BeWell: a smartphone application to monitor, model and promote wellbeing"||Nicholas D. Lane, Mashfiqui Mohammod, Mu Lin, Xiaochao Yang, Hong Lu, Shahid Ali, Afsaneh Doryab, Ethan Berke, Tanzeem Choudhury, Andrew T. Campbell||5th international ICST conference on pervasive computing technologies for healthcare||2011~05||http://www.cs.dartmouth.edu/~campbell/papers/bewell_pervhealth.pdf|
|"BeWell: Sensing sleep, physical activities and social interactions to promote wellbeing"||Nicholas D Lane, Mu Lin, Mashfiqui Mohammod, Xiaochao Yang, Hong Lu, Giuseppe Cardone, Shahid Ali, Afsaneh Doryab, Ethan Berke, Andrew T Campbell, Tanzeem Choudhury||Mobile Networks and Applications, volume 19, issue 3||2014~06||http://niclane.org/pubs/bewell_monet.pdf|
|"BodyBeat: A Mobile System for Sensing Non-Speech Body Sounds"||Tauhidur Rahman, Alexander Adams, Erin Carroll, Bobby Zhou, Huaishu Peng, Mi Zhang, Tanzeem Choudhury||International Conference on Mobile Systems, Applications and Services (MobiSys), New Hampshire, USA||2014~06||http://pac.cs.cornell.edu/pubs/body-beat-mobisys-2014.pdf|
|"Ear-Phone: A Context-Aware Noise Mapping using Smart Phones"||Rajib Rana, Chun Tung Chou, Nirupama Bulusu, Salil Kanhere, Wen Hu||CoRR, abs/1310.4270||2013||http://arxiv.org/pdf/1310.4270v1.pdf|
|"A continental-scale tool for acoustic identification of European bats"||Walters, Charlotte L.; Freeman, Robin; Collen, Alanna; Dietz, Christian; Fenton, M. Brock; Jones, Gareth; Obrist, Martin K.; Puechmaille, Sebastien J.; Sattler, Thomas; Siemers, Bjoern M.; Parsons, Stuart; Jones, Kate E.||Journal of Applied Ecology||2012||http://onlinelibrary.wiley.com/doi/10.1111/j.1365-2664.2012.02182.x/full|
|"Microphones as Sensors: Teaching old Microphones New Tricks"||Not specified||The Economist||2013~06||http://www.economist.com/news/technology-quarterly/21578518-sensor-technology-microphones-are-designed-capture-sound-they-turn-out|
|"MusicalHeart: A Hearty Way of Listening to Music"||Shahriar Nirjon, Robert F. Dickerson, Qiang Li, Philip Asare, and John A. Stankovic, Dezhi Hong, Ben Zhang, Xiaofan Jiang, Guobin Shen, and Feng Zhao||SenSys’12||2012~11||https://www.cs.virginia.edu/people/faculty/pdfs/MusicalHeart.pdf|
|"Crowdsourcing of Pollution Data using Smartphones"||Matthias Stevens & Ellie D’Hondt||UbiComp ’10||2010~09||http://soft.vub.ac.be/Publications/2010/vub-tr-soft-10-15.pdf|
|"NoiseSPY: A Real-Time Mobile Phone Platform for Urban Noise Monitoring and Mapping"||Eiman Kanjo||Mobile Networks and Applications, volume 15||2009~09||http://link.springer.com/article/10.1007%2Fs11036-009-0217-y#page-1|
|"Towards Personal Stress Informatics: Comparing Minimally Invasive Techniques for Measuring Daily Stress in the Wild"||Phil Adams, Mashfiqui Rabbi, Tauhidur Rahman, Mark Matthews, Amy Voida, Geri Gay, Tanzeem Choudhury, Stephen Voida||8th International Conference on Pervasive Computing Technologies for Healthcare||2014||http://pac.cs.cornell.edu/pubs/stress-pervasive-health-2014.pdf|
|"Use of Mobile Phones as Intelligent Sensors for Sound Input Analysis and Sleep State Detection"||Krejcar, Ondrej; Jirka, Jakub; Janckulik, Dalibor||Sensors 11(6)||2011||http://www.mdpi.com/1424-8220/11/6/6037/htm|
|"SoundSense: Scalable Sound Sensing for People-Centric Applications on Mobile Phones"|
Hong Lu, Wei Pan, Nicholas D. Lane, Tanzeem Choudhury and Andrew T. Campbell
|"SpiroSmart: Using a Microphone to Measure Lung Function on a Mobile Phone"||Eric C. Larson, Mayank Goel, Gaetano Boriello, Sonya Heltshe, Margaret Rosenfeld, Shwetak N. Patel||UbiComp’ 12||2012~09||http://abstract.cs.washington.edu/~shwetak/papers/SpiroSmart.CR.Final.pdf|
|"StressSense: Detecting Stress in Unconstrained Acoustic Environments using Smartphones"||Hong Lu, Mashfiqui Rabbi, Gokul Chittaranjan, Denise Frauendorfer, Marianne Schmidt, Andrew Campbell, Daneil Gatica-Perez, and Tanzeem Choudhury||Proceedings of Ubicomp 2012 ||2012~09||http://www.cs.dartmouth.edu/~campbell/ubicomp-2012.pdf|
|"A Survey of Mobile Phone Sensing"||Nicholas D. Lane, Emiliano Miluzzo, Hong Lu, Daniel Peebles, Tanzeem Choudhury, and Andrew T. Campbell||IEEE Communications Magazine||2010~09||http://www.cs.dartmouth.edu/~campbell/papers/survey.pdf|
|"Acoustic Scene Classification"||D. Barchiesi, D. Giannoulis, D. Stowell and M. D. Plumbley.||Accepted by IEEE Signal Processing Magazine||to be published||https://danielebarchiesi.files.wordpress.com/2014/09/asc.pdf|
|"Participatory noise mapping works! An evaluation of participatory sensing as an alternative to standard techniques for environmental monitoring"||Ellie D’Hondt, Matthias Stevens, An Jacobs||Pervasive and Mobile Computing 9(5)||2013~10||http://www.ademloos.be/sites/default/files/partnoisemaps.pdf|
|"Participatory noise pollution monitoring using mobile phones"||Nicolas Maisonneuve, Matthias Stevens and Bartek Ochab||Information Polity 15||2010||ftp://progftp.vub.ac.be/tech_report/2010/vub-tr-soft-10-14.pdf|
|"The era of ubiquitous listening dawns"||David Talbot||MIT Technology Review||2013~08||http://www.technologyreview.com/news/517801/the-era-of-ubiquitous-listening-dawns/|