By Geir Turtum and Torgeir Lien
Idea
We want to make an alternative interface for controlling smartphones. It's annoying having to take the phone out of the pocket for every little thing. We want to make it possible to perform simple predetermined actions on the phone by simply knocking on the phone while it's in our pocket.
Use-cases where it's nice to not have to take out the phone:
- While in meeting: silencing a ringing phone or sending a predetermined message
- Start recording sound
- Control music apps
- Control a stop watch
- For any use when it's inappropriate to be using the phone
Goal
At the end of this project we want an interface which is reliable enough for everyday use and we will have two example applications using this interface. The project must be done and ready for presentation by March 16. when at least one example app will be presented to our class.
The first app will be a distance measuring app for golf, two knocks for marking the start GPS location for where the shot was taken from, and then two knocks for marking where the ball landed. The app will then read back to the user the distance the ball has traveled using text to speach.
The second app will be an app for controlling media-playback in smartphones. It will be possible to stop, pause or go to next song by knocking on the phone. Should ideally work for most music playing applications like the built in player, Spotify, Pandora etc.
Main technical challenges
- Properly detecting when the user is knocking on the phone
- Giving feedback to the user
- Making this happen while the phone is in a locked state
- Creating an interface which is reliable enough, no false detections!
Solutions to some of the technical challenges
- Fusing accelerometer and sound recording to detect the signature of a knock to the side of the phone (maybe also gyroscope)
- Launching the app as a service to make it run in background and during lock state
- Feedback via text-to-speech, vibration or simple sounds
Progress
February 7 - The project has been presented to the class and our professor has accepted it.
February 8 - We have determined some of the most important challenges and modules in our project and divided them between us.
February 9 - Programmed several simple apps for testing and getting used to: accelerometer, microphone, vibration motor, text-to-speech, controlling media apps etc. Started looking at what kind of data we can get from the accelerometer and microphone and ways to analyze these.
February 10 - We have now implemented two prototype apps for the alternative interfaces. One the measures the sound amplitude to detect knocks on the phone, and one which detects spikes in accelerometer data due to movement in the z-axis of the phone.
Detecting knock-spikes is easy enough, the difficult part is removing all the noise so we don't get false positives. It seems very difficult to detect actual -single- knocks vs a wide range of other noise. However, we found that when we look for a pattern like knock-wait-knock, both prototypes are actually already pretty robust. Both apps are now separately able to reliable detect knocks without triggering from other things while the phone is in the pocket etc. The accelerometer app is now simply using a absolute high-pass filter with a simple event-timer to detect successive knocks. The sound measure app returns the highest measured noise in some interval (a knock produces a saturation) and then waits some time for another knock. These app have also been used to simply vibrate on event, as well as pause and resume multimedia-apps on our phones.
We are now considering adding a state machine for detecting of knock-patterns, so that we can easily add more complex patterns later. We also want to optimize each of the apps and make them even more robust, or finding better ways to analyze data before we merge them together. When we are happy with the interface itself we will move over to making example apps and maybe even putting them on Google Play if they are any good!
February 14 - With both the accelerometer and volume "knock" sensing done we have now combined the two sensors to make the app more robust. Now each of the sensors has to agree that a knock has happened before a knock event is triggered. See below for the state machine that combines the two sensors to detect a knock.
February 18 - Created the state machine that is responsible for detecting knocking patterns. Now we are able to start and stop the built in music player by knocking twice on our pocket. Next step is to make the knocking interface more robust and start working on the distance measuring app.
February 28 - Distance measuring using GPS is up and running. Next task is to work on design and adding the KnockKnock interface to the app.
March 3 - The music controlling app is now able to pause/play and switch to next song seamlessly for some mainstream media apps. Tested and working for Android Media Player, Spotify and Pandora. NOT working for: Last.fm, but then again not even headset buttons work for that app.
March 8 - Extended the knock detector to detect both double and tripple knocks. It works pretty well right now and is easily extensible to the other applications through an abstract function giving the number of taps. The sound control app uses this functionality.
Interface flowchart
March 10 - The knock detector has been optimized to make it easier to use. Also the GPS application is pretty much done.
March 13 - Further optimized parameters and altered some things in the state machine to make the interface much more reliable. Also created final presentation slides.
March 14 - Held final presentation. See link below for slides. Project is complete and we are happy with the results. The interface is working as we intended. Things to improve:
- Make it even more reliable.
- Add calibration routine
- Make it easier to add interface to applications.
Contact us: Torgeir Lien (samma@github / sloooth@gmail.com) and Geir Turtum (geirivtu@github)
Link to project proposal presentation slides:
https://docs.google.com/presentation/d/1M8Bzie2UMnzLqXpyUimqu77IS7XygZtEv5jetCrQHeQ/edit?usp=sharing
Link to final project presentation
https://docs.google.com/presentation/d/1r8U2fuztg3u0K_vKCm5WPM-e785fHtYuIEjjNv0Iopc/edit?usp=sharing