Tom Hume

If you happen to be in Worthing on February 28th, I'll be reprising my Making Sense of Sensors talk from Future of Mobile 2011; possibly with some updates courtesy of the Master's and some reading I've been doing around the topic since. More details here.

One of the courses I'm really enjoying right now is Pervasive Computing. It involves playing with hardware (something I've never done to any real degree), ties into the trend of miniaturising or mobilising computing, and humours an interest I developed last year about the potential for mass use of sensors, and spoke about at Future of Mobile.

Dan Chalmers, who runs the Pervasive Computing course, has us playing with Phidgets in lab sessions, and very kindly lets us borrow some kit to play with at home, so I've had a little pile of devices wired up to my laptop for the last few days. The lab sessions are getting us used to some of the realities of working with sensors in the real world: notionally-identical sensors can behave differently, there are timing issues when dealing with them, and background noise is ever-present. At the same time we're also doing a lot of background reading, starting with the classic Mark Weiser paper from 1991 (which I'm now ashamed I hadn't already read), and moving through to a few discussing the role sensor networks can play in determining context (a topic I coincidentally wrote a hypothetical Google project proposal for as part of Business & Project Management, last term).

I've been doing a bit of extra homework, working on an exercise to implement morse code transmission across an LED and a light sensor: stick text in at one end, it's encoded into ITU Morse, flashed out by the LED, picked up by the light sensor, and readings translated back first into dots and dashes, then text. It's a nice playground for looking at some of those issues of noise and sensor variation, and neatly constrained: I can set up simple tests, have them fired from my laptop, and record and analyse the results quite simply.

Here's what the set-up looks like:

Note that the LED and light sensor are jammed as close together as I could get them (to try and minimise noise and maximise receipt of the signal). When I'm running the tests, I cover the whole thing to keep it dark. I have run some tests in the light too, but the lights in my home office aren't strong enough to provide a consistent light level, and I didn't want to be worrying about whether changes in observed behaviour were down to time of day or my code.

First thing to note is the behaviour of that LED when it's read by a light sensor. Here's a little plot of observed light level when you turn it on, run it for a second, then turn it off. I made this by kicking the light sensor off, having it record any changes in its readings, then turning the LED on, waiting a second, and turning it off. Repeat 200 times, as the sensor tends to only pick up a few changes in any given second. Sensor reading is on the Y axis, time on the X:

A few observations:

This is all very low-level stuff, and I'm enjoying learning about this side of sensors - but most of the work for this project has been implementing the software. I started out with a dummy transport which simulated the hardware in ideal circumstances: i.e. stick a dot or dash onto a queue and it comes off just fine. That gave me a good substrate on which to implement and test my Morse coding and decoding, and let me unit test the thing in ideal conditions before worrying about hardware.

The Phidgets API is really simple and straightforward: no problems there at all.

Once I got into the business of plugging in hardware, I had to write two classes which deal with real-world messiness of deciding if a given signal level means the bulb is lit or not. I used a dead simple approach for this: is it nearer the top or bottom of its range, and has it changed state recently? The other issue is timing: Morse relies on fixed timing widths of 1 dot between morse symbols, 3 between morse characters and 7 between words… but when it takes time to light and unlight a bulb, you can't rely on these timings. They're different enough that I could be slightly fuzzy ("a gap of 4 or fewer dots is an inter-character gap", etc.) and get decent results. There should be no possibility of these gaps being too short - but plenty of opportunity (thanks to delays in lighting, or signals travelling from my code to the bulb) for them to be a little slow.

I didn't implement any error checking or protocol to negotiate transmission speed or retransmits; this would be the next step, I think. I did implement some calibration, where the LED is lit and a sensor reading taken (repeat a few times to get an average for the "fully lit" reading).

I ran lots of tests at various speeds (measured in words per minute, used to calculate the length in milliseconds of a dot), sending a sequence of pangrams out (to ensure I'm delivering a good alphabetic range) and measuring the accuracy of the received message by calculating its Levenshtein distance from the original text. Here's the results, with accuracy on the Y axis (lower is fewer errors and thus better) and WPM on the X:

You can see two sets of results here. The blue dots are with the sensor and LED touching; the green ones are with sensor and LED 1cm apart. You can see that even this small distance decreases accuracy, even with the calibration step between each test.

Strange how reliability is broadly the same until 50WPM (touching sensors) or 35WPM (1cm apart), then slowly (and linearly) gets worse, isn't it? Perhaps a property of those speed-ups/slow-downs for the bulb.

So, things I've learned:

Things I still don't get:

And in future, I'd like to try somehow taking into account the shape of that lighting/dimming curve for the bulb - it feels like I ought to factor that into the algorithm for recognising state changes in the bulb. Also, some error correction or a retransmit protocol would increase accuracy significantly, or let me run faster and recover from occasional issues, giving greater throughput overall.

Update: I've stuck the source for all this on Github, here.

Something is becoming clear as I do more work on my alarm clock design project (which had its first user test today): prototyping touch UIs that rely on more expressive motions than single touches is tricky. In particular I'm working with direct manipulation of a clock, and want to test single-finger dragging gestures around a dial, and multi-touch gestures to expand or contract the selection of a time period represented on that dial.

It's hard to do this sort of thing on paper: if a gesture like a finger-drag results in a visible change of state (e.g. "click the centre of the wheel and it turns green") then you're shuffling stationary around, which is disruptive if you're trying to see how the interface feels, or testing it with someone else.

It's also hard to do digitally. Looking around, I see people who've had success using Keynote for prototyping on tablets - it seems to offer good fidelity and be fast to work with, but lacks the level of detail I'm wanting to play with. I want to be able to fiddle with the sizes of touch areas, ratios of on-screen motion to finger movement, etc., either to see if it feels good or tweak it until it is.

I've used the iMockups product to play with putting screens together on an iPad, and quite liked it - but again, whilst it's great for chaining screens together it lacks fine-grained control. Perhaps my expectations of what flexibility and control a prototyping tool should offer are out-of-line, and I'm mixing up my fidelities - trying to hi-fi stuff with lo-fi tools.

Nicholas Zambetti's LiveView product looks brilliant for taking digital stuff from a desktop and presenting it on a mobile screen; but I'd still need something at the desktop end which eases the process of putting together touch-responsive interfaces.

Given that I'm at a bit of a dead end, I'm going to try taking a very small part of the interface, mocking it up with actual code (so it works on a real device), and seeing if I can test that. My reasoning being that if I can take the most complicated bits, prototype and test them in isolation, I can produce components which individually perform well, test the transitions in and out of them on paper, and validate the UI that way. At the back of my mind is an idea around recording input from a real user and replaying it against differently sized interaction areas, to find ideal sizes, but that'll be later I think.

Let's see…