Four Months of Alternative Controllers

How do you interact with a computer?

Chances are the first thing that comes to your mind is the keyboard and mouse, or a keyboard and laptop trackpad. Those more used to mobile devices might point out the touchscreen. Technically-savvy readers might point out the other sensors and systems that are typically included in today’s computers and mobile devices, like cameras, microphones, or motion detectors. Gamers might hold up the button-filled game gamepad as their interaction method of choice.

But are those the only ways? What other methods of interaction are there? Can we get creative with them?

Those are the kinds of questions I was asked during the Spring 2016 semester, when I signed up for NYU Tandon’s Beyond the Joystick course. Taught by Kaho Abe, it’s a course based around rapid prototyping of alternative computer interfaces, with a focus on interactive creative systems such as games and art exhibitions. It tasked us with creating a new controller, and associated game or application, nearly every single week over the course of the semester.

I’m going to take you on a quick whirlwind tour of a few of the projects I built, going a bit into the motivations behind each, talking a bit about the kind of sensors used, and relating some of the lessons I learned.

Tiny Tennis

This was the first project I made for the course, where our assignment was to create some kind of a game just out of LEDs and buttons. I made this:

To describe what’s going on here: one player “serves” the “ball” by pressing the button, and the two players have to volley the “ball” back and forth. You return the “ball” by pressing on the button while it’s the green areas on your side of the “field”, and critically, the further back the “ball” is in the green area—in other words, the longer you wait before pressing the button—the faster the “volley” becomes. The first player to score three points wins.

The wiring may look complex, but it’s not. All it is is a set of ten LEDs wired up to the output pins of the Arduino, and two buttons wired up to two input pins. It really just was a combination of a lot of simple circuits.

Most of the game’s complexity was in the software, but even that was less than 200 lines of code. The key lesson learned here is that you can create fun experiences without using complex hardware, even with extremely limited input and output systems.

The hardware for the later projects was more interesting, though.

The Magical Daintiness Fan

By the time I got to this project, we had started to use more unconventional input methods. For this one, we had to create an unusual musical instrument using a piezoelectric sensor, a sensor that generates an electric signal when it vibrates.

This kind of sensor is already used in alternative game controllers today. For example, the drum kits in Rock Band use a piezo sensor inside them to tell when you hit them. And you can do fun things with them if you attach them to a surface like a cardboard box.

For our purposes, rather than attaching the sensor to a stationary surface, me and my partner for this project, Allison Cole, decided to attach the sensor to an object that you waved around, and let air resistance do the work of activating the sensor for you.

We were inspired by old Looney Tunes cartoons, where nearly every action the characters do is accompanied by a musical sound effect. What we settled on was the idea of a dainty young beauty with a handheld fan, a flute trill playing while she waves the fan.

That’s what we did: we made a fan. One that, thanks to Allison’s wonderful craftmanship, came out looking like it came directly out of the cartoons we were inspired by.

And as you waved the fan, a flute trill would play from the attached computer, and a heart displayed on the screen would grow bigger and bigger.

The communication between the computer and the Arduino was accomplished using a serial connection via USB, and the graphics were done using Processing, which is wonderful for rapid prototyping of small visual tools like this.

If there’s one thing to take away from this, it’s that theming matters. At its mechanical core, this is just a stick that makes a noise when you wave it. But by clothing it in that specific idea from childhood cartoons, you end up with something that’s plenty of fun to play with.

The Secret Agent is You

By the middle of the term, we’d been working with a wide variety of sensors: bend sensors, light sensors, potentiometers, microphones, and more. We were given a midterm project over spring break that had two major requirements: it had to be done on a minimal budget, and it had to combine two or more of the sensors we’d used so far.

Pairing with Allison again, we again looked back at potential points of inspirations, and this time we settled on the classic bomb defusal scene seen in so many crime thrillers. You know what I’m talking about:

The idea here was that the player, playing the part of the agent in the field, would have to defuse the “bomb”, which was actually a cardboard box covered in metallic-looking wrapping with circuitry on the inside. The player would be getting instructions from Mission Control on the screen in front of them, and would rely not only on those instructions, but on the people around them as well.

Like I said, we were required to use at least two different types of sensors. We met that requirement and more:

• In keeping with the theming we were going for, the primary mode of interaction was to have the player actually cutting real wires in order to advance the “mission”. This added a bit of setup time between players, as we had to replace the wires, but it was too cool an idea not to use given what we were going for.

• We had a piezo sensor on the inside of the box, that would trip if it detected impact or vibration. This would cause the bomb to explode if you treated it too roughly.

• And last, but perhaps most interesting, we had an actual blindfold attached to the system by a wire. This blindfold could detect whether you were wearing it, by virtue of a light sensor that would rest against your head when you put it on. The bomb would explode if you weren’t wearing this “eye protection”. What this meant for the player was that rather than looking at the wires, you’d have to rely on the people around you to guide your hand and tell you which wires to cut, as they’d be the only ones that could see Mission Control’s instructions.

Now, unfortunately, with this project, I have a bit of an issue: somehow, through the course of creating it, we never took any pictures of the “bomb” itself. I also couldn’t find where it ended up after the semester was over, so unfortunately the video of the on-screen interface above is the best I really have to show you.

Regardless of that, one of the major successful points of this project was that it succeeded in hiding the sensors from the player, and as such, the whole thing had a bit of a sense of magic to it. This was one of the things that Kaho emphasized throughout the course: you have to preserve the magic in the experience.

Relax

Back in my Scenes from New York post, I showed a picture of Bobst Library, the main study space at NYU. Here’s that picture again:

I mentioned that the last project was during midterm season, and this definitely influenced my choice of project this time around. Our assignment was to create an exhibit or game that used an infrared distance sensor and was meant for display in a particular space. Knowing that Bobst is a place many students go to study during the stressful exam periods, the talented Michael Kim and I chose to make a project geared towards helping those stressed students relax.

The intention was that it would be displayed off to the side of the front lobby of Bobst near the main entrance, on a television screen mounted to the wall, and that it would use the distance sensor to detect when a curious student comes up to it. When one of them comes by to take a look at it, it would guide the student through a short deep breathing exercise, complete with custom-built relaxing music composed by Michael.

The lesson to take from this project was, of course, to consider the location and context in which your works will be displayed.

But as it turns out, our choice to display the project in Bobst was even more appropriate that I had initially known. You see how, in the picture of Bobst above, the center of the building is a large empty space that cuts across all floors? When presenting this project in class, Kaho told me that they had to add extra fencing on the walkways around it, because there were some incidents in which some students were getting so stressed during exams as to be suicidal, and this was giving them a method to carry it out that was far too close at hand.

I was shocked, since I’d never heard this story before, but I took it as a sign that even these seemingly small problems can become big ones. Maybe small solutions like this could help, even if just a little.

Don’t Wake the Baby

So, let me describe to you a scenario. You’re a new parent with a young baby. You need to step out of the house for an errand, and you’ve spent the last half hour putting the baby to sleep so that your partner could have a bit of rest.

Just when you’re about to head out the door, you realize that you left your keys in the baby’s room. The door of the baby’s room is very squeaky. Your baby is a light sleeper.

You’d better be careful getting those keys.

This project was done with both Allison and Michael, and it was our first exposure to accelerometers. We put one in the door handle, and made the intensity of the door’s squeaks proportional to the magnitude of the acceleration it measured. Jerk the door open too quickly, and the baby wakes up.

This is a pretty blunt way of using an accelerometer, given that it can measure acceleration across three independent axes. There are a lot of people who even try to use accelerometers for positional tracking. But the problem is that accelerometers aren’t very precise: past a certain point of acceleration, they “cap out”, and you end up not being able to tell how far or fast the accelerometer is moving.

Given that this was a one-week project, like most of these other projects listed here, we decided to stick with a simpler design that didn’t require such subtle input or extensive testing. In the end, limiting our scope got us something enjoyable and decently fun to play with.

Not-so-stranger

This was our final project for the course, and our most ambitious.

So far in the course, I’d done projects that made people feel happy, relaxed, and focused. This time around, we made a project that was meant to make you feel anxious.

More specifically, we chose to try to make a project that put people in a nerve-wracking situation, and get them to overcome that anxiety, which would hopefully lead to a pretty empowering experience.

And what kind of anxiety is great to overcome? Social anxiety.

So this was the plan: we’d put Arduinos in a small pouch on the player’s hip. These Arduinos would be wired to a glove equipped with conductive tape on the outside, which would be used as a capacitative sensor—a sensor that detects skin contact. And your mission, as an agent of Not-So-Stranger, would be to go out and collect handshakes from random strangers around NYU Tandon’s quad or cafeteria. Two players would compete to see who can get the most handshakes in the allotted time.

The Arduino would detect when you were shaking hands (or even high-fiving) with someone, and you’d be given a fixed time limit, after which you’d come back, plug both player’s Arduinos into the computer, the results would be loaded into the system, and the winner would be displayed.

The results were… mixed.

These were a couple of lessons I learned from this project and its playtesting sessions.

This was a lot scarier than anticipated

Even among our group members, most people didn’t want to play this game twice. As much as I had hoped this would be an empowering game to play, it didn’t quite get there. In particular, rejection—when people declined to shake your hand—was devastating to most players.

When you look at what they typically do in exposure therapy for anxiety patients—an extreme example to be sure, but a representative one in how the experts get people to face their fears—our mistake becomes clearer. Typically, in exposure therapy, you start out with everything being in a very safe, controlled environment, and gradually increase exposure to whatever it is you’re scared of. Here? We were throwing people in the deep end from the start. There was a lot of to lose (in terms of awkwardness and fear) and comparatively little to gain (you won the game).

Context matters a lot

With that said, it ended up being really dependent on where you did this. We tested this a few different times in different places: outside in the quad, in a playtesting day for the game design students, and in the Tandon cafeteria.

The quad was one of the worst places for this. You ended up having to approach people who had somewhere to be, and people were more suspicious of being approached by random people on the street. There was a lot of rejection there.

The cafeteria was sort of a middle ground. People weren’t in a rush, they were mostly more open-minded young students, and you might even find someone you know.

The playtesting day was also decent. The people there were all people involved in game development in one way or another, people who’ve taken classes on doing things like developing LARPs and board games, and so they were used to a little quirkiness.

I get the feeling that this would work best in some sort of convention setting, where people are looking to meet and interact with new people, and where people have a high tolerance for a little bit of weird. That last bit is important, because…

People were suspicious

Picture this: you’re minding your own business with your friends somewhere in the university when some person wearing fingerless gloves for no apparent reason decides to come up and engage with you in conversation. Maybe they explain that they’re doing some weird handshake challenge, or maybe they just talk to you about something inane. Then, they stretch out their hands to shake yours, and you might even see that they’ve got random wires hooked up to their gloves. What would you expect?

A lot of people actually expected that we were pranking them, or that the gloves would shock them, or something along those lines. It took a lot of effort and bravery to deal with this kind of natural suspicion and it’s one of the things that contributed to how emotionally hard it was to play this game.

Allison suggested that we lean into the awkwardness by making it completely ridiculous, like wearing a big, Mickey-Mouse-style white glove and playing up the silliness of the whole situation. I also think that would have worked better, but it would have involved far more time and budget than we had room for. Ultimately, we worked with the time allotted to us.

But it also worked

It did accomplish one thing though: it succeeded in people out of their comfort zone and challenged them to do something that most people haven’t done before, which is ask favours of random strangers. Ultimately, despite the difficulties, it did end up being an interesting experiment.

The takeaway

Ultimately this was a collection of often silly, usually quirky, mostly fun experiences and games. But I think the variety in the different experiences that the other students and I were able to come up with in a course like this says something about the technological possibilities we have open to us these days.

I think it should surprise no one to hear that I spend a lot of time in front of a computer, given my line of work. Like I said in my last post:

In this time of my life, the computer is my work, my school, and my entertainment. It’s a big part of my social life, since so many of my friends are far from me, […] and it was even my romance, during the time when I was in a long-distance relationship last year.

It’s a bit much, honestly, and it’s a hard trap to avoid.

But that actually takes for granted the benefits that the computer brings. It’s my school because I can access practically all human knowledge on it. It’s my work because it allows me to create wonderful dynamic experiences and useful tools for people. It’s my social life because the communication medium of the internet allows me to stay close to distant friends and loved ones, instead of losing them entirely when I move away.

The underpinning of that complaint, though, is that the computer takes this wide breadth of human experience and compresses it down to whatever you can do by staring at a screen and pushing buttons:

• Learning is distilled down to reading text on a screen and watching videos, rather than conversing and experiencing things.

• Entertainment becomes more about watching, reading, and playing games with rigid systems than about exploring the world around us.

• Friendship and romance becomes more about words and pictures, about text messages and Skype calls, than about actions, gestures, and touch.

• And my work? My work becomes all about facilitating all of that.

But it doesn’t have to be that way. We’re neck-deep in a race towards miniaturization of computers, and we’ve been seeing a Cambrian explosion of sensors that allow us to break out of the simple touch-screen-and/or-keyboard limitations of before. These sensors are surprisingly cheap, and they open up a world of new interactions that are very different from anything we could have done ten or fifteen years ago.

The further away we move from being hunched over our laptops and smartphones, the closer we get to a future where we can harness the power of computers in a far more human way.

When the computer becomes invisible, all that remains will be us.