Home, smart home

By Taylor Alexander, Co-Founder of Flutter Wireless

As founder of Flutter Wireless, a company that is building new hardware for the internet of things and connected devices movement, I spend a lot of time thinking about how this new technology will affect our lives. Right now computers are all relatively separate workstations, with tasks isolated to one individual machine. We may check email on our phone and on our desktop, but only recently have companies begun making it fluid to switch between the two. As our software advances and connectivity becomes more widespread and robust, we will begin to see programs that run across multiple machines simultaneously. I’d love to open an app on my phone and stream music to every device with speakers in my house, for example, rather than needing to buy a “home speaker system”. Ultimately, I see our home networks evolving into a single computing entity with many access points. A common home or cloud access point could provide services across multiple devices simultaneously. I could send one stream simultaneously to my living room TV and my kitchen tablet, for example, so I can catch up on a TV show while preparing dinner. As our homes become more connected, we will have increasing freedom with how we use computing to improve our lives, and entirely new possibilities will come out of these new use cases. Below is a story I wrote imagining a time maybe a decade from now, when the connected home is perhaps as commonplace as self-driving cars.

I hope you enjoy it, and that it prompts you to dream of what else a connected home can do for you.

I live in a connected home. Every electronic thing in my house is controlled by the home system. Not toasters or blenders or the fridge — not things that only sensibly need physical access. Those things have their own local user interface, though some may report back to the home. The microwave, for example, communicates photos of food to the server for analysis, but you can only turn it on from its front panel. The interface panel is just a touch-oled with images for its interface controlled by the home. In default mode it just has 3 buttons, and they change based on what I put in. Put in my favorite mug with a clear liquid and you just get a big “hot water” button. The house interface on my phone shows graphs that prove that the cook time it chose is optimal based on my use of this cup in this microwave every morning since I started my new job, but honestly… I never look at it, since it never fails. Usually when I’m using the house interface on my phone, it’s to control the music or change the channel.

I took a YouTube class in the living room last month, and the inductive charging in my new coffee table means that I could leave the interface open for the whole hour of class without draining its battery. The home has a local content stream it can serve to any audio or video device with a speaker box or cheap HDMI streamer. The audio channel let’s me do things like play music, talk with my friends, or control the lights and temperature.

I also have interface pads in the rooms. Interface pads are like the interface on the microwave – they have a touch-oled and an audio system for voice interaction. Four microphones mean it can pick up quiet conversation even with the fan on, and it blocks out other sounds like the TV like they aren’t there. This makes it feel like the system is in my head. I’ll mutter to myself “I wonder if I turned off the coffee pot”, and the system sometimes butts in and tells me. Usually I have to address the house to get it to listen, but I’m running some software that let’s me play back my ramblings when I am deep in thought, so right now it’s live all the time. This lets it answer questions without having to repeat myself. If I think out loud, sometimes the house is a pretty good assistant.

We call ours Hiro, and while he can’t tell me everything without a manual query at a terminal, he’s pretty good at answering basic questions about the world. Anything with a clear answer like… how deep is the English channel, how much money did I spend last month, or who won the gaming competition last week… those questions Hiro answers well. Of course he’s also great for taking notes for me and reading them back so I can edit them. He’ll read anything I want. He’s been reading me Steinbeck and Plato lately, and in the mornings I’ll usually have him read the news. Last night I streamed live ocean sounds from a beach in Madagascar as I slept.

In the mornings I read my emails on the terminal in the kitchen while I stir my coffee. I keep work emails out of the morning routine, but read what my mom is up to over a bagel and eggs. I fill my foodbox once a week and it serves up a hot bagel and fresh eggs every morning. It only fits a few types of meals but it’s enough for all my breakfast and lunch for a week, and using it beats rummaging through cold storage for all the pieces. It will slice bagels and fruit, even core an apple, and it has refrigerated dispensers for eggs, cream cheese, peanut butter and jelly, even mustard and mayonnaise. It has a small compartment for fresh meat and cheese, so I make sandwiches for lunch. The machine prepares the bread and washes itself, just like it does with my morning bagel, egg, and yogurt. It tracks the age of each perishable, and the deliveryman brings by fresh food weekly for things like meat and eggs. It even breaks the eggs and cooks them, and stores the shells in an oxygen free environment with the apple cores, until I empty the canister.

I charge a tablet on the kitchen table, and use it to watch last night’s news footage. I use the house interface app, which shows me stuff I probably want to watch. Anything I don’t want to watch on the tablet screen I can throw to any TV too. I am studying be a paramedic, so I’ll usually stream class to both displays at night when I’m cooking and cleaning. There are so many times where I need my hands for one thing but can use my mind and voice for another. The tablet was pretty good for that before, but with Hiro I don’t need to bring anything with me. I can wander to the other room mid voice chat without ever losing my train of thought. When I talk to friends, its like they’re in the same room and follow me around. With Hiro’s chat interface I can log into voice chat rooms with friends. Its like we’re sitting in a room together, either quietly working, having a meeting, or just watching the news together. I feel like I always have my friends with me.

A computer block and a storage block that I keep in the office control the whole system. All my home computers store data on the storage block, and the computer block runs Hiro’s software. We have phone and tablet apps along with interface panels, and cheap HDMI dongles on the TV. Voice is usually handled by the interface panel most rooms have. But there is a voice-only interface panel that is the cheapest. It skips the touch display on the large interface for a four-button fob and voice control. You can plug headphones and speakers into that one for a custom speaker setup, but by default the internal speaker is pretty good. It still has four microphones so we usually don’t use an external for that, just output.

It cost about a two grand for the whole system, but that’s the lights, computer, audio tactile pucks and 4tb storage brick. I saved up for one summer when I was in college and got this system. Its been around for a few years so the CPU takes longer to recognize my food scans from the microwave than the new models, but its a few milliseconds difference – 250 maybe – I don’t worry about stuff like that.

All in all, my connected home system was the best purchase I made since switching to a self-driving car.

 

Simplifying smart home programming

A team of computer science researchers from Brown and Carnegie Mellon universities has adapted a method of programming known as “trigger-action” to more effectively communicate with IoT smart home devices.

According to Brown University’s Kevin Stacey, the trigger-action paradigm is already gaining steam across the Internet, particularly on the website IFTTT.com (If This, Then That), which helps users automate tasks across various Internet services.

 More specifically, users create “recipes” using simple if-then statements — for example: “If somebody tags me in a Facebook photo, then upload it to Instagram.”

The website interfaces with both service providers, with the action occurring automatically each time it’s triggered.

Although IFTTT.com started out as a tool to link websites, it recently added the capability to command a number of Internet-connected devices, such as Belkin’s WeMo power outlet and the Philips Hue lightbulb. Ultimately, the website prompted Brown’s Professor Michael Littman to consider adapting the trigger-action model for home automation.

“As a programming model, it’s simple and there are real people using it to control their devices,” Littman explained. “But the question we asked [was]: Does it work for the [home automation] tasks people want to do, or is it perhaps too simple?”

To find out, the researchers asked workers on Mechanical Turk, Amazon’s crowdsourcing marketplace, what they might want a hypothetical smart home to do. The team subsequently evaluated answers from 318 respondents to see if those activities would require some kind of programming, and, If so, whether the program could be expressed as triggers and actions.

The next step? Determining how well users could actually design “recipes” to accomplish tasks. To do so, the team employed two interfaces designed by McManus, one of the undergraduate researchers, enlisting Mechanical Turkers to create recipes with the interfaces.

“We based both [interfaces] on ‘If This Then That,’” McManus said. “But then we made one of them slightly more complex, so you could add multiple triggers and multiple actions.”

The study showed that participants were able to use both interfaces — the simpler one and the one with multiple triggers and actions — fairly well. Participants who didn’t have any programming experience performed just as well on the tasks as those who did. Simply put, the results suggest that trigger-action programming is flexible enough to do what people want a smart home to do – and simple enough that non-programmers can use it.

Melwyn Pak, one of the Brown undergraduates on the project, finds that encouraging.

“People are more than ready to have some form of finer control of their devices,” he said. “You just need to give them a tool that allows them to operate those devices in an intuitive way.”

Littman, who has been studying end-user programming of electronic devices for several years, concurs.

“We live in a world now that’s populated by machines that are supposed to make our lives easier, but we can’t talk to them,” he concluded. “Everybody out there should be able to tell their machines what to do. This is our attempt to start thinking about how to bridge that gap.”