2016 was a big year for Smart Homes. A good number of existing products like Amazon’s Echo and Philip’s Hue range gained traction while a host of new products entered the market, most noticeable Google Home. Last year in Fall, I had written a paper outline the challenges a contemporary smart home faced and this is the part which goes in detail about the three big ones: Chaos, Control and Context.* For the full paper go here.

*This is presented as is from the paper. For newer updates, thoughts and discussion about Smart homes go to my medium blog.

The first 4 Sections of the paper were

  1. Introduction
  2. Visions of Smart Home
  3. Smart Home Infrastructure
  4. Legacy Infrastructures
    These were followed by the key section of the challenges of smart home and which was wrapped with a conclusion after.
  5. Challenges of the smart home

Smart homes promise to augment our living conditions with added functionality, better communications and enhanced awareness of our environments and the actors in it. Like all the technologies preceding it, increasing adoption will be disruptive and take unexpected paths as time progresses and technologies evolve. They will be appropriated and adapted in unforeseen ways. Edwards and Grinter27 give the example of landline and wireless phones to showcase this point and how relationships between technologies and people evolve in spaces, private and shared, through routine. They caution designers to pay heed to the subtle, complex and unarticulated or ill-articulated routines of domestic spaces. This is one of seven challenges they list concerning evolution of smart homes within technical, social and pragmatic domains. Keeping these challenges in mind, I examined the current state of smart homes to find a unique intersection of complications worth classifying from the perspective of a smart home occupant.

5.1 Chaos

The sheer number of electrical and electronic devices at home these days can be staggering. From kitchen appliances to gadgets around the living room, the total number of devices at home including smaller ones like light bulbs and chargers could be well over a hundred. If one includes more passive objects that don’t need power supply the numbers can run into several hundreds. Possessing so many things can be chaotic and things that are small can often get lost. Smart homes imagine a future where not only the connected devices can be remotely managed they also allow objects deemed important can be instantaneously located and saving the more disorganized of us a lot of time. Those spare bulbs can be found in an instant, saving us the pain of searching the whole basement trying to remember where it was placed. While they help us manage the chaos of the many things around us, there are elements of home turning smart itself to be chaotic.

5.1.1 Customer, choice and setup

Smart homes, now and in the near future, are an expensive choice. Those who own these products are individuals who would be seen as early adopters and they often are from high income households. As with all technology products, they are willing to experiment and find the right fit to get their homes just right. However, an increasing number of available products along with growing awareness of smart homes and a strong marketing push are helping expand the consumer base. It is transitioning from a hobbyist market to a lifestyle segment and slated to grow in sales rapidly1.

Not surprisingly, some of the big smart bulb manufacturers are those companies which have historically had considerable expertise with home lighting like Philips, GE, Cree, OSRAM, Lutron and TCP. These traditional lighting equipment manufacturers are also facing competition from startups like LIFX, Illumi, Misfit, Elgato etc, which are startups focusing only on either light bulbs or a range of computing technologies including wearables and other IoT products.

The resourcefulness of the big, established players and ingenuity of smaller startups are driving forces. The competition is great for the industry focusing on the consumer means attention is paid to innovation, quality, and lower costs. These bulbs vary in cost from $15 a piece for a fixed brightness soft white bulb to $99 for a variable color and brightness portable light. Besides price, the customers need to choose which bulbs they would like to buy based on what smartphones they use (based on app support) and current (or future) purchases of smart home products. This should, technically, make the user move towards brands which are supported by a greater number of platforms and platforms with a wider brand support. However, users often choose the brand they trust the most or the one with better advertising.

After making the complex decision  of which bulbs and how many, the customer, who is now poorer by a few hundred dollars, would believe setting them would be a simple job and they would work like magic in a few minutes. That’s the impression the movies and ads project. In reality, it is a long and often frustrating process setting them up, followed by the harder task of thinking and setting up routines and instruction sets. I shall address those factors further on. Visual representation of smart homes in popular culture and product advertisements never showcase the work involved in setting them up or during moments of breakdown.

While bulbs light up as soon as they are screwed in and switched on, tapping into their real functionality of remote, timed and conditional-based access needs thought, time and effort. The setup process is often neither obvious nor uniform across products. Most of them require you to download the brand app, setup an account and use the smartphone to either create a connection between the router and the bulb (Like LIFX) or setup the brand hub, link it to the router and then use it to detect bulbs (like Philips and Belkin). Setting up platform hub to discover and control the hub and bulbs needs another set of instructions which again vary widely based on platforms and brand.

There is usually a limitation on how many lights can be supported by the hub (It is 50 for the Philips Hue) and where it is located. For a particularly large home, one hub might not be enough. Some other smart bulbs like the LIFX don’t need a hub and connect to the network through the Wi-Fi. This comes with its own problem of taking up an individual IP address on the home network. While this might not be an issue to begin with (the number of devices which can sit on a single local network created by Wi-Fi router is limited to 256) it can quickly add up as more and more devices add up. In dense urban areas, there is already Wi-Fi connectivity issue with the 2.4Ghz spectrum being heavily loaded. This can lead to potential delay and failures in signal communication. Another deterrent of setting up smart home devices is keying in Wi-Fi passwords each time a new system/device is connected. This becomes exceptionally tough with secure but complex passwords which need to be changed from time to time.

Designers have to consider that these technologies are overwhelmingly complex and understood by few, even among “early adopters”28. They have to design systems which are easy to setup, use and maintain in an environment where no specialist is available at home to solve these problems. Edwards and Grinter27 suggested the appliance and utility models for designers where smart home devices can be designed in such a way that they are either work with utter simplicity or come with solutions for remote diagnosis, administration and upgrades. The Google Onhub router and Amazon’s Mayday service are two examples to follow for these models.

5.1.2 Standards, platforms and consortiums

Like any other infrastructure, the smart home devices use existing standards to comply with the installed base of home and software infrastructures while bringing their own new set of standards for their functioning. The brands usually choose a set of standards which will have lower production costs, they have access or licenses to and which might suit their target customers best. The brands are under no obligation to integrate their products with other services, to follow any specific standards or have open, accessible APIs. This makes it possible for them to lock down the consumer to their own products or those which belong to their preferred partners or platforms.

As the IoT market is growing, several brands are coming together to form consortiums31 which enable them to create and work on uniform standards. These standards can in turn be used to make their products compatible with each other and achieve the plug and play conditions technology industry idealizes but is an expectation in most infrastructural setups. As consortiums are still competing and evolving with companies free to changing their affiliations, these consortiums are like lines in the sand, constantly shifting till a clear victor emerges. This is a constant characteristic of emerging technologies.

Philips has a range of smart bulbs under the brand name ‘Hue’. They sell a wide variety of bulbs that come in different shapes, sizes and varying features. All of these need a hub to connect to the internet and other platforms like Echo or Smartthings. In the third quarter of 2015 when Apple launched Homekit, Philips released a new hub with Homekit compatibility and recommended iOS users among their customers to upgrade to this new piece of hardware (which cost 40$ for existing and 60$ for new customers) for greater functionality. This is unfair for the users.

There is a need for impromptu interoperability27 for smart home and IoT devices. This means not only the devices should be able to interconnect but do so with no prior planning. The Internet has had large scale success because it has been largely built on open source technologies and open standards. By similarly embracing such open technologies, IoT can grow leaps and bounds by allowing both corporations and Makers to operate and innovate simultaneously.

5.1.3 Software, development and lifecycles

Devices at home are traditionally upgraded only after a few years or when they breakdown. The short upgrade cycle with incremental benefits is a sort of planned obsolescence technology industry is used to from laptops to smartphones but totally unheard of in domestic environments. Even if the energy savings, the luxury and convenience provided by the smart home devices do add up to cost it has to be seen if, in the current climate, it is really worth the effort in constant upgrades, unease of owning redundant systems after paying top dollar and the invisible but obvious impact on environment. Smart home companies need to take into consideration of making devices which can be modular and upgraded with newer components. The tech industry’s practice of constant product upgrades might need rethinking32. On the flip side, the perk of being partly physical and partly virtual in construction is the scope for software updates and introduction of better and more feature filled interfaces without always needing a hardware update.

Development and support become new variables in smart homes. Wink was one of the first major platform hubs and was very similar to Smartthings in how it operated. It is owned by Quirky Inc which collaborated with established players like GE to put out smart home products and also helped foster innovation by creating community powered product invention. While Smartthings got acquired by Samsung in August 201535, Quirky filed for Chapter 11 bankruptcy and is planning to sell the Wink platform34. It is undetermined how long the support for Wink apps (and ecosystem) will continue and if any development will occur. Once a dominant player, it is now trying hard to survive and the customers are left in a lurch. This is not uncommon in the technology industry where fortunes turn around very quickly. So if the wink platform is shut down, all the GE Link bulbs (which only work with a wink hub) might need to be replaced for a smoother, safer workflow. While it could be easy to continue using a standalone, standards based plug and play like a toaster, dysfunctional things in the context of a (smart) home where devices and routines are interlinked can upset and put off even an enthusiastic customer. This is especially problematic since the sphere of operation is in a shared space, where the young and the old, technically literate and illiterate function on a daily basis. One cannot even be sure how long the big companies support software and product releases. While hardware comes in with warranties and support, no such promises exist for software, even the ones essential for the hardware, especially when not paid for. This gives rise to plethora of possible bugs, hacking, and security issues as the platforms these apps and hubs themselves ride on are constantly upgraded themselves. Reliability for both hardware and software aspects of smart home technologies becomes a critical concern27.

The aforementioned impromptu interoperability is applicable in this scenario but in the form of data. Clearly defined ownership of user’s own data and open data flows between brands through APIs and other means can allow users to move from one ecosystem to another without losing on the ‘labor’ hitherto invested.

5.2 Control

5.2.1 How to control?

In the modern smart home, smart phone becomes the epicenter for user control. In the daily workflow there is an overwhelming importance for the smartphone to the point that it becomes a critical point of failure. While it is fairly safe to imagine that every smart home customer is a savvy smart phone user, it overlooks the several complications that this concept of design smart home brands have.

After plugging in the smart bulb (or device), the instructions are to download the app on the smartphone for setup. The app is also the primary way to turn on/off, adjust settings and create instruction sets for the devices. As the advertisements of these products x demonstrate this can be easy and swift way to control lights in the house. While it does help when one is not at home and wants to check up on it or when someone is otherwise occupied and unwilling to move, it is also true that pulling out the smartphone from one’s’ pants/bag, finding that specific app to switch the lights on or off each time isn’t always time efficient. Often, it can be slower than walking over to the switch and flipping it. There are also scenarios where loss or even temporary failure to find the smartphone can upset the daily routines. Forgetting the phone in the bedroom and walking into the kitchen downstairs to get some water isn’t a good idea anymore. The user would have no way to turn the lights on. The separation anxiety from owning an already overbearing, seemingly omnipotent, all important smartphone is going to be greater than ever. Those who don’t like the idea of smartphones or are incapable to own one for various reasons are automatically deprived of access to the smartness of the home.

Different apps are required to control different devices and platforms, but app switching is a terrible pain. This is where platform hubs like Smartthings, Homekit and Echo come into play. Platform apps can make managing of the smart home easier by centralizing the controls of all the (supported) devices. Platforms can even reduce the dependence of the smart home with their ability to respond to voice commands. If one owns a mobile device by Apple Siri, the iOS virtual assistant will take commands and control the light bulbs and other devices setup in Apple’s HomeKit platform. Amazon’s Echo37 is a very capable multidirectional mic cum Bluetooth speaker sitting in the house which acts similarly through its virtual assistant, Alexa. By adding “skills”36 to the Alexa interface, voice commands can be used to change the status of the devices connected to the platform. While the smartphone is not completely dispensable, voice interactions make giving instructions and getting information much easier and brings us closer to the fantastic vision of a smart home.

While voice interactions are much easier than navigating to specific screens within apps on a smartphone, this is still tied down to having a smart mobile device or a Bluetooth speaker placed at certain parts of the home. Strangely the functionality doesn’t extend to all the mics available around the home. Apple Homekit doesn’t work with Apple TV, which has a remote and takes voice instructions too, but only for the television. Alexa doesn’t work via the smartphone app when one needs to interact with it when far away from the Echo. But it does work with the Fire TV, which is Amazon’s Smart TV companion and they also have a voice remote to be purchased separately. While the latter may be yet another device to carry around the home it can be useful tool for guests, kids and others who have temporary or minimal access to the home or Echo. Some brands of light bulbs have speakers, cameras and Wi-Fi repeaters in them33. By also adding a mic to the mix and platform connectivity a key problem can be solved. The smart bulb is on the path to become smarter with newer developments. It seems only logical that in a paradigm of ubiquitous computing all smart devices which are already connected to a network should also become conduits for control. While voice and speech recognition has made progress by leaps and bounds over the last few years, it has to still iron out the chinks to include people with varying accents and speech patterns.

By creating various geographic scenarios like homes, rooms, zones and custom settings like routines, alerts, scenes platform hubs can effectively manage dozens of devices at home. You can call out to Alexa or Siri and switch of all the lights in the basement, switch on the dryer and ask it turn the coffee machine on at 6 am. Google with its Now platform is not far at all from creating this functionality too, as a lot of smart home devices also “Work with Nest”. One can reasonably expect this to be rolled out in the near future. If and when it launches, all android devices, which form more than 80% of the smartphone market right now instantly become controllers38. Coupled with apps which can be designed to sit on wearable devices like Apple Watch and Android Wear, controlling the smart home becomes much more manageable. Some smart home brands already are developing for these platforms. Wearables39 in the form of activity tracker and smart watches which perform dual roles of monitoring the body and acting as secondary displays for smartphones can now become instruments of control for the smart home too. Adding voice command based support can further improve their usefulness and reduce the use of a smartphone. With an embedded mic and the virtue of always being on the person they can be very effective agents of smart home control.

Gestures are also an effortless and silent way to control the smart home. Kinect Sensor, Leap motion controller40 and SingleCue41 are devices which use gesture for very different uses. While they are in the new to the market and there hasn’t been large uptake of these technologies yet, integration with other services and devices could become a distinct possibility. There are a large number of prototypes for gestural control using wearables as well. It is inevitable for them to be error prone and systems have to be aware of this tendency.42 With reduction of errors in ubiquitous computing systems involving touch, speech and gestural interfaces make smart home much more natural.

One of the charms of having smart bulbs in your house is the notion that you don’t have to be physically present to turn them on or turn them off. One can check in if any lights were left switched on the way to the airport43. As more lights become smart and respond to touchscreen presses flipping of a switch becomes less desirable. Those bulbs which are not connected seem like a burden each time they are used creating a monopoly of smartphone control. This can be problematic since there are chances of the house staying dark if the router or hub or other network infrastructure broke down. Smart devices need to be equipped with failsafe protocols to operate in a manner that while the network is down, they still operate in at least a semi or non-smart fashion. By having some instruction sets stored natively on a device and on a hub (platform and brand) some basic functionality can be preserved in spite of a severed connection to the cloud.

5.2.2 Who has the controls?

Once the smart home is set up and all the lights and appliances are working seamlessly with the apps and hub, traditional light switches do not perform any meaningful function in the daily workflow. For tactile operations one will need connected switches or control panels. This might seem redundant but useful for those without access to the controls of the lights, like young kids or guests or anyone who refuses to use a smartphone. They can’t turn them on without the access and if they do use the normal switch to turn the ones already off, the bulbs are off the network and unavailable for anyone. This is a critical issue of control in the smart home where boundaries are created between those who have permissions and those who don’t. These permissions physically limited before based on ad hoc location and intent. Now they are assigned. Permissions are granted to those access to app (which can be installed on a supported smartphone) and account login details. Guests and temporary residents are left with no agency. They could install (yet another) app on their phone and be given access to the home network. This not only has a high cost of effort, it also brings up the questions of security. Giving someone who is not living on the premises permanent access to light bulbs in your home is not a good idea. Temporary (time and location based) access is a solution but no apps really support it. Most apps don’t even support multiple accounts either, if you need to have distinct access control for your living and working space, it becomes very difficult. Also, no app has provisions for custom roles. The app remains blind to the power structure, locational agency and domestic hierarchy. The kids have the same access as the parents. The enthusiastic husband who replaces the light bulbs with smart ones, might not understand the logics of organization of his wife’s system. The teen might prank his younger siblings by fiddling with the virtual switches. Several scenarios good and bad shall emerge as the smart homes evolved. App developers need to take into consideration the social roles of their technology and its interactions. Platforms which work via voice could support instructions of users but there is no authentication factor involved there since the devices don’t discern based on voice but follow instructions blindly. With Siri, while the authentication can come from the unlocking of the smartphone, Alexa doesn’t have any such authentication. But the presence of mics as discussed in 5.2.1 can allow kids, elders, guests and anyone else without an access to smart home controls via smartphone.

5.2.3 What to control?

Sal in Weiser’s example had an impression of what is going on around her through ambient lighting and peripheral display of information from window panes to computer screens. This kind of peripheral information in the world of computing is still limited to laptops and mobile devices through various notifications and alerts which are audio visual and haptic. Speakers and displays which are tied to traditional media consumption sources are not harnessed to the new smart devices at home. The television doesn’t have a pop up on the screen to let the user know that the cake is baked. The kitchen speaker doesn’t ring and forward the call when the phone is left behind in the bedroom upstairs.

People already use physical objects to help track (post its, notes) and experience (printing, modeling) virtual objects and work. IOT enables a virtual way of mapping physical world. Flic and Bttn are Wi-Fi connected buttons45 which can execute various virtual actions based on how they are pressed. They are not tethered to traditional computing devices like laptops, tablets or smartphones. On the other side, presence tags44 like Tile and Trackr keep a virtual track of less often used yet valuable objects like passports, or even retrieve objects prone to be lost or stolen like bikes and house keys. Web services like IFTTT and Zapier can not only bridge devices on various ecosystems but also utilized the APIs of virtual services to connect the physical world to the virtual one. An update of the critical document your team is working on could change the light color to red. Your lunch meeting can send you a reminder via an announcement by the virtual assistant like Siri/Alexa a few minutes in advance so you can prepare to leave and reach on time.

That said, there is the fear of bringing work more seamlessly into the home46 and partaking in conspicuous consumption/production in the late modern age. Work is already being expected to be done all day and these technologies can exacerbate the lack of work/life balance. Social protocols need to be developed to make it acceptable to not work after hours.

Wearables are an interesting center of this amalgam where the physical world is constantly mapped with a range of sensors and the virtual world is physically represented through haptic and audio visual cues. The site of this amalgam is on the user and it codifies his or her world while giving him control of it at the same time. With few quick taps of the smartwatch or using the mic can let him control the home.

5.3 Context

But why control the smart home at all? Shouldn’t the light bulbs know when to go on and switch off? With motion and presence sensors light can turn themselves on and off when people enter and leave the room. Having routines and presets which automatically switch on the lights in the evening and switch them off morning. Having them a certain way (scenes) for dinner and in a different way for a party can also be something that can be preprogrammed. But routines change, preferences alter and situations may vary causing the same presets to be undesirable. Having greater context would help the smart home be more aware of its users and their states.

5.3.1 What is context?

While automatic sensors, voice control, and pre-programmed instructions make smart homes a dream out of science fiction, a real smart home can do better. It doesn’t need explicit instructions each time. It can learn behaviors and integrate with other services which are not necessarily in the home. It uses various data from the sensors around the house and information from the objects owned and used by the dwellers. The identity of individuals interacting with the home, their preferences and permissions are given importance. By monitoring routines and habits around the house, common instructions given and their changing natures, and using collective data smart homes can create contextual changes with a scope for permission.

Dey et al. define context as “any information that can be used to characterize the situation of entities” and elaborate it as “typically the location, identity, and state of people, groups, and computational and physical objects”47. Context makes actions and interactions meaningful. General markers of context within ubiquitous computing are location, identity, and time. Knowing the use cases of the device, nearby devices and user preferences can drastically improve the contexts with which smart devices can operate. A smart light bulb in the living room can start flashing in red when the timer you have set on Amazon’s Echo for baking a cake in the oven goes off. Knowing the users’ location and the accessible devices has helped create a noticeable alert. Right now using a combination of some custom software and tags, something like that is possible but the companies haven’t built that into the programming.

5.3.2 Dynamic and configurable context

Traditional notion of context in computing is a technical one. Dourish categorizes this technical notion of context as positivist and representational where it is treated as information which can be encoded and represented in software systems48. In this worldview where context is objective and reduced to broad statistical trends, it is found to be fixed, stabled and separable from the activity which it describes or related to. By taking it so, context is able to be encoded and remains static. However Dourish wants context to be seen as a dynamic property which is a relation between objects and/or activities. He sees it as something that arise from the activity itself than being predefined. By employing phenomenological methodologies, Dourish, Abowd and Mynatt42, Tolmie et al49 have found that context is to be found in the activities of everyday life and is grounded in the rules, expectations and norms of particular activities and habits. Context becomes meaningful or relevant through its relationship to practices that are engaged by the people which use it which are not necessarily the one conceived by the designer.

Situating it in a contemporary technology where most of context exists as preprogrammed information and technologies are not completely interactive with each other, a great way to adopt Dourish’s interactive model of context48 is by providing open APIs and granular controls of features. This can be an information overload for the average user. This is where to begin with preprogrammed scenarios which can be customizable will help hand hold the user to great detail. Smartphones and Facebook are good examples to drive home this point. They have evolved from having limited and basic customizability and privacy set to having intricate settings with variable permissions for every application and its corresponding actions.  As long as designers and manufacturers plan hardware with long-term development in mind, software upgrades can increase the granularity of users’ inputs without creating factors of obsolescence.

Context is fluid and will vary from every interaction to interaction. By building feedback mechanisms into the user experience, small nudges and pushes from the device and gestures and commands from the user, devices can have a rich and varied set of information with a better estimate of context every time. Context will have richer data set by understanding not just fixed time stamps, duration and schedule but also sequence, rhythm, and synchronization.

Building on the example from before, the lamp on the work desk can blink when an important document has been uploaded by a colleague and the room speaker can announce when it is time to leave home so as to not be late to the movie with your friend. This can be made possible by a platform hub like Google Now which can read your email can talk to your home devices. However, the user has to decide which document and which sender is important and which devices should alert him for each of the above examples individually. Behavioral analysis based on usage data has already helped learning algorithms determine which information would useful for the user at what time. This could be a starting point for the user to be given a model to work with which he can tweak and regulate anytime. Some like Nest combine user behavior and tap on the larger data set available and adjust to temperature outside. If not every home, every street could have its own weather station and hence forecast too with consolidated readings and weather data. And this can in turn help the Nest perform better.

Context-related information like most algorithms used to make decisions on behalf of the people are currently black boxed. People often do not know why exactly certain suggestions were made and how they could tweak them. Google Now and Amazon display cards and products respectively indicating which previous actions are showcasing these new information.  By allowing systems to display their evolving definition of context for a certain scenario and allowing people to modify them in a noninvasive way through highly functioning user interfaces can help improve suggestions and results.