• Gunther Van Geetsom

  • Software Engineer

Personalizing the Smart Mirror

As is tradition at Foreach, every few months we organize Ship It Days. During this event all the developers at Foreach get 48 hours to come up with a small but shippable product, preferably a cool one. Our team took it upon ourselves to make some upgrades to the smart mirror that was made during a previous event. We wanted it to be possible for a user to create their own personal dashboard that would appear whenever they identified themselves to the mirror.

They would be able to customize their dashboard by choosing the components it contained, for example a clock or the weather for a specific location, and how they were organized. This way the information most important to the user could easily be accessed.

Technologies & Frameworks

We wanted to use the opportunity of this Ship It Days project to experiment with a bunch of new technologies like: Polymer, RFID cards and Amazon Lex. Of course, we also used a number of frameworks to make our lives as developers a bit easier. We will briefly discuss which ones we used, what they are, how we used them and what we thought of them.

But before we could get started on something new, we had to solve a bug that had been plaguing the existing mirror. Sometimes the Raspberry PI would would show an error. This was caused by the wireless connection that randomly disconnected. Because of the limited time window and the complexity of the issue, we decided to write a script that checked every minutes if it could connect to the internet. If it couldn’t connect to the internet five consecutive times, the smart mirror automatically restarted.

Polymer

Polymer is a quick and easy way to display a large variety of standard HTML web components, while still providing enough flexibility to allow for custom components. It has a standardised way to develop components, as well as a marketplace that offers several components for anyone to use.

We created the three following components:

Weather component

We made some small modifications to the this weather component.

 the weather component

Notably we changed the position of the week forecast to the right because our blocks are more horizontal oriented.

Slack chat integration

This component shows the last three messages from a special purpose Slack channel called ‘Narcissus’. At the moment this is used for general announcements and greeting people. To get the latest chat data, we could reuse the chat controller created in the previous iteration.

 Slack chat

One improvement for the future is to allow the user to select the channel. Or, even better, to detect who is standing before the mirror and show the messages from their chosen Slack channel automagically!

Finding my way home

One of the many bright ideas we had, was to show a component that displayed how long it would take to get home and what the best route is. For example Antwerp-Brussels on a tuesday morning gives this result:

Maps component

We used a Google Maps component that will display the closest route from the company’s location to the location the user has defined in his dashboard settings using the Google Maps Directions API. This API also provides us with the distance and the travel-time-by-car data we wanted to show.

In the future we could extend the dashboard settings by allowing a user to define more specific parameters of the directions API, such as the transit mode, in order to more accurately customize the component to the user’s wishes.

User identification

We needed a way to identify the person standing in front of the mirror. At first we tried to use Bluetooth, as virtually everybody at Foreach has a smartphone with Bluetooth enabled. But this turned out to be harder than expected. We were able to link and detect the signal strength of Bluetooth devices on our Raspberry PI. This would allow our smart mirror to automatically detect the closest linked Bluetooth device and automatically show a personal dashboard on the mirror. On new Android devices, however, we noticed that a new connection can only be made if the screen is turned on. On iOS devices, we simply couldn’t get a connection, even after hours of trying. So, we sought out two alternative ways to log in to our mirror: RFID and voice commands (Amazon Lex).

Radio-Frequency Identification (RFID)

Since Bluetooth was not working out for us, we went and bought some RFID cards and an RFID scanner, which we promptly connected to the Raspberry PI that drives the smart mirror.

The RFID cards worked very well. We connected our RFID scanner to our Raspberry PI via Serial Peripheral Interface bus (SPI) following this connection schema:
SDA <--> 24

SCK <--> 23

MOSI <--> 19

MISO <--> 21

IRQ <--> UNUSED

GND <--> 6

RST <--> 22

3.3V <--> 1


Then, we activated SPI by issuing the command raspi-configThis opened a menu in the command line where you could configure your Raspberry PI. Selecting the option SPI in the ‘Advanced Options’ menu activated the SPI interface. After activating the SPI interface, we rebooted our Raspberry PI using sudo rebootWe used Python to interact with the SPI interface, so we had to install the python-dev libraries: sudo apt-get install python-devThen, we downloaded a library from GitHub to talk to the SPI interface with git clone https://github.com/mxgxw/MFRC522-python.gitFinally, we then altered the library, so that we could easily read and write a user ID on an RFID card. This allowed every owner of an RFID card to log into the mirror and show his personal dashboard.

Amazon Lex

To interact with the mirror using voice commands, we experimented with Amazon Lex, the Amazon Web Service (AWS) used to build Alexa. We started by creating a new Bot on AWS with the following configuration:

The goals of this bot was simple: retrieve the name of the person standing in front of the mirror. Therefore, we only ask for the name of the person and we configured some sample utterances accordingly. In these sample sentences, you see the name portion highlighted in blue, because we defined it as a custom slot later on.

The name slot uses a custom type, FE_FirstName, which contains a fixed set of the first names of all our employees. By declaring all possible names in advance, Lex could more easily recognize the different names.

Once the bot was tested, built and published, we could start integrating it in our Polymer frontend. Luckily, in the Amazon documentation, there is already an example on how to make the browser capture voice with the MediaDevices API and send it to our bot.  A detailed explanation can be found on the AWS blog, and code on GitHub.

Across (WebCmsModule and UserModule)

We immediately opted to use Across to kick-start our development. Combining WebCmsModule and UserModule allowed us to quickly create both a dashboard for every Foreacher (which is nothing more than a simple page being rendered), and the tools necessary to start customizing those dashboards through the use of components.

As WebCmsModule allows us to define our own components, we could quickly integrate other frameworks such as Polymer, to create a variety of components that would come in handy for the end users.

Results

We're able to identify people using a badge. We use the Across framework to show the dashboard for that user, which contains their chosen configurable Polymer components. Polymer allows us to quickly add some standard components to our dashboard and change existing components to suit our needs.

To create the Slack component, we used the weather component as base and we made small changes until it showed the chat messages. Much of the CSS is similar between these two components. This is also one of the downsides, however, as there is duplicate CSS stored in the different components. While Polymer is a great framework, it takes some time to get into the flow; we mostly learned by trial and error and by adjusting existing components.

Also a mirror

When we showed the results of our efforts to our colleagues, they were excited to see their new mirror. We held an RFID card against the scanner, but the card was not recognized. A second attempt was successful, luckily, and the dashboard of Gunther appeared with all of his personal components. Sander did a demo of the Amazon bot and, when a carefully selected volunteer introduced himself, the system responded with a funny hardcoded text (that’s not fit for publication). Yes, we had time for an easter egg.

Concluding Thoughts

Though our first plan of automatically recognizing a user using Bluetooth didn’t work out, we did have some other ideas, which worked out great: RFID and voice recognition.

Because we chose to use Polymer, we got a bunch of premade components, and we were able to allow everyone to create their own favourite components to show on their dashboard.

There is still one important item left on our todo-list, however: UX. Because we didn’t have enough time left and because we didn’t have a real front-ender in the team, we just kept the default styling of the Polymer components. Turns out this isn’t the best styling for a smart mirror.

In the end we learned a lot and we’re proud of what we’ve achieved in 48 hours. Our predetermined goals were reached: we have a way to create personal dashboards and the user can be recognized using RFID or voice recognition.

Still, we have our work cut out for us for the next version of the smart mirror.


Check out the other Ship It Days projects: WAY person locator and the Awesome Logo of Foreach Animation.

 

Written with a great deal of assistance by Robin Cominotto, Steven Gentens, Stefan Pauwels, Wouter Van Hecke, Sander Van Loock & Jesse Van Rooy.