Facebook's Mark Zuckerberg shows off his self-coded smart home AI Jarvis

Jarvis, the Man Friday

Zuckerberg built, or rather coded Jarvis using Python, PHP and Objective C. Along with this, he incorporated artificial intelligent techniques such as speech recognition, face recognition, natural language processing and reinforcement learning.

Along the way, Zuckerberg encountered a couple of issues. Firstly, he had to get his array of different devices - Crestron smart system, thermostat and doors, a Sonos system with Spotify, a Samsung TV, a Nest cam, and Facebook's systems - to talk to each other in a simple language. Then he had to make sure that the same command from different people in the house - Priscilla (his wife), Max (his son) and himself - brought about the results.

Zuckerberg notes how he spent just 100 hours on this project in 2016 and how he ended up "with a pretty capable system". The Facebook CEO may release it to the public if he can find more functionality that isn't tied specifically to his own setup.

Furthermore, this might even be the basis for a future product from Facebook. A lot of what Jarvis can do isn't new. Amazon's Echo and Google's Home systems can do a lot of features that Jarvis has incorporated. Facebook already has two companies dealing with AI.

Facebook AI Research (FAIR) that deals with the academic community and Applied Machine Learning that ties AI into Facebook's products. For what we know, Facebook's engineers are exploring some kind of AI assistant product. An Alexa (Amazon) competitor.

Messenger bot and iPhone app

Where the company might differ from others is in these open-ended requests. As Zuckerberg says, "At this point, I mostly just ask Jarvis to "play me some music" and by looking at my past listening patterns, it mostly nails something I'd want to hear. If it gets the mood wrong, I can just tell it, for example, "that's not light, play something light", and it can both learn the classification for that song and adjust immediately.

"It also knows whether I'm talking to it or Priscilla is, so it can make recommendations based on what we each listen to. In general, I've found we use these more open-ended requests more frequently than more specific asks. No commercial products I know of do this today, and this seems like a big opportunity."

Zuckerberg demoed the Messenger bot he built as a front end for the system to various people in the past week. He could use just his iPhone, type in a few commands, and get the lights turned on and off. All without having to get up from wherever he is seated.

He's also built a system to respond to multiple voice commands and this is done via a custom iOS app that he made. In the initial stages, there will be a lot more hiccups than should be there but eventually the app will get stronger and instantly recognise what is being said.

For now, Zuckerberg prefers text commands over voice commands. This is purely because he doesn't want to disturb anyone else in the house.

Music is where it's at

The most interesting aspect of the AI system is the different musical tastes that everyone has.

Getting an AI system to learn that would be revolutionary. Just take this passage from his Facebook post to understand: "Music is a more interesting and complex domain for natural language because there are too many artists, songs and albums for a keyword system to handle. The range of things you can ask it is also much greater.

"Lights can only be turned up or down, but when you say "play X", even subtle variations can mean many different things. Consider these requests related to Adele: "play someone like you", "play someone like Adele", and "play some Adele". Those sound similar, but each is a completely different category of request.

"The first plays a specific song, the second recommends an artist, and the third creates a playlist of Adele's best songs. Through a system of positive and negative feedback, an AI can learn these differences".

Morgan Freeman over Jarvis

With the system being inspired by the Iron Man movies, the obvious choice for a voice would have been Robert Downey, Jr. (aka Tony Stark) or even the man behind Jarvis, Paul Bettany.

That isn't the case though. With Zuckerberg posing the question as to who should be the voice behind the AI system to the public in October, the outright winner wasn't actually a surprise.

Through that post which garnered over 50,000 comments, Morgan Freeman, the voice of Lucius Fox from the Batman franchise beat out the likes of Downey Jr, Arnold Schwarzenegger, Benedict Cumberbatch, Neil deGrasse Tyson and others. Freeman will be replacing the standard female robotic voice.

Zuckerberg had met Freeman at the Breakthrough Awards and according to Fast Company, he called him up afterward. Facebook hasn't confirmed whether any remunerations went to Freeman who will be recording a limited set of commands.

Zuckerberg is planning to publish a summary of what he is working on and has worked on. This reflects Facebook's philosophy about keeping all the work done open-source, especially that in the field of AI. Zuckerberg will keep tinkering with the system and hopefully one day we'll see the light of it in a commercial product. That will only spur other companies to improve their systems.

In the end, it's a win-win for consumers. Artificial intelligent systems definitely are the future and the the potential is limitless.