Blog post coauthored by Carlotta Riesser.

As you may know, Sonar Festival is one of the biggest electronic music festivals in Europe and takes place every June in Barcelona. Alongside the festival, Sonar organizes a technical challenge as part of the Sonar +D. For this event, Sonar has partnered with companies who define new challenges. Several groups get together in order to work on these subjects.

Deezer is once again part of this challenge for the third year running.

Our challenge

This year, we tried to pick an actual product issue encountered at Deezer.

One of Deezer’s core values is catalog diversity. Today, we have over 43 million tracks to offer, and will expand in the near future… This is great, but also represents a big challenge for Deezer.

Curation, personalization and recommendation are key to us in order to define the best possible user experience, which is to offer the perfect tailored content, in a specific context.

Understanding the usage context is essential.

Home context is probably one of the most complex ones, as it is influenced by a lot of different things:

• Activities: cooking, taking care of family, work, working out…
• Moods: lazy, active, sleepy, happy, sad…
• Connected devices and IOT present in homes: TV, speaker, lights, PC, mobile…
• The time of day, weather, season etc.

Although the hack is defined by the team, Deezer introduced the topic to give a direction.

When a user is at home, how can we detect the context in which they are listening to music and recommend music accordingly?

We engaged the challengers on this topic and encouraged them to come up with cool ideas.

The set-up

At first, we anxiously waited for applicants. For a more interesting challenge we made a wishlist with complementary profiles: a UX designer, a mobile developer and a data scientist.

We really wanted to bootstrap the project before the Sonar took place to anticipate specific needs on our side (hardware or software). Our past experiences have proved that starting on site with a little preparation beforehand is far more productive.

Then it happened! Sonar sent us an email with five challengers all with impressive credentials. We talked internally about asking for more applicants but quickly realized that this was enough. We got ready for our first video call.

Unfortunately it didn’t go smoothly because of technical issues: we desperately tried to find the right tool (Hangout, Slack, Skype?) to reduce lags and sound issues like sudden echos, sound drops, or background noises. It took us half of the first meeting to efficiently talk to each other. When we finally managed to communicate, we were pleased to discover that each of the challengers was already bringing ideas to the table. From this call onwards we narrowed down the subject, and took action on our side about who to include and what equipment to use.

The project

The project’s goal is to capture the listener’s environment thanks to various sensors, to identify the activity they are currently engaging in, and to recommend the adequate music. The retained activities were: relaxing, working out, cooking, sex and study.

The sensors captured temperature, humidity (from a Raspberry Pi + sensors), accelerometer, proximity, light intensity (from a smartphone), heart rate, steps, activity (from a Fitbit provided by Deezer), weather and time (from external APIs).

Deezer, for the first time, opened a recommendation API to the challengers, to help them leverage the power of our algorithms, and to be able to improve them thanks to fresh new data. This data helped the challengers to train models, for instance.

On site, the atmosphere was studious. When we joined our team — and met them for the first time — they were already working alongside the other teams. The impressive congress hall was tall wide and pretty hot in Barcelona especially during the heatwave. We all prepared to spend a few sweaty days at Sonar! We opened our laptops and joined the hack.

During these three days, a model was trained to classify audio tracks into the five chosen activities. Another was trained to detect activity from the sensor’s data (well… simulated to be more accurate!). All of the pieces were put together around an Electron’s app. The whole concept, baptised Yin, describes this enhanced listening experience.

[UPDATE]: The video of the presentation is available here:

Next steps

The Sonar +D challenge was again a great experience.

We achieved what we wanted:

• A project which answers to a very concrete Deezer product issue.
• A great collaboration with great profiles.
• A proof of concept which inspires Deezer’s work and strategy.

With this proof of concept the first step is to create in-house awareness on context aware recommendation systems.

Secondly, we hope that this challenge will be the starting point of an internal product initiative and that we will begin to focus on contextualized recommendations.