The use of Reinforcement Learning to Personalize User Experience at Zynga
By Mehdi Ben Ayed and Patrick Halina from Zynga’s ML Engineering Team
Introduction to Series
Designing user experiences is a difficult art. Compared to other applications, video games provide designers a huge canvas to work with. At Zynga, we’re constantly thinking of innovative ways to maximize our user’s experience while playing our games.
In this multi-part series, Zynga’s ML Engineering Team will discuss how deep reinforcement learning (RL) is used in production to personalize many aspects of Zynga’s games. The use of deep reinforcement learning has proven to be successful at increasing key metrics such as user retention and user engagement. These articles are an extension of our presentations at the Toronto Machine Learning Summit 2019 and Spark Summit 2020.
In this first article, we will:
- Give an introduction to deep reinforcement learning
- Explain why it is a good candidate for powering personalization systems
- Present a high-level overview of its use at Zynga
The rest of the series will cover the following:
- Part 2: Deep Dive into an RL Application at Zynga
- Part 3: RL Technology and Infrastructure
- Part 4: Training Challenges with RL (Part 4)
- Part 5: Tips on Ramping Up for and RL Application
About Zynga
Zynga is one of the leading developers of mobile games in the world. Its portfolio has some of the world’s most popular mobile games, like Words with Friends, Empires and Puzzles, Toon Blast, CSR2, Poker, and many more. Zynga’s mission is to connect people from everywhere in the world through games.
The Need to Personalize User Experiences
Building successful applications for a wide range of users is challenging. How difficult should the daily challenge be? What game mode should we recommend? Each user is different, which makes it difficult to build an application that fits everyone’s needs. The difference can be due to the player’s demographics, location, or even their style of play. As a result, building personalized user experiences has the potential to increase user engagement and retention.
While there are many knobs and levers to control during a user’s session, personalization can be boiled down to a common pattern: given a user’s state, select an action in order to maximize a long term reward.
These are examples of typical state, actions, and rewards in our personalization problems at Zynga:
- State: The user’s context, composed of a collection of features, e.g. how long they’ve played the game for, what level they’re on, performance history in the last few levels, etc.
- Action: The aspect of the application being controlled. It could be the difficulty of the next level, the time to send a message, game modes to recommend, etc.
- Reward: This is the key performance indicator (KPI) we want the personalization to improve. For Zynga, some of those metrics are engagement metrics such as the number of levels played by the player and session length.
#deep-learning #recommendations #reinforcement-learning #machine-learning #personalization #deep learning