Central Chat Service for Mobile Games

Last 3 quarters, my small and wonderful team and I spent most of our time designing and building several services that many different mobile games could use, instead of building their own. Out of the many services we built, my favorite was the Central Chat Service. It was a fun project, I got to do a lot of architecture and design work, and I learned a lot about building scalable systems and AWS. In this post, I will share some high level details about the chat service and its architecture.

We already had a chat service. It was running on AWS, but was originally deployed on servers running on Rackspace. It started with the right motivations but had become very complex and a pain to manage: several pagerduty alerts a week, poor performance, lost messages to name a few. The biggest issues in my opinion: it used many systems like ActiveMQ, MongoDB, Memcached, ZooKeeper, etc. that required constant maintenance, and it had become complex because many custom features were added to support games that had long been shut down. Scaling it up and down was a manual process. Since we were on AWS, we wanted to use managed services as much as possible. There were some disagreements about the use of managed services with cost as the main concern. While it’s true that managed services cost more, the saving outweigh the time DevOps and engineering teams spend to keep these up and running and for regular maintenance.

Here are the goals for the chat service:

Support different types of chat: direct messages and chatrooms (guild, event and global)
Make the system extensible for custom features or metadata e.g. stickers. We said no to custom game-specific features and gave them configuration that they could extend.
Moderation (configurable, off by default)
Chat history (optional, retention period defined per game)
Support up to 1,000,000 concurrent users
Low latency message delivery all over the world
Efficient use of mobile battery and data
Abuse control, rate limits and protection against attacks
Keep it simple and painless to manage and operate with a small team.

Architecture at a glance

[Image removed]

Here are some key decisions that we made.

Custom binary protocol for client-server communication

For client<->server communication, we decided to use a custom binary protocol (TCP, TLS). It was developed in-house by the SDK engineering team and was already integrated into the central SDK that majority of our games used. The alternatives were to use WebSockets or HTTPS with JSON or XML. We decided to go with our protocol because it was available and it way more efficient than alternatives for our use case without the overhead.

Servers close to players

The previous chat service was deployed in the us-east-1 AWS region (Northern Virginia). This was fine for most users in the US, but every message from players across the continent and oceans had to travel to Virginia and back. We noticed latencies 300-600ms higher than in the US. This is a lot, and we wanted to reduce it as much as possible.

So we decided to place connection servers, called ‘Edge Servers’, all over the world (US, Europe, Asia Pacific and South America; AWS regions). Edge servers hold persistent TCP connections with players and forward messages to the chat service or from chat service to players. In other words, they hold players’ connections and act as the proxy. Based on our load testing, we chose compute optimized instances each getting its own Elastic IP. (We also decided to use multiple US regions, so west coast players connect to servers in Oregon instead of a furtheraway server on the east coast. Also, each region can have more than one edge server depending on the load).

Placing edge servers near players was an improvement but it wasn’t enough if the chat service was in US region. Imagine 2 players in Singapore chatting with each other. Their messages would go to the nearest edge server in Asia which is just a few kilometers away, but each message would have to travel over 15,000 kilometers to Virginia and then back (30,000 kms for a single message). With edge servers, this was an improvement in terms of speed because instead of traveling 30,000 kms over public internet, messages travelled over very fast AWS network (AWS backbone) but each message still had to cross 30,000 kms and speed of light limits how fast it could go. So after running some performance tests, we decided to deploy chat service is every region that had edge servers. Now, for our 2 hypothetical players in Singapore, each message travelled less than 100 kms instead of 30,000. (100 kms is my rough guess from Google but it’s in the ballpark.) When we co-located edge and chat servers, we saw on average 150-250ms latency improvements when players were in the same region.

One database region

We decided to keep the database in a single region (us-east-1) to avoid complexity of maintaining multiple databases and synchronizing them. This was a tradeoff which meant that database writes would cross oceans and continents for players outside of the US. However, we accepted this tradeoff because:

Some games didn’t need chat history and those that did were willing to tolerate slightly higher latency (seconds) for writes
Chat service didn’t wait for the database write before sending message to players (database writes were asynchronous and didn’t block message delivery).

Routing table in Redis

For the chat service to route messages to the right recipients (chat rooms), it needed a way to know which edge servers had player connected to particular chat rooms. We could have used a database for this, but we wanted the lookups to be fast (~1ms) and avoid the overhead of database writes (cost), and prevent consistency issues (DynamoDB has eventual consistency by default). We decided to use Redis to hold the routing table. The Redis cluster (shared with other services) was in us-east-1. Chat services would query Redis directly to find out which edge servers have players connected to a particular chat room. Chat services also update the routing table in Redis when players leave or join a chat room (All reads and writes were single key operations).

Every message needs the routing table, but we didn’t want every message to cross the ocean for a lookup. So chat services in all regions pull the routing table, cache it locally and keep checking it periodically every few seconds. The cached copy can be a little out of date but we accepted this to favor simplicity.

SQS and DynamoDB

We decided to get rid of ActiveMQ and use SQS instead. The choice was easy: SQS is managed by Amazon so there’s nothing for us to manage.

For the database to store chat messages (clients could request message history when they first joined a chat room), we decided to go with DynamoDB. DynamoDB allowed us to keep the system extensible and support game-specific metadata more efficiently, plus AWS had given us early access to the TTL feature, which meant we could remove messages automatically. My primary concerns were spikes, hot partitions (think global or small number of very popular chat rooms with lots of players), and unexpected cost, but I’ll cover how we addressed these in a separate post.

A thin client

We decided to do very little work on the mobile client. The client was only responsible for de-duplicating and ordering messages using the message ID. For message IDs, we chose Twitter ID instead of UUID because we wanted the ID to be time-sortable.

Java & Spring Boot

We decided to use Java and Spring Boot because that was the prevalent language and framework at the company. An engineer pushed hard to use Rust instead of Java, and while I agreed with him that Rust would give much better performance, no one on the team except him knew Rust, so it was a no-go. (I was tempted because load tests showed how much better it would be but at the same time, I wanted anyone on the team to be able to debug issues at 1am when the pager goes off). We could have also used Node.js which I was in favor of for building Edge Server because it would have handled network I/O more efficiently (non-blocking by default) compared to Java but team didn’t have Node experience and the engineer who coded in Rust on weekends refused to even look at JavaScript much less touch it! So Java it was.

Load balancers

The chat service were behind a AWS ELB, load balancer (the load balancer was internal to us and wasn’t reachable from outside of AWS). Because Chat Services were stateless, simple round-robbin traffic distribution worked and we would add or remove instances (scale up or down) very easily with auto-scaling groups. Edge servers in a particular region only needed to know how to reach the load balancer.

Edge servers were different: they didn’t have a load balancer in front of them because SDKs connected directly and connections were long lived. This meant that we had to build our own safeguards against common TCP attacks, which is a whole topic for a different post (and a humbling experience for me).

The high-level flow

SDK (client) connection flow

The SDK embedded within games sends a request to the Geo Service (separate service) to get addresses for the nearest ‘Edge’ servers that it could connect to.
The Geo Service uses MaxMind to resolve the IP to a geo location. It then returns a list of the nearest Edge servers (Elastic IPs) to the client. There’s some business logic involved here: the nearest edge server might be overloaded so it may return the second best.
The initial response is in priority order: the client should attempt to connect to the first suggested server, then the 2nd, and so on. Clients were allowed to cache the response. For the next attempts, the clients chose a random server from the first 5 options, because the list could no longer be considered prioritized.
The Geo Service was located in the us-east-1 region and had its own domain name. Clients from Europe had to cross the ocean to get a response, but that’s fine because it’s called less frequently (only when clients are first connecting).
Once the SDK identifies an edge server, it opens a persistent TCP connection with the server over TLS.
The SDK then exchanges some messages to prove that it’s our SDK and not some random guy connecting, and to identify who the player is. (For security reasons, I won’t go into details because it would make it a little easier for hackers to break it.)
Each Edge Server also has its own SQS queue, in the same region. When a new edge server comes up, it spins up its own SQS queue, subscribes it to the SNS topic for global chat, and registers itself with the chat service.

Sending a message flow

The player sends a message over the persistent TCP connection. Edge server receives it and sends it to the chat service (same region via internal load balancer.)
The chat service gives the message a Twitter ID (actually a slight variant of it just to double-triple make sure that IDs are unique).
The chat service then looks up the room in its cache (routing table) to find which edge servers have players connected to that room.
It sends the message to the SQS queues of those edge servers.
Asynchronously, it also writes the message to DynamoDB (I’ll describe the design in a separate post on how we avoided hot paritions because in celebrity games, few rooms or the global chat will get very, very popular.)
Each edge server reads its queue (HTTP long poll to SQS) and pushes the message to its players.
The SDK de-duplicates against running list of message IDs and also sorts them by ID since the ID is time-sortable.

Direct chat

Rooms are the basic unit of operation and we made direct chats work using rooms too. Every player who connects gets their own room. In the previous chat service, this was handled by a player service that managed a directory of players and their connections but we opted for a simpler approach.

Global chat

Some games have global chat rooms. We supported this by creating smaller chat rooms and letting VIP players push messages to all smaller chatrooms. This was the pattern that the previous chat service used and it worked well so we decided to keep it.

Lessons learned

I really wanted to share specific numbers on latency, throughput and costs, but I’d need my company’s permission which I don’t have. In general, we reduced latency meaningfully for both international and US players. One game studio, launching a celebrity social network, expected 50,000-100,000 concurrent players when the celebrity was online. They ran large load tests with BlazeMeter, simulating sudden spikes as well as long soak tests. For load tests, everything auto-scaled except the edge servers. We added a pool of 25 servers. We could have used fewer servers but we landed at this number based on our load tests and the fact that we wanted to limit connections to each server to avoid a failure from disconnecting a large numbers of players at once and handling retry requests over a short period of time.

DynamoDB and spikes: Our biggest lesson was handling sudden spikes in DynamoDB. We used an open-source tool to manage DynamoDB scaling automatically, but it couldn’t keep up with sudden spikes. To be fair, the tool was not the issue. DynamoDB doesn’t give you extra capacity instantly, because it sometimes has to add partitions i.e. more nodes to increase throughput. On top of that, AWS limited how many times a day we could lower capacity, so once we scaled up for a spike, we often paid for the extra capacity for hours.

We worked with our amazing AWS team to fine tune our settings. We scaled DynamoDB up as soon as we saw traffic building, which meant paying for more capacity than we needed most of the time. But we didn’t go extreme here for costs and also a large spike of users coming from a tweet that the celebrity posts would still not work. So we added another SQS queue to absorb writes whenever DynamoDB started throttling as well as queue workers that wrote them to DynamoDB later. This added some complexity, but it was worth it because it removed a source of failure: a spike would delay or not return all of the past messages, but it will not result in backups with servers getting slow or crashing and it also will never lose a message. Overall, I was happy with DynamoDB’s design and performance, but not with the costs and the fact that this was something for us to monitor (PagerDuty!).

In hindsight, for the traffic we actually had, a single beefy, MySQL RDS instance (with a standby) would have been cheaper and would have worked just as well (we already added a queue to absorb spikes and could have done batch writes to MySQL). To be fair, I did push for DynamoDB partly because of its features: it’s managed by AWS, TTL, flexible schema. But also because I wanted to use it and learn it, coming from managing Cassandra clusters at my previous job and I had read DynamoDB paper and was super curious about it at scale.

SQS and SNS: We load tested SQS and SNS and were happy with their throughput and reliability. They were the biggest contributor to latency on our side, adding 10s of milliseconds. Direct edge to chat connections e.g. protobuf would have removed delays associated with queuing but we decided to keep using the boring queue pattern to keep things simple. We also tested Redis pub/sub, and it decreased latency significantly. We decided not to use it for the time being, because it would have meant more load tests to adjust our Redis cluster, which was shared by all our services. SQS has better message retention compared to Redis Pub/Sub but we could have relaxed that requirement given real-time nature of most chat messages plus the fact that history was stored when games needed it. Or may be these are just excuses and the real reason was that Redis Pub/Sub wasn’t discussed until the project was tested and shipped. (Note: For some reason, we had a big discount from AWS on SQS to cost wasn’t a concern for these).

Custom features: We made messages extensible to avoid adding custom game-specific code later. Each message could carry optional metadata defined by game studios (we did place some reasonable upper limits mostly to keep each read within 4 KB which is 1 RCU in DynamoDB). Chat service didn’t use the metadata itself and simply passed it to servers. It was a welcome feature: one studio built stickers within a day with it and another liked our low latency so much that they built a mini game in-chat (users had to smash a button and each hit was recorded!) entirely using chat messages (metadata).

Single points of failures: The system had two main single points of failure: Redis and us-east-1. We decided to accept the latter telling ourselves that it an entire AWS region is down, we have bigger problems anyways (up until a few months ago, I’d have also probably said: ‘oh these failures are so rare, don’t worry about them and keep it simple’). To fix the first, we added caching and if Redis was permanently down, the system would short-circuit and start broadcasting messages (via SNS topic) to all edge servers (their SQS queues). To be clear, I’m not suggesting was accepting us-east-1 failure is the right choice. Our wonderful DevOps team is actively working on making our entire infrastructure run in more than 1 region and tolerate regional failures for critical services so we’ll address this soon.

All games are not equal: The chat service treats all players for all games the same. In reality, we had a lot of small games that had many players but made very little revenue. We also had a very small number of very popular games that had mostly predictable peaks, but no always. The system didn’t have a good way of degrading gracefully in emergency situations and prioritizing games that generated more revenue.

There must have been several other lessons but I’ll end the post here. In future posts, I will share how we load tested various AWS instances (general, memory and compute optimized and how we tweaked the OS and Java to allow more TCP connections per server) to find the right type of instance of Edge and Chat servers, how we designed DynamoDB tables and partition keys to avoid partitions and in very high level terms, how we handled security and prevented common attacks at the Edge Server level. Thanks for reading!

[Removed links as part of deadlink clean-up]