Home > Latest Thinking > Learn > Running gaming platforms at scale in Kubernetes

Running gaming platforms at scale in Kubernetes

Author

Date

Overview

Kasna cloud engineer Sai Maduri elaborates on a kube demo running Agones and OpenMatch that he built for the Google Cloud Summit Sydney.

As we all embrace Kubernetes for its extensibility and open source community, now it’s also enabling gaming platforms under its hood.

Google, through its https://games.withgoogle.com/ initiative, has released two open source frameworks for running gaming platforms at scale in Kubernetes.

- Agones Game Servers https://agones.dev/site/
- OpenMatch https://open-match.dev/site/

At the Google Cloud Summit Sydney in 2019, the Kasna team demonstrated these two tools to the crowd, and here we share what we created.

When running Game Servers, one needs to consider running them at scale, optimising cost and enhancing the experience of players with quickest game server allocation, ease of matchmaking and strong player community.

Here’s how the Google tech addresses these considerations.

Agones Game Server will help to run game servers at scale managing the cost and low latency allocation of game servers as containers in Kubernetes.

OpenMatch will help to customise the gaming community experience with match-making, ranking allocations among players and game strategy choice. This would enable us to enhance player communities.

Xonotic and Game Server overview

I’ll use an open source game called Xonotic to explain how Agones and OpenMatch can embrace similar kind of games among more players communities.

Xonotic is an arena style first person shooter game, you can either play in local mode or join any of the community servers to play online. There are various weapons, game modes and maps. I believe this would be the ideal game with minimum parameters (game modes and maps) to create different combinations for game players to create a game session for players in a multiplayer game scenarios.

Let me explain what I mean by combinations using parameters. Xonotic has different game modes and almost every massively multiplayer online game would have something similar. For example, Deathmatch is where teams win when they make more kills, Capture the Flag is where teams win if they capture more flags. There are different Maps which has its own advantages and disadvantages in each game mode. Players ranking, experience and weapons are also important parameters.

If your game allocations involve high experience players and low experience players in the same game session it is highly probable low experience players will find the game disadvantaging and will never comeback. Similarly high experience players will not find game challenging at all which is really important to keep players experience interesting. We should also be able to create right teams as players login as quickly as possible and allocate a game servers to those ideal teams.

This is what I embraced in games such as Tanki Online, which I used to play years ago and such strategies are followed in latest popular mobile games such Walking War Robots (it has more combinations of parameters though which makes it more interesting).

Let’s look at how these two open source tools from Google help us with this.

Deploy into GKE

First let’s create a Kubernetes cluster. GKE is ideal and simple, so I will use GKE to demonstrate this but you can run in any Kubernetes cluster.

Clone this repo https://github.com/kasna-cloud/gameswithgoogle and just run

The above make task should create a gke cluster, setup kube auth, install agones system, install open-match and add firewall rules to allow udp traffic from 7000-8000 for instances with network tags game-server . It will take a few mins to setup all the services for Agones and OpenMatch.

Now the core services from Agones and OpenMatch are all up and running. Lets first explore Agones and gameservers.

Agones

The Deployment scripts and Kubernetes manifest files for this post are available at mainrepourl open folder agones-game-servers/xonotic.

It has Docker file to create a container with xonotic game server which can be downloaded from https://xonotic.org/download/ and also a go program main.go, which will invoke the xonotic executable and utilises agones sdk to health check the game server.

We can deploy a simple game server by running

It builds a Docker image with xonotic server and pushed to gcr repository (required update makefile var REPOSITORY with your gcloud project id).

The agones-game-servers/xonotic/gameserver.yaml file custom resource definition to deploy an Agones game server type as defined with the Kubernetes manifest as:

Once it is executed you should be able to run this command and see the IP address (one of the gke nodes public ip) and port (in the range 7000-8000) which we allowed in the firewall rules to be allowed to connect from the world.

You can launch the xonotic game in your local computer and go the multi-player menu and enter your ip address and port to establish the connection to the game server you just created in agones. It is a single same server at this point. I’ll explain allocations and autoscaling game servers in the later sections.

OpenMatch

In the above step we only created one game server with a single container and we are the only player in the server. There are many public servers already in the multiplayer menu in the xonotic game and it doesn’t seem much different than a standalone server. The xonotic executable itself has all the necessary configurations to run own server or join existing public game servers but still it lacks a strong community of players.

The reasons are:

It doesn’t do any kind of matchmaking with teams and those public servers are always sitting idle, less players are engaged in each of those servers.
Player levels or experience is not considered to enable ideal matchmaking with similar levels or choice of game mode to create intense game matches.

These things needs to be stripped out of the actual game servers and make match-making logic out of the gameservers to enhance player experience. That is where OpenMatch shines.

I am not going to re-code the xonotic game to make it modular. But I’ll explain how we can create logic in OpenMatch to create groups of players and allocate gameserver containers on demand from the Agones system for different groups of players.

Let’s assume we have 200 players logged in at one point in time and we can allocate 5 people in each team. Making 10 players in each game server with two opposite teams playing a match. Out of these 200 players, let’s assume we have these following characteristics

Player Levels

Player Mode Preferences

You can see the above tables looks simple to make various combinations and assign them to any game server.

But let’s assume out of the 10 level 5 players, six of them are interested in DeathMatch at that point in time and only four players are interested in Capture the Flag Mode. We cannot just put all the 10 players in one gameserver by randomly choosing the game mode or we cannot just bring level 1 players and combine with level 5 players as it would cause bad game experience for both levels of players.

Instead of player level, we can consider another parameter such as experience to bring some of level 4 players whose experience in closer to level 5 to make teams with level 5. Matchmaking is really important for multiplayer games to create larger and continuous gaming communities.

We will look at how we can create a simple matchmaking by creating matches with the following properties:

There are only 10 players in each match
Each match has five players each with two rosters (red and blue teams in each match)
All 10 players in a match with same player level

Let’s first look at what OpenMatch Platform Provides and which parts we need to implement to create matchmaking:

Only the Green Component in the diagram is what we deployed in GKE until now, which is the core services. Let’s deploy a sample client and match function which will create fake players in batches every 15 seconds with random player levels and a matchmaker function to group the players and create matches with requirements which we defined above.

The Kubernetes manifest files I’m using to deploy for this post are using the Docker images from my gcr registry. However, you can look at the code for the frontend implementation to create tickets, Match Functions to create Matches from Pools of players and Director to assign game servers to tickets in the repository at https://github.com/kasna-cloud/gameswithgoogle/tree/master/open-match-example

The above command should deploy the Kubernetes manifest with a client application and a matchmaking function. We can look at the pods running by looking up running pods with command

The last two pods should be something similar to below, with some other random suffixes for your pods:

openmatchclient-ff7b64f6c-dj7fk 1/1 Running 0 3m

openmatchfunction-6df64cb567-7wqbd 1/1 Running 0 3m

openmatchclient-ff7b64f6c-dj7fk is the pod simulating client applications such as ticket creation, fetching matches from backend, and director to allocate matches to gameservers(randomly generated IP addresses at this stage).

openmatchfunction-6df64cb567-7wqbd is the pod which is running our matchmaking logic.

We can access this through a browser using the below command via a proxy at localhost:51507. You should be able to see how a simulation is running to use fake players to create tickets into Agones via the client application and how it is getting responses of matches with our desired match configuration. That is, each match with 10 players, all players at the same level and two teams in each match.

You can look at the OpenMatch function pod logs to see how it is allocating tickets from pools to matches.

It creates matches super quick using redis indexes and we can customise the pooling functions on tickets using various FilterFunctions available in the open-match sdk (explore the given code to understand how it works).

As this stage the IP addresses allocated to matches are randomly generated. Now le’s see how we can integrate Agones to dynamically allocate gameservers to each match from openmatch in super quick way.

Agones autoscaling

In a previous section, we created a single game server in Agones. now as OpenMatch has started to create us matches with teams. We need to create game servers instantly to allocate a match in it. Agones has a concept called fleet to run gameservers at scale. we can create a feel for xonotic using the command

This applies the agones-game-servers/xonotic/fleet.yaml which creates a fleet of gameservers. We can also create a fleet autoscaler with the definition agones-game-servers/xonotic/fleetautoscaler.yaml

This particular feel autoscaler is to using a buffer type of scaling size of 5 (there are other interesting scaling strategies to look for check agones.dev), which will always ahead maintain five gameservers available for us to allocate to a match.

There is another manifest agones-game-servers/xonotic/gameserverallocation.yaml which allocates a gameserver from the fleet.

This should allocate a gameserver from the fleet to allocated status, and the fleetautoscaler will create another gameserver in Ready Status. A gameserver that is allocated will have status ‘Allocated’.

At this stage we created a gameserver allocation manually, how can we link this to the director in our openmatch code? We’ll link them both in the next section.

Agones allocator service with OpenMatch director

Agones sdk provides us capabilities to create allocations through api thus enabling us to create a service which we can deploy as a Kubernetes service and can be consumed by the director service in our openmatch client to continuously request gameserver allocation on the fly as we create matches from teams.

The code for the allocator service is provided at https://github.com/kasna-cloud/gameswithgoogle/tree/master/agones-game-servers/allocator-service

We can deploy the Kubernetes manifest to create the service in our cluster with the following command

This creates a service for the allocator. Now let’s use this in our director in open-match

Open the simulator of this new client to see actual game server IP addresses from Agones being allocated to matches now

Just to visualise the scaling of tickets requests, matches, gameserver allocations and gameservers running I’m sharing the following screenshots from grafana for both agones and open-match here.

OpenMatch Tickets and Assignments:

Agones Allocations and GameServers:

Average latency is 371 ms to allocate new gameserver to matches. If you see the Gameservers fleet is maintained with a buffer of around 5 instances in the graph.

Finally…

Don’t forget to destroy cluster!