If i’m going to write a server-authoritative multiplayer game, it would be nice to have a decent way to deploy it. Especially to deploy servers around the world in response to player demand, to autoscale and keep latency down.

Modifying the Lightyear spaceships example

I took my spaceships example from the lightyear repo, and repackaged it with the following changes:

Split into a client, shared, and server crates
Changed to only support webtransport – no steam, no plain UDP, etc.
Made server headless - no gui option
Added Dockerfile to build the server; produces a ~30mb image (thanks distroless)
Added Dockerfile to build wasm client; produces an image based on nginx to serve our index.html and wasm assets.
Added Github actions to build and push the server and wasm client docker images

At this point it was easy enough to run the server and client containers on a single server, and I started to look for a way to autodeploy servers to multiple locations on demand. I’m not going to review the various options or discuss the pros and cons other than to say there are a few to choose from, and I decided to go with Edgegap.

My bevygap-spaceships repo contains this repackaged version of the spaceships demo.

Edgegap – the basics

You push the gameserver docker image to Edgegap’s container registry, so they are able to deploy gameservers for you at will.
You configure the metadata for your game – max players per server, time an empty server stays online for, etc.
When players want to play, you use Edgegap’s API to create a Session, which is a list of at least 1 player IP addresses.
For every Session, Edgegap will either return the details for an already-running server, that has capacity, in a sensible location, for the IPs in the session, or take a few seconds to deploy another gameserver for you, before returning the deployment details. An edgegap deployment refers to a single instance of your gameserver container, and includes amongst other things the ip:port clients need to make a connection
You pass the ip:port along to the game client, which makes the connection to the server.

There’s a bit of plumbing needed to wire all this up so it’s easy to use in bevy, which I will now explain.

Bevygap: a toolkit for running server/client games on Edgegap

Allow me to introduce the various components:

Bevygap component diagram

Messaging layer

NATS, running from a docker container, listening using TLS

Gameservers need to be able to talk to the matchmaker(s), and a bit of key/value storage is useful for configuration. I chose to use NATS, which supports various messaging patterns like pub-sub, along with some key/value storage. My NATS server listens on a public IP, so that the gameservers can connect to it from Edgegap’s infrastructure.

Matchmaker

A rust async tokio binary

When a player wants to connect, the matchmaker talks to the edgegap API to create a session, and await the results before returning the deployment details (the gameserver ip:port etc). The matchmaker exposes this functionality as a service on the NATS bus.

Matchmaker HTTPd

A rust async tokio binary using the Axum webserver

This is the public websocket interface to the matchmaker. It speaks JSON to the game clients, and passes on the query via NATS to the matchmaker service.

Bevygap Server Plugin

A bevy plugin for the gameserver

At startup, this will make a connection to your NATS server, and read the various environment variables set by Edgegap. This tells us things like the geographic location of the server, the public ip:port, and how to fetch additional context data. Check out the edgegap docs for injected variables for more.

Bevygap Client Plugin

A bevy plugin for the game client, wasm and native

This is mostly concerned with connecting to our matchmaker websocket over http, and requesting to play. Once a successful response is returned, the plugin will modify the Lightyear settings with the returned server ip and port, the lightyear ConnectToken, and in the case of wasm clients, the certificate digest for the WebTransport connection. It then signals the game to make the connection to the lightyear game server.

As part of this, I wrote bevy_nfws, a no-frills websocket client library for bevy that works on native and wasm. There are a few feature-rich websocket libraries for bevy already, but many expect you to also be running bevy on the server end of the connection, which I’m not.

Modifying a Lightyear game to use bevygap

Code changes to the server and client are minimal. My example game toggles them with a bevygap cargo feature, so I can still do local development without using the matchmaker.

Server

Your lightyear gameserver needs two changes to support bevygap: adding the BevygapServerPlugin, and delaying listening on the socket until bevygap reports it is ready. Any failure to connect to NATS or read the edgegap ENVs during startup will cause a panic by design.

impl Plugin for BevygapSpaceshipsServerPlugin {
    fn build(&self, app: &mut App) {
      // Add the bevygap server plugin which will connect to NATS, read the environment, etc
      app.add_plugins(BevygapServerPlugin::self_signed_digest(cert_digest));
      // only start listening once bevygap setup complete
      app.observe(start_listening_once_bevygap_ready);
    }
}

/// Without bevygap, you'd just call `start_server()` in a Startup system.
/// We defer this until bevygap setup is complete.
fn start_listening_once_bevygap_ready(_: Trigger<BevygapReady>, mut commands: Commands) {
    info!("Lightyear server listening, bevygap reported ready");
    commands.start_server();
}

WebTransport, WASM Clients, and TLS Certificate Digests

My gameserver generates a WebTransport-specification-compliant self-signed TLS cert on startup. This is fully supported as long as you provide the certificate digest to the browser before it attempts a connection, and comply with the following part of the spec:

The certificate MUST be an X.509v3 certificate as defined in RFC5280.
The key used in the Subject Public Key field MUST be one of the allowed public key algorithms. (I’m using ECDSA P-256)
The current time MUST be within the validity period of the certificate as defined in Section 4.1.2.5 of RFC5280.
The total length of the validity period MUST NOT exceed two weeks.

The bevygap system automatically stores the certificate digest in NATS and passes it along to clients in the matchmaker response.

Client

Add the bevygap client plugin, after inserting BevygapClientConfig, specifying:

the URL to the matchmaker websocket, eg: wss://matchmaker.example.com/matchmaker/ws
the name of the game, as configured in edgegap
the version of the game, as configured in edgegap

Instead of calling Lightyear’s commands.connect_client(), you use commands.bevygap_connect_client() which will call lightyear’s connect_client() for you after receiving a response from the matchmaker and modifying the connection parameters.

You can watch for changes to BevygapClientState to see progress.

impl Plugin for BevygapSpaceshipsClientPlugin {
    fn build(&self, app: &mut App) {
      // ..
      app.insert_resource(BevygapClientConfig {
          matchmaker_url: get_matchmaker_url(),
          game_name: "bevygap-spaceships".to_string(),
          game_version: "1".to_string(),
          ..default()
      });

      app.add_plugins(BevygapClientPlugin);
      app.add_systems(Startup, connect);

      // You can monitor changes to matchmaking state like this:
      // app.add_systems(
      //     Update,
      //     on_bevygap_state_change.run_if(state_changed::<BevygapClientState>),
      // );
      // ..
    }
}

/// This opens the websocket to the matchmaker url, and requests to play.
/// Once a successful response arrives, it calls lightyear's commands.connect_client() fn.
fn connect(mut commands: Commands) {
  commands.bevygap_connect_client();
}

Cost and Scaling

If you follow the Edgegap setup instructions in The Bevygap Book you’ll see that I set the “Empty Time To Live” to 10 minutes, and select autodeploy. This means server instances are terminated after 10 mins of having no connected players, and automatically started where they are needed.

Since Edgegap charge for the running time and bandwidth servers use, you aren’t billed for the time you have zero instances running.

This means as well as scaling up, it scales down to almost zero when there are no players. There is a small fixed cost for a low powered server that runs NATS and the matchmaker service. I run those services on the machine that hosts this website.

Using it

You can set this all up with an Edgegap free trial account. The installation sections in The Bevygap Book shows how I set up NATS (and the offical NATS docs are excellent), along with Edgegap, and all the required environment variables.

If you try it out, let me know in the #lightyear channel on Bevy’s Discord.

Live Demo

Is the live demo of bevygap-spaceships working? Hopefully! In chrome at least..

Richard Jones