What do you do when your favorite nostalgic game shuts down?
You reverse-engineer it and write your own infrastructure for it with 0 documentation!
overview
The original server for the game, Enemy On Board by Windwalk Games, was shut down in 2021. The game relies on servers for everything: progression, games, data, etc. - the game will not even load if servers are down.
reverse engineering
The game is written in Unity and is compiled with Mono - Unity games can be compiled with Mono, or IL2CPP. Mono games are completely unobfuscated, whilst IL2CPP games are converted to machine code, which is very hard to read. Since Unity games are made with C#, then compiled to bytecode (MSIL, Microsoft Intermediate Language), it's very easy to read the MSIL code.
In fact, to read a Unity Mono game's code, one can go into the game's folder, then the x_Data (where x is the name of the game) folder, then the Managed folder and the Assembly-CSharp.dll can be found (along with many other DLL dependencies of the game) that can be easily opened with software such as dotPeek, dnSpy, ILSpy. The game's entire unobfuscated MSIL code can be read there.
The mentioned software not only read the MSIL code, but also have features to re-convert them to easy-to-read C# code, which eases the reverse engineering process. After that, they also allow decompiling the project to add some code, which could, for example, load a DLL that you write. However, this approach was not used - I instead used MelonLoader, which allows loading mods without direct DLL editing for Unity along with some utils, such as a console window, preinstalled libraries, etc.
writing the mod
MelonLoader was used to develop a mod, the Andromeda Mod, that helped me understand what the game expects - by using the Harmony library provided by it, one is able to create patches for specific methods of another DLL; being prefixes (running before the method, able to reject running it), postfixes (running after the method, being able to modify the returned result) and transpilers (being able to directly modify IL instructions and add your own or delete existing ones).
I used a postfix on Unity's UnityWebRequest helper class' SendWebRequest, which allowed me to see every http request going through Unity. This was used in order to accomplish a custom DevTools-like Network tab, which helped to understand what the game was expecting as well as what it was sending to the now-shutdown Windwalk servers. After this and looking around on the game's code, I was able to understand what would be required for a server to accomodate for the game.
writing the server
Now that I had some data, I could start creating a server, which is the Andromeda.Core server. I used Python's FastAPI module to create a test server - then patched the game using my mod to switch from Windwalk's server to my localhost server. Surprise: I finally got past the "Couldn't contact Windwalk servers: please try again. The game will now exit" screen. Nothing was functional yet - progress, characters, etc. so I had to keep trying. After a lot of datamining, I found that the game had GUIDs for each character and items were stored in a key, after which I added a feature to the mod to export them to JSON. The server could now read these jsons, and after a little manual tweaking, each player now had access to their own characters and such.
Now that I was done with core server logic and main menu, I could go into actual gameplay. This was a lot more complicated to make - I did not have access to a "dedicated server build". By patching heavily the game client, I was able to create a headless dedicated server that supported arguments, being managed by the core server, heartbeats, etc. Even the main game didn't have heartbeats!
Production
After completing the server and mod, I could start deploying to prod. However, this produced some challenges
- Regional matchmaking
- Cost of hosting and how to minimize it
- Which platform to use
I finally set on Google Cloud. Google Cloud had these advantages other platforms didn't have
- Suspended machines instead of entirely stopped - necessary for fast start/stop of machines
- Per-minute billing for cost-efficiency
- No extensive tax registration
Although it is pricier than other VPS providers (Vultr, Hetzner, etc.) these advantages, since the lobbies are not 24/7, allowed to reduce the cost significantly. To bring the testing environment to prod, I needed to minimize the amount of VMs, so I developed a simple program called Andromeda.Orchestrator which receives requests from the core server, deployed on Cloudflare Workers.
The Orchestrator spins up Docker instances pulled from a Docker image containing an installer of the game using SteamCMD + MelonLoader & Andromeda. To avoid being DMCA'd, as I can't distribute game files, it downloads the game using SteamCMD, and to avoid long boots where the game is redownloaded, they are binded to on-disk volumes that prevent useless reinstallation. Only the mod's version is checked on boot with file metadata.
The Orchestrator is hosted on the GCP VM - if the Core cannot reach it, it first checks GCP to make sure it's up - if it's not, it starts it, which starts the billing. Then, an ephmeral IPv4 is assigned to the machine, which is free compared to a static IPv4. The Orchestrator machine updates Cloudflare DNS records (Dynamic DNS) with its new IP, and the Core machine reads the IP from GCP to assign players to the proper IP for TCP connection. After 2h of inactivity, Core suspends the proper GCP instance. Multiple Orchestrators exist and are spinned up by Core regionally, using the player's settings.
The Orchestrator Docker images for the game uses Wine, in order to avoid using VMs for overhead and Windows licensing prices.
steam issues
The game would not work at first in prod. I thought it was a Linux issue, but when tested on my local Linux machine, everything worked fine. After a LOT of debugging, I found the issue: the game expects Steam installed on the target machine, as if it was a player. A few patches allowed me to change from Facepunch Steamworks' SteamClient to SteamServer, and then, a full-fledged production-ready server was created. An admin panel was added for the game, including reports; banning; a very extensive log system with Steam ID filtering, date filtering, auto lookup; and stats.
distribution
For easy setup, I forked the MelonLoader installer - I could've created something from scratch, but using the existing MelonLoader installer which finds steam games automatically, installs MelonLoader cuts about 80% of the job I had left. I added support to auto download the Andromeda mod, disabling the debug console MelonLoader opens automatically, and I had a finished product that was easy to distribute and install in a single click.
My GitHub has all 4 repositories, being Andromeda.Core, Andromeda.Mod, Andromeda.Installer and Andromeda.Orchestrator. The master repository, Andromeda, contains the in-production version of each mod as a submodule. There is a full CI/CD pipeline for each repository - Andromeda.Mod is automatically built using dependencies privately distributed through Andromeda.Core and distributed through releases, where the installer grabs them; Andromeda.Installer is automatically built from releases; Andromeda.Orchestrator's host-image is published in the GitHub Container Registry (GHCR) automatically, and the master repository grabs artifacts on release.
The installer fetches new versions from the master repository's releases and bleeding edge from Andromeda.Mod.
The Andromeda Discord Server gained a decent following almost instantly. In all, the project was a success. We achieved:
- Production-grade servers
- CI/CD for automatic deployment and auto-updating + keeping clients up-to-date
- A discord server automatically linked to the game, showing lobbies
- A full revival of the game!