AiTiles by Dodam
Introduction
Hello Falcon BMS community,
I’m very proud to present AiTiles, an AI upscaling of the default terrain tiles in Falcon BMS. These tiles have a 1024x1024 resolution, up from 512x512 of the original tiles. Tiles are 1x1km in BMS, so this means that they have a resolution of ~1m / px, up from the original ~2m / px, or 4 times as many pixels.
This is a WIP, and I will be fine-tuning the network in the coming weeks to see what I can tweak to improve the quality. This post will be updated as newer versions come out. The current revision is Rev. 3, released on 2021-05-25. Please provide feedback if you have any!
Description
The upscaled tiles were made by using a GAN (generative adversarial network), trained on 3.6 terabytes of 1ft resolution natural colour images publicly available from the USGS EROS (United States Geological Survey - Earth Resources Observation and Science Center), from the High Resolution Orthoimagery dataset.
In simple terms, I taught one neural network (the upscaler) to take 2m resolution aerial images and upscale it to 0.5m resolution (while maintaining the original image as much as possible), and another neural network (the detector) to figure out which images were the originals and which ones were the upscaled versions. Over time, they learned from each other, and the upscaler learned to make 0.5m resolution images from 2m ones that are indistinguishable from the photos that were taken at 0.5m resolution to begin with, filling in the details as necessary. I am only releasing the 1m version (which is downsampled from the 0.5m version) because the difference in quality seems marginal (currently, anyway). The architecture (SPUN-GAN) is my own creation that is very loosely based on ESR-GAN, that allows for more context sensitivity, feature localisation, and edge preservation.
Screenshots
Here’s a few closeups of how the new tiles look. The left image is the original BMS terrain tile, the centre is the original tile with bilinear upsampling, and the right image is the new AiTiles tile.
(Note: These were taken at 0.5m resolution rather than 1m resolution.)
Download Link
Click (2.33 GB compressed, 9.09 GB uncompressed)
Installation
- Rename the folder ‘Falcon BMS\Data\TerrData\Korea\texture\texture_polak’ to whatever you want – this will be your backup
- Extract the downloaded archive
- Copy the folder ‘aitiles_rev3_texture_polak’ into ‘‘Falcon BMS\Data\TerrData\Korea\texture’
- Rename ‘aitiles_texture_polak’ to ‘texture_polak’
Q & A
Why AiTiles and not AITiles with a capital I?
The name is an homage to HiTiles / HiTilesAF - I think HiTiles was one of the first online purchases I made as a teenager.
How is this different from just bilinear upsampling?
In bilinear upsampling, all you’re doing is averaging 4 pixels with slightly different weights to get a higher resolution image, but the image doesn’t contain any more information. Neural nets, on the other hand, can generate more information – effectively, the network “sees” something that looks like a house or a road at low resolution, and fills in the details to make it look like the high resolution house or a road that it’s seen at high resolution.
Why do some images look “painted” or pixelated when close up?
Upscaling based on the kind of limited data like the BMS terrain textures is a really hard problem on several fronts: firstly, there’s very little data – the 2m resolution means that a typical American home occupies about 6x6 pixels, and a car is 2x1 pixels. Secondly, there is a huge variation in buildings, roads, and terrain, the details of which are lost at such a low resolution. In addition, there are things in the BMS terrain textures (low buildings with blue slate roofs or massive high-rise condominium complexes) that simply do not exist in the training data – the network has never seen these things, and doesn’t know what to do with them (as far as it knows, blue patches next to development is an outdoor swimming pool). Finally, not all BMS terrain textures look the same even amongst themselves (from what I can tell, some are actual photos, some are painted, some are painted on photos, and some are alpha blends of the other types – some even have block artifacts at 4m resolution, which I’m guessing were introduced when an artist made 512x512 textures based on 256x256 textures in a hurry). Together, these factors mean that the network has to make its best guess about what the high resolution data looks like from any number of things it could have been, and what it does is to locally guess something that looks reasonable. If you look at tiles that look particularly bad, you’ll find that they are often tiles where the artist took half of a photo of a city and blended it in Photoshop into a painted image of a forest – there’s no way that the network can learn to deal with that sort of artificial blending without additional handling (I might do that in a future version). Anyway, it might look a little weird when it’s very close up, but I think overall it still looks better than the original.
Will this affect my frame rates?
It will probably be marginal at best, but I cannot say for sure. I’m using 3 4K monitors (11520x2160) on an RTX 3090, and I haven’t noticed any difference in performance.
I don’t notice any difference.
At really low altitudes, the effects from anisotropy (views are slanted, textures are flat) will still make the textures look questionable no matter the resolution. At high altitudes, you won’t notice too much of a difference – on a standard 4K monitor (3840x2160) with the default 60 deg FoV, looking straight down from 18,000 ft AGL, you’re looking at ~5.7 km of terrain over 3840 pixels, which is about 1.5m / px resolution. It’ll be hard to tell the difference between 1m / px and 2m / px at that sort of scale; you’ll notice a difference below 6,000 ft AGL on a 4K display. If you’re playing on a 1080p display, you’ll probably need to be below 4,000 ft AGL.
Can you do this for the terrain textures in other theatres?
Gladly, but with a caveat: The upscaler was trained based on 2 terabytes of images of the U.S. east of the Appalachians; 1.5 terabytes of images of the Rockies and the Sierra Nevadas, as well as the deserts of Nevada, Utah, Arizona, and California; 173 gigabytes of images of metropolitan LA, Indianapolis, Chicago, Philadelphia, and St. Louis; 68 gigabytes of images of airports around the U.S. If your theatre looks different from all of these areas, the upscaler may not know what to do with your textures. Furthermore, the network hasn’t seen many really high mountains, because the images are aerial rather than satellite (which is why I was able to get them for free - if these had been satellite images, they’d have cost ~100 million dollars depending on the vendor). If you are a theatre developer or someone who can give me permission to do so, let me know in the comments or PM me and I’ll upscale your tiles for you.
Why not use something off the shelf?
I figured it’d be more fun to build this myself. Also, this is something very specific (aerial images) with its own goals (“visual quality metric” for photographs that other networks use is likely to be nearly completely useless for satellite imagery), and generic image enhancement nets didn’t seem to generalise super well. For example, using DeepAI’s Super Resolution API definitely helps the textures look clean, but it doesn’t know what a house or a road looks like from the sky, resulting in something that looks a little strange. I think other textures would benefit greatly from using off-the shelf solutions, though!
Is the code available somewhere?
It will be at some point, but I might want to write a paper on the network architecture so I’m keeping it private for now.
Licence
Whatever licence the BMS team has on their terrain textures applies, but other than that, use the textures however you want – USGS data has no restrictions, and this work is in public domain. Accreditation would be nice if you end up including my work in a theatre, but is unnecessary. I’d be incredibly happy if the BMS team ended up including these textures in the default installation.
Acknowledgements
Jonathan Zung, Princeton University Math Department, for being a great friend and helping me come up with a reasonable network architecture.
Zetta AI, whose open-source code policy let me adapt code I originally wrote for electron microscopy of brains to this project.
Malc, for answering my questions about the terrain textures in BMS.
Data available from the U.S. Geological Survey.