AiTiles [Rev. 3] - AI Upscaled Terrain Tiles for Falcon BMS

[Dodam]

AiTiles by Dodam

Introduction

Hello Falcon BMS community,

I’m very proud to present AiTiles, an AI upscaling of the default terrain tiles in Falcon BMS. These tiles have a 1024x1024 resolution, up from 512x512 of the original tiles. Tiles are 1x1km in BMS, so this means that they have a resolution of ~1m / px, up from the original ~2m / px, or 4 times as many pixels.

This is a WIP, and I will be fine-tuning the network in the coming weeks to see what I can tweak to improve the quality. This post will be updated as newer versions come out. The current revision is Rev. 3, released on 2021-05-25. Please provide feedback if you have any!

Description

The upscaled tiles were made by using a GAN (generative adversarial network), trained on 3.6 terabytes of 1ft resolution natural colour images publicly available from the USGS EROS (United States Geological Survey - Earth Resources Observation and Science Center), from the High Resolution Orthoimagery dataset.

In simple terms, I taught one neural network (the upscaler) to take 2m resolution aerial images and upscale it to 0.5m resolution (while maintaining the original image as much as possible), and another neural network (the detector) to figure out which images were the originals and which ones were the upscaled versions. Over time, they learned from each other, and the upscaler learned to make 0.5m resolution images from 2m ones that are indistinguishable from the photos that were taken at 0.5m resolution to begin with, filling in the details as necessary. I am only releasing the 1m version (which is downsampled from the 0.5m version) because the difference in quality seems marginal (currently, anyway). The architecture (SPUN-GAN) is my own creation that is very loosely based on ESR-GAN, that allows for more context sensitivity, feature localisation, and edge preservation.

Screenshots

Here’s a few closeups of how the new tiles look. The left image is the original BMS terrain tile, the centre is the original tile with bilinear upsampling, and the right image is the new AiTiles tile.

(Note: These were taken at 0.5m resolution rather than 1m resolution.)

Download Link
Click (2.33 GB compressed, 9.09 GB uncompressed)

Installation

1. Rename the folder ‘Falcon BMS\Data\TerrData\Korea\texture\texture_polak’ to whatever you want – this will be your backup
2. Extract the downloaded archive
3. Copy the folder ‘aitiles_rev3_texture_polak’ into ‘‘Falcon BMS\Data\TerrData\Korea\texture’
4. Rename ‘aitiles_texture_polak’ to ‘texture_polak’

Q & A

Why AiTiles and not AITiles with a capital I?
The name is an homage to HiTiles / HiTilesAF - I think HiTiles was one of the first online purchases I made as a teenager.

How is this different from just bilinear upsampling?
In bilinear upsampling, all you’re doing is averaging 4 pixels with slightly different weights to get a higher resolution image, but the image doesn’t contain any more information. Neural nets, on the other hand, can generate more information – effectively, the network “sees” something that looks like a house or a road at low resolution, and fills in the details to make it look like the high resolution house or a road that it’s seen at high resolution.

Why do some images look “painted” or pixelated when close up?
Upscaling based on the kind of limited data like the BMS terrain textures is a really hard problem on several fronts: firstly, there’s very little data – the 2m resolution means that a typical American home occupies about 6x6 pixels, and a car is 2x1 pixels. Secondly, there is a huge variation in buildings, roads, and terrain, the details of which are lost at such a low resolution. In addition, there are things in the BMS terrain textures (low buildings with blue slate roofs or massive high-rise condominium complexes) that simply do not exist in the training data – the network has never seen these things, and doesn’t know what to do with them (as far as it knows, blue patches next to development is an outdoor swimming pool). Finally, not all BMS terrain textures look the same even amongst themselves (from what I can tell, some are actual photos, some are painted, some are painted on photos, and some are alpha blends of the other types – some even have block artifacts at 4m resolution, which I’m guessing were introduced when an artist made 512x512 textures based on 256x256 textures in a hurry). Together, these factors mean that the network has to make its best guess about what the high resolution data looks like from any number of things it could have been, and what it does is to locally guess something that looks reasonable. If you look at tiles that look particularly bad, you’ll find that they are often tiles where the artist took half of a photo of a city and blended it in Photoshop into a painted image of a forest – there’s no way that the network can learn to deal with that sort of artificial blending without additional handling (I might do that in a future version). Anyway, it might look a little weird when it’s very close up, but I think overall it still looks better than the original.

Will this affect my frame rates?
It will probably be marginal at best, but I cannot say for sure. I’m using 3 4K monitors (11520x2160) on an RTX 3090, and I haven’t noticed any difference in performance.

I don’t notice any difference.
At really low altitudes, the effects from anisotropy (views are slanted, textures are flat) will still make the textures look questionable no matter the resolution. At high altitudes, you won’t notice too much of a difference – on a standard 4K monitor (3840x2160) with the default 60 deg FoV, looking straight down from 18,000 ft AGL, you’re looking at ~5.7 km of terrain over 3840 pixels, which is about 1.5m / px resolution. It’ll be hard to tell the difference between 1m / px and 2m / px at that sort of scale; you’ll notice a difference below 6,000 ft AGL on a 4K display. If you’re playing on a 1080p display, you’ll probably need to be below 4,000 ft AGL.

Can you do this for the terrain textures in other theatres?
Gladly, but with a caveat: The upscaler was trained based on 2 terabytes of images of the U.S. east of the Appalachians; 1.5 terabytes of images of the Rockies and the Sierra Nevadas, as well as the deserts of Nevada, Utah, Arizona, and California; 173 gigabytes of images of metropolitan LA, Indianapolis, Chicago, Philadelphia, and St. Louis; 68 gigabytes of images of airports around the U.S. If your theatre looks different from all of these areas, the upscaler may not know what to do with your textures. Furthermore, the network hasn’t seen many really high mountains, because the images are aerial rather than satellite (which is why I was able to get them for free - if these had been satellite images, they’d have cost ~100 million dollars depending on the vendor). If you are a theatre developer or someone who can give me permission to do so, let me know in the comments or PM me and I’ll upscale your tiles for you.

Why not use something off the shelf?
I figured it’d be more fun to build this myself. Also, this is something very specific (aerial images) with its own goals (“visual quality metric” for photographs that other networks use is likely to be nearly completely useless for satellite imagery), and generic image enhancement nets didn’t seem to generalise super well. For example, using DeepAI’s Super Resolution API definitely helps the textures look clean, but it doesn’t know what a house or a road looks like from the sky, resulting in something that looks a little strange. I think other textures would benefit greatly from using off-the shelf solutions, though!

Is the code available somewhere?
It will be at some point, but I might want to write a paper on the network architecture so I’m keeping it private for now.

Licence

Whatever licence the BMS team has on their terrain textures applies, but other than that, use the textures however you want – USGS data has no restrictions, and this work is in public domain. Accreditation would be nice if you end up including my work in a theatre, but is unnecessary. I’d be incredibly happy if the BMS team ended up including these textures in the default installation.

Acknowledgements

Jonathan Zung, Princeton University Math Department, for being a great friend and helping me come up with a reasonable network architecture.
Zetta AI, whose open-source code policy let me adapt code I originally wrote for electron microscopy of brains to this project.
Malc, for answering my questions about the terrain textures in BMS.
Data available from the U.S. Geological Survey.

[SemlerPDX]

Awesome work, man! Well done!

[1508AD]

I have only had a quick look around but it looks noticably more detailed. Well done.

I havent really had a chance to check performance on it but I didnt notice anything.

[hoover]

Very cool mate, thanks for sharing! Great to see the “power of open source” at work here again. It would be great to see the results in a theatre like EMF!

All the best & keep up the great work,

Uwe

[Dodam]

Thanks guys – a tweak I made seems to be working quite well, so I hope to be able to release a significantly improved Rev 1 tomorrow.

Rev 1 will have a fair bit more sharpness in the image instead of tending to create overly smoothed images with low local contrast.

[dema]

I’d love to see a tweak like this in Balkans

[ctn808]

First, Thx Dodam!!
Dema, in Balkan newer airbase are made in 1024rez.
Of course, everybody can try personally to upscale any theater.
I will give try too.
Of course, this will increase dramatically ( about double I believe) the size of Texture folder.
Anyway project like this one is always welcome

[dema]

Some Balkans airports are beautiful. Others have runways and taxi textures with low Res. I can live with that. I’d focus the work on farms textures honestly.
I noticed the time spent on this project is not so long. We know a new terrain engine is coming but not soon.

[hoover]

I think it’s a really clever strategy to “let the AI do all the work”. I’m certainly aware of the fact that many person-moons were spent on the development of the original brain scan image enhancement network, but to “coax” this network into doing work on something completely different (insert funny Monty Python video here) like the BMS KTO tileset ist just genius

All the best,

Uwe

[sasah320]

Amazing work. Very clever!!

[molnibalage]

Pls. make somebody comparison screenshots.

[vAiCon]

Actually, Arty converted KTO tiles into 1k*1k res already,
The change is minor compare to the size and demand from the GPU\CPU to process those in sim

[Migbuster]

Thanks I mean that is mightily impressive work!

Anyone who has manually worked on Terrain tiles knows the time they take and the massive limitations of them due to that. Concepts using forms of AI are definitely the way forward.

[Dodam]

@hoover:

I think it’s a really clever strategy to “let the AI do all the work”. I’m certainly aware of the fact that many person-moons were spent on the development of the original brain scan image enhancement network, but to “coax” this network into doing work on something completely different (insert funny Monty Python video here) like the BMS KTO tileset ist just genius

Oh, it was just one me-month for this. I started working on this to as an idea to save space (we have petabytes of images for structural connectomics and it’d be nice to not have to store everything at full resolution, or at least not have to fetch them at full resolution – storage is expensive), but then realised that the requirements for EM and BMS Tileset were similar:

1. Dataset is very limited domain but there’s huge variability within that domain.
2. Context awareness is crucial to getting the correct image.
3. Good feature localisation that’s stable with respect to semantic segmentation at multiple scales.
4. Off-the-shelf loss used by neural nets to supplement the adversarial loss with regularisation are likely to be irrelevant.

@vAiCon:

Actually, Arty converted KTO tiles into 1k*1k res already,
The change is minor compare to the size and demand from the GPU\CPU to process those in sim

I did come across that post when I was searching the forums, but I think he was using Topaz (which is for photographs) – I don’t know how well that would / did generalise to terrain textures, though. Do you have them somewhere?

Also, BMS isn’t that demanding even on modest hardware in 2021 (unless it’s in inclement weather or something) – there’s certainly diminishing returns, but I think this is still worth the increase in size (2k x 2k probably won’t be for most people, though, and definitely not for ITO).

@Migbuster:

Anyone who has manually worked on Terrain tiles knows the time they take and the massive limitations of them due to that. Concepts using forms of AI are definitely the way forward.

The reasonable things that were done to save time (e.g. blending cities and forests with masks, drawing two skinny grey rectangles for a freeway, upsampling 256x256 to 512x512 and working with that as the base and not paying attention to the blocks) are the things that end up causing issues with the net, though there’s ways I can think of around it.

I think it wouldn’t be super hard to make / teach another network to generate tiles based on some sort of semantic encoding (“here’s elevation data, where we want cities, towns, roads, freeways, forests, fields, farms, water – come up with tiles that look reasonable”) IF there are consistent tiles for the network to learn from AND good labelling, but I’m likely being way overoptimistic.

As an unrelated side note, the KTO will never be super realistic for me because I don’t think anybody would take the time to implement the rooftop GOPs (Not to mention the FA-50 Fighting Eagle, though that only entered service in 2012.) Many skyscrapers – including residential high-rises – have AAA batteries on the top. (https://namu.wiki/w/%EB%B9%8C%EB%94%A9%20GOP)

[tad]

Hi Dodam,
How similar is your method to this one:
https://mrl.cs.nyu.edu/projects/image-analogies/

[Dodam]

@tad:

Hi Dodam,
How similar is your method to this one:
https://mrl.cs.nyu.edu/projects/image-analogies/

That’s an ancient website and an ancient method (20 years in machine learning is a verrrry long time) – the paper doesn’t use any convolutions, nonlinear activations, or pooling, which are all hallmarks of modern deep learning. (I don’t know what year these started becoming popular – I know that some of them have existed since the 90s, at least.)

It’s a precursor, but I think about the only similarity is “filters are learned” – this is what my method is ripping off, though I have made pretty significant changes: https://arxiv.org/abs/1809.00219; it’s capable of doing some amazing things for game textures in general.

[tad]

It’s a precursor, but I think about the only similarity is “filters are learned”

No, I fully understand that the link in my post is “ancient”. I wanted just to confirm that I understood your method with some added clarity.

[Jackal]

Pretty nice initiative this yours, Dodam.

Thanks for sharing and your commitment. As just seems, major changes will happen in theater making in the next future, but until then… oh, who cares, after then, we will see how to apply all what we learned in these years about.

I’m just curious now to see how things will go even with my RTX 2080Ti, “ancient” if compared with your powerful 3090 (good purchase, BTW, well done this too!)

With best compliments and regards.

[Dodam]

@tad:

No, I fully understand that the link in my post is “ancient”. I wanted just to confirm that I understood your method with some added clarity.

The “analogy” in this sense would be (2m res aerial photography : 0.5m resolution aerial photography) :: (512px terrain tiles : ?), but with a lot more going into the sausage.

I’m doing what’s called a “zero-shot generalisation”, which is just to say that the network never sees the terrain tiles in learning until it’s asked to actually upscale it for production – this is obviously necessary since I don’t have high resolution terrain tiles I can give the network as “this is what it should look like in the ideal case”. This relies on the terrain tiles looking similar to actual aerial photography, though there are techniques to help with generalisation. I might add another network that just focuses on photorealism (“take this 2048x2048 terrain tile, and make it look like a 2048x2048 aerial photograph”) in the future – the way to do that is to use another GAN, though I would really need to sit down and think about what the best way to do that would be since there’s only so many tiles.

Something that’s cool is that the night tiles are also a zero-shot generalisation – the SPUN-GAN has never seen a night tile (high resolution orthoimagery isn’t available for night scenes), but it seems know what to do with night tiles without being explicitly taught.

[Dodam]

@Jackal:

Thanks for sharing and your commitment. As just seems, major changes will happen in theater making in the next future, but until then… oh, who cares, after then, we will see how to apply all what we learned in these years about.

Heh – if nothing else, I hope this can reduce the workload for theatrebuilders. How are they changing?

Also, if you need an accurate OOB for ROK with approximate locations at the battalion level, I’m happy to translate that from what’s available publicly on Korean wikis. There’s also a bunch of spelling errors / border inaccuracies in the KTO.