Dedicated Server CPU issues
-
Salut @Mav-jp
Rouge asked me to take part in the conversation as I have already investigated issues with our server.
I can explain a bit our current practice for MP flights. Maybe we are doing things in a suboptimal way or have issues with our setup, the point is to have some guidance, and thanks a lot for taking time to read our messages.
Note : I joined the squadron early 2022, so I can only give info about our practice for the last 12 months. I will let Rouge comment on this fact, but I am under the impression that our MP experience has degraded since 4.36 came out.
But … also … over the year 2022, the squadron grew pretty fast. It is quite common for us now to do MP flight with 12-16 pilots. It is our target at least. Maybe this could explain difficulties instead of incriminating changes in 4.36 or 4.37. Maybe it is time to improve our hosting practice or setup with regards to this increased number of pilots ?We do use RDP to connect to the server and start the session.
Usually, we put the server in a C-130 flight that stays on the ground for the whole session (once in 3d we just leave it on ramp).
I am not sure whether we always put that C-130 close to the area of our mission or not, and whether this has an impact on MP stability/performance (I read about the bubble).In 4.36 we noticed weird missile and AI flights trajectories. Very jittery/vibrating doing ‘curly ribbon stuff’ visible in ACMI. Also visible in 3d when dogfighting the AI for instance.
I noticed that the server was constant 100% CPU and attempted to minimize the BMS window as I suspected software rendering to eat all the ressources.
And indeed, reducing the window did significantly lower CPU usage and made the funky vibrating trajectories disappear.
I heard about an option with a specific d3d dll that inhibits render, but did not try it. Is it the best practice ?Now, in 4.37, and especially last wednesday, we encountered new kinds of problem.
The one in the ACMI Rouge just sent. And also IA missiles missing their targets at the last minute although they were in ideal conditions.
I will try and send another ACMI on that last example.Here are our server’s specs :
CPU is Xeon E3-1245V2 @ 3.40Ghz (8 logical cores)
RAM is 32 Go
No graphical card, software rendering
Windows 2012 R2
Network over 1Gb/sIn the squadron, we have never-ending debates about :
- whether looking for a server with a graphical card would change something. Currently I doubt it, as I think reducing the window does inhibit rendering quite efficiently (at least the CPU is not overloaded anymore). There is no point in spending CPU cycles rendering on the server anyway, right ?
- we fiddled a lot with bandwith settings on the clients side. We tried putting everybody at 70% of their maximum BW, and also putting everybody at a 2048/1024 cap. No clear consensus on which is the best.
You asked about whether we use RDP : does this have an impact ? Should we try :
- using VNC instead
- and/or closing the RDP session once the server is in 3d for the whole mission duration ?
Thanks a lot for your help and for reading this long message. I wanted to give all the context and history …
-
-
@pyhg i think BMS and rdp has been a bad marriage for years but indeed things seem to have gotton worse. This was also noted within the dev team and a very skilled coder managed to tackle the issue. If all goes well, this fix should be in a next update
-
Sorry but I had to separate this post from the rest as this is completely out of scope compared to the AIM-120 discussion…
Of course a server without GPU must not render in 3D otherwise, you’ll end up having massive performance issues. The dll obviously helps with that but U1 should also contain something to help on that front (no need for dll anymore).
Meanwhile, a server that is unresponsive will give you a bad experience overall…
For now, try the dll blank rendering from: https://github.com/UOAF/bmsdedi
Cheers
-
Ok I understand about forking the ‘server topic’, we intially reacted to the missile topic as our concern is mainly over missile behaviour in MP currently.
So, I will also post here the additional ACMI details I wanted to share.
Here is the ACMI I was referring to.
https://we.tl/t-8juqTjGjDZIn it, we can see two things :
- the issue Rouge already reported with the IA missile rotating in the air.
- I was flying the F-16 labelled Pgen, doing RTB litterally on fumes, so high altitude, pretty slow. I saw a 29 spike in my 6 on the RWR, and then an inbound missile. Not much I could do as I could not afford to lose altitude. I was certain to die, but survived. When I look at the ACMI I find it quite a miracle how the missile just takes over me and then dives instead of killing me. I was quite an easy target : slow, high, not manoeuvering, the missile is around Mach 2 when it reaches my position …
-
Thanks.
In the meantime, do you think we should favour VNC ?
I do not believe we have ‘hypervisor/console’ access to the server. So for us it is either RDP or VNC for now. -
Thanks, we will try with the DLL.
But reducing the window as we currently do puts the CPU well below 100%. So currently I would not say we are CPU-bottlenecked anymore. -
@pyhg said in Dedicated Server CPU issues:
CPU well below 100%. So currently I would not say we are CPU-bottlenecked anymore.
You can’t think of it quite so simply … games like BMS don’t spread work perfectly evenly over all cores. There’s a main-loop thread, and render-loop thread… and threads dedicated to things like network I/O and input handling, and audio etc.
With 8 cores, you will mostly likely see 1 core maxed out to 100% and the rest much less, around 20-30%. So overall CPU usage will be, say, 35%. But you’re still CPU bottlenecked because there’s always 1 thread that’s going to be taking the longest, each turn of the loop. Make sense?
Anyway, yes definitely use the bmsdedi dll. If your server is a cloud virt you can probably save a lot of money… 4 cores and 16GB is probably sufficient.
-
We are considering trying another VPS provider, for the sake of having a comparison point with our current server.
Would you say more than 16Gb RAM is overkill if only BMS is running to host MP ?
Also, in terms of BW, is 400Mbps enough for 16 pilots, or should we look for more ?The provider I am currently looking at has a product with 6 cores (probably too much for just BMS process), 16Gb RAM and 400Mbps.
The offer above is 8cores, 30Gb RAM, 600Mbps. I am not sure it is necessary in our situation ? -
Yes, I understood the point about single CPU core usage vs. averaged load (as displayed in the task manager). I will check again but I am pretty sure that when I did the minimization tests, I specifically checked the impact on the load of a single core. But that was some time ago, let’s check again.
Anyway, we have to test the dll clearly.
-
@pyhg after dropping in bmsdedi … next session, keep an eye on TaskMgr memory tab – the overall “commit charge” value will tell you how much RAM the whole system is using.
-
Thanks ! Now we have many things to test and look for at the next session
-
@pyhg Hi Pyhg.
very interesting thread. Please keep us update on the test results.
Many thanks!
Gundam
-
Wait for U1…
We have stuff coming for dedicated servers
-
@MaxWaldorf OK MaxW! Great news.
Gundam
-
Ok thanks for the hint !
In the meantime we can already :
-
experiment with the bmsdedi.dll over some time
-
run missions hosted by one of the pilots’ PC. This is not ideal in our mind because it means relying on the availability of one person and the stability of his session (we tend to believe that hosting in a plane that stays on ramp is less prone to crashes than hosting by someone doing a lot of stuff in the air, playing with the TGP, etc…). But at least that can help us identifity whether or not our server is a factor in the weird stuff we see with the missiles. If we do not see this stuff when hosting on one of our PCs, that’s important info we need to gather.
-
-
@pyhg On your server, please run BMS with the
-logfps
flag. After every session, this will save aframerate.csv
file in youruser\logs
directory. You should use this FPS log to guide optimizations. If you can, please post that frame rate log along with further ACMIs from your multiplayer sessions.Ideally, server FPS should be well over 40 fps. If it gets below ~15 FPS, it seems like the AI stops working correctly. If it drops into the single digits, then you start to see aircraft doing backflips, AI-launched missiles no longer can hit their targets, etc.
As others have said, CPU load in Windows is probably not a good metric. The dll should help with FPS no matter what, even with the window as small as possible.
You can also try placing the following line in your .cfg file:
set g_sNonCompatibleChanges “all-PARALLEL_DRAW_OBJLIST-PARALLEL_DRAW_PLATFORM”
(Edit: gave you the wrong setting, correct one below, my apologies!)set g_sCpuPerfOptimizations "all-PARALLEL_DRAW_OBJLIST-PARALLEL_DRAW_PLATFORM"
to see if that makes an FPS improvement.
-
I am adding 2 more points for our case.
2 persons in our group have very low upload (less than 1Mbps).
And someone does not have the ports opened.
Do you think it can have an impact on the AIM-120 behavior? -
-
@Hebus missiles are linked to each client… SO yes…
-
Sounds like a smart way to correlate observed behaviours with ‘hard metrics’. Thanks a lot for this, we will implement this tracing option.