24/7 Falcon Online CO-OP server
-
There will be more specifics to say about this but please keep in mind that we made a trade-off decision in releasing 4.33.1 that specifically concerns MP games with scale of 10’s of players. The BMS team is aware of some specific scaling issues that can result in uneven gameplay and game code crash (CTD) events. Obviously this is less than ideal. However, we chose to release anyway knowing that a very significant portion of the community would get enjoyment and stable results out of 4.33.1 as it stands now, even if some of the larger-server-ops folks might experience frustration.
We’ve actually learned quite a bit and found some new avenues for exploration quite recently. Those investigations will continue and with luck will bear fruit in due course. We did not feel close enough to meaningful changes to address stability and scaling issues, as yet, so as to warrant holding up 4.33.1 from release. The things I’m referring too are inherent in 4.33 as well, btw, it’s not newly introduced issues per se.
We did make a couple of point fixes for issues (for example the g_bSkipAggregationBWCheck fix which we believe will help with ghost flights and the like) but more significant changes to address what we have learned will need to wait a while yet.
Now, please understand that I’m not going to try and get into more details or respond to questions here. Others more familiar with the issues will start separate thread(s) to share what we know and what we think we know and what we think are best-known-methods for improving your chances of stable MP experiences with larger population MP games.
For now, I just wanted to let you know that this is a concern for the team and to temper any frustration you might have coming if you are at some point bitten by these kinds of issues: in that regard, please know that we are aware and working on solutions.
Are you stating BMS 4.33 U1 is not suitable (ie. unstable) for Multi-Player sessions exceeding 10 players?
-
Note
The things I’m referring too are inherent in 4.33 as well, btw, it’s not newly introduced issues per se.
-
I’m a bit dissapointed to hear this because that very F-O campaigns with large number of players still keep me close to BMS and I believe I’m not the only one. There are lots of guys like me loving proper Force vs Force events.
But I hope You guys (BMS Devs & F-O devs) will tighten cooperation even closer than ever before to solve all the MP problem. F-O has also very talented folks and it’s a Great platform to hunt those bugs. Fingers crossed.
-
Are you stating BMS 4.33 U1 is not suitable (ie. unstable) for Multi-Player sessions exceeding 10 players?
Let me put it this way: we know of specific behaviors that players can engage in that, in scenarios with geographically spread out sessions that number in the 10-20 range, will likely result in at least blue-pause events and potentially CTD events. I’m not making any value statement about suitability. For smaller numbers of players at a time these behaviors are rock solid stable all day and twice on Sunday. Since, as has already been pointed out twice now, this isn’t new with 4.33.1 I can say this: 4.33.1 is no less suitable than 4.33 and probably more suitable owing to the tweaks around the edges of the larger issues such as the g_bSkipAggregationBWCheck. Does that help clarify??
-
For further clarification and empirical results of testing please see the very latest (updated post Update 1) BMS Manual p96 and the supporting annex with the actual results on p280.
-
Since, as has already been pointed out twice now, this isn’t new with 4.33.1 I can say this: 4.33.1 is no less suitable than 4.33 and probably more suitable owing to the tweaks around the edges of the larger issues such as the g_bSkipAggregationBWCheck. Does that help clarify??
Yes, it does. Thank you Boxer.
-
Let me put it this way: we know of specific behaviors that players can engage in that, in scenarios with geographically spread out sessions that number in the 10-20 range, will likely result in at least blue-pause events and potentially CTD events. I’m not making any value statement about suitability. For smaller numbers of players at a time these behaviors are rock solid stable all day and twice on Sunday. Since, as has already been pointed out twice now, this isn’t new with 4.33.1 I can say this: 4.33.1 is no less suitable than 4.33 and probably more suitable owing to the tweaks around the edges of the larger issues such as the g_bSkipAggregationBWCheck. Does that help clarify??
For further clarification and empirical results of testing please see the very latest (updated post Update 1) BMS Manual p96 and the supporting annex with the actual results on p280.
From what I can gather from your post (and after reading p.96 + annex) the issue is not specifically code related but a symptom of limited upload bandwidth available on the client and server’s end. A limit which in practice may cause the entire session to become unstable due to bandwidth saturation of either client or server (or both).
So if I understand correctly…
If we assume all clients have sufficient bandwidth ie 2048 or higher, and have this set as their connection speed while at the same time connected to a server which has enough bandwidth to adequately cope with the client load, then the caution you are stating is not relevant?
I say this because at FO we have an active solution (not publicly released) which measures each client’s available bandwidth and latency before providing access to the server.
As we can filter clients which do not meet the minimum requirements, this would shield the server from conditions where the code is vulnerable to timing issues and be “rock solid” as you stated.
Not sure what type of connection BMS are testing on but at FO we have a 1GBs pipe available so server bandwidth is not an issue. All that is required would be to filter clients.
-
Not sure what type of connection BMS are testing on but at FO we have a 1GBs pipe available so server bandwidth is not an issue. All that is required would be to filter clients.
Actually the server is running on a 2 gigabit pipe not 1, both of them. And they are hosted in a Data center on a gaming PC which is using a server grade LAN PCIe Card.
-
Actually the server is at 2 gigabit pipe not 1, both of them. And they are hosted in a Data center on a gaming PC which is using a server grade LAN PCIe Card.
-
+1 Khronik LOL!
-
Haha! Nice one Aphex!
I told You guys F-O has a good “platform”.
-
From what I can gather from your post (and after reading p.96 + annex) the issue is not specifically code related but a symptom of limited upload bandwidth available on the client and server’s end. A limit which in practice may cause the entire session to become unstable due to bandwidth saturation of either client or server (or both).
So if I understand correctly…
If we assume all clients have sufficient bandwidth ie 2048 or higher, and have this set as their connection speed while at the same time connected to a server which has enough bandwidth to adequately cope with the client load, then the caution you are stating is not relevant?
I say this because at FO we have an active solution (not publicly released) which measures each client’s available bandwidth and latency before providing access to the server.
As we can filter clients which do not meet the minimum requirements, this would shield the server from conditions where the code is vulnerable to timing issues and be “rock solid” as you stated.
Not sure what type of connection BMS are testing on but at FO we have a 1GBs pipe available so server bandwidth is not an issue. All that is required would be to filter clients.
Foreword : MP is really really complicated, so what I’m saying is to be taken as “our current understanding of the problem”, not “gospel truth”. especially since I cannot be described as an expert in MP at all. That said :
Server BW was never really a problem in our testings, it is more on the client side that stuff is a problem. But all around, BW management is far from optimal - and if you factor in the rather high BW requirements of Falcon itself, things get messy whenever you have high number of clients.
To paraphrase the manual, BW setting of clients are used in two places :
- locally, client use it to limit what they send,
- server also uses it to limit what they send to each client. Server scales what it send based on min(BW of all clients). Server hosts all AIs, so it does have quite a lot to send, and scaling it too much can lead to problems : gear status not correctly displayed, radar lock on IAs not working, etc. (this showed up in testing).
What is important to understand is that for a typical P2P client, the BW setting is subsequently subject to 2 contradictory requirements.
- first requirement : you need to scale what you send, depending on what your connection can upload. So according to this, putting a BW setting above your real UL rate is a bad thing. Especially with a lot of chaff/flares/munitions/etc.
- second requirement : you need to put a BW setting high enough that allows the server to send you enough information. If the server has a ton of AIs to deal with, you need to put a BW setting quite high (typically, 1024 for KTO.).
These two requirements cannot be conciliated if there are too many AIs in the server (2nd requirement forces BW to be high) AND too many clients to be dealt with (without scaling, you would overflow your pipe, so you need a BW setting below your UL rate).
“Too many” being highly dependent on client upload rate and how busy the situation is on the server side. This is why for small scale MP flights, stuff usually works just fine, but big events are problematic.As a result, you can try the following advices :
- COOP servers with a big AI environment need to limit the number of players, and tell people to put 1024. This way, you dont need much scaling for clients and 1st requirement is less severe. (Most VFW’s case here)
- pure PvP servers need to drastically lower the number of AI units, especially moving ones. So the strict minimum of bataillons, the strict minimum of SAMs, the strict minimum of AI flights. This way, 2nd requirement go away, and you tell people to put their real UL rate to satisfy 1st requirement.
Hopefully, the appendix on the manual will help organizers and squadron evaluate in which case they are and what kind of missions they need to be designing.
Now, for the questions that are bound to come :
-
surely, there is room for improvement ?
Well yes, obviously. The explanation above should make you raise your eyebrows quite a bit. But this stuff is NOT easy. This is quite close to the heart of the sim, and making changes here is not exactly trivial. So… wait and see. -
why is 4.33 different than 4.32 ?
That is an excellent question, to which at least I dont have a real answer. BW requirements were similar, BW management was similar. So I’m inclined to think big events were really at the edge of what 4.32 could take, and 4.33 strained this slightly over the limit. A bit sketchy, but that’s all I have.
-
@ l3crusader thanks for the detail post much appreciated. We at Falcon Online has been using the connection bandwidth 1024 for clients for the last 4 years or more. What we did for our events was that we had all the members do a speed test to a close location to the server itself, but in some ways it was limited because it was not to the same location. But now the Speed test.net finally hosted the speed test in the same data center so now we can get very accurate upload / download and ping on each client.
I think at FO we will have better results because we can maintain the quality of the client’s connection to the server itself. Upload speed higher then .50Mbps and Ping lower then 200 ms is the new standard.
As far as i have seen with the results 98% of the clients are above this. Here’s the thread if you wanna look at it: http://falcon-online.org/forum/index.php?topic=2222.0
I have my fingers crossed because Battle For Balkans event is gonna be huge and i am expecting 64 people to connect and fly on it. COOP server is online and i urge and invite all you guys to come and fly on it specially the BMS Dev guys and that means everyone. I know we had our differences with Bad Boy and some others but i think it’s time to bury the hatchet.
So again it is a open invitation to fly and work together towards a common goal.
-
- why is 4.33 different than 4.32 ?
That is an excellent question, to which at least I dont have a real answer. BW requirements were similar, BW management was similar. So I’m inclined to think big events were really at the edge of what 4.32 could take, and 4.33 strained this slightly over the limit. A bit sketchy, but that’s all I have.
RakNet was updated because Enet wasn’t as good as we were expected (if i remember well)
BB
- why is 4.33 different than 4.32 ?
-
Burn-In test with stock update 1 are planned on the F-O server coming sunday 12:00 Zulu. More info available on F-O for VFW’s and squad’s able to participate.
-
CO-OP server live stream starting in 5 mins here:
-
4.33 U1 Burn-In Test moved to 12 Zulu / UTC sunday !!!
-
Test starts in 45 minutes. If able to contribute, show up on Falcon-Online teamspeak, (ts3.falcon-online.org no pass), and we’ll get you sorted.
-
This post is deleted! -
Pfft, far too early for me on a Sunday morning here EST.