Falcon SharedMemory via Python - FalconSharedMemoryAreaString
-
I would like to access the Falcon BMS shared memory via Python. After searching online, I have found a way to access some stuff. What I’m still struggling with is the FalconSharedMemoryAreaString. I want to get the various things it exposes, like the base directory, keyfile in use, etc. Does anyone have any pointers how to accomplish this?
Here’s how far I’ve gotten with the FalconSharedMemoryAreaString:
import mmap import ctypes import ctypes.wintypes class StringData(ctypes.Structure): _fields_ = ( ("VersionNum", ctypes.wintypes.UINT), ("NoOfStrings", ctypes.wintypes.UINT), ("dataSize", ctypes.wintypes.UINT), ("data", ctypes.wintypes.CHAR) ) class StringStruct(ctypes.Structure): _fields_ = ( ("strId", ctypes.wintypes.UINT), ("strLength", ctypes.wintypes.UINT), ("strData", ctypes.wintypes.CHAR) ) buff = mmap.mmap(-1, ctypes.sizeof(StringData), "FalconSharedMemoryAreaString") data = StringData.from_buffer(buff) print(data.data)
I have no idea how to reconstruct the stuff in data.data (no C++ knowledge and the explanation in the header file is way over my head).
-
@justonce01 Not sure I’m good enough with Python to help, directly, but I can maybe help interpret the C/C++ code in FlightData.h for you, if you have specific questions.
Generally speaking … the background here is they exposed
std::string
andstd::vector
in a binary-interface boundary, which where I come from is very horrible, bad and wrong.Those types are C++ constructs which may vary from compiler to compiler … different std library vendors, or from version to version of the same compiler. So, ie. it might not even be possible for other C++ programmers to (easily) parse that stuff out of shared mem, using FlightData.h header file … let alone C# or Python or whatever.
Thankfully, they included a little example helper function (lines 883-919), to unpack those gooey C++ data structures.
That helper function is written in old-school C… which has a much more dependable, unambiguous binary layout and behavior. Here’s a quick decoder ring to get you started.
‘size_t’ => just pretend it says “integer”
‘const’ => meaningless for a binary interface boundary; so just ignore that
‘char*’ => this is a pointer to a block of memory – the memory address where something resides … it may be the start of an array of characters … but it could literally be anything … ‘char’ is 1-byte so (char*) is often used as a convention to mean “pointer to something without any assumptions about its size, or alignment, or type or structure”memcpy((char*)&B, (char*)&A, L);
you have to kinda read it backwards … it means “copy L bytes of memory from A to B” … (the ‘&’ operator is the “pointer-to-thing” operator, btw)
So, for example on line 891
memcpy((char*)&result.VersionNum, (char*)inputbuffer + offset, sizeof(result.VersionNum));
translation => “copy 4 bytes from memory at ‘inputbuffer+offset’ into the memory location of ‘result.VersionNum’ … pretending the data is an array of 1-byte characters even though it’s obviously not, but this will avoid any possibility of alignment-mismatch on different cpu architectures” lol can you believe this is how we used to write code, for many decades?
-
@justonce01 said in Falcon SharedMemory via Python - FalconSharedMemoryAreaString:
I would like to access the Falcon BMS shared memory via Python. After searching online, I have found a way to access some stuff. What I’m still struggling with is the FalconSharedMemoryAreaString. I want to get the various things it exposes, like the base directory, keyfile in use, etc. Does anyone have any pointers how to accomplish this?
Here’s how far I’ve gotten with the FalconSharedMemoryAreaString:
import mmap import ctypes import ctypes.wintypes class StringData(ctypes.Structure): _fields_ = ( ("VersionNum", ctypes.wintypes.UINT), ("NoOfStrings", ctypes.wintypes.UINT), ("dataSize", ctypes.wintypes.UINT), ("data", ctypes.wintypes.CHAR) ) class StringStruct(ctypes.Structure): _fields_ = ( ("strId", ctypes.wintypes.UINT), ("strLength", ctypes.wintypes.UINT), ("strData", ctypes.wintypes.CHAR) ) buff = mmap.mmap(-1, ctypes.sizeof(StringData), "FalconSharedMemoryAreaString") data = StringData.from_buffer(buff) print(data.data)
I have no idea how to reconstruct the stuff in data.data (no C++ knowledge and the explanation in the header file is way over my head).
It looks like you’re on the right track with using the mmap and ctypes libraries to access the FalconSharedMemoryAreaString. The StringData and StringStruct classes you’ve defined match the structure of the shared memory area as defined in the header file, so you’re able to read the raw data into a buffer.
To extract the specific pieces of information you’re interested in (base directory, keyfile in use, etc.), you will need to parse the data field of the StringData structure. The data field is a raw byte buffer that contains a series of StringStruct structures, one for each string in the shared memory area. To extract the information from the data field, you will need to iterate through the buffer, interpreting each byte as a StringStruct and reading the strId, strLength, and strData fields.
Here’s an example of how you could extract the base directory from the data field:
# Constants for string IDs BASE_DIRECTORY_ID = 1 KEYFILE_ID = 2 # Iterate through the buffer, interpreting each byte as a StringStruct offset = 0 for i in range(data.NoOfStrings): string = StringStruct.from_buffer(data.data, offset) offset += ctypes.sizeof(StringStruct) # Check the strId field to determine what information the string contains if string.strId == BASE_DIRECTORY_ID: base_directory = string.strData[:string.strLength].decode() print(base_directory) elif string.strId == KEYFILE_ID: keyfile = string.strData[:string.strLength].decode() print(keyfile)
Note that the example above uses constants to define the string IDs that correspond to the base directory and keyfile in use. You will need to determine the correct string IDs to use based on the information in the header file. Additionally, the example uses the .decode() method to convert the raw bytes in the strData field to a string.
Seems like decent advice … truth be told, it’s not mine. I just copied your question into ChatGPT Everything above this line was written by an ai.
Wanted to share that because ChatGPT just helped me this week to make use of a BMS SharedMem .dll provided by JanJan in my WPF App in C#.
You can literally go back and forth with ChatGPT on things like this, if an error is thrown, copy it to ChatGPT and it will adjust any previous examples so they do work. Not all code examples are correct, sometimes they include some pseudo and sometimes its obvious like an example missing a type cast when declaring a variable that needs one (instead of just “var”).
Still, it’s an invaluable tool, a 24-7 assistant that knows way more about all programming languages that we do, and can bounce ideas off it, too. “Here is my main loop, can I do anything to improve it?”
…limit is 2000 characters per question, so can’t copy a whole big page of code, but if it’s small enough (like yours above) you can. Can also share blocks of code, because conversation recalls previous answers/questions (on a per-chat session basis). You can create a new chat anytime, and open old chats and continue where you left off.
That’s my testimonial - ChatGPT helped me streamline the creation of a WPF App in a week that would have taken me a month of researching and testing each concept including the Shared Mem reader we created together.
-
@SemlerPDX ChatGPT knows about Falcon BMS shared memory? Wild. I guess there’s enough on github for it…
-
@airtex2019 It knows “of” it, but doesn’t have all the information contained within the header files… but could copy them (in parts) to a chat, and then ask ChatGPT how to make use of them.
-
-
@SemlerPDX
Wow, I didn’t expect that, thanks.I have tried various things, but it’s still not working. Using your example, I am getting a ValueError:
Traceback (most recent call last): File "C:\Users\Me\Downloads\a.py", line 24, in <module> string = StringStruct.from_buffer_copy(data.data, offset) ValueError: Buffer size too small (1 instead of at least 16 bytes)
I think the problem is in the data itself. It seems to contain only b’\x00’. I think the issue is related to the data type.
ctypes.wintypes.CHAR
seems to be the wrong choice. According to the comment in the header it says:If you parse the data yourself, it should be handled as char[dataSize] by external apps
Unfortunately, ChatGPT didn’t provide much useful help with that.
Does anyone know what else I could try? -
import mmap import ctypes import ctypes.wintypes class StringStruct(ctypes.Structure): _fields_ = ( ("strId", ctypes.wintypes.UINT), ("strLength", ctypes.wintypes.UINT), ("strData", ctypes.c_char_p) ) class StringData(ctypes.Structure): _fields_ = ( ("VersionNum", ctypes.wintypes.UINT), ("NoOfStrings", ctypes.wintypes.UINT), ("dataSize", ctypes.wintypes.UINT), ("data", ctypes.POINTER(StringStruct)) ) buff = mmap.mmap(-1, ctypes.sizeof(StringData), "FalconSharedMemoryAreaString") data = StringData.from_buffer(buff) string_struct_array = ctypes.cast(data.data, ctypes.POINTER(StringStruct * data.NoOfStrings)) for i in range(data.NoOfStrings): print(ctypes.string_at(string_struct_array.contents[i].strData, string_struct_array.contents[i].strLength))
This is what I have now, but
string_struct_array[0].strId
orstring_struct_array[0].strData
isn’t outputting anything. What did I do wrong? -
@justonce01 AFAIR you need to read in the entire struct, since it is one consecutive piece of memory. I can’t really tell from this if you do that? Then you can read the individual elements, aka memory segments you need and cast to particular type.
Also, are you in 3D in BMS while reading ?
-
you need to read in the entire struct, since it is one consecutive piece of memory.
I think this is that part? data.data should be holding the struct, right? Sorry if these are stupid questions. I only know what seems to be the basics of Python (the books never mentioned anything about this “type” of programming).
string_struct_array = ctypes.cast(data.data, ctypes.POINTER(StringStruct * data.NoOfStrings))
Also, are you in 3D in BMS while reading ?
Yes. I checked with the bundled shared memory reader and it’s displaying everything correctly.
-
@justonce01 said in Falcon SharedMemory via Python - FalconSharedMemoryAreaString:
you need to read in the entire struct, since it is one consecutive piece of memory.
I think this is that part? data.data should be holding the struct, right?
As far as I can tell, you allocate memory for only 4 fields - from a quick glance at the flightdata.h it has a lot more. But maybe I don’t quite understand the Python here
-
@justonce01 Did you see my earlier reply? Sorry if not helpful.
I was trying to say, I don’t think you’re going to be able to parse this blob with declarative interop – you’re going to have to write procedural code to walk that packed sequence of length-prefixed-but-also-null-terminated-char[] strings.
The comment on line 879 is the salient point.
// Note that this can NOT be treated as StringStruct[NoOfStrings] by external apps, due to the flexible size of StringStruct!
[Edit: I see https://docs.python.org/3/library/ctypes.html#variable-sized-data-types has a little bit to say on how to handle variable-sized types. But not sure it’s sufficient in this case.]
-
@jayb
It’s just the FalconSharedMemoryAreaString part of it. I’m not sure, but I don’t think that all the other areas have to be mapped as well every time (at least judging from the examples I have seen online).@airtex2019
Yes, but I couldn’t make much out of it. I will consider that and see if it will bring me closer to solving this. Thanks. -
@justonce01 again I don’t know Python well enough to say if something is impossible … but even C# which has the most comprehensive C/C++ interop support I know of… I don’t think it’s possible to handle a dynamically-sized struct composed of array of dynamically-sized structs, containing embedded null-term strings.
The C# code from lightningstools had to resort to a for-loop to unpack this section.
-
@justonce01 have you seen this? It may do everything you need?
https://github.com/nmeier/simscript/blob/master/modules/falcon.py
-
@S3NTRY
Yes, that one has been pretty helpful, I basically copied everything from there. But unfortunately that one is lacking the FalconSharedMemoryAreaString part as well. There’s an open issue regarding that, perhaps someone with more knowledge might eventually chime in and figure it out.