Is there a way to customize the way a binary writer writes out to files so that I could read the file from a C++ program?
Eg:
myBinaryWriter.Write(myInt);
myBinaryWriter.Write(myBool);
And in C++:
fread(&myInt, 1, sizeof(int), fileHandle);
fread(&myBool, 1, sizeof(bool), fileHandle);
EDIT: From what I can see, if the length of a string is small enough to fit into one byte then that's how it writes it, which is bad if I want to read it back in in C++.
If you want to guarantee binary compatibility, possibly the easiest approach from c# is to ditch binary writer and just use a stream to write the bytes yourself. That way you get full control of the output data
Another approach would be to create an assembly that can write the data using c++/cli, so you can get direct compatibility with c++ from managed code.
You have some options you can choose:
Writing it byte-byte by yourself. this is possibly the worst option. it requires more work in both sides (the serializing and the de-serializing sides).
Use some cross-platforms serializers like Protobuf. it has ports for almost any platform, including C# (protobuf-net) and C++. and it's also easy to use and has really good performance.
you can use Struct and convert it yo byte array using Marshal.PtrToStructure (if you need the other way, you can use Marshal.StructureToPtr). there are plenty of examples in the internet. If you can use Managed CPP, you can use the same object so you can change the struct in one place and it will change in both places.
if you use Managed CPP, you can use the built-in serializers, such as BinaryFormatter...
Related
I am recieving binary stream from an application I am running in Python.
From the binary stream, I want to create a C# object that is inside the stream in byte array.
How do I deserialise the object and retrieve the object from the binary stream?
We can ignore that it's a python application. I am more interested in how binary streaming works.
You seem to think that all languages automatically use the same serialization scheme.
This is not so.
It is not even theoretically possible, because different programming languages have different notions of what it means to be an object.
If you are specifically interested in how to read a Python serialized stream in C#, then ask that. Otherwise, this question is unanswerable because it is based on a false premise.
FOLLOW UP - Out of curiosity, I did some searching for a Python pickle reader in C#. Nothing in the first 3 pages of search results ... though there was a reference to a pickle reader in C++.
Just to add you a little general info:
In C#/.Net there's a general approach to serialize objects to NOT a binary form, because a binary form needs a lot of protocol-like headers to - note - include the metadata, and this causes the receiver to have to know the .Net/CLR inner structure very well.
Instead, today, the objects are usually serialized to XML (when type information is crucial) or JSON formats (when only data matters), so that any receiver may read them quite easily, and more often - any 3rd party may easily generate new object-like data that our application may "just deserialize", regardless of who generated it and on what platform.
However, binary serialization is still used. XML/JSON data, even if compressed, is still usually larger than the binary image. However, the binary serialization is strictly used when we want the data to not be published to the outside world, or if we somehow magically know that it will be only processed on .Net with use of our assemblies.
C# object
C# does not have objects; it's a .Net object.
Secondly we absolutely CANNOT ignore that it's a Python application, because that implies that it's likely it's not running on .Net and therefore the .Net binary format is not native to your Python runtime. That's not to say that it's not possible for the .Net serialization to be available to you in this case, because if you're running IronPython - the .Net python implementation - then you can simply use the Binary serialization APIs from within that and get the .Net object that was serialized.
If, however, it's Python running on a different platform, then you can decode the information in the binary stream, for that you need to know the format, and for that go straight to the horse's mouth and read through the Binary Format Data Structure spec from MSDN.
This will, of course, require (quite a lot) more work!
If the project you're working on allows you to change the way that the original object is serialized, then I strongly suggest changing over to XML serialization or something similar - that is designed to be portable.
I'm trying to write a simple reader for AutoCAD's DWG files in .NET. I don't actually need to access all data in the file so the complexity that would otherwise be involved in writing a reader/writer for the whole file format is not an issue.
I've managed to read in the basics, such as the version, all the header data, the section locator records, but am having problems with reading the actual sections.
The problem seems to stem from the fact that the format uses a custom method of storing some data types. I'm going by the specs here:
http://www.opendesign.com/files/guestdownloads/OpenDesign_Specification_for_.dwg_files.pdf
Specifically, the types that depend on reading in of individual bits are the types I'm struggling to read. A large part of the problem seems to be that C#'s BinaryReader only lets you read in whole bytes at a time, when in fact I believe I need the ability to read in individual bits and not simply 8 bits or a multiple of at a time.
It could be that I'm misunderstanding the spec and how to interpret it, but if anyone could clarify how I might go about reading in individual bits from a stream, or even how to read in some of the variables types in the above spec that require more complex manipulation of bits than simply reading in full bytes then that'd be excellent.
I do realise there are commercial libraries out there for this, but the price is simply too high on all of them to be justifiable for the task at hand.
Any help much appreciated.
You can always use BitArray class to do bit wise manipulation. So you read bytes from file and load them into BitArray and then access individual bits.
For the price of any of those libraries you definitely cannot develop something stable yourself. How much time did you spend so far?
i am writing a project in C#
i wanna save a class in binary file and then read that file it in C
i wanna know how can i do it without serialize and deserialize
please help me
You are talking about cross-platform serialization.
A few options:
serialize it as text (xml, json); text is still binary, after all - and simple
serialize it manually
use a third party cross-platform serializer
But whatever you do, don't use BinaryFormatter. The reason I stress this is that it is probably the first thing you'll see if you search for C# binary serialization, but is entirely inappropriate for your purposes. The format is proprietary, and includes type information that only makes sense from .NET (not really from unmanaged C).
I'm quite attached to "protocol buffers" as a serialization API, and there are both C# and C versions here.
Saving the state of an object to a file means serializing it.
Reading the state of an object from a file means deserializing it.
You have to use serialization/deserialization to do what you want.
Since you need to do this across different languages, using the built in serializers would probably not be very helpful.
You can use one of the XML serializers for the C# part, but then would have to parse the XML out in c.
Another option is to write your own custom serizlizer to do this. This way you have full control over the file format.
Do you want to save a class? This is not possible since classes are compiled into assemblies (exe,dll) in .net.
I think what you want is to save the state of an object or better suited, a struct to a file.
You can write all fields of the class to a file using the BinaryWriter class. Also you can have a look at this.
I presume you mean you want to have a C# application write a file. Then have a separate C/C++ application read that file? On that assumption, in C# you'll need to look into the System.IO namespace, and specifically the FileStream class.
On a side note, I'd really recommend writing a C# Class Library project that handles this read/write via .NET serialization classes and then invoke it nativly from your C# code, and use COM ([assembly: ComVisible(true)]) to access your .NET code from your C/C++ code.
I use the excellent FileHelpers library when I work with text data. It allows me to very easily dump text fields from a file or in-memory string into a class that represents the data.
In working with a big endian microcontroller-based system I need to read a serial data stream. In order to save space on the very limited microcontroller platform I need to write raw binary data which contains field of various multi-byte types (essentially just dumping a struct variable out the serial port).
I like the architecture of FileHelpers. I create a class that represents the data and tag it with attributes that tell the engine how to put data into the class. I can feed the engine a string representing a single record and get an deserialized representation of the data. However, this is different from object serialization in that the raw data is not delimited in any way, it's a simple binary fixed record format.
FileHelpers is probably not suitable for reading such binary data as it cannot handle the nulls that show up and* I suspect that there might be unicode issues (the engine takes input as a string, so I have to read bytes from the serial port and translate them into a unicode string before they go to my data converter classes). As an experiment I have set it up to read the binary stream and as long as I'm careful to not send nulls it works quite well so far. It is easy to set up new converters that read the raw data and account for endian foratting issues and such. It currently fails on nulls and cannot process multiple records (it expect a CRLF between records).
What I want to know is if anyone knows of an open-source library that works similarly to FileHelpers but that is designed to handle binary data.
I'm considering deriving something from FileHelpers to handle this task, but it seems like there ought to be something already available to do this.
*It turns out that it does not complain about nulls in the input stream. I had an unrelated bug in my test program that came up where I expected a problem with the nulls. Should have investigated a little deeper first!
I haven't used filehelpers, so I can't do a direct comparison; however, if you have an object-model that represents your objects, you could try protobuf-net; it is a binary serialization engine for .NET using Google's compact "protocol buffers" wire format. Much more efficient than things like xml, but without the need to write all your own serialization code.
Note that "protocol buffers" does include some very terse markers between fields (typically one byte); this adds a little padding, but greatly improves version tolerance. For "packed" data (i.e. blocks of ints, say, from an array) this can be omitted if desired.
So: if you just want a compact output, it might be good. If you need a specific output, probably less so.
Disclosure: I'm the author, so I'm biased; but it is free.
When I am fiddling with GPS data in the SIRFstarIII binary mode, I use the Python interactive prompt with the serial module to fetch the stream from the USB/serial port and the struct module to convert the bytes as needed (per some format defined by SIRF). Using the interactive prompt is very flexible because I can read the string to a variable, process it, view the results and try again if needed. After the prototyping stage is finished, I have the data format strings that I need to put into the final program.
Your question doesn't mention anything about why you have a C# tag. I understand FileHelpers is a C# library, but I that doesn't tell me what environment you are working in. There is an implementation of Python for .NET called IronPython.
I realize this answer might mean you have to learn a new language, but having an interactive prompt is a very powerful tool for any programmer.
I can use write(&stName,sizeof(stName),&FileName) and define a same struct in other program to read the file(XXX.h) when i use C, But I want do the same use C# and I should not use the unsafe mode. How do to solve the problem?
Edit:
thanks all. I will to try them
Edit:
Now if I want to use C write the Struct to file.h and use C# to read the struct from file.h, may I have chance solve that and not to count the offset? Because count the offset is not a good answer when I want to add some variable or other struct in the struct.
Look at the ISerializable interface and Serialization in general.
Even in C, this is a dangerous thing to do IMO. If you use a different compiler, operating system, architecture etc you can very easily "break" your data files - you're absolutely relying on the layout of the data in memory. It's a bit like exposing fields directly instead of properties - the in-memory layout should be an implementation detail which can be changed without the public form (the file) changing.
There are lots of ways of storing data, of course. For example:
Binary serialization (still pretty fragile, IMO)
XML serialization
Google's Protocol Buffers
Thrift
YAML
Hand-written serialization/deserialization e.g. using BinaryReader and BinaryWriter
There are balances in terms of whether the file is human readable, speed, size etc. See this question for answers to a similar question about Java.
You should take a look at the NetDataContractSerializer. You can markup those portions of the struct that you wish to serializer and use a file stream to write them out.
Look at the StructLayoutAttribute
Use Managed C++ or C++/CLI. It can read your .h file struct. It can read and write using:
read(in, &structure, sizeof(structure));
write(out, &structure, sizeof(structure));
and it can transfer that data very simply to anything else in .NET.
You'll have to convert each member of the struct individually using Bitconverter.convert(). This works well when your struct holds numeric data types, but you might have to do something more complex when using structs that contain more complicated data types like strings, arrays, and collections. For purposes like this, you will want to check out the .Net serialization facilities that other have mentioned.
You can look into Google Protocol buffers as well. You may not want to add another dependency into your code, but it's meant to allow you to do this sort of thing.