I'm working on c# windows service that handles firebird database requests. My problem occurs at random moments (sometimes after 5 minutes, sometimes after just 4 calls to database), when I try to deserialize object on client application. It happens though only at specific position (stops at 18th byte in 54 byte array). Rest of the time the function returns a proper result.
I'm using this function to serialize single object
public byte[] ObjectToByteArray(Object obj)
{
if (obj == null)
return null;
MemoryStream fs = new MemoryStream();
BinaryFormatter formatter = new BinaryFormatter();
formatter.Serialize(fs, obj);
fs.Seek(0, SeekOrigin.Begin);
byte[] rval = fs.ToArray();
fs.Close();
return rval;
}
I am not serializing any custom classes, only strings and numeric types (firebird api returns them as objects though).
I use this to deserialize:
public object ByteArrayToObject(Byte[] Buffer)
{
BinaryFormatter formatter = new BinaryFormatter();
MemoryStream stream = new MemoryStream(Buffer);
stream.Position = 0;
object rval = formatter.Deserialize(stream); <--- this thing drives me nuts.
stream.Close();
return rval;
}
and main fnct in client aplication. Sorry for ugly code,
public List<object[]> ByteToList(byte[] data, int[] pomocnicza)
{
//pomocnicza table contains size of (original) particular column of list in bytes
int size_row = 0;
foreach (int i in pomocnicza)
{ size_row += i; }
List<object[]> result = new List<object[]>();
int iterator = 0;
for (int i = 0; i < data.Length / size_row ; i++)
{
object[] zxc = new object[3];
int l = pomocnicza.Length/4;
for (int j = 0; j < l; j++)
{
byte[] tmp = new byte[pomocnicza[j*4]];
System.Array.Copy(data, iterator, tmp, 0, pomocnicza[j*4]);
object ffs = ByteArrayToObject(tmp);
zxc[j] = ffs;
iterator += pomocnicza[j*4];
}
result.Add(zxc);
}
return result;
}
What is baffling me is that it works in most cases, but inevitably causes to throw an error. Thing that it happens on random makes pinpointing it harder. Please help.
#EDIT
This is how I read the input:
public List<object[]> RetrieveSelectData(FbConnection dbConn, string SQLCommand)
{
using (var command = dbConn.CreateCommand())
{
command.CommandText = SQLCommand;
using (var reader = command.ExecuteReader())
{
var rows = new List<object[]>();
while (reader.Read())
{
var columns = new object[reader.FieldCount];
reader.GetValues(columns);
rows.Add(columns);
}
return rows;
}
}
}
and then serialize with this function
public byte[] ListToByte(List<object[]> lista, out int[] rozmiary)
{
int size= 0;
rozmiary = new int[lista[0].Length];
for (int i = 0; i < lista[0].Length; i++)
{
byte[] test = this.ObjectToByteArray(lista[0][i]);
size+= test.Length;
rozmiary[i] = test.Length;
}
size*= lista.Count;
byte[] result = new byte[size];
int index = 0;
for (int i = 0; i < lista.Count; i++)
{
for (int j = 0; j < lista[i].Length; j++)
{
byte[] tmp = this.ObjectToByteArray(lista[i][j]);
tmp.CopyTo(result, index);
index += tmp.Length;
}
}
return result;
}
If you are using above deserializing methods & also call them while getting stream from clientstream OR other streams.... skip it. try to use directly those streams with formatter. Like Below :
NetworkStream clientStream = client.GetStream();
Object src = (Object)formatter.Deserialize(clientStream);
I have found the bug. The code above works fine, but care for encoding in some cases(!), so feel free to use it.
The problem laying in another part of a program, where I mistyped and send 4 bytes BUT the client app was told to receive 8, so in most cases it filled it in with zeros, but sometimes it got it from next pack of data.
It was #Marc Gravell and his blog that made me look over and over again to eventually find the source.
Related
I have an object array containing array of a different type that is not known at compile time, but turns out to be int[], double[] etc.
I want to save these arrays to disk and I don't really need to process their contents online, so I looking for a way to cast the object[] to a byte[] that I then can write to disk.
How can I achieve this?
You may use binary serialization and deserialization for Serializable types.
using System.Runtime.Serialization.Formatters.Binary;
BinaryFormatter binary = BinaryFormatter();
using (FileStream fs = File.Create(file))
{
bs.Serialize(fs, objectArray);
}
Edit: If all these elements of an array are simple types then use BitConverter.
object[] arr = { 10.20, 1, 1.2f, 1.4, 10L, 12 };
using (MemoryStream ms = new MemoryStream())
{
foreach (dynamic t in arr)
{
byte[] bytes = BitConverter.GetBytes(t);
ms.Write(bytes, 0, bytes.Length);
}
}
You could do it the old fashioned way.
static void Main()
{
object [] arrayToConvert = new object[] {1.0,10.0,3.0,4.0, 1.0, 12313.2342};
if (arrayToConvert.Length > 0) {
byte [] dataAsBytes;
unsafe {
if (arrayToConvert[0] is int) {
dataAsBytes = new byte[sizeof(int) * arrayToConvert.Length];
fixed (byte * dataP = &dataAsBytes[0])
// CLR Arrays are always aligned
for(int i = 0; i < arrayToConvert.Length; ++i)
*((int*)dataP + i) = (int)arrayToConvert[i];
} else if (arrayToConvert[0] is double) {
dataAsBytes = new byte[sizeof(double) * arrayToConvert.Length];
fixed (byte * dataP = &dataAsBytes[0]) {
// CLR Arrays are always aligned
for(int i = 0; i < arrayToConvert.Length; ++i) {
double current = (double)arrayToConvert[i];
*((long*)dataP + i) = *(long*)¤t;
}
}
} else {
throw new ArgumentException("Wrong array type.");
}
}
Console.WriteLine(dataAsBytes);
}
}
However, I would recommend that you revisit your design. You should probably be using generics, rather than object arrays.
From here:
List<object> list = ...
byte[] obj = (byte[])list.ToArray(typeof(byte));
or if your list is complex type:
list.CopyTo(obj);
I have a class in a dll which parses a file and returns a Stream which represents an FAT image (or any other)
My problem is when there is any other image the class creates about 3702 (on average) null bytes at the beginning of the stream.
So I have to edit the stream first and then save it to a file.
I have a code already but it works slow.
[Note : fts is the returned FileStream.]
BufferedStream bfs = new BufferedStream(fts);
BinaryReader bbr = new BinaryReader(bfs);
byte[] all_bytes = bbr.ReadBytes((int)fts.Length);
List<byte> nls = new List<byte>();
int index = 0;
foreach (byte bbrs in all_bytes)
{
if (bbrs == 0x00)
{
index++;
nls.Add(bbrs);
}
else
{
break;
}
}
byte[] nulls = new byte[nls.Count];
nulls = nls.ToArray();
//File.WriteAllBytes(outputDir + "Nulls.bin", nulls);
long siz = fts.Length - index;
byte[] file = new byte[siz];
bbr.BaseStream.Position = index;
file = bbr.ReadBytes((int)siz);
bbr.Close();
bfs.Close();
fts.Close();
bfs = null;
fts = null;
fts = new FileStream(outputDir + "Image.bin", FileMode.Create, FileAccess.Write);
bfs = new BufferedStream(fts);
bfs.Write(file, 0, (int)siz);
bfs.Close();
fts.Close();
Now, my question is :
How can I remove the nulls more efficiently and faster than the above code?
Instead of pushing bytes onto a List you could simply loop through your stream until you find the first non-null byte and then just copy the array from there using Array.Copy.
I would think about something like this (untested code):
int index = 0;
int currByte = 0;
while ((currByte = bbrs.ReadByte()) == 0x00)
{
index++;
}
// now currByte and everything to the end of the stream are the bytes you want.
I am try to do some code using BinaryWriter and Then BinaryReader.
When I wanna write I use method Write().
But the problem is that between two lines of Write method there appears a new byte which is in ASCII table in decimal 31 (sometines 24).
You can see it on this image:
You can see that byte at index 4 (5th byte) is of ASCII decimal value 31. I didnt insert it there. As you can see 1st 4 bytes are reserved for a number (Int32), next are other data (some text mostly - this is not important now).
As you can see from the code i write:
- into 1st line a number 10
- into 2nd line text "This is some text..."
How come came that 5th byte (dec 31) in between??
And this is the code I have:
static void Main(string[] args)
{
//
//// SEND - RECEIVE:
//
SendingData();
Console.ReadLine();
}
private static void SendingData()
{
int[] commandNumbers = { 1, 5, 10 }; //10 is for the users (when they send some text)!
for (int i = 0; i < commandNumbers.Length; i++)
{
//convert to byte[]
byte[] allBytes;
using (MemoryStream ms = new MemoryStream())
{
using (BinaryWriter bw = new BinaryWriter(ms))
{
bw.Write(commandNumbers[i]); //allocates 1st 4 bytes - FOR MAIN COMMANDS!
if (commandNumbers[i] == 10)
bw.Write("This is some text at command " + commandNumbers[i]); //HERE ON THIS LINE IS MY QUESTION!!!
}
allBytes = ms.ToArray();
}
//convert back:
int valueA = 0;
StringBuilder sb = new StringBuilder();
foreach (var b in GetData(allBytes).Select((a, b) => new { Value = a, Index = b }))
{
if (b.Index == 0) //1st num
valueA = BitConverter.ToInt32(b.Value, 0);
else //other text
{
foreach (byte _byte in b.Value)
sb.Append(Convert.ToChar(_byte));
}
}
if (sb.ToString().Length == 0)
sb.Append("ONLY COMMAND");
Console.WriteLine("Command = {0} and Text is \"{1}\".", valueA, sb.ToString());
}
}
private static IEnumerable<byte[]> GetData(byte[] data)
{
using (MemoryStream ms = new MemoryStream(data))
{
using (BinaryReader br = new BinaryReader(ms))
{
int j = 0;
byte[] buffer = new byte[4];
for (int i = 0; i < data.Length; i++)
{
buffer[j++] = data[i];
if (i == 3) //SENDING COMMAND DATA
{
yield return buffer;
buffer = new byte[1];
j = 0;
}
else if (i > 3) //SENDING TEXT
{
yield return buffer;
j = 0;
}
}
}
}
}
If you look at the documentation for Write(string), you'll see that it writes a length-prefixed string. So the 31 is the number of characters in your string -- perfectly normal.
You should probably be using Encoding.GetBytes and then write the bytes instead of writing a string
for example
bw.Write(
Encoding.UTF8.GetBytes("This is some text at command " + commandNumbers[i])
);
When a string is written to a binary stream, the first thing it does is write the length of the string. The string "This is some text at command 10" has 31 characters, which is the value you're seeing.
You should check the documentation of methods you use before asking questions about them:
A length-prefixed string represents the string length by prefixing to
the string a single byte or word that contains the length of that
string. This method first writes the length of the string as a UTF-7
encoded unsigned integer, and then writes that many characters to the
stream by using the BinaryWriter instance's current encoding.
;-)
(Though in fact it is an LEB128 and not UTF-7, according to Wikipedia).
The reason this byte is there because you're adding a variable amount of information, so the length is needed. If you were to add two strings, where would you know where the first ended and the second began?
If you really don't want or need that length byte, you can always convert the string to a byte array and use that.
Ok, here is my edited code. I removed BinaryWriter (while BinaryReader is still there!!), and now it works very well - no more extra bytes.
What do you thing? Is there anytihng to do better, to make it run faster?
Expecially Im interesting for that foreach loop, which read from another method that is yield return type!!
New Code:
static void Main(string[] args)
{
//
//// SEND - RECEIVE:
//
SendingData();
Console.ReadLine();
}
private static void SendingData()
{
int[] commands = { 1, 2, 3 };
// 1 - user text
// 2 - new game
// 3 - join game
// ...
for (int i = 0; i < commands.Length; i++)
{
//convert to byte[]
byte[] allBytes;
using (MemoryStream ms = new MemoryStream())
{
// 1.st - write a command:
ms.Write(BitConverter.GetBytes(commands[i]), 0, 4);
// 2nd - write a text:
if (commands[i] == 1)
{
//some example text (like that user sends it):
string myText = "This is some text at command " + commands[i];
byte[] myBytes = Encoding.UTF8.GetBytes(myText);
ms.Write(myBytes, 0, myBytes.Length);
}
allBytes = ms.ToArray();
}
//convert back:
int valueA = 0;
StringBuilder sb = new StringBuilder();
foreach (var b in ReadingData(allBytes).Select((a, b) => new { Value = a, Index = b }))
{
if (b.Index == 0)
{
valueA = BitConverter.ToInt32(b.Value, 0);
}
else
{
sb.Append(Convert.ToChar(b.Value[0]));
}
}
if (sb.ToString().Length == 0)
sb.Append("ONLY COMMAND");
Console.WriteLine("Command = {0} and Text is \"{1}\".", valueA, sb.ToString());
}
}
private static IEnumerable<byte[]> ReadingData(byte[] data)
{
using (MemoryStream ms = new MemoryStream(data))
{
using (BinaryReader br = new BinaryReader(ms))
{
int j = 0;
byte[] buffer = new byte[4];
for (int i = 0; i < data.Length; i++)
{
buffer[j++] = data[i];
if (i == 3) //SENDING COMMAND DATA
{
yield return buffer;
buffer = new byte[1];
j = 0;
}
else if (i > 3) //SENDING TEXT
{
yield return buffer;
j = 0;
}
}
}
}
}
OK so I have converted a file into a binary format using BinaryWriter. The format is:
number of ints, followed by the ints.
So the code will be something like:
readLineOfNumbers() {
count = read();
int[] a = read(count ints);
return a;
}
Do I use a BinaryReader? The closest thing I can see there is to read everything into a byte[], but then how do I make that an int array? This all has to be done very efficiently as well. I need buffering and so on.
If you use BinaryWriter to create file it makes sense to read it using BinaryReader
Something like:
private static int[] ReadArray(BinaryReader reader)
{
int count = reader.ReadInt32();
int[] data = new int[count];
for (int i = 0; i < count; i++)
{
data[i] = reader.ReadInt32();
}
return data;
}
I don't know of anything within BinaryReader which will read an array of integers, I'm afraid. If you read into a byte array you could then use Buffer.BlockCopy to copy those bytes into an int[], which is probably the fastest form of conversion - although it relies on the endianness of your processor being appropriate for your data.
Have you tried just looping round, calling BinaryReader.ReadInt32() as many times as you need to, and letting the file system do the buffering? You could always add a BufferedStream with a large buffer into the mix if you thought that would help.
int[] original = { 1, 2, 3, 4 }, copy;
byte[] bytes;
using (var ms = new MemoryStream())
{
using (var writer = new BinaryWriter(ms))
{
writer.Write(original.Length);
for (int i = 0; i < original.Length; i++)
writer.Write(original[i]);
}
bytes = ms.ToArray();
}
using (var ms = new MemoryStream(bytes))
using (var reader = new BinaryReader(ms))
{
int len = reader.ReadInt32();
copy = new int[len];
for (int i = 0; i < len; i++)
{
copy[i] = reader.ReadInt32();
}
}
Although personally I'd just read from the stream w/o BinaryReader.
Actually, strictly speaking, if it was me I would use my own serializer, and just:
[ProtoContract]
public class Foo {
[ProtoMember(1, Options = MemberSerializationOptions.Packed)]
public int[] Bar {get;set;}
}
since this will have known endianness, handle buffering, and will use variable-length encoding to help reduce bloat if most of the numbers aren't enormous.
I have a long array. How to write this array to a binary file?
Problem is that if I convert it into byte array some values are changed.
The array is like:
long array = new long[160000];
Give some code snippet.
The BinaryFormatter will be the easiest.
Also valuetypes (I assume this is what you mean by long), serializes very efficiently.
var array = new[] { 1L, 2L, 3L };
using (var stream = new FileStream("test.bin", FileMode.Create, FileAccess.Write, FileShare.None))
using (var writer = new BinaryWriter(stream))
{
foreach (long item in array)
{
writer.Write(item);
}
}
How are the values changed? And an array of long can be copied into an array of byte very quickly, no need for serialization.
static void Main(string[] args) {
System.Random random = new Random();
long[] arrayOriginal = new long[160000];
long[] arrayRead = null;
for (int i =0 ; i < arrayOriginal.Length; i++) {
arrayOriginal[i] = random.Next(int.MaxValue) * random.Next(int.MaxValue);
}
byte[] bytesOriginal = new byte[arrayOriginal.Length * sizeof(long)];
System.Buffer.BlockCopy(arrayOriginal, 0, bytesOriginal, 0, bytesOriginal.Length);
using (System.IO.MemoryStream stream = new System.IO.MemoryStream()) {
// write
stream.Write(bytesOriginal, 0, bytesOriginal.Length);
// reset
stream.Flush();
stream.Position = 0;
int expectedLength = 0;
checked {
expectedLength = (int)stream.Length;
}
// read
byte[] bytesRead = new byte[expectedLength];
if (expectedLength == stream.Read(bytesRead, 0, expectedLength)) {
arrayRead = new long[expectedLength / sizeof(long)];
Buffer.BlockCopy(bytesRead, 0, arrayRead, 0, expectedLength);
}
else {
// exception
}
// check
for (int i = 0; i < arrayOriginal.Length; i++) {
if (arrayOriginal[i] != arrayRead[i]) {
throw new System.Exception();
}
}
}
System.Console.WriteLine("Done");
System.Console.ReadKey();
}