I am receiving some raw data from audio input via System.Runtime.InteropServices.Marshal.Copy() and this is stored in a byte[]. The actual data in that byte[] I know to be of type Int16. I would like to treat this byte[] as if it was an array of type Int16 and loop over its elements, process it and at the end of this I would like to pass this altered array as byte[] into another function.
For those who wonder what I'm trying to do: I would like to try adding an echo effect to the incoming sound (from mic) by simply adding up the wave data from previous buffer with the wave data from the current buffer, mixing it in at a lower volume.
EDIT: the byte[] array may be storing a int16 number 258 as a pair of bytes [1][2]
0000 0001 0000 0010
You can't, as far as I'm aware.
I suggest you use Buffer.BlockCopy to copy the data into a short[], work on it, and then copy it back. Of course, that assumes that the bytes are arranged in an appropriate endianness... but it's likely that they will be.
Alternatively... use the overload of Marshal.Copy which accepts a short[] to copy the data into to start with.
You can use BitConverter to treat the bytes as Int16 elements using BitConverter.ToInt16. You can also use Buffer.BlockCopy to copy the array of bytes to an array of Int16s. The first is more memory efficient and the second is more speed efficient.
Related
I have asked this question over the last 2 years and am still looking for a good way of doing this. What I am doing is as follows:
I have a WPF/C# application which has been developed over the last 3 years. It takes a real time stream of bytes over a UDP port. Each record set is 1000 bytes. I am getting 100 of these byte records per second. I am reading the data and processing it for display in various formats. These logical records are sub-commutated.
The first 300 bytes are the same each logical record contain a mixture of Byte, Int16, UInt16, Int32 and UInt32 values. About 70% of these values are eventually multiplied by an least significant bit to create a Double. These parameters are always the same.
The second 300 bytes are another mixture of Byte, Int16, UIn32, Int32 and UInt32 values. Again about 70% of these values are multiplied by an LSB to create a Double. These parameters are again always the same.
The last segment is 400 bytes and sub-commutated. This means that the last part of the record contains 1 of 20 different logical record formats. I call them Type01...Type20 data. There is an identifier byte which tells me which one it is. These again contain Byte, Int, UInt data values which need to be converted.
I am currently using hundreds of function calls to process this data. Each function call takes the 1000 byte array as a parameter, an offset (index) into the byte array to where the parameter starts. It then uses the BitConverter.ToXXX call to convert the bytes to the correct data type, and then if necessary multiply by an LSB to create the final data value and return it.
I am trying to streamline this processing because the data stream are changing based on the source. For instance one of the new data sources (feeds) changes about 20 parameters in the first 300 bytes, about 24 parameters in the second 300 bytes and several in the last sub-commutated 400 bytes records.
I would like to build a data dictionary where the dictionary contains the logical record number (type of data), offset into the record, LSB of data, type of data to be converted to (Int16, UInt32, etc) and finally output type (Int32, Double, etc). Maybe also include the BitConverter function to use and "cast it dynamically"?
This appears to be a exercise in using Template Classes and possibly Delegates but I do not know how to do this. I would appreciate some code as in example.
The data is also recorded so playback may run at 2x, 4x, 8x, 16x speeds. Now before someone comments on how you can look at thousands of parameters at those speeds, it is not as hard as one may think. Some types of data such as green background for good, red for bad; or plotting map positions (LAT/LON) over time lend themselves very well for fast playback to find interesting events. So it is possible.
Thanks in advance for any help.
I am not sure others have an idea of what I am trying to do so I thought I would post a small segment of source code to see if anyone can improve on it.
Like I said above, the data comes in byte streams. Once it is read in a Byte Array it looks like the following:
Byte[] InputBuffer = { 0x01, 0x00, 0x4F, 0xEB, 0x06, 0x00, 0x17, 0x00,
0x00, 0x00, ... };
The first 2 bytes are an ushort which equals 1. This is the record type for this particular record. This number can range from 1 to 20.
The next 4 bytes are an uint which equals 453,455. This value is the number of tenths of a seconds. Value in this case is 12:35:45.5. To arrive at this I would make the following call to the following subroutine:
labelTimeDisplay.Content = TimeField(InputBuffer, 2, .1).ToString();
public Double TimeField(Byte[] InputBuffer, Int32 Offset, Double lsb)
{
return BitConverter.ToUInt32(InputBuffer, Offset) * lsb;
}
The next data field is the software version, in this case 23
labelSoftwareVersion.Content = SoftwareVersion(InputBuffer, 6).ToString();
public UInt16 SoftwareVersion(Byte[] InputBuffer, Int32 Offset)
{
return BitConverter.ToUInt16(InputBuffer, Offset);
}
The next data field is the System Status Word another UInt16.
Built-In-Test status bits are passed to other routines if any of the 16 bits are set to logic 1.
UInt16 CheckStatus = SystemStatus(InputBuffer, 8);
public UInt16 SystemStatus(Byte[] InputBuffer, Int32 Offset)
{
return BitConverter.ToUInt16(InputBuffer, Offset);
}
I literally have over a thousand of individual subroutines to process the data stored in the array of bytes. The array of bytes are always fixed length of 1000 bytes. The first 6 bytes are always the same, identifier and time. After that the parameters are different for every frame.
I have some major modifications coming the software which will redefine many of the parameters for the next software version. I still have to support the old software versions so the software just gets more complicated. My goal is to find a way to process the data using a dictionary lookup. That way I can just create the dictionary and read the dictionary to know how to process the data. Maybe use loops to load the data into a collection and then bind it to the display fields.
Something like this:
public class ParameterDefinition
{
String ParameterNumber;
String ParameterName;
Int32 Offset;
Double lsb;
Type ReturnDataType;
Type BaseDataType;
}
private ParameterDefinition[] parms = new ParameterDefinition[]
{
new ParameterDefinition ( "0000","RecordID", 0, 0.0, typeof(UInt16), typeof(UInt16)),
new ParameterDefinition ( "0001", "Time", 2, 0.1, typeof(Double), typeof(UInt32)),
new ParameterDefinition ( "0002", "SW ID", 6, 0.0, typeof(UInt16), typeof(UInt16)),
new ParameterDefinition ( "0003", "Status", 8, 0.0, typeof(UInt16), typeof(UInt16)),
// Lots more parameters
}
My bottom line problem is getting the parameter definitions to cast or select the right functions. I cannot find a way to link the "dictionary" to actual data ouputs
Thanks for any help
Using a data dictionary to represent the data structure is fine, as long as you don't walk the dictionary for each individual record. Instead, use Reflection Emit or Expression trees to build a delegate that you can call many many times.
It sounds like you are manually deserializing a byte stream, where the bytes represent various data types. That problem has been solved before.
Try defining a class that represents the first 600 bytes and deserialize that and deserialize it using Protocol Buffer Serializer (that implementation is by SO's own Marc Gravell, and there is a different implementation by top SO contributer Jon Skeet).
Protocol buffers are language-neutral, platform-neutral, extensible way of serializing structured data for use in communications protocols and data storage. You define how you want your data to be structured once, then you can use special generated source code to easily write and read your structured data to and from a variety of data streams and using a variety of languages. You can even update your data structure without breaking deployed programs that are compiled against the "old" format.
Source, as well as a 3rd implementation I have not personally used.
For the last 300 bytes, create appropriate class definitions for the appropriate formats, and again use protocol buffer to deserialize an appropriate class.
For the final touch-ups (e.g. converting values to doubles) you can either post-process the classes, or just have a getter that returns the appropriate final number.
I want to use serialport.readline() because it's a blocking call and the data I'm getting has 0x0D 0x0A (CR LF) at the end. However, what I want is an Hex String instead of an Ascii representation.
For example, the device I am communication with is sending byte arrays like {0xff,0xff,0x45,0x0D,0x0A}. I want to simply print out in my program just like this: 0xff, 0xff,0x45. Readline() kindly trims out LF and CR.
I though about using serialport.read(buff[]...) by specifying how many bytes I want to read. But it didn't work very well because if I am reading too fast half of the array will be 0x00 and if I am reading too slow, there will be an overflow for the com port. I don't want to lose any bytes.
I tried to convert what I got from serialport.readline() to a byte array but the hex string I got usually turns into 0x3f. Code is like this:
var line = string.Join(",", mySerialPort.ReadLine().Select(c => ((Byte)c).ToString("X")).ToArray());
I changed the encoding a few times (ASCII, UTF8,UNICODE) but still no go.
Is there any way to convert the non-Ascii String I got from readline() into a byte array?
It sounds like you shouldn't be reading it as text data at all.
You're fundamentally dealing with binary data, so use the overload of Read which takes a byte array rather than a char array. (Or call ReadByte repeatedly.)
Any time you try to treat arbitrary binary data as if it's text, you're going to have a bad experience.
It sounds like you've already tried this, but done it badly:
But it didn't work very well because if I am reading too fast half of the array will be 0x00
That suggests you're ignoring the return value of Read, which says how many bytes have actually been read. You should have something like:
int bytesRead = port.Read(buffer, 0, buffer.Length);
// Now use the portion of buffer which is from 0 (inclusive) to
// bytesRead (exclusive).
I have recently started learning C# Networking and I was wondering how would you tell if the received Byte array is a file or a string?
A byte array is just a byte array. It's just got data in.
How you interpret that data is up to you. What's the difference between a text file and a string, for example?
Fundamentally, if your application needs to know how to interpret the data, you've got to put that into the protocol.
A byte array is just a byte array. However, you could make the original byte array include a byte that describes what type it is (assuming you are the originator of it). Then you find this descriptor byte and use it to make decisions.
Strings are encoded byte arrays; files can contain strings and/or binary data.
ASCII strings use byte values between 0-127 to represent characters and control codes. For UTF8 people have written validation routines (https://stackoverflow.com/a/892443/884862).
You'd have to check the array for all of the string encoding characteristics before you could assume it's a binary file.
edit Here's an SO question about classifying a file type Using .NET, how can you find the mime type of a file based on the file signature not the extension using a signature (first X bytes) of the file to determine it's mimetype.
No you can't. Data is data, you must layer on top of your network communication form of protocol, it will need to say something like: "If the first byte I see is a 1 the next four bytes represent a int, if I see a 2 read the next byte and that is the length of the text string that follows that..."
A much easier solution than inventing your own protocol is use a prebuilt one that gives you a higher level abstraction like WCF so you don't need to deal with byte arrays.
Not quite a "file", an array contains data. You should loop through that array and write the data,
Try this:
foreach(string data in array)
{
Console.WriteLine(data);
}
Now, if it doesn't contain strings, but data, you can simply use a
foreach(var data in array)
{
Console.WriteLine(data.ToString());
}
What I try to do:
I want to store very much data in RAM. For faster access and less memory footprint I need to use an array of struct values:
MyStruct[] myStructArray = new MyStruct[10000000000];
Now I want to store unsigned integer values with one, two, three or four bytes in MyStruct. But it should only use the less possible memory amount. When I store a value it one byte it should only use one byte and so on.
I could implement this with classes, but that is inappropriate here because the pointer to the object would need 8 bytes on a 64bit system. So it would be better to store just 4 bytes for every array entry. But I want to only store/use one/two/three byte when needed. So I can't use some of the fancy classes.
I also can't use one array with one bytes, one array with two bytes and so on, because I need the special order of the values. And the values are very mixed, so storing an additional reference when to switch to the other array would not help.
Is it possible what want or is the only way to just store an array of 4 byte uints regardless I only need to store one byte, two byte in about 60% of the time and three bytes in about 25% of the time?
This is not possible. How would the CLR process the following expression?
myStructArray[100000]
If the elements are of variable size, the CLR cannot know the address of the 100000th element. Therefore array elements are of fixed size, always.
If you don't require O(1) access, you can implement variable-length elements on top of a byte[] and search the array yourself.
You could split the list into 1000 sublists, which are packed individually. That way you get O(n/2000) search performance on average. Maybe that is good enough in practice.
A "packed" array can only be searched in O(n/2) on average. But if your partial arrays are 1/1000th the size, it becomes O(n/2000). You can pick the partial array in O(1) because they all would be of the same size.
Also, you can adjust the number of partial arrays so that they are individually about 1k elements in size. At that point the overhead of the array object and reference to it vanish. That would give you O(1000/2 + 1) lookup performance which I think is quite an improvement over O(n/2). It is a constant-time lookup (with a big constant).
You could get close to that what you want if you are willing to sacrifice some additional CPU time and waste additional 2 or 4 bits per one stored value.
You could just use byte byte[] and combine it with BitArray collection. In byte[] you would then just sequentially store one, two, three or four bytes and in BitArray denote in binary form (pairs of two bits) or just put a bit to value 1 to denote a new set of bytes have just started (or ended, however you implement it) in your data array.
However you could get something like this in memory:
byte[] --> [byte][byte][byte][byte][byte][byte][byte]...
BitArray --> 1001101...
Which means you have 3 byte, 1 byte, 2 bytes etc. values stored in your byte array.
Or you could alternatively encode your bitarray as binary pairs to make it even smaller. This means you would vaste somewhere between 1.0625 and 1.25 bytes per your actual data byte.
It depends on your actual data (your MyStruct) if this will suffice. If you need to distinguish to which values in your struct those bytes really corresponds, you could waste some additional bits in BitArray.
Update to your O(1) requirement:
Use another index structure which would store one index for each N elements, for example 1000. You could then for example access item with index 234241 as
indexStore[234241/1000]
which gives you index of element 234000, then you just calculate the exact index of element 234241 by examining those few hundred elements in BitArray.
O(const) is acheieved this way, const can be controlled with density of main index, of course you trade time for space.
You can't do it.
If the data isn't sorted, and there is nothing more you can say about it, then you are not going to be able to do what you want.
Simple scenario:
array[3]
Should point to some memory address. But, how would you know what are dimensions of array[0]-array[2]? To store that information in an O(1) fashion, you would only waste MORE memory than you want to save in the first place.
You are thinking out of the box, and that's great. But, my guess is that this is the wrong box that you are trying to get out of. If your data is really random, and you want direct access to every array member, you'll have to use MAXIMUM width that is needed for your number for every number. Sorry.
I had one similar situation, with having numbers of length smaller than 32 bits that I needed to store. But they were all fixed width, so I was able to solve that, with custom container and some bit shifting.
HOPE:
http://www.dcc.uchile.cl/~gnavarro/ps/spire09.3.pdf
Maybe you can read it, and you'll be able not only to have 8, 16, 24, 32 bit per number, but ANY number size...
I'd almost start looking at some variant of short-word encoding like a PkZip program.
Or even RLE encoding.
Or try to understand the usage of your data better. Like, if these are all vectors or something, then there are certain combinations that are disallowed like, -1,-1,-1 is basically meaningless to a financial graphing application, as it denotes data outsides the graphable range. If you can find some oddities about your data, you may be able to reduce the size by having different structures for different needs.
I got some problem transfering Data via the BinaryWriter.
When I try to send
bw.Write(0x1a);
bw.Write(0xf8);
bw.Write(0x05);
It gets in the output to 0x00 - via
Client2Server._mainSock.Send(ms.ToArray());
What is causing this problem?
Greetings
You are writing 3 integers here. Integers take 4 bytes, and in the cases shown, 3 of them are going to be zeros. Send bytes instead:
bw.Write((byte)0x1a);
of course, if you are writing bytes, then BinaryWriter is overkill - you could just use the Stream.