I am currently using System.Collections.Concurrent.BlockingCollection, and it´s very good with what it´s doing.
However it seems that it only keeps the reference of an object.
So if i have one byte[] object, which is written to and added to the Queue 100 times.
And after it reaches 100, i want to read all those, i will only get 100 copies of the current data "byte[]" holds.
Hope that explains it, at least it seems it´s doing this from my tests.
So if it´s doing this, is there another one that can keep copies of the data and just add it and add it till i read it?
Like for example, i would have 100 byte[] files, write it to a MemoryStream in the correct order, then i can read them in that order.
Though a Memory Stream isn´t what i would prefer to use, but works as an example.
Here is my code:
try
{
Thread.Sleep(100);
for (int i = Queue.Count; i <= Queue.Count; i++)
if (Queue.TryTake(out AudioData, 300))
{
if (Record)
waveWriter.Write(AudioData, 0, AudioData.Length);
}
}
catch (Exception e)
{
if (e is ArgumentNullException)
return;
}
Here is the part which receives the data
using (ms = new MemoryStream(TcpSize))
using (var tt1 = tcplisten.AcceptTcpClient())
{
ReceiveData = new byte[TcpSize];
tt1.NoDelay = true;
using (var tcpstream = tt1.GetStream())
while (connect)
{
if (Record)
Queue.Add(ReceiveData);
tcpstream.Read(ReceiveData, 0, TcpSize);
waveProvider.AddSamples(ReceiveData, 0, TcpSize);
}
}
You may wonder why i use a for loop and all that for writing, but it´s just there for debug purposes. I wanted to test if the objects in the Queue was copies, cause if so, it shouldn´t matter when i write it, but it does which means it must be reference.
Thanks
If you want to queue copies of the data, just make a copy and then queue the copy.
Queue.Add((byte[])ReceiveData.Clone());
But I think you also need to sort out the fact that you're writing the data to the queue before filling the buffer...
Alternatively, create a new buffer on each iteration and queue that instead:
while (connect)
{
ReceiveData = new byte[TcpSize];
tcpstream.Read(ReceiveData, 0, TcpSize);
waveProvider.AddSamples(ReceiveData, 0, TcpSize);
if (Record)
Queue.Add(ReceiveData);
}
Related
I am writing an application that needs to write messages to a USB HID device and read responses. For this purpose, I'm using USBHIDDRIVER.dll (https://www.leitner-fischer.com/2007/08/03/hid-usb-driver-library/ )
Now it works fine when writing many of the message types - i.e. short ones.
However, there is one type of message where I have to write a .hex file containing about 70,000 lines. The protocol requires that each line needs to be written individually and sent in a packet containing other information (start, end byte, checksum)
However I'm encountering problems with this.
I've tried something like this:
private byte[] _responseBytes;
private ManualResetEvent _readComplete;
public byte[][] WriteMessage(byte[][] message)
{
byte[][] devResponse = new List<byte[]>();
_readComplete = new ManualResetEvent(false);
for (int i = 0; i < message.Length; i++)
{
var usbHid = new USBInterface("myvid", "mypid");
usbHid.Connect();
usbHid.enableUsbBufferEvent(UsbHidReadEvent);
if (!usbHid.write(message)) {
throw new Exception ("Write Failed");
}
usbHid.startRead();
if (!_readComplete.WaitOne(10000)) {
usbHid.stopRead();
throw new Exception ("Timeout waiting for read");
}
usbHid.stopRead();
_readComplete.Reset();
devResponse.Add(_responseBytes.ToArray());
usbHid = null;
}
return devResponse;
}
private void ReadEvent()
{
if (_readComplete!= null)
{
_readComplete.Set();
}
_microHidReadBytes = (byte[])((ListWithEvent)sender)[0];
}
This appears to work. In WireShark I can see the messages going back and forth. However as you can see it's creating an instance of the USBInterface class every iteration. This seems very clunky and I can see in the TaskManager, it starts to eat up a lot of memory - current run has it above 1GB and eventually it falls over with an OutOfMemory exception. It is also very slow. Current run is not complete after about 15 mins, although I've seen another application do the same job in less than one minute.
However, if I move the creation and connection of the USBInterface out of the loop as in...
var usbHid = new USBInterface("myvid", "mypid");
usbHid.Connect();
usbHid.enableUsbBufferEvent(UsbHidReadEvent);
for (int i = 0; i < message.Length; i++)
{
if (!usbHid.write(message)) {
throw new Exception ("Write Failed");
}
usbHid.startRead();
if (!_readComplete.WaitOne(10000)) {
usbHid.stopRead();
throw new Exception ("Timeout waiting for read");
}
usbHid.stopRead();
_readComplete.Reset();
devResponse.Add(_responseBytes.ToArray());
}
usbHid = null;
... now what happens is it only allows me to do one write! I write the data, read the response and when it comes around the loop to write the second message, the application just hangs in the write() function and never returns. (Doesn't even time out)
What is the correct way to do this kind of thing?
(BTW I know it's adding a lot of data to that devResponse object but this is not the source of the issue - if I remove it, it still consumes an awful lot of memory)
UPDATE
I've found that if I don't enable reading, I can do multiple writes without having to create a new USBInterface1 object with each iteration. This is an improvement but I'd still like to be able to read each response. (I can see they are still sent down in Wireshark)
I'm having some trouble with a simple TCP Read/Write application where I need to write a command to a device/host. Normally I can do this using a stream.Write() command however with this particular device, it seems to send an initial welcome message back (PJLINK 0) before any command can be sent to it. I can send the commands fine using PuTTY but when using C# I think my connection is closing before I can get my command through.
So my question would be how can I adjust my code below to receive that welcome message and then send my command back (I don't need to read a response) without the TcpClient closing the connection early?
Any help would be greatly appreciated.
using (tcpClientA = new TcpClient())
{
int portA = 4352;
if (!tcpClientA.BeginConnect("10.0.2.201", portA, null, null).AsyncWaitHandle.WaitOne(TimeSpan.FromSeconds(1.0)))
{
throw new Exception("Failed to connect.");
}
while (tcpClientA.Connected)
{
using (streamA = tcpClientA.GetStream())
{
if (type == "raw")
{
// Buffer to store the response bytes.
byte[] writeBufferC = Encoding.ASCII.GetBytes("%1 INPT 32$0D"); //Command I need to send
byte[] readBufferC = new byte[tcpClientA.ReceiveBufferSize];
string fullServerReply = null;
using (var writer = new MemoryStream())
{
do
{
int numberOfBytesRead = streamA.Read(readBufferC, 0, readBufferC.Length);
if (numberOfBytesRead <= 0)
{
break;
}
writer.Write(writeBufferC, 0, writeBufferC.Length);
} while (streamA.DataAvailable);
fullServerReply = Encoding.UTF8.GetString(writer.ToArray());
Console.WriteLine(fullServerReply.Trim());
}
}
}
}
}
Update 1
Removed the BeginConnect and Async methods.
using (tcpClientA = new TcpClient())
{
int portA = 4352;
tcpClientA.Connect("10.0.2.201", portA);
while (tcpClientA.Connected)
{
using (streamA = tcpClientA.GetStream())
{
if (type == "raw")
{
byte[] readBufferC = new byte[tcpClientA.ReceiveBufferSize];
byte[] writeBufferC = Encoding.ASCII.GetBytes("%1 INPT 31$0D"); //Command I need to send
string fullServerReply = null;
using (var writer = new MemoryStream())
{
do
{
streamA.Read(readBufferC, 0, readBufferC.Length); //First read
writer.Write(writeBufferC, 0, writeBufferC.Length); //Send command
} while (streamA.DataAvailable);
fullServerReply = Encoding.UTF8.GetString(readBufferC.ToArray());
Console.WriteLine(fullServerReply.Trim());
tcpClientA.Close();
}
}
}
}
}
DataAvailable does not tell you how much data will be sent in the future by the remote side. It's use is almost always a bug. Here, it causes you to randomly exit the loop early.
Read, until you have all the bytes you expect or until the stream is being closed.
Is this a line-based protocol? Instantiate a StreamReader and draw entire lines from the stream.
while (tcpClientA.Connected) accomplishes nothing. Even if it returns true, the connection could be lost 1 nanosecond later. Your code has to deal with that anyway. It should be while (true). This is not a bug, it just shows weak TCP understanding so I point it out.
Remove all usages of ReceiveBufferSize. This value means nothing of significance. Instead, use a fixed buffer size. I find that 4096 works well with not very high throughput connections.
numberOfBytesRead <= 0 should be ==0. Again, not a bug but you don't seem to understand exactly what the API does. This is dangerous.
In the updated code you're not using the return value of streamA.Read which is a bug. You have tried to fix that bug by trimming off the resulting \0 chars. That's just treating the symptoms and is not a true fix.
You need a socket tutorial. This carnage comes because you are not relying on best practices. Socket reading loops are actually rather simple if done right. This code is a collection of what can go wrong.
I have a NetworkStream that I use to get data from another program. The data arrives as a Byte[64], which I then Enqueue to a ConcurrentQueue so that another thread can dequeue for analysis later.
The queue is instantiated:
ConcurrentQueue<byte[]> fifo = new ConcurrentQueue<byte[]>();
then I Enqueue all the data being sent:
Byte[] bytesIn = new Byte[64];
int i;
while ((i = stream.Read(bytesIn, 0, bytesIn.Length)) != 0)
{
fifo.Enqueue(bytesIn);
}
If I then look at (during debug) the data in fifo it turns out that every byte[64] contained therin is identical to the latest bytesIn. How do I ensure that the arrays I'm adding to fifo are the values and not the pointers (if that's the correct terminology)?
Enqueue a copy of the array instead. You can use the ToArray extension for this.
while ((i = stream.Read(bytesIn, 0, bytesIn.Length)) != 0)
{
var received = bytesIn.Take(i).ToArray();
fifo.Enqueue(received);
}
I also used Take to trim the buffer, and copy only the bytes that were received.
Alternatively, as suggested by #hvd in the comments, using Array.Copy will be faster
while ((i = stream.Read(bytesIn, 0, bytesIn.Length)) != 0)
{
var received = new byte[i];
Array.Copy(bytesIn, 0, received, 0, i);
fifo.Enqueue(received);
}
I think the main issue here is your misunderstanding of adding a reference type to the queue which you have declared outside your while-loop.
If you take a close look at the code you provided, you can see you only declare bytesIn once. You enqueue the bytesIn, and then rewrite the value of the array. The array is, however, still the same object as before and can thus not be queued again, hence it changes the array to the new value.
So what's it what we actually want to do? We want to;
Read the stream
Put the output in a new array-object
Enqueue the new object
Which is exactly what #dcastro does, but I'll strip down the code for you;
while ((
i = stream.Read(bytesIn, 0, bytesIn.Length)) != 0 //read the contents of the
//stream and put it in
//bytesIn, if available
)
{
var received = new byte[i]; //Create a new, empty array, which we are
//going to put in the queue.
Array.Copy(bytesIn, 0, received, 0, i); //Copy the contents of bytesIn into our new
//array. This way we can reuse bytesIn while
//maintaining the received data.
fifo.Enqueue(received); //Enqueue the new array and thus saving it.
}
For more information, perhaps you should read about Reference types.
I am writing a telnet server using the async Begin/End methods. The issue that I am having is determining what within my buffer is actual data and what is not. Network coding is a bit new to me, but I've tried to research this and have not been able to find a answer.
public bool Start(IGame game)
{
// Get our server address information.
IPHostEntry serverHost = Dns.GetHostEntry(Dns.GetHostName());
IPEndPoint serverEndPoint = new IPEndPoint(IPAddress.Any, this.Port);
// Instance the server socket, bind it to a port and begin listening for connections.
this._ServerSocket = new Socket(serverEndPoint.Address.AddressFamily, SocketType.Stream, ProtocolType.Tcp);
this._ServerSocket.Bind(serverEndPoint);
this._ServerSocket.Listen(this.MaxQueuedConnections);
this._ServerSocket.BeginAccept(new AsyncCallback(Connect), this._ServerSocket);
return true;
}
private void Connect(IAsyncResult result)
{
var player = new BasePlayer();
try
{
player.Game = this.Game;
player.Connection = this._ServerSocket.EndAccept(result);
lock (this.Connections)
{
this.Connections.Add(player);
}
// Pass all of the data handling for the player to itself.
player.Connection.BeginReceive(player.Buffer, 0, player.BufferSize, SocketFlags.None, new AsyncCallback(player.ReceiveData), player);
// Fetch the next incoming connection.
this._ServerSocket.BeginAccept(new AsyncCallback(Connect), this._ServerSocket);
}
catch (Exception)
{
}
}
and then the player.ReceiveData..
public void ReceiveData(IAsyncResult result)
{
int bytesRead = this.Connection.EndReceive(result);
if (bytesRead > 0)
{
// TODO: Parse data received by the user.
//Queue the next receive data.
this.Connection.BeginReceive(this.Buffer, 0, this.BufferSize, SocketFlags.None, new AsyncCallback(ReceiveData), this);
var str = System.Text.Encoding.Default.GetString(Buffer);
}
else
{
this.Disconnect(result);
}
}
So when I call BeginReceive, I need to provide a buffer of a predetermined size. In doing that, I end up with unused bytes in my buffer array. They all have the value of 0, so I am assuming that I can loop through the array and build a new one starting at index 0 and working until I hit a value of 0.
I imagine there is a better way to do this? Can someone please point me in the right direction as to how I should determine what the data is within my buffer or perhaps a way that I can do this without having to use a predetermined buffer size.
So when call BeginReceive, I need to provide a buffer of a predetermined size. In doing that, I end up with unused bytes in my buffer array. They all have the value of 0, so I am assuming that I can loop through the array and build a new one starting at index 0 and working until I hit a value of 0.
No, that's not what you should do. Instead, in your callback (ReceiveData) you're already calling EndReceive - and the result of that is the number of bytes you read. That's how much of the buffer you should use.
However, you should copy the data you've read out of the buffer before you call BeginReceive again, otherwise you may end up with the next bit of data overwriting the just-read data before you get to use it.
So something like:
string text = Encoding.ASCII.GetString(Buffer, 0, bytesRead);
Connection.BeginReceive(this.Buffer, 0, this.BufferSize, SocketFlags.None,
new AsyncCallback(ReceiveData), this);
I would not suggest that you use Encoding.Default to convert the bytes to text - instead, you should decide which encoding you're using, and stick to that. If you use an encoding which isn't always one-byte-per-character, you'll end up in a slightly trickier situation, as then you might end up receiving a buffer with part of a character. At that point you need to keep a Decoder which can maintain state about partial characters read.
this is a continuation of part 3
Write file need to optimised for heavy traffic part 3
as my code changed somewhat i think it is better to open a new thread.
public class memoryStreamClass
{
static MemoryStream ms1 = new MemoryStream();
static MemoryStream ms2 = new MemoryStream();
static int c = 1;
public void fillBuffer(string outputString)
{
byte[] outputByte = Encoding.ASCII.GetBytes(outputString);
if (c == 1)
{
ms1.Write(outputByte, 0, outputByte.Length);
if (ms1.Length > 8100)
{
c = 2;
Thread thread1 = new Thread(() => emptyBuffer(ref ms1));
thread1.Start();
}
}
else
{
ms2.Write(outputByte, 0, outputByte.Length);
if (ms2.Length > 8100)
{
c = 1;
Thread thread2 = new Thread(() => emptyBuffer(ref ms2));
thread2.Start();
}
}
}
void emptyBuffer(ref MemoryStream ms)
{
FileStream outStream = new FileStream(string.Format("c:\\output.txt", FileMode.Append);
ms.WriteTo(outStream);
outStream.Flush();
outStream.Close();
ms.SetLength(0);
ms.Position = 0;
Console.WriteLine(ms.Position);
}
there are 2 things i have changed changed from the code in part 3.
the class and method is changed to non-static, the variables are still static tho.
i have move the memorystream reset length into the emptyBuffer method, and i use a ref parameter to pass the reference instead of a copy to the method.
this code compiled fine and runs ok. However, i run it side by side with my single thread program, using 2 computers, one computer runs the single thread, and one computer runs the multithread version, on the same network. i run it for around 5 mins. and the single threaded version collects 8333KB of data while the multithread version collects only 8222KB of data. (98.6% of the single thread version)
its first time i have do any performance comparison between the 2 version. Maybe a should run more test to confirm it. but base on looking the code, any masters out there will point out any problem?
i haven't putting any code on lock or threadpooling at the moment, maybe i should, but if the code runs fine, i dont want to change it and break it. the only thing i will change is the buffer size, so i will eliminate any chance of the buffer fill up before the other is emptied.
any comments on my code?
The problem is still static state. You're clearing buffers that could have data that wasn't written to disk.
I imagine this scenario is happening 1.4% of the time.
ms1 fills up, empty buffer1 thread started, switch to ms2
empty buffer1 is writing to disk
ms2 fills up, empty buffer2 thread started, switch to ms1
empty buffer1 to disk finishes
ms1 is cleared while it is the active stream
When doing multi-threaded programming, static classes are fine but static state is not. Ideally you have no shared memory between threads and your code is entirely dependent on it.
Think of it this way -- if you're expecting a value to consistently change, it's not exactly static is it?