I have noticed a problem where the .Completed event of a SocketAsyncEventArgs seems to stop firing. The same SAEA can fire properly and be replaced in the pool several times, but eventually all instances will stop firing, and because the code to replace them in the pool is in the event handler, the pool empties.
The following circumstances are also apparently true:
1) It seems to only occur when a server side socket sends data out to one of the connected clients. When the same class is connecting as a client, it doesn't appear to malfunction.
2) It seems to occur under high load. The thread count seems to creep up until eventually the error happens.
3) A test rig under similar stress appears never to malfunction. (It's only 20 messages per second, and the test rig has been proven to 20K)
I'm not going to be able to paste the rather complicated code, but here is a description of my code:
1) The main inspiration is this: http://vadmyst.blogspot.ch/2008/05/sample-code-for-tcp-server-using.html. It shows how to hook up a completion port using an event, how to get different sized messages over the TCP connection, and so on.
2) I have a byte buffer in which all SAEAs have a piece, that doesn't overlap.
3) I have an object pool of SAEAs, based on a blockingcollection. This throws if the pool is empty for too long.
4) As a server, I keep a collection of sockets returned from the AcceptAsync function, indexed by the endpoint of the client. A single process can use one instance as a server as well as multiple instances as clients (forming a web). They share the data buffer and pool of SAEAs.
I realise it's hard to explain this; I've been debugging it for an entire day and night. Just hoping someone has heard of this or has useful questions or suggestions.
At the moment, I am suspecting some sort of thread exhaustion, leading to the SAEAs not being able to call the completion. Alternatively, some sort of buffer problem on the outgoing buffer.
So, another day of debugging and finally I have an explanation.
1) The SAEAs were not firing the completed event because they were unable to send more. This is revealed by Wireshark to be due to the TCP window emptying. (TCP ZeroWindow)
2) The TCP window was emptying because the networking layer was passing an event up the stack that took too long to complete, ie there's no producer/consumer between the network layer and the UI. Thus the network op would have to wait for the screen draw before sending the ACK.
3) The event that took too long was a screen draw in an event handler on the GUI. The test rig was a console window (one that summarized incoming messages), so that's why it didn't cause a problem at much higher load. It's normal not to redraw the screen on each message, but this was happening because the project isn't quite done yet. The redraw rate would have been fixed later.
4) The short term solution is simply to make sure there's no GUIs holding up the show. A more robust solution might be to create a producer/consumer at the network layer.
Related
I'm trying to implement a basic UDP client. One of its functions is the ability to probe computers to see if a UDP server is listening. I need to scan lots of these computers quickly.
I can't use the Socket.BeginReceiveFrom method and run a timeout waiting for it to complete, because callbacks may occur after the timeout is over, and seeing as many computers are being probed quickly, I found that later callbacks ended up using modified data as a new probe was already underway when the callback was finally invoked.
I can't use the Socket.ReceiveFrom method and set a Socket.ReceiveTimeout because the SocketException being thrown+handled takes a long time (not sure why, I'm not running much code to handle it), meaning it takes about 2 seconds per computer rather than 100ms like hoped.
Is there any way of running a timeout on a synchronous call to ReceiveFrom without using exceptions to determine when the call has failed/succeeded? Or is there a tactic I've not yet taken that you think could work?
Any advice is appreciated.
I decided to rewrite the probe code using TCP.
However, I later discovered the Socket.ReceiveFromAsync method which, seeing as it only receives a single datagram per call, would have made life easier.
I am writing an application that uses OpenNETCF.IO.Serial (open source, see serial code here) for its serial communication on a Windows CE 6.0 device. This application is coded in C# in the Compact Framework 2.0. I do not believe the issue I am about to describe is specifically related to these details, but I may be proven to be wrong in that regard.
The issue I am having is that, seemingly randomly (read as: intermittent issue I cannot reliably duplicate yet), data will fail to transmit or be received until the device itself is rebooted. The Windows CE device communicates with a system that runs an entirely different application. Rebooting this other system and disconnecting/reconnecting communication cables does not appear to resolve this issue, only rebooting the Windows CE device.
The only sign of this issue occurring is a lack of a TxDone event from OpenNETCF firing (look for "TxDone();" in OpenNETCF.IO.Serial class), and no data being received, when I know for a fact that the connected system is sending data.
Any character value from 1 - 255 (0x01 - 0xFF) can be sent and received in our serial communication. Null values are discarded.
My serial settings are 38400 Baud, 8 data bits, no parity, 1 stop bit (38400, 8n1). I've set the input and output buffer sizes to 256 bytes. DataReceived event happens whenever we receive 1 or more characters, and transmission occurs when there's 1 or more bytes in the output buffer, since messages are of variable length.
No handshaking is used. Since this is RS422, only 4 signals are being used: RX+, RX-, TX+, TX-.
I receive a "DataReceived" event, I read all data from the input buffer and make my own buffer in my code to parse through it at my leisure outside of the DataReceived event. When I receive a command message, I send an quick acknowledgment message back. When the other system receives a command message from the Windows CE device, it will send a quick acknowledgment message back. Acknowledgment messages get no further replies since they're intended as a simple "Yep, got it."
In my code, I receive/transmit through multiple threads, so I use the lock keyword so I'm not transmitting multiple messages simultaneously on multiple threads. Double checking through code has shown that I am not getting hung up on any locks.
At this point, I am wondering if I am continuously missing something obvious about how serial communication works, such as if I need to set some variable or property, rather than just reading from an input buffer when not empty and writing to a transmit buffer.
Any insight, options to check, suggestions, ideas, and so on are welcome. This is something I've been wrestling with on my own for months, I hope that answers or comments I receive here can help in figuring out this issue. Thank you in advance.
Edit, 2/24/2011:
(1) I can only seem to recreate the error on boot up of the system that the Windows CE device is communicating with, and not every boot up. I also looked at the signals, common mode voltage fluctuates, but amplitude of the noise that occurs at system boot up seems unrelated to if the issue occurs or not, I've seen 25V peak-to-peak cause no issue, when 5V peak-to-peak the issue reoccurred).
Issue keeps sounding more and more hardware related, but I'm trying to figure out what can cause the symptoms I'm seeing, as none of the hardware actually appears to fail or shutdown, at least where I've been able to reach to measure signals. My apologies, but I will not be able to give any sort of part numbers of hardware parts, so please don't ask the components being used.
(2) As per #ctacke's suggestion, I ensured all transmits were going through the same location for maintainability, the thread safety I put in is essentially as follows:
lock(transmitLockObj)
{
try
{
comPort.Output = data;
}
[various catches and error handling for each]
}
(3) Getting UART OVERRUN errors, in a test where <10 bytes were being sent and received on about a 300msec time interval at 38400 Baud. Once it gets an error, it goes to the next loop iteration, and does NOT run ReadFile, and does NOT run TxDone event (or any other line checking procedures). Also, not only does closing and reopening the port do nothing to resolve this, rebooting the software while the device is still running doesn't do anything, either. Only a hardware reboot.
My DataReceived event is as follows:
try
{
byte[] input = comPort.Input; //set so Input gets FULL RX buffer
lock(bufferLockObj)
{
for (int i = 0; i < input.Length; i++)
{
_rxRawBuffer.Enqueue(input[i]);
//timer regularly checks this buffer and parses data elsewhere
//there, it is "lock(bufferLockObj){dataByte = _rxRawBuffer.Dequeue();}"
//so wait is kept short in DataReceived, while remaining safe
}
}
}
catch (Exception exc)
{
//[exception logging and handling]
//hasn't gotten here, so no point in showing
}
However, instantly after the WriteFile call did timed out the first time in the test was when I started getting UART OVERRUN errors. I honestly can't see my code causing a UART OVERRUN condition.
Thoughts? Hardware or software related, I'm checking everything I can think to check.
Everything sounds right, but your observations kind of show that they're not.
Since you've stated that you're sending from multiple threads, the first thing I'd do is put in some sort of mechanism for sending where all send requests come into one location before calling out to the serial object instance. Sure, you say that you've ensured you have thread safety, but serializing these calls through one location would help reinforce that (and make the code a bit more maintainable/extensible).
Next I'd probably add some temp handling in the Serial lib to specifically set an event or break in the debugger when you've done a Tx but the TxDone event doesn't fire within some bounding period. It's always possible that the Serial lib has a bug in it (trust me, the author of that code is far from infallible) where some race condition is getting by.
Thank you everyone who responded. We've found that this actually appears to be hardware-related. I'm afraid I can't give more information than this, but I thank everyone who contributed possible solutions or troubleshooting steps.
I have a program that is used to talk to hardware over rs232. This software is used to display a stream of data that is pushed over the rs232 from the hardware as fast as it can be. The problem I am running into is that over time the private memory assigned to the program explodes, and will very rapidly crash the program. If I disable the hardware from sending data for about 2 minutes, then the software can clear out the memory, but only if I pause the data stream.
I am using the DataReceived event from the SerialPort, and this appears to be where the problem is at, because it will cause a memory spike even if the DataReceived function does nothing inside it. The only thing I can come up with is that every time this event is raised it creates a new thread to run, and it is happening so fast that the computer doesn't have time to run GC while the data is coming in.
Is there a more efficient way to pull data off a SerialPort object? I only care about a string when I receive a "NewLine"?
Thanks,
John Vickers
DataReceived is executed on a different thread. I had problems with really fast data and this event caused me problems. Because of that, I created one thread and read the data myself:
while (this.serialPort.IsOpen)
{
int b = this.serialPort.ReadByte();
if (b != -1)
{
// data is good here
}
}
But like the others said, without any code sample, there isn't much we can help you with.
This is very unusual but it is technically possible. SerialPort uses threadpool threads to call the DataReceived event handler. As soon as it receives one or more bytes, it grabs a TP thread to notify your app. There's a lock in the event generation code, only one thread can call your event handler at a time.
A potential failure mode here is that one of these calls, likely the first one, enters a loop in your code from which it never exits. If you haven't set up the Handshake property, the device can keep sending and triggering more TP calls, all of them blocking on that lock.
Diagnose this from the Debug + Windows + Threads window. If my guess is accurate then you should see a large number of threads listed here. One of them should be inside your DataReceived event handler, double-click it and look at the call stack to see where it is stuck. The memory you are seeing consumed is eaten by the stacks of these threads, one megabyte each.
Another possibility is that your DataReceived event handling code is very slow, possibly by calling Control.Invoke(). Slow enough to not be able to keep up with the device. You now really do need to use the Handshake property to setup flow control. Or fix whatever makes it so slow. There should also be a very large number of ErrorReceived events btw, be sure to implement it so you can see this stuff going wrong.
There's an upper limit on the number of TP threads that can be running at the same time. It is rather generous, 250 times the number of cores. That can easily consume half a gigabyte of memory on a typical dual-core machine.
Just to revive this issue.
I'm seeing a massive memory leak when using the DataReceived event.
I am using a USB 3G modem which provides a serial modem interface. I wrote a tiny program that just opens the serial port and connects to the DataReceived event. The event handler is just an empty method.
If you yank out the dongle memory starts to leak at about 10MB per second. No exception is thrown.
Spinning up a new thread and using the synchronous Read(...) method solved the problem for me. I now get an exception when I yank out the dongle that I can handle and no memory leaks.
I developed a state-driven serial port programming language in C# and I believe it really solves nearly all of the serial port problems that everybody encounters with.
Would you please try it with the following simple state and check memory leaks ?
state Init
recv();
$len = length($DATA_PACKET);
if("$len > 0") {
log($DATA_PACKET, Debug);
}
end state
Screen shots
Project homepage
Download
If you have any questions, please feel free to ask.
I've written an application which reads from a serial device at a very fast rate. The serial port object however, fails to fire the DataReceieved event after about 20 min of continuous operation. Disconnecting and reconnecting the serial programmatically allows the event to work again but for only another 20 min.
I tried using DiscardInBuffer after every DataReceived event and this has appeared to have solved the problem. But the method consumes a lot of cpu time and is degrading the performance of the application. MSDN mentions that the method "Discards data from the serial driver's receive buffer.", but fails to suggest when it should be used.
When and how should DiscardInBuffer be used, and am I using it in the correct context for my particular problem ?
Edit:
After implementing the ErrorReceived event, the event data returned indicated the event type was an "RXOver".
After more investigations, it appears my problem was more a fundamental issue. Since the data was flooding in at a hot pace, the SerialPort buffer needed to be cleared or processed continuously to prevent the "RXOver" error. I achieved this by reading into another buffer during the DataReceived event and processed it in another separate thread.
From my understanding the DiscardInBuffer should only be used selectively to clear the contents of the ports for initialisation purposes, such as before opening a port. The process of clearing the driver buffer does take some time to complete and therefore should be used wisely in a performance orientated application.
Two ideas come to mind. The first (horrible) idea: call DiscardInBuffer every 15 to 20 minutes instead of after every DataReceived event. The second (somewhat better) idea: call DiscardInBuffer when you receive the ErrorReceived event, which you should be handling.
I'm an embedded programmer trying to do a little bit of coding for a communications app and need a quick start guide on the best / easiest way to do something.
I'm successfully sending serial data packets but need to impliment some form of send/ response protocol to avoid overflow on the target system and to ensure that the packet was received ok.
Right now - I have all the transmit code under a button click and it sends the whole lot without any control.
What's the best way to structure this code , i.e sending some packets - waiting for response .. sending more .. etc etc until it's all done, then carrying on with the main program.
I've not used threads or callbacks or suchlike in this environment before but will learn - I just need a pointer to the most straigtforward ways to do it.
Thanks
Rob
The .NET serialport uses buffers, learn to work with them.
Sending packets that are (far) smaller than the Send-buffer can be done w/o threading.
Receiving can be done by the DataReceived event but beware that that is called from another thread. You might as well start your own thread and use blocking reads from there.
The best approach depends on what your 'packets' and protocol look like.
I think to have a long experience about serial comm, both MCU and PC-based.
I strongly UNSUGGEST the single-thread based solution, although it is very straigthful for light-speed testing, but absolutely out for final releases.
Surely you may choose among several patterns, but they are mostly shaped around a dedicated thread for the comm process and a finite-state-machine to parse the protocol (during receiveing).
The prevoius answers give you an idea to how build a simple program, but it might depends on the protocol specification, target device, scope of the application, etc.
there are of course different ways.
I will describe a thread based and an async operation based way:
If you don't use threads, your app will block as long as the operation is performing. This is not what a user is expecting today. Since you are talking about a series of sending and receiveing commands, I would recommend starting the protocol as a thread and then waiting for it to finish. You might also place an Abort button if neccesary. Set the ReadTimeout values and at every receive be ready to catch the exception! An introducing into creating such a work thread is here
If you want to, use Async Send/Receive functions instead of a thread (e.g. NetworkStream.BeginRead etc.). But this is more difficult because you have to manage state between the calls: I recommend using a Finite State Machine then. In fact you create an enumeration (i.e. ProtocolState) and change the state whenever an operation has completed. You can then simply create a function that performs the next step of the protocol with a simple switch/case statement. Since you are working with a remote entity (in your case the serial target system), you always have to consider the device is not working or stops working during the protocol. Do this by starting a timeout timer (e.g. set to 2000ms) and start it after sending each command (assuming each command will get a reply in your protocol). Stop it if the command was received successfully or on timeout.
You could also implement low-level handshaking on the serial port; set the serial port's Handshake property to rts/cts or xon/xoff.
Otherwise (or in addition), use a background worker thread. For simple threads, I like a Monitor.Wait/Pulse mechanism for managing the thread.
I have some code that does read-only serial communications in a thread; email me and I'll be happy to send it to you.
I wasn't sure from your question if you were designing both the PC and embedded sides of the communication link, if you are you might find this SO question interesting.