I've been working on learning to deliver and display data in "real-time" over websockets. Real-time, in this context, just means that the data is being read from a sensor (camera with image processing) every .03 to 1 seconds. Each data point consists of a time and a value (t,v) which are encoded as doubles (although the time is always an integer in this case, I'm not assuming it will be).
The server side uses Alchemy Websockets implementation (C#) as I found it very easy to understand/modify for my purposes.
I'm leveraging the websockets examples found here and here as well as the examples included with Alchemy.
I'm using HighCharts to display the data in "real-time", but I also have it printing to a div for debug purposes (independent example so they don't interfere with each other).
So much of it already works pretty well, but there's an obvious problem that happens when I send data too quickly (to be clear, sending the data about a point every second or two results in a nice graph which doesn't appear to have any problems - the problems become more pronounced the faster I call the alchemy server "send" function).
The data appears to be coming in in the wrong order resulting in an interesting "mangled" effect.
I'm going to start delving into the packet order contained on the server side buffer (the server is instructed to send a certain number of "historical" points when a new user connects and it is already running - this results in a pronounced problem like the one shown above) as well as the client side receive order by looking at the timestamps.
The error is inconsistent in that each time I reload the page it results in a different "mangled" data set. This makes me suspect the communication over websockets to be responsible or something involving the alchemy server.
I will attach the full code if necessary, but right now it's rather messy so I am more looking for troubleshooting ideas.
I've gathered this is not expected behavior for a web socket as it is built on TCP.
Any suggestions/ideas for things to look at?
Thanks!
Edit: I ran another test to check how many data points were out of order each time I refreshed the page. The numbers are as follows:
1 2 3 25 6 5 10 11 96 2 8
Very inconsistent (never 0). Certainly interesting!
This result was taken by excluding the charting component and only using websockets and an array to store the data.
Update:
I decided I'd start analyzing the order things come in in and it does appear to be randomly receiving out of order points using an identical data set. I implemented an "insert" function which will take into account out of order packets. The result (plus a little theme change) looks pretty good!
Open question remains: Is it expected that a websocket can deliver information out of order? Or is there something wrong with my implementation on the server side (or Alchemy). I will investigate further when I have time.
SOLUTION!
I figured it out! After a lot of testing, I realized that my Connection object (which is responsible for watching a data set for new data and sending it as is appropriate given how the connection is configured) was implemented using a Timer object. This was something I took from an example (normally I just use the Thread object for most things asynchronous).
As the Timer object speeds up, it starts executing asynchronously with it's previous calls to it's Tick function. This means that very occasionally, one call to it's Tick function will happen a little faster than another (due to the latency in the Alchemy Send function). This causes minor out of order problems.
I switched the implementation of the communication loop from a Timer object to a Thread object, thus enforcing synchronization, and the out of order packets went away!
Websockets use TCP, and TCP guarantees that data is delivered in order.
I would hope that the browser fires websocket message events in order as well. In my tests, this seemed to be the case.
Using this simple Node app to test:
var sio = require('socket.io')
, http = require('http')
, express = require('express')
, app = express()
, srv = http.createServer(app)
, io = sio.listen(srv)
, fs = require('fs')
, idx = fs.readFileSync('index.html').toString()
;
app.get('/', function(req, res) {
res.send(idx);
});
var i = 0;
setInterval(function() {
io.sockets.emit('count', i++);
}, 1);
srv.listen(888);
This simply sends websocket messages as fast as it can with a number that's incremented each message. The client:
<script src="/socket.io/socket.io.js"></script>
<script>
var last = 0;
var socket = io.connect('/');
socket.on('count', function(d) {
if (d-1 != last) console.warn('out of order!', last, d);
last = d;
});
</script>
Throws a console warning if it receives a message that contains a number that is not one more than the previous message.
In Chrome and Firefox, I saw zero out-of-order messages.
I also tried blocking for a while in the message received event (for (var i = 0; i < 1000000; i++) { }) to simulate work that would cause messages to queue up. Message events still fired in order.
In other words, it's something else in your setup. Most likely, the Alchemy server is actually sending the messages in a different order.
Do not use an object like Timer which has asynchronous callbacks when the task is synchronous. Use a Thread and run the communication loop that way.
I don't know when the issue was posted. I also have similar problem. I use Alchemy Client send small data, then there is no problem. There are a lot example for chat Service. But when I send some file more than 4 KB (not precisely), the problem take place. I try to find what happened. I wrote a program sent numbers from 0-7000 by Alchemy client, and received from UserContext.DataFrame(onreceive), there will happen that DataFrame.ToString will get extra "\0\0\0\0" on about position 508. Then, after this position, the data ordering will be wrong. I used the 2.2.1 from nuget. And I read version 2.0 on GiHub. The source code is not workable. So, to old and no reference value.
Related
I have about 10.000 jobs that I want to be handled by approx 100 threads. Once a thread finished, the free 'slot' should get a new job untill there are no more jobs available.
Side note: processor load is not an issue, these jobs are mostly waiting for results or (socket) timeouts. And the amount of 100 is something that I am going to play with to find an optimum. Each job will take between 2 seconds and 5 minutes. So I want to assign new jobs to free threads and not pre-assign all jobs to threads.
My problem is that I am not sure how to do this. Im primarily using Visual Basic .Net (but C# is also ok).
I tried to make an array of threads but since each job/thread also returns a value (it also takes 2 input vars), I used 'withevents' and found out that you cannot do that on an array... maybe a collection would work? But I also need a way to manage the threads and feed them new jobs... And all results should go back to the main-form (thread)...
I have it all running in one thread, but now I want to speed up.
And then I though: Actually this is a rather common problem. There is a bunch of work to be done that needs to be distributed over an amount of worker threads.... So thats why I am asking. Whats the most common solution here?
I tried to make it question as generic as possible, so lots of people with the same kind of problem can be helped with your reply. Thanks!
Edit:
What I want to do in more detail is the following. I currently have about 1200 connected sensors that I want to read from via sockets. First thing I want to know is if the device is online (can connect on ip:port) or not. After it connects it will be depending on the device type. The device type is known after connect and Some devices I just read back a sensor value. Other devices need calibration to be performed, taking up to 5 minutes with mostly wait times and some reading/setting of values. All via the socket. Some even have FTP that I need to download a file from, but that I do via socket to.
My problem: Lot's of waiting time, so lot's of possibility to do things paralel and speed it up hugely.
My starting point is a list of ip:port addresses and I want to end up with a file with that shows the results and the results are also shown on a textbox on the main form (next to a start/pause/stop button)
This was very helpfull:
Multi Threading with Return value : vb.net
It explains the concept of a BackgroundWorker which takes away a lot of the hassle. I am now trying to see where it will bring me.
I have 2 programs. 1 server and 1 client.
In the client it goes something like this:
while(statement)
{
networkstream.write(data);
}
And in the server it goes something like this:
while(statement)
{
while(statement)
{
ReceiveData;
}
Do other stuff;
}
So, while the client can write to the network stream really fast, the server still has to attend to the data before he can read some more.
What happens when the client has already made 4 laps of the loop containing the write, while the server has still only read 1 time for example.
Is there a way of letting the client know when he can make another write?
And also what happens when the client make several '.write'? does the server keep them all and reads them all or does the data that has been sent get overwriten?
Hopefully you can understand my question. Edit the question title if you desire.
There is a whole stack of layers between your .write and the actual wires. The layers do all kinds of stuff, from error correction to making sure that your network traffics does not collide with others etc.; non the least, they provide you with buffering and blocking (if you sent too much or - on the receiving side - there is no data yet).
To answer your question, somewhere along the line will be a buffer (a chunk of memory) where your bytes are written to. Until they are sent along to the next stop along the way, it will block (wait/hang...). These calls will travel down the stack, up on the receiving side, and then down the other side and up yours again. So when your write returns, you can be reasonably sure that all is well, unless you get back an error code, exception or however your error handling works.
If the server is slow when processing your requests, it will not get to read as quick as you'd like, so the buffers between you and them will fill up, at which point writing stops.
There are plenty of different buffers on different layers as well, a full answer would be pretty complex. https://en.wikipedia.org/wiki/OSI_model
I'm not sure how to approach this. I am hesitant about showing my code because it's a university assignment. I need some place to start.
I'm making a TCP card game with four players and a server. Every 100ms, a player asks for an update from the server using a background worker. The server accepts a client connection and reads in a Enumeration value (sent as an Int32) that tells it the action it wants server to send it (update, card, player, etc) and a value that is read in based on the Enumeration value (Recieving an Update Enumeration means it needs to read in a Int32 next). The server sends back a response based on the Enumeration read in.
Here's where the problem occurs. I have a custom built computer (2500K processor, Win8 x64) and when I execute the program on it, it will loop forever, accepting client requests and sending the appropriate response back. Exactly as expected! However, on my laptop (Levono YogaPad, Win8 x64) the back and forth exchange lasts for around 30-50 requests and then deadlocks. It's always at the same spot. The server has read in the Enumeration and is awaiting for the second value. The client is past the part of sending the enum and value and is waiting for the results. It is always stable on my desktop and always deadlocks on my laptop. I even slow the program down to update every second and it still deadlocks. I'm not sure what to do.
I've built the program on each computer. I've built the program on my desktop and ran it on my laptop and it still deadlocks. Does anyone have any suggestions?
Many thanks!
You are lucky that the code hangs on one machine before you send the assignment in and it hangs on your teachers machine. You are also lucky that the problem is reproducible, so better find out where exactly it hangs. Without having access to the code I have the following wild guesses:
you forgot to do proper error handling in all places and now it busy loops because of an unexpected error
it hangs inside a read where you try to read N bytes but the peer sends only K<N bytes
These are wild guesses, but without having access to even the basic structure of your program you cannot probably expect anything more.
I'm currently working on an application that communicates with an electronic device via the SerialPort.
This communication is done in a half-duplex fashion, where the application is the master and the device is the slave. The master needs to send a message to the device, and the device needs to respond before the next message is sent.
If the message doesn't receive a response, the application needs to resend it.
The content of the next message is dependent on the result of the current message. i.e. each new message has an incremented sequence number, and sometimes data for the next message is taken from the reponse of the current one.
To send messages I use an interface to System.IO.Ports.SerialPort. When messages are received a SerialDataReceivedEventHandler is fired.
What's the best way for me to manage this? Is there a pattern that I can base this on?
I have worked on something similar in the past. This is the basic structure of the messaging I have used:
Application: Send Command Seq #1 -->
<-- Device: Acknowledge command, seq #1 (including any command-specific response)
If the device didn't acknowledge within 1 second, the same command would be resent, same sequence number. This "retry" sequence would happen 3 times and then it would time out, and retire the command.
The sequence number from the application side would increment by one for every command it sent that was not a retry. It would loop back to sequence number 1 after hitting 99.
The application will have to keep track of its state based on what command(s) is "in-flight" between it and the device, and what kind of response it has received. It can identify the response to a specific command by the sequence number the device puts in its acknowledgement.
To keep it simple to transition from state to state you can make it so that there is only ever one active command and don't move on until that one has been ack'ed or timed out and retired.
Shane Wealti approach is right but I think what you might be really asking yourself, should you use threads? Even though it is a master/slave scenario, should you have a listening thread? When do you listen to the port and when do you send?
The simplest approach - no threads
You don't need to use threads in this scenario due to master/slave configuration, all you have is two functions.
SendCommand(char * bfr)
{
}
RecieveCommand(char * bfr)
{
}
SendCommand( txBfr );
RecieveData( rxBfr );
// process receive buffer, prepare new command
SendCommand( txbfr );
RecieveCommand( rxBfr );
// and so on
The approach is the most simple one and totally functional. However since there are no threads and say your RecieveData() times out in one second, you GUI will not be responsive in that second. Note that you are not listening to the port all the time but only when are expecting a reply.
I might edit this to add Comprehensive approach using threads later but don't have the time right now.
I am writing an application that uses OpenNETCF.IO.Serial (open source, see serial code here) for its serial communication on a Windows CE 6.0 device. This application is coded in C# in the Compact Framework 2.0. I do not believe the issue I am about to describe is specifically related to these details, but I may be proven to be wrong in that regard.
The issue I am having is that, seemingly randomly (read as: intermittent issue I cannot reliably duplicate yet), data will fail to transmit or be received until the device itself is rebooted. The Windows CE device communicates with a system that runs an entirely different application. Rebooting this other system and disconnecting/reconnecting communication cables does not appear to resolve this issue, only rebooting the Windows CE device.
The only sign of this issue occurring is a lack of a TxDone event from OpenNETCF firing (look for "TxDone();" in OpenNETCF.IO.Serial class), and no data being received, when I know for a fact that the connected system is sending data.
Any character value from 1 - 255 (0x01 - 0xFF) can be sent and received in our serial communication. Null values are discarded.
My serial settings are 38400 Baud, 8 data bits, no parity, 1 stop bit (38400, 8n1). I've set the input and output buffer sizes to 256 bytes. DataReceived event happens whenever we receive 1 or more characters, and transmission occurs when there's 1 or more bytes in the output buffer, since messages are of variable length.
No handshaking is used. Since this is RS422, only 4 signals are being used: RX+, RX-, TX+, TX-.
I receive a "DataReceived" event, I read all data from the input buffer and make my own buffer in my code to parse through it at my leisure outside of the DataReceived event. When I receive a command message, I send an quick acknowledgment message back. When the other system receives a command message from the Windows CE device, it will send a quick acknowledgment message back. Acknowledgment messages get no further replies since they're intended as a simple "Yep, got it."
In my code, I receive/transmit through multiple threads, so I use the lock keyword so I'm not transmitting multiple messages simultaneously on multiple threads. Double checking through code has shown that I am not getting hung up on any locks.
At this point, I am wondering if I am continuously missing something obvious about how serial communication works, such as if I need to set some variable or property, rather than just reading from an input buffer when not empty and writing to a transmit buffer.
Any insight, options to check, suggestions, ideas, and so on are welcome. This is something I've been wrestling with on my own for months, I hope that answers or comments I receive here can help in figuring out this issue. Thank you in advance.
Edit, 2/24/2011:
(1) I can only seem to recreate the error on boot up of the system that the Windows CE device is communicating with, and not every boot up. I also looked at the signals, common mode voltage fluctuates, but amplitude of the noise that occurs at system boot up seems unrelated to if the issue occurs or not, I've seen 25V peak-to-peak cause no issue, when 5V peak-to-peak the issue reoccurred).
Issue keeps sounding more and more hardware related, but I'm trying to figure out what can cause the symptoms I'm seeing, as none of the hardware actually appears to fail or shutdown, at least where I've been able to reach to measure signals. My apologies, but I will not be able to give any sort of part numbers of hardware parts, so please don't ask the components being used.
(2) As per #ctacke's suggestion, I ensured all transmits were going through the same location for maintainability, the thread safety I put in is essentially as follows:
lock(transmitLockObj)
{
try
{
comPort.Output = data;
}
[various catches and error handling for each]
}
(3) Getting UART OVERRUN errors, in a test where <10 bytes were being sent and received on about a 300msec time interval at 38400 Baud. Once it gets an error, it goes to the next loop iteration, and does NOT run ReadFile, and does NOT run TxDone event (or any other line checking procedures). Also, not only does closing and reopening the port do nothing to resolve this, rebooting the software while the device is still running doesn't do anything, either. Only a hardware reboot.
My DataReceived event is as follows:
try
{
byte[] input = comPort.Input; //set so Input gets FULL RX buffer
lock(bufferLockObj)
{
for (int i = 0; i < input.Length; i++)
{
_rxRawBuffer.Enqueue(input[i]);
//timer regularly checks this buffer and parses data elsewhere
//there, it is "lock(bufferLockObj){dataByte = _rxRawBuffer.Dequeue();}"
//so wait is kept short in DataReceived, while remaining safe
}
}
}
catch (Exception exc)
{
//[exception logging and handling]
//hasn't gotten here, so no point in showing
}
However, instantly after the WriteFile call did timed out the first time in the test was when I started getting UART OVERRUN errors. I honestly can't see my code causing a UART OVERRUN condition.
Thoughts? Hardware or software related, I'm checking everything I can think to check.
Everything sounds right, but your observations kind of show that they're not.
Since you've stated that you're sending from multiple threads, the first thing I'd do is put in some sort of mechanism for sending where all send requests come into one location before calling out to the serial object instance. Sure, you say that you've ensured you have thread safety, but serializing these calls through one location would help reinforce that (and make the code a bit more maintainable/extensible).
Next I'd probably add some temp handling in the Serial lib to specifically set an event or break in the debugger when you've done a Tx but the TxDone event doesn't fire within some bounding period. It's always possible that the Serial lib has a bug in it (trust me, the author of that code is far from infallible) where some race condition is getting by.
Thank you everyone who responded. We've found that this actually appears to be hardware-related. I'm afraid I can't give more information than this, but I thank everyone who contributed possible solutions or troubleshooting steps.