We Keep Coding

Asynchronous Processing of Data

At the minute I am trying to put together an asynchronous tcp server to receive data which I then want to process, extracting values and inserting to sql server.
The basic concept I thought would be best is once the data is received and confirmed as the entire message, the message should then be passed of to some sort of collection to await processing on a FIFO basis, which will parse the values and insert them to sql server. I suppose this is whats known as the consumer/producer pattern.
I have been doing some looking into the best collection / way of doing this and have so far seen the BlockingCollection,ConcurrentCollection and BufferBlock using async/await and i think this may be the way to go but to be honest im not sure.
The best example i have found is on Stephen Cleary's blog in particular this article,
http://blog.stephencleary.com/2012/11/async-producerconsumer-queue-using.html
My main reservations are that I in no way want to slow down or interrupt the receiving of messages which to me would suggest using the multiple producer/consumer example which can be seen at the above link, but what i want to know is;
Am i correct in this assumption or is there a more suitable way of doing this in my scenario.
And if im correct in my assumption could anyone suggest the best way of implementing this taking into consideration my use case.
Any and all help is much appreciated.

At the minute I am trying to put together an asynchronous tcp server to receive data which I then want to process, extracting values and inserting to sql server.
There's a common pitfall with this kind of scenario. It is usually wrong to report success back to the client when the work has yet to be done. Most of the time I've seen this design, it's because of an efficiency "requirement" self-imposed by the developer, not by the client or for technical reasons. So first, take a step back and make absolutely sure that you do want to return a "successful completion" message to the client when the operation has not actually completed yet.
If you are sure that's what you want to do, then there's another question you must ask: is it acceptable to lose requests? That is, after you tell the client that the operation successfully completed, will the system still be stable if the operation does not actually ever complete?
The answer to that question is usually "no." At that point, the most common architectural solution is to have an out-of-process reliable queue (such as an Azure queue or MSMQ), with an independent backend (such as an Azure worker role or Win32 service) that processes the queue messages. This definitely complicates the architecture, but it is a necessary complication if the system must return completion messages early and must not lose messages.
On the other hand, if losing messages is acceptable, then you can keep them in-memory. It is only in this case that you can use one of the in-memory producer/consumer types mentioned on my blog. This is a very rare situation, but it does happen from time to time.

In general, I would avoid using BlockingCollection and friends for this sort of work. Doing so encourages you to architect the entire system into a single process, which is the enemy of scalability and reliability.
I second Stephen Cleary's suggestion of using an out-of-process queue to manage the work. I disagree that this necessarily complicates the architecture, though - in fact, I think it can make things quite a bit simpler. Specifically, a major complication of the original requirement ("put together an asynchronous tcp server") disappears. Asynchronous TCP servers are a pain in the butt to write and easy to screw up - why not just skip that part altogether and be free to focus all of your energy on the post-processing code?
When I built a system like this, I used a Redis List as the task queue. Tasks were serialized to JSON, and clients would add their task to the queue with a RPUSH command. Worker processes retrieve the next task from the queue BLPOP, do their thing, then go back to waiting for the next task.
Advantages:
No locks. All synchronization comes for free from Redis (or whatever task queue you choose).
Everything in the system is single-threaded. Multi-threading is hard.
I'm free to spin up as many worker processes as I want, across as many nodes as I want.

Write custom events that can be used by 3rd party applications

Is it possible to write custom events that can be handled by 3rd party applications?
We have an existing app and we're finding that many people that use the app are using sql triggers to custom-write functionality of their own certain when things happen in our app.
This has led to some instances where our own processes have slowed down due to shoddy 3rd party Triggers that block our app.
I was thinking we could make this easier for 3rd party devs if we could raise events that they could handle in their own services or apps instead of having to use triggers.
That way we'd lose the blocking because we can just fire the event and continue. Also their slowness/potential crashes would happen outside of our process.
A) is this a reasonable approach?
B) Is this possible? Can I scope an event beyond the scope of my app?
EDIT
I have since found other related questions to be of interest:
wcf cross application communication
Interprocess pubsub without network dependency
Listen for events in another application (This seems very close to what I'm after)
I guess I'm looking for the simplest approach but if we wanted to adopt this method across a number of other apps within our company we'd have some further challenges:
We have some older apps in vb6 and delphi - from those I'd just like to be able to listen for their events in my (or 3rd party) newer C# apps or services.
For now, I'll look at:
Managed Spy and http://pubsub.codeplex.com

No, events are only usable by code that's loaded into your own process. If you don't trust these people now, you really don't want to expose yourself to shoddy code that you load into your own process and throws unhandled exceptions that terminate your app. You'll get the phone call, not them. Besides, they'll use such an event to run code that slows down your app.
In general, anything you do with a dbase will run with an entirely unpredictable amount of overhead. Not just because of triggers added by others, the dbase server could simply be bogged down by other work and naturally slow down over time as it stores more and more data. Make sure that doesn't make your app difficult or unpleasant to use, dbase operations typically must run on a worker thread or be done asynchronously with, say, BeginExecuteXxxx(). And make it obvious in your UI that progress is stalled by the dbase server, not by any code that you wrote. Saves you from having to do a lot of explaining.

What you're looking to do is basically to send messages to other processes. For this, you need some sort of IPC mechanism. Since it sounds like you'll have multiple listeners to each message, a mailslot is probably the best way. Unfortunately, .NET doesn't have built-in support for mailslots, so you'll have to use P/Invoke.
If you're looking for a built-in solution, then you could use named pipes, WCF, .NET Remoting, or bare TCP or UDP. With any of these, though, you'll have to loop through all of your listeners and send the message one at a time to each of them, which is not that big of a deal, but maintaining the separate connections is a little more of a hassle.
Note that with WCF and .NET Remoting, you're pretty much limiting your clients to using .NET as well. If your clients might be native or some other platform, then mailslots, named pipes, and TCP/IP are your best bet.

C# is there an easy way to get into TCP Socket Programming?

I've worked on a program that uses databases to send small messages from one PC to another. What I've done is put the database in a shared folder, have the program on the other PC connect to it (via a Path, no less), and there it is, a simple and easy way to get messages to and fro PCs on a network. Not the best option, but it's just homework, and the quick and dirty approach got me a grade.
But now the homework is done, and I'd like to improve upon what I did. The problem with the program is in the deployment stage. There are too many folders / installation paths and administrative / sharing issues regarding pathing directly to a database on a shared folder.
So the good folks here in stackoverflow advised me to try Socket Programming, which I think is a bit out of my league. But you never know...
Also, I'm aware of the difference between Sync and Async socket programming. One blocks, the other doesn't. The program I'm working on is a simple turn-based game, so I thought Synchronous might be good enough, since if it's not your turn, you really can't do anything. The issue however is that the program is treated as "not responding". I tried asynchronous, but ran into problems with threading, something I consider WAY out of my league.
Logically, the program is simple. One host, one client. Upon client connection, host sends data. Then client receives, send out its own data. And so on, until one player loses.
I'm sorry to say only .NET 2.0 is installed in my school. No WCF or anything. Also, it must be done in C# Windows Forms, so XNA is out.
So, I'd like to ask... is there an easy way to get into Socket Programming? Any guides / sample projects that can help? Pre-made codes that can be studied, and adapted?
Majority of the samples I found and adapted are chat applications, which I thought good enough, but making it modular simply breaks it.

The chat application examples you encountered should be enough. It is not clear to me what you refer to as "making it modular".
What you need is to design a protocol to be sent over the connection, an agreement of rules so to say, so that one knows what the other is talking about. So instead of sending plain text (chat) you can send the following:
0x03 (length of the message)
0x0A (move command in this fictional protocol)
0x02 (parameter 1 of the command, X coordinate in this case, it's all defined in the protocol design)
0x05 (parameter 2 of the command, Y coordinate in this case, it's all defined in the protocol design)
Now it's entirely up to you what happens after you received and interpreted the data. Personally I would go for the Async solution, since it leaves your program to do other stuff (graphics?). And it's more easily adaptable in code, in my experience.

I've made some classes which can be used to transport objects over a socket using the BinaryFormatter.
Here are some tests for my BinaryTransport class:
http://fadd.codeplex.com/SourceControl/changeset/view/67972#1055425
The actual class:
http://fadd.codeplex.com/SourceControl/changeset/view/67972#1054822
Do note that it's a while ago that I wrote them. I just noticed some small bugs. But either use them or just study the classes to learn more.

I remember when I started with socket communication in C# I tried to implement a simple chat program between a client and a server and then between multiple clients. Here is the tutorial that I was reading then: http://www.codeproject.com/KB/IP/TCPIPChat.aspx
If you want the full code I can upload my final project and you can study the code. It also uses multithreading so you can see how to handle this situation in GUI applications.
Side note: Wow, that database idea is the craziest thing I've seen in terms of PC-to-PC communication. Well done!

One interesting, useful and easy exercise you can do to learn about sockets (which C# makes it easier even) was creating a TCP-based logger.
During development every programmer needs a way to know what's happening under the hood at certain points. Without a logger you would normally write something like:
Console.WriteLine( "blah" );
which results in a dull, unfiltered, unorganized string thrown to the output window.
I created a TCP-based logger very easily using sockets. In one hand you have a separate Winforms application (the server), which is in charge of listening to incoming messages and beautifully displaying them on a rich-content control. In the other hand, you write a very simple class (the client) with a single function like:
public static class MyConsole
{
public static void WriteLine( string message, string whatever )
{
// send to the net
if( mTcpSocket.Connected )
mTcpSocket.Send( message );
// in case the server is not there we still have regular output
Console.WriteLine( message );
}
}
I created this logger once and have been using it ever since. Furthermore, given its tcp nature, with minor changes on the server side I've been successfully using it from different languages, as C# and Java, and now using it from ActionScript.

need advice for type of TCP server to cater for this type of application

The requirement of the TCP server:
receive from each client and send
result back to same client (the
server only do this)
require to cater for 100 clients
speed is an important factor, ie:
even at 100 client connections, it should not be laggy.
For now I have been using C# async method, but I find that I always encounter laggy at around 20 connections. By laggy I mean taking around almost 15-20 seconds to get the result. At around 5-10 connections, time to get result is almost immediate.
Actually when the tcp server got the message, it will interact with a dll which does some processing to return a result. Not exactly sure what is the workflow behind it but at small scale you do not see any problem, so I thought the problem might be with my TCP server.
Right now, I thinking of using a sync method. Doing so, I will have a while loop to block the accept method, and spawn a new thread for each client after accept. But at 100 connections, it is definitely overkill.
Chance upon IOCP, not exactly sure, but it seems to be like a connection pool, as the way it handles tcp is quite like the normal way.
For these TCP methods I am also not sure whether it is a better option to open and close connection each time message needs to be passed. On average, message are passed from each client at around 5-10 min interval.
Another alternative might be to use a web, (looking at generic handler) to form only 1 connection with the server. Any message that needs to be handled will be passed to this generic handler, which then sends and receive message from the server.
Need advice from especially those who did TCP in large scale. I do not have 100 PC for me to test out, so quite hard for me. Language wise C# or C++ will do, I'm more familar with C#, but will consider porting to C++ for the speed.

You must be doing it wrong. I personally wrote C# based servers that could handle 1000+ connections, sending more than 1 message per second, with <10ms response time, on commodity hardware.
If you have such high response times it must be your server process that is causing blocking. Perhaps contention on locks, perhaps plain bad code, perhaps blocking on external access leading to thread pool exhaustion. Unfortunately, there are plenty of ways to screw this up, and only few ways to get it right. There are good guidelines out there, starting with the fundamentals covered in Rick Vicik's High Performance Windows Programming articles, going over the SocketAsyncEventArgs example which covers the most performant way of writing socket apps in .Net since the advent of Socket Performance Enhancements in Version 3.5 and so on and so forth.
If you find yourself lost at the task ahead (as it seems you happen to be) I would urge you to embrace an established communication framework, perhaps WCF with a net binding, and use the declarative service model programming of WCF. This way you'll piggyback on the WCF performance. While this may not be enough for some, it will get you far enough, much further than you are right now for sure, with regard to performance.

I don't see why C# should be any worse than C++ in this situation - chances are that you've not yet hit upon the 'right way' to handle the incoming connections. Spawning off a separate thread for each client would certainly be a step in the right direction, assuming that workload for each thread is more I/O bound than CPU intensive. Whether you spawn off a thread per connection or use a thread pool to manage a number of threads is another matter - and something to determine through experimentation and also whilst considering whether 100 clients is your maximum!

Serial Comms programming structure in c# / net /

I'm an embedded programmer trying to do a little bit of coding for a communications app and need a quick start guide on the best / easiest way to do something.
I'm successfully sending serial data packets but need to impliment some form of send/ response protocol to avoid overflow on the target system and to ensure that the packet was received ok.
Right now - I have all the transmit code under a button click and it sends the whole lot without any control.
What's the best way to structure this code , i.e sending some packets - waiting for response .. sending more .. etc etc until it's all done, then carrying on with the main program.
I've not used threads or callbacks or suchlike in this environment before but will learn - I just need a pointer to the most straigtforward ways to do it.
Thanks
Rob

The .NET serialport uses buffers, learn to work with them.
Sending packets that are (far) smaller than the Send-buffer can be done w/o threading.
Receiving can be done by the DataReceived event but beware that that is called from another thread. You might as well start your own thread and use blocking reads from there.
The best approach depends on what your 'packets' and protocol look like.

I think to have a long experience about serial comm, both MCU and PC-based.
I strongly UNSUGGEST the single-thread based solution, although it is very straigthful for light-speed testing, but absolutely out for final releases.
Surely you may choose among several patterns, but they are mostly shaped around a dedicated thread for the comm process and a finite-state-machine to parse the protocol (during receiveing).
The prevoius answers give you an idea to how build a simple program, but it might depends on the protocol specification, target device, scope of the application, etc.

there are of course different ways.
I will describe a thread based and an async operation based way:
If you don't use threads, your app will block as long as the operation is performing. This is not what a user is expecting today. Since you are talking about a series of sending and receiveing commands, I would recommend starting the protocol as a thread and then waiting for it to finish. You might also place an Abort button if neccesary. Set the ReadTimeout values and at every receive be ready to catch the exception! An introducing into creating such a work thread is here
If you want to, use Async Send/Receive functions instead of a thread (e.g. NetworkStream.BeginRead etc.). But this is more difficult because you have to manage state between the calls: I recommend using a Finite State Machine then. In fact you create an enumeration (i.e. ProtocolState) and change the state whenever an operation has completed. You can then simply create a function that performs the next step of the protocol with a simple switch/case statement. Since you are working with a remote entity (in your case the serial target system), you always have to consider the device is not working or stops working during the protocol. Do this by starting a timeout timer (e.g. set to 2000ms) and start it after sending each command (assuming each command will get a reply in your protocol). Stop it if the command was received successfully or on timeout.

You could also implement low-level handshaking on the serial port; set the serial port's Handshake property to rts/cts or xon/xoff.
Otherwise (or in addition), use a background worker thread. For simple threads, I like a Monitor.Wait/Pulse mechanism for managing the thread.
I have some code that does read-only serial communications in a thread; email me and I'll be happy to send it to you.

I wasn't sure from your question if you were designing both the PC and embedded sides of the communication link, if you are you might find this SO question interesting.