I am working on a project where I am to extract information continually from multiple servers (fewer than 1000) and write most of the information into a database. I've narrowed down my choices to 2:
Edit: This is a client, so I will be generating the connections and requesting information periodically.
1 - Using the asynchronous approach, create N sockets to poll, decide whether the information will be written into the database on the callback and put the useful information into a buffer. Then write the information from the buffer using a timer.
2 - Using the multithreading approach, create N threads with one socket per thread. The buffer of the useful information would remain on the main thread and so would the cyclic writing.
Both options use in fact multiple threads, only the second one seems to add an extra difficulty of creating each of the threads manually. Are there any merits to it? Is the writing by using a timer wise?
With 1000 connections async IO is usually a good idea because it does not block threads while the IO is in progress. (It does not even use a background thread to wait.) That makes (1) the better alternative.
It is not clear from the question what you would need a timer for. Maybe for buffering writes? That would be valid but it seems to have nothing to do with the question.
Polling has no place in a modern async IO application. The system calls your callback (or completes your IO Task) when it is done. The callback is queued to the thread-pool. This allows you to not worry about that. It just happens.
The code that reads data should look like this:
while (true) {
var msg = await ReadMessageAsync(socket);
if (msg == null) break;
await WriteDataAsync(msg);
}
Very simple. No blocking of threads. No callbacks.
In answer to the "is using a timer wise" question, perhaps it is better to make your buffer autoflush when it reaches either a certain time, or a certain size. This is the way the in-memory cache works in the .NET framework. The cache is set to both a maximum size and a maximum stale-ness.
Resiliancy on failure might be a concern, as well as the possibility that peak loads might blow your buffer if its an in-memory one. You might consider making your buffer local but persistent - for instance using a MSMQ or similar high speed queue technology. I've seen this done successfully, especially if you make the buffer write async (i.e. "fire and forget") it has almost no impact on the ability to service the input queue, and allows the database population code to pull from the persistent buffer(s) whenever it needs to or whenever prompted to.
Another option is to have a dedicated thread whose only job is to service the buffer and write data to the database as fast as it can. So when you make a connection and get data, that data is placed in the buffer. But you have one thread that's always looking at the buffer and writing data to the database as it comes in from the other connections.
Create the buffer as a BlockingCollection< T >. Use asynchronous requests as suggested in a previous answer. And have a single dedicated thread that reads the data and writes it to the database:
BlockingCollection<DataType> _theQueue = new BlockingCollection<DataType>(MaxBufferSize);
// add data with
_theQueue.Add(Dataitem);
// service the queue with a simple loop
foreach (var dataItem in _theQueue.GetConsumingEnumerable())
{
// write dataItem to the database
}
When you want to shut down (i.e. no more data is being read from the servers), you mark the queue as complete for adding. The consumer thread will then empty the queue, note that it's marked as complete for adding, and the loop will exit.
// mark the queue as complete for adding
_theQueue.CompleteAdding();
You need to make the buffer large enough to handle bursts of information.
If writing one record at a time to the database isn't fast enough, you can modify the consumer loop to fill its own internal buffer with some number of records (10? 100? 1000?), and write them to the database all in one shot. How you do that will depend of course on your server. But you should be able to come up with some form of bulk insert that will reduce the number of round trips you make to the database.
For option (1) you could write qualifying information to a queue and then listen on the queue with your database writer. This will allow your database some breathing space during peak loads and avoid the requests backing up waiting for a timer.
A persistent queue would give you some resilience too.
Related
so im writing a udp server and client for the first time for a 1v1's game.
My idea is to have the server handling first connections made and creating a new thread every time 2 new players connect to handle all communication between them.
A typical client message would have the threadIndex (i have an array of threads), playerId (which player it came from) and whatever they need to be done.
Is it possible to receive the packet on all threads and analyze if its meant for them? Would this be efficient? How should i approach this?
The suitable approach depends of nature of server tasks, but creating a new thread for every pair of players is not the best idea probably. Basically lets imagine, that your server mostly performs:
I/O bound tasks. In other words most of time it waits for some I\O
opertiton - network respond, query to database or disk operation. In
this case you probably need asynchorous model, when all your
connections are handled in the same thread. It would be efficient
because you actually don't have much to do in your own code. I suppose
you more likely have kinda I/O bound tasks. For example you just need to route messages between players and push\pull some data from DB. All routed messages will have an Id of the game(between to plyers), so you will never miss any of them, and they won't be missent. Take a look on this video to see the ideas and goals of asynchronous approach.
CPU bound tasks. Here server must compute something, perform heavy algorithms or process huge amount of data. In this case you probably need multithreading, but again thread per players pair may not be the most suitable approach, because it is not well scaleable and eats too much resourses. If you have some heavy CPU tasks, try to hanlde them in queue with a set of background workers. And then push the messages in asynchronous manner. Take a look on producer-consumer implementation with BlockingCollection.
You may have a combination of two cases, and of cource you can combine the approaches above. Also see questions 1, 2, 3. Try and return with specific questions. Hope it helps.
I'm playing with SocketAsyncEventArgs and IO Completion Ports.
I've been looking but I can't seem to find how .NET handles race conditions.
Need clarification on this stack overflow question:
https://stackoverflow.com/a/28690948/855421
As a side note, don't forget that your request might have completed synchronously. Perhaps you're reading from a TCP stream in a while loop, 512 bytes at a time. If the socket buffer has enough data in it, multiple ReadAsyncs can return immediately without doing any thread switching at all. [emphasis mine]
For the sake of simplicity. Let's assume one client one server. The server is using a IOCP. If the client is a fast writer but server is a slow reader, does IOCP mean the kernel/underlying process can signal multiple threads?
1 So, socket reads 512 bytes, kernel signals a IOCP thread
2 Server processes new bytes
3 socket receives another X bytes but server is still processing previous buffer
Does the kernel spin up another thread? SocketAsyncEventArgs has a Buffer which by definition is: "Gets the data buffer to use with an asynchronous socket method." So the buffer should not change over the lifetime of the SocketAsyncEventArgs if I understand that correctly.
What's preventing SocketAsyncEventArgs.Buffer from getting corrupted by IOCP thread 2?
Or does the .NET framework synchronize IOCP threads? If so, what's the point of spinning up a new thread then if IOCP thread 1 blocks the previous read?
I've been looking but I can't seem to find how .NET handles race conditions.
For the most part, it doesn't. It's up to you to do that. But, it's not clear from your question that you really have a race condition problem.
You are asking about this text, in the other answer:
If the socket buffer has enough data in it, multiple ReadAsyncs can return immediately without doing any thread switching at all
First, to be clear: the method's name is ReceiveAsync(), not ReadAsync(). Other classes, like StreamReader and NetworkStream have ReadAsync() methods, and these methods have very little to do with what your question is about. Now, that clarified…
That quote is about the opposite of a race condition. The author of that text is warning you that, should you happen to call ReceiveAsync() on a socket that already has data ready to be read, the data will be read synchronously and the SocketAsyncEventArgs.Completed event will not be raised later. It will be the responsibility of the thread that called ReceiveAsync() to also process the data that was read.
All of this would happen in a single thread. There wouldn't be any race condition in that scenario.
Now, let's consider your "fast writer, slow reader" scenario. The worst that can happen there is that the first read, which could take place in any thread, does not complete immediately, but by the time the Completed event is raised, the writer has overrun the reader's pace. In this case, since part of handling the Completed event is likely to be calling ReceiveAsync() again, which now will return synchronously, an IOCP thread pool thread will get tied up looping on the calls to ReceiveAsync(). No new thread is needed, because the current IOCP thread is doing all the work synchronously. But it does prevent that thread from handling other IOCP events.
All that will mean though, is that if you have some other socket the server is handling and which also needs to call ReceiveAsync(), the framework will have to ensure there's another thread in the IOCP thread pool available to handle that I/O. But, that's a completely different socket and you would necessarily be using a completely different buffer for that socket anyway.
Again, no race condition.
Now, all that said, if you want to get really confused, it is possible to use asynchronous I/O in the .NET Socket API (whether with BeginReceive() or ReceiveAsync() or even wrapping the socket in a NetworkStream and using ReadAsync()) in such a way that you do have a race condition for a particular socket.
I hesitate to even mention it, because there's no evidence in your question this pertains to you at all, nor that you're even really interested in having this level of detail. Adding this explanation could just confuse things. But, for the sake of completeness…
It is possible to have issued more than one read operation on a socket at any given time. This would be somewhat akin to double- or triple-buffered video display (if you're familiar with that concept). The idea being that you might still be handling a read operation while new data comes in, and it would be more performant to have a new read operation already in progress to handle that data before you're done handling the current read operation.
This sounds great, but in practice because of the way Windows schedules threads, and in particular does not guarantee a particular ordering of thread scheduling, if you try to implement your code that way, you create the possibility that your code will see read operations completed out of order. That is, if you for example call ReceiveAsync() twice in a row (with two different SocketAsyncEventArgs objects and two different buffers, of course), your Completed event handler might get called with the second buffer first.
This isn't because the read operations themselves complete out of order. They don't. Hence the emphasis on "your" above. The problem is that while the IOCP threads handling the IO completions become runnable in the correct order (because the buffers are filled in the order you provided them by calling ReceiveAsync() multiple times), the second IOCP thread to become runnable could wind up being the first thread to actually be scheduled to run by Windows.
This is not hard to deal with. You just have to make sure that you track the buffer sequence as you issue the read operations, so that you can reassemble the buffers in the correct order later. All of the async options available provide a mechanism for you to include additional user state data (e.g. SocketAsyncEventArgs.UserToken), so you can use this to track the order of your buffers.
Again, this is not common. For most scenarios, a completely orderly implementation, where you only issue a new read operation after you're completely done with the current read operation, is completely sufficient. If you're worried at all about getting a multi-buffer read implementation correct, just don't bother. Stick with the simple approach.
Problem I'm tasked to resolve is (from my understanding) a typical producer/consumer problem. We have data incoming 24/7/365. The incoming data (call it raw data) is stored in a table and is unusable for the end user. We then select all raw data that has not been processed and start processing one by one. After each unit of data is processed, its stored in another table and is now ready to be consumed by the client application.
The process from loading the raw data till persisting processed data takes 2 - 5 seconds on average. But its highly dependent on the third party web services that we use to process the data. If the web services are slow, we are no longer processing data as fast as we're getting it in and accumulate backlog, hence causing our customers to loose live feed.
We want to make this process a multithreaded one. From my research I can see that the process can be divided into three discreet parts:
LOADING - A loader task (producer) that runs indefinitely and loads unprocessed data from DB to BlockingCollection<T> (or some other variation of a concurrent collection). My choice of BlockingCollection is due to the fact that it is designed with Producer/Consumer pattern in mind and offers GetConsumingEnumerable() method.
PROCESSING - Multiple consumers that consume data from the above BlockingCollection<T>. In its current implementation I have a Parallel.ForEach loop through GetConsumingEnumerable() that on each iteration starts a task with two task continuations: First step of the task is to call a third party web service, wait for the result and output the result for the second task to consume. Second task does calculations based on the first task's output and outputs the result for the third task, which basically just stores that result into the second BlockingCollection<T> (this one being an output collection). So my consumers are effectively producers too. Ideally each unit of data that has been loaded by the task 1 would be queued for processing in parallel.
PERSISTING - A single consumer runs against the second BlockingCollection mentioned above and persists processed data into database.
Problem I'm facing is the item number 2 from the list above. It does not seem to be fast enough (just by using Parallel.ForEach). I tried inside Parallel.ForEach instead of directly starting a task with continuation, start a wrapping thread that will in turn start the processing task. But this caused OutOfMemory exception, because thread count went out of control and reached 1200 very soon. I also tried scheduling work using ThreadPool with no avail.
Could you please advise if my approach is good enough for what we need done, or is there a better way of doing it?
If the bottleneck is some 3rd party service and this will not handle parallel execution but will queue your request then you cannot do a thing.
But first you can try this:
use the ThreadPool or Tasks (those will use ThreadPool too) - don't fire up Threads yourself
try to make your request async instead of using the thread exclusively
run your service/app through an performance profiler and check where you are "wasting" your time
make a spike/check for the 3rd party service and see how it handles parallel requests
think about caching the answers from this service (if possible)
That's all I can think of without further info right now.
I recently faced a problem which was very much similar to yours,
Here's what i did, hope it might help:
It seems like your 1st and 3rd part are rather simple, and can be
managed on their respective threads without any problem,
The 2nd part must firstly be started on a new thread, Then use System.Threading.timer, to make your web-service calls,
the method that calls the web-service passes the response(result) to the processing method by Invoking it asynchronously and letting it process the data at it's own pace,
this solved my problem, i hope it helps you too, if any doubts ask me, i'll explain it here...
We need to write some data into a file at about 100ms interval for about 30 minutes and then wait for a while, and repeat again. Our application is a C#, .net 3.5 application. The data is small for each write , less than 1MB. Right now we get a thread from the Threadpool and let that thread to write into the file each time when a new data is received (at about 100ms interval).
There is another way to do this, I think. We can get a thread from Threadpool at beginning and keep that thread running during the entire writing session. When that thread finished a write, let it wait for next signal to get updated data from a shared place and write again. The downside of this way is we need to synchroized the shared data object to make sure it will not be overwriten by the the new data if the writting is slower. then it may slow down the communication which the data is transferred from another system.
I don't have time to write code test them yet. do you think it is worthy to test them? or it is obviously one way is better than another?
You can use a producer/consumer pattern, so a thread can have the file locked all the time and you don't need to open and close the FileStream.
Here an example: http://www.yoda.arachsys.com/csharp/threads/deadlocks.shtml
(in the "More Monitor methods" section)
If you think that you might have contention writing to your file, then a single thread writer is a good idea. Otherwise, there is no real problem dispatching a single operation to a miscellanous thread imo.
Everything that I read about sockets in .NET says that the asynchronous pattern gives better performance (especially with the new SocketAsyncEventArgs which saves on the allocation).
I think this makes sense if we're talking about a server with many client connections where its not possible to allocate one thread per connection. Then I can see the advantage of using the ThreadPool threads and getting async callbacks on them.
But in my app, I'm the client and I just need to listen to one server sending market tick data over one tcp connection. Right now, I create a single thread, set the priority to Highest, and call Socket.Receive() with it. My thread blocks on this call and wakes up once new data arrives.
If I were to switch this to an async pattern so that I get a callback when there's new data, I see two issues
The threadpool threads will have default priority so it seems they will be strictly worse than my own thread which has Highest priority.
I'll still have to send everything through a single thread at some point. Say that I get N callbacks at almost the same time on N different threadpool threads notifying me that there's new data. The N byte arrays that they deliver can't be processed on the threadpool threads because there's no guarantee that they represent N unique market data messages because TCP is stream based. I'll have to lock and put the bytes into an array anyway and signal some other thread that can process what's in the array. So I'm not sure what having N threadpool threads is buying me.
Am I thinking about this wrong? Is there a reason to use the Async patter in my specific case of one client connected to one server?
UPDATE:
So I think that I was mis-understanding the async pattern in (2) above. I would get a callback on one worker thread when there was data available. Then I would begin another async receive and get another callback, etc. I wouldn't get N callbacks at the same time.
The question still is the same though. Is there any reason that the callbacks would be better in my specific situation where I'm the client and only connected to one server.
The slowest part of your application will be the network communication. It's highly likely that you will make almost no difference to performance for a one thread, one connection client by tweaking things like this. The network communication itself will dwarf all other contributions to processing or context switching time.
Say that I get N callbacks at almost
the same time on N different
threadpool threads notifying me that
there's new data.
Why is that going to happen? If you have one socket, you Begin an operation on it to receive data, and you get exactly one callback when it's done. You then decide whether to do another operation. It sounds like you're overcomplicating it, though maybe I'm oversimplifying it with regard to what you're trying to do.
In summary, I'd say: pick the simplest programming model that gets you what you want; considering choices available in your scenario, they would be unlikely to make any noticeable difference to performance whichever one you go with. With the blocking model, you're "wasting" a thread that could be doing some real work, but hey... maybe you don't have any real work for it to do.
The number one rule of performance is only try to improve it when you have to.
I see you mention standards but never mention problems, if you are not having any, then you don't need to worry what the standards say.
"This class was specifically designed for network server applications that require high performance."
As I understand, you are a client here, having only a single connection.
Data on this connection arrives in order, consumed by a single thread.
You will probably loose performance if you instead receive small amounts on separate threads, just so that you can assemble them later in a serialized - and thus like single-threaded - manner.
Much Ado about Nothing.
You do not really need to speed this up, you probably cannot.
What you can do, however is to dispatch work units to other threads after you receive them.
You do not need SocketAsyncEventArgs for this. This might speed things up.
As always, measure & measure.
Also, just because you can, it does not mean you should.
If the performance is enough for the foreseeable future, why complicate matters?