I have a multithreaded application that spawns threads for several hardware instruments. Each thread is basically an infinite loop (for the lifetime of the application) that polls the hardware for new data, and activates an event (which passes the data) each time it collects something new. There is a single listener class that consolidates all these instruments, performs some calculations, and fires a new event with this calculation.
However, I'm wondering if, since there is a single listener, it would be better to expose an IEnumerable<> method off these instruments, and use a yield return to return the data, instead of firing events.
I'd like to see if anybody knows of differences in these two methods. In particular, I'm looking for the best reliability, best ability to pause/cancel operation, best for threading purposes, general safety, etc.
Also, with the second method is it possible to still run the IEnumerable loop on a separate thread? Many of these instruments are somewhat CPU-bound, so ensuring each one is on a different thread is vital.
This sounds like a very good use case for the Reactive Extensions. There's a little bit of a learning curve to it but in a nutshell, IObservable is the dual of IEnumerable. Where IEnumerable requires you to pull from it, IObservable pushes its values to the observer. Pretty much any time you need to block in your enumerator, it's a good sign you should reverse the pattern and use a push model. Events are one way to go but IObservable has much more flexibility since it's composable and thread-aware.
instrument.DataEvents
.Where(x => x.SomeProperty == something)
.BufferWithTime( TimeSpan.FromSeconds(1) )
.Subscribe( x => DoSomethingWith(x) );
In the above example, DoSomethingWith(x) will be called whenever the subject (instrument) produces a DataEvent that has a matching SomeProperty and it buffers the events into batches of 1 second duration.
There's plenty more you could do such as merging in the events produced by other subjects or directing the notifications onto the UI thread, etc. Unfortunately documentation is currently pretty weak but there's some good information on Matthew Podwysocki's blog. (Although his posts almost exclusively mention Reactive Extensions for JavaScript, it's pretty much all applicable to Reactive Extensions for .NET as well.)
It's a close call, but I think I'd stick to the event model in this case, with the main decider behing that future maintenance programmers are less likely to understand the yield concept. Also, yield means the code processing each hardware request is in the same thread as the code generating the requests for processing. That's bad, because it could mean your hardware has to wait on the consumer code.
And speaking of consumers, another option is a producer/consumer queue. Your instruments can all push into the same queue and your single listener can then pop from it do whatever from there.
There's a pretty fundamental difference, push vs pull. The pull model (yield) being the harder one to implement from the instrument interface view. Because you'll have to store data until the client code is ready to pull. When you push, the client may or may not store, as it deems necessary.
But most practical implementations in multi-threading scenarios need to deal with the overhead in the inevitable thread context switch that's required to present data. And that's often done with pull, using a thread-safe bounded queue.
Stephen Toub blogs about a blocking queue which implements IEnumerable as an infinite loop using the yield keyword. Your worker threads could enqueue new data points as they appear and the calculation thread could dequeue them using a foreach loop with blocking semantics.
I don't think there's much difference performance-wise between the event and yield approach. Yield is lazy evaluated, so it leaves an opportunity to signal the producing threads to stop. If your code is thoughtfully documented then maintenance ought to be a wash, too.
My preference is a third option, to use a callback method instead of an event (even though both involve delegates). Your producers invoke the callback each time they have data. Callbacks can return values, so your consumer can signal producers to stop or continue each time they check in with data.
This approach can give you places to optimize performance if you have a high volume of data. In your callback you lock on a neutral object and append incoming data to a collection. The runtime internally uses an ready queue on the lock object, so this can serve as your queuing point.
This lets you choose a collection, such as a List<T> with predefined capacity, that is O(1) for appending. You can also double-buffer your consumer, with your callback appending to the "left" buffer while you consolidate from the "right" one, and so forth. This minimizes the amount of producer blocking and associated missed data, which is handy for bursty data. You can also readily measure high-water marks and processing rates as you vary the number of threads.
Related
I've got an application where there are several threads that provide data, that needs to go through some heavy math. The math part needs a lot of initialization, afterwards it's pretty fast - as such I can't just spawn a thread every time I need to do the calculation, nor should every source thread have its own solver (there can be a LOT of such threads, beyond a certain point the memory requirements are obscene, and the overhead gets in the way or processing power).
I would like to use a following model: The data gathering and using threads would call to a single object, through one thread-safe interface function, like
public OutData DoMath(InData data) {...}
that would take care of the rest. This would involve finding a free worker thread (or waiting and blocking till one is available) passing by some means the data in a thread safe manner to one of the free worker threads, waiting (blocking) for it to do its job and gathering the result and returning it.
The worker thread(s) would then go into some sleep/blocked state, until a new input item would appear on its interface (or a command to clean up and die).
I know how to do this by means of various convoluted locks, queues and waits in a very horrible nasty way. I'm guessing there's a better, more elegant way.
My questions are:
Is this a good architecture for this?
Are there commonly used elegant means of doing this?
The target framework is .NET 4.5 or higher.
Thank you,
David
The math part needs a lot of initialization, afterwards it's pretty fast - as such I can't just spawn a thread every time I need to do the calculation, nor should every source thread have its own solver (there can be a LOT of such threads, beyond a certain point the memory requirements are obscene, and the overhead gets in the way or processing power).
Sounds like a pool of lazy-initialized items. You can use a basic BlockingCollection for this, but I recommend overriding the default queue-like behavior with a stack-like behavior to avoid initializing contexts you may not ever need.
I'll call the expensive-to-initialize type MathContext:
private static readonly BlockingColleciton<Lazy<MathContext>> Pool;
static Constructor()
{
Pool = new BlockingCollection<Lazy<MathContext>>(new ConcurrentStack<Lazy<MathContext>>());
for (int i = 0; i != 100; ++i) // or whatever you want your upper limit to be
Pool.Add(new Lazy<MathContext>());
}
This would involve finding a free worker thread (or waiting and blocking till one is available)
Actually, there's no point in using a worker thread here. Since your interface is synchronous, the calling thread can just do the work itself.
OutData DoMath(InData data)
{
// First, take a context from the pool.
var lazyContext = Pool.Take();
try
{
// Initialize the context if necessary.
var context = lazyContext.Value;
return ... // Do the actual work.
}
finally
{
// Ensure the context is returned to the pool.
Pool.Add(lazyContext);
}
}
I also think you should check out the TPL Dataflow library. It would require a bit of code restructuring, but it sounds like it may be a good fit for your problem domain.
Investigate Task Parallel Library. It has a set of methods for creating and managing threads. And such classes as ReaderWriterLock, ManualResetEvent
and their derivatives may help in synchronizing threads
Don't use locks. This problem sounds nice for a proper nearly lock free approach.
I think what you need to look into is the BlockingCollection. This class is a powerful collection for multiple consumers and producers. If you think about using it with Parallel.ForEach you may want to look into writing your own Partitioner to get some more performance out of it. Parallel contains a couple of very nice methods if you only need a couple of threads for a relatively short time. That sounds like something you need to do. There are also overloads that provide initialization and finalization methods for each spawned thread along with passing thread local variables from one stage of the function to the next. That may really help you.
The general tips apply here of cause too. Try to split up your application in as may small parts as possible. That usually clears things up nicely and the ways how to do things become clearer.
All in all from what you told about the problem at hand I do not think that you need a lot of blocking synchronization. The BlockingCollection is only blocking the consumer threads until new data is ready to be consumed. And the producer if you limit the size...
I can't think of anything beyond that out of the top of my head. This is a very general question and without some specific issues it is hard to help beyond that.
I still hope that helps.
You've pretty much described a thread pool - fortunately, there's quite a few simple APIs you can use for that. The simplest is probably
await Task.Run(() => DoMath(inData));
or just call Task.Run(() => DoMath(inData)).GetAwaiter().GetResult() if you don't mind blocking the requesting thread.
Instead of starting a whole new thread, it will simply borrow a thread from the .NET thread pool for the computation, and then return the result. Since you're doing almost pure CPU work, the thread pool will have only as much threads as you really need (that is, about the same (or double) amount as the number of CPU cores you have).
Using the await based version is a bit trickier - you need to ensure your whole call chain returns Tasks - but it has a major advantage in avoiding the need to keep the calling thread alive while you wait for the results to be done. And even better, if you make sure the original thread is also a thread-pool thread, you don't even need the Task.Run - the threads will be balanced automatically. Since you're only doing synchronous work anyway, this turns your whole problem into simply avoiding any manual new Thread, and using Task.Run(...) instead.
First, create a pool of N such "math service objects" that are heavy. Then, guard usage of that pool with a new SemaphoreSlim(N, N). Accessing those objects is then as easy as:
SemaphoreSlim sem = ...;
//...
await sem.WaitAsync();
var obj = TakeFromPool();
DoWork(obj);
Return(obj);
sem.Release();
You can vary this pattern in many ways. The core of it is the pool plus a semaphore that can be used to wait if the pool is empty at the time.
I am very excited about using Rx in production application; where I will be listening to incoming notification updates coming from different channel.
I will be writing Rx query on top this stream where I will throttling using .Window() operator. Subscriber (In my case it is ActionBlock) will process this data in blocking fashion; (i.e it will not spawn Task from ActionBlock). Keeping above in mind if data comes at much faster rate than what my Subscriber can consume then what will happen to incoming data. Does Rx query uses any buffer internally; will it get overflowed ?
The phenomenon you're referring to is called Back Pressure, and the Rx team is currently exploring different ways to handle this situation. One solution might be communicating back-pressure back to the Observable so that it might "slow down".
To alleviate back-pressure, you could use lossy operators such as Throttle or Sample.
Timothy's answer is mostly right, but it is possible to have back-pressure occur on a single thread. This can happen if you use asynchronous code. In that sense, back-pressure is related to synchronization and scheduling, not threading (recall that by default Rx is single threaded).
If you run into a scenario where events are being produced faster than they can be consumed, and you're not using a lossy operator to alleviate the back-pressure, those items are usually being scheduled/queued/buffered, which can lead to a lot of memory allocation.
Personally, this has not been an issue for me, since usually events are processed faster than they are yielded, or loss of events is simply not an option, and therefore the extra memory consumption is inevitable.
This is actually down to the implementation of individual operators, but the built-in ones will buffer on a per-subscription basis - so a slow consumer won't block other subscribers, unless of course subscribers are sharing threads.
On a related note, Rx doesn't always protect the Rx grammar; for example, it is your responsibility to ensure you don't make concurrent calls to OnNext on a Subject. You can use Observable.Synchronize() to fix this.
If the subscriber is processing on the same thread as the emitting observable, the data cannot come faster than the subscriber can consume.
IObservable<int> data = ...;
var subscription = data.Subscribe(n => Console.WriteLine(n));
In this example, each int that emits out of data will be written to the console before the next int is emitted.
If the subscription crosses threads, then the above doesn't hold.
If I have 1 thread for my MMORPG server running a async socket and async packet handler, and in that 1 thread I have a static World that contains all entities in the game.
Would there be any threading issues if say, the async packet handler recieves an Attack message, resulting in a search of the entities in the world to figure out the target.
At the same time the static World Proc method is increasing the size of the Dictionary containing the monster entities adding extra monsters that spawned.
If this is all on the same thread, will the server explode?
will the server explode?
Yes, you can run into problems ("explode") because the async stuff is running on a different thread (even though you didn't create that thread explicitly) and it might access a shared object (world) at the same time as your main thread. Many datastructures (including the Dictionary) are not designed for this scenario and might crash or return the wrong answer.
The typical approach is to use locks to protect your shared objects: take the lock before modifying it, do whatever modification, and then release the lock. This way, only one thread at a time accesses the world (and its dictionary) and so everything remains consistent. Explosion averted.
Another way would be to switch to a more synchronous form of networking, perhaps for example avoiding completion handlers and instead waiting to hear from each of the players, and then acting on the inputs. This can be done very simply, but the simple way has drawbacks: any one slow player can slow the whole thing down. So sadly you're probably going to have to deal with some complexity, one way or another.
If I give answer in one line. Server will explode. As network activity and game logic is in same thread. And as you have mentioned you will be needing high network usage.
I seriously like that if you have a look to F#. It has all the things that you needed. As far as I got it from question. And few things are like collection change, and async is by default in language. Even Nodejs it is also worth trying. But again it all depends on requirement. Though I try to explain few keywords that may help you to take decision.
Non-blocking : It means thread will not be block on events. It will not wait for function to wait for another function to execute. But that doesn't mean you can't block it. In any case it is a single thread.
Async: It is some what like that. But in C# 5 async comes with keyword so you don't have to do threading part of programming.
Parallel Processing: In game development parallel processing is important. Now, that you can do with multiple thread or just use TPL.
In the case of UI based (where there are many objects) game I highly recommended that you separate processing thread and UI thread to improve user experience. Else FPS will go down while you are processing data.
Let me know if any further information needed.
Server will not go down if you take little care of it.
If on the same thread, then no. If you are doing all the work mentioned on a single thread, then there's no issue. However, as stated, if you are accessing a "shared" object instance across threads, then yes, there will be an issue, and locking will be required (using a "lock(){...}" block).
As your user base increases, you will have to keep an eye on the number of threads generated, or event messages if using a non-blocking event model for incoming requests.
On a different, yet related note, keep an eye on this C# based MMO server (with scripting support): https://dreamspace.codeplex.com/ - it may become a big help to MMO game creators very soon (will support Construct 2 by default).
Everything that I read about sockets in .NET says that the asynchronous pattern gives better performance (especially with the new SocketAsyncEventArgs which saves on the allocation).
I think this makes sense if we're talking about a server with many client connections where its not possible to allocate one thread per connection. Then I can see the advantage of using the ThreadPool threads and getting async callbacks on them.
But in my app, I'm the client and I just need to listen to one server sending market tick data over one tcp connection. Right now, I create a single thread, set the priority to Highest, and call Socket.Receive() with it. My thread blocks on this call and wakes up once new data arrives.
If I were to switch this to an async pattern so that I get a callback when there's new data, I see two issues
The threadpool threads will have default priority so it seems they will be strictly worse than my own thread which has Highest priority.
I'll still have to send everything through a single thread at some point. Say that I get N callbacks at almost the same time on N different threadpool threads notifying me that there's new data. The N byte arrays that they deliver can't be processed on the threadpool threads because there's no guarantee that they represent N unique market data messages because TCP is stream based. I'll have to lock and put the bytes into an array anyway and signal some other thread that can process what's in the array. So I'm not sure what having N threadpool threads is buying me.
Am I thinking about this wrong? Is there a reason to use the Async patter in my specific case of one client connected to one server?
UPDATE:
So I think that I was mis-understanding the async pattern in (2) above. I would get a callback on one worker thread when there was data available. Then I would begin another async receive and get another callback, etc. I wouldn't get N callbacks at the same time.
The question still is the same though. Is there any reason that the callbacks would be better in my specific situation where I'm the client and only connected to one server.
The slowest part of your application will be the network communication. It's highly likely that you will make almost no difference to performance for a one thread, one connection client by tweaking things like this. The network communication itself will dwarf all other contributions to processing or context switching time.
Say that I get N callbacks at almost
the same time on N different
threadpool threads notifying me that
there's new data.
Why is that going to happen? If you have one socket, you Begin an operation on it to receive data, and you get exactly one callback when it's done. You then decide whether to do another operation. It sounds like you're overcomplicating it, though maybe I'm oversimplifying it with regard to what you're trying to do.
In summary, I'd say: pick the simplest programming model that gets you what you want; considering choices available in your scenario, they would be unlikely to make any noticeable difference to performance whichever one you go with. With the blocking model, you're "wasting" a thread that could be doing some real work, but hey... maybe you don't have any real work for it to do.
The number one rule of performance is only try to improve it when you have to.
I see you mention standards but never mention problems, if you are not having any, then you don't need to worry what the standards say.
"This class was specifically designed for network server applications that require high performance."
As I understand, you are a client here, having only a single connection.
Data on this connection arrives in order, consumed by a single thread.
You will probably loose performance if you instead receive small amounts on separate threads, just so that you can assemble them later in a serialized - and thus like single-threaded - manner.
Much Ado about Nothing.
You do not really need to speed this up, you probably cannot.
What you can do, however is to dispatch work units to other threads after you receive them.
You do not need SocketAsyncEventArgs for this. This might speed things up.
As always, measure & measure.
Also, just because you can, it does not mean you should.
If the performance is enough for the foreseeable future, why complicate matters?
What exactly do I need delegates, and threads for?
Delegates act as the logical (but safe) equivalent to function-pointers; they allow you to talk about an operation in an abstract way. The typical example of this is events, but I'm going to use a more "functional programming" example: searching in a list:
List<Person> people = ...
Person fred = people.Find( x => x.Name == "Fred");
Console.WriteLine(fred.Id);
The "lambda" here is essentially an instance of a delegate - a delegate of type Predicate<Person> - i.e. "given a person, is something true or false". Using delegates allows very flexible code - i.e. the List<T>.Find method can find all sorts of things based on the delegate that the caller passes in.
In this way, they act largely like a 1-method interface - but much more succinctly.
Delegates: Basically, a delegate is a method to reference a method. It's like a pointer to a method which you can set it to different methods that match its signature and use it to pass the reference to that method around.
Thread is a sequentual stream of instructions that execute one after another to complete a computation. You can have different threads running simultaneously to accomplish a specific task. A thread runs on a single logical processor.
Delegates are used to add methods to events dynamically.
Threads run inside of processes, and allow you to run 2 or more tasks at once that share resources.
I'd suggest have a search on these terms, there is plenty of information out there. They are pretty fundamental concepts, wiki is a high level place to start:
http://en.wikipedia.org/wiki/Thread_(computer_science)
http://en.wikipedia.org/wiki/C_Sharp_(programming_language)
Concrete examples always help me so here is one for threads. Consider your web server. As requests arrive at the server, they are sent to the Web Server process for handling. It could handle each as it arrives, fully processing the request and producing the page before turning to the next one. But consider just how much of the processing takes place at hard drive speeds (rather than CPU speeds) as the requested page is pulled from the disk (or data is pulled from the database) before the response can be fulfilled.
By pulling threads from a thread pool and giving each request its own thread, we can take care of the non-disk needs for hundreds of requests before the disk has returned data for the first one. This will permit a degree of virtual parallelism that can significantly enhance performance. Keep in mind that there is a lot more to Web Server performance but this should give you a concrete model for how threading can be useful.
They are useful for the same reason high-level languages are useful. You don't need them for anything, since really they are just abstractions over what is really happening. They do make things significantly easier and faster to program or understand.
Marc Gravell provided a nice answer for 'what is a delegate.'
Andrew Troelsen defines a thread as
...a path of execution within a process. "Pro C# 2008 and the .NET 3.5 Platform," APress.
All processes that are run on your system have at least one thread. Let's call it the main thread. You can create additional threads for any variety of reasons, but the clearest example for illustrating the purpose of threads is printing.
Let's say you open your favorite word processing application (WPA), type a few lines, and then want to print those lines. If your WPA uses the main thread to print the document, the WPA's user interface will be 'frozen' until the printing is finished. This is because the main thread has to print the lines before it can process any user interface events, i.e., button clicks, mouse movements, etc. It's as if the code were written like this:
do
{
ProcessUserInterfaceEvents();
PrintDocument();
} while (true);
Clearly, this is not what users want. Users want the user interface to be responsive while the document is being printed.
The answer, of course, is to print the lines in a second thread. In this way, the user interface can focus on processing user interface events while the secondary thread focuses on printing the lines.
The illusion is that both tasks happen simultaneously. On a single processor machine, this cannot be true since the processor can only execute one thread at a time. However, switching between the threads happens so fast that the illusion is usually maintained. On a multi-processor (or mulit-core) machine, this can be literally true since the main thread can run on one processor while the secondary thread runs on another processor.
In .NET, threading is a breeze. You can utilize the System.Threading.ThreadPool class, use asynchronous delegates, or create your own System.Threading.Thread objects.
If you are new to threading, I would throw out two cautions.
First, you can actually hurt your application's performance if you choose the wrong threading model. Be careful to avoid using too many threads or trying to thread things that should really happen sequentially.
Second (and more importantly), be aware that if you share data between threads, you will likely need to sychronize access to that shared data, e.g., using the lock keyword in C#. There is a wealth of information on this topic available online, so I won't repeat it here. Just be aware that you can run into intermittent, not-always-repeatable bugs if you do not do this carefully.
Your question is to vague...
But you probably just want to know how to use them in order to have a window, a time consuming process running and a progress bar...
So create a thread to do the time consuming process and use the delegates to increase the progress bar! :)