I need to implement a persistent queue in my C# service. It polls data from an external source. After it received a unit of data, it shall send it to a server. If the sending fails, it shall write it to disk in a queue-manner and try to resend it with an interval but also continue to poll the data and thus keep fill up the queue. I need to save it to disk because the network can fail and meanwhile the server can be shutdown. Resulting in a restart of the service and thus in-memory queue deleted. (Of course no polling will be made during reboot but the data during the network failure before will be lost).
I have now solved this problem by implementing a queue in Sql CE. After it polls the data, it directly writes it to the sql ce database, another thread then reads (peek) the database and tries to send the data. If it managed to send it, the message gets dequeued. I feel this solution is quite heavy and not very efficient.
Do anyone have experience with similar scenario and tips of how to implement it in a better way?
Related
I am currently self-hosting a SignalR server in a WPF Application. In this application I need to call a method on the client at least 500 times per seconds. Right now I'm calling the method on the client side on each change. The CPU on the client side is way to high. The object I'm transferring contains about 20 base properties.
My requirement is that I cannot lose or skip any messages. But I can send the notifications as a list in bursts each second.
I'm not sure which is going to perform the best: short and fast or long and rare.
I would buffer the information serverside (only storing changes) and wait for the client to ask for new data. Each time the client asks for data, the server send the information in one packet (binary). When the client finished processing the data, it asks for new data. This way you prevent writing too much data on the socket so it doesn't block. The length of the queue is an indication of the transport/processing of the client. When the queue grows too large (serverside) your network isn't fast enough or your client can't process that much data.
I use this method on my software which is connected to a PLC sending current states of robot angles/positions
Another way is using UDP, but it is lossy, so not very usefull in your situation.
At the moment i am putting together an asynchronous tcp server, everything seems to be coming together but im now at the stage where i need to figure out what to do with the data once received (I should mention that it will be used for receiving data primarily and will possibly never send anything to the clients).
As it is written asynchronously i don't particularly want to do any processing of the data in the server application itself (in the handler in which the data is received) to ensure that it will perform as optimally as possible, though eventually the data needs to processed and submitted to various sql tables to be of some use.
As part of a previously asked question here on SO
Asynchronous Processing of Data
Stephen Clearly had pointed out that to ensure no messages are lost due to power failure, system failure etc i should look into some kind of message queue.
In doing so i have seen various ways of doing this, one of which being using SQL server as the host to the queue.
What im wondering is using the SQL Service Broker and a Queue going to be any quicker than doing a normal insert to a table which contains only a UID, The Data (byte array no bigger than 1024 bytes) and a processed flag? And if not what is the fastest insert to use in C#
The processing of said data will probably take place in another application on the same server which will also receive the data and host the sql server if it makes any difference.
Any advice or thoughts will be much appreciated!
If you can afford the license fee I would recommend NServiceBus. If you can't afford it, consider MassTransit. Both of which will manage the message queue for you. They both support multiple queue types such as:
MSMQ
RabbitMQ
ActiveMQ
Azure
Implementing your own queuing system in SQL Server is a poor choice in the long run. Been there, got the t-shirt.
I have a windows service written in C# that reads from MSMQ and based on the type of the message it assigns them to Agents that process that message in a worker thread. The application starts with no agents and are created dynamically at runtime as messages arrive in the MSMQ
Here is a basic figure of how it works:
If the agent worker thread is busy doing work the message is queued to its local queue. So far so good. But if for some reason if the service is stopped, the local queue content is lost.
I am trying to figure out what could be the best way to handle this scenario. Right now the local queues are a System.Concurrent.ConcurrentQueue. I could probably use a Sql Ce db or some other persistent storage, but i am worried about performance. The other thing in my mind is to read from MSMQ only when agents are ready to process message, but the problem is that I don't know what message the MSMQ will contain.
What possible approaches can I take on this issue?
Your design is basically implements the following pattern: http://www.eaipatterns.com/MessageDispatcher.html
However, rather than using actual messaging you are choosing to implement the dispatcher in multithreaded code.
Rather, each processing agent should be an autonomous process with it's own physical message queue. This is what will provide message durability in case of failure. It also allows you to scale simply by hosting more instances of the processing agent.
I have built a similar system dependent on Redis. The idea is that it provides memory - fast data access isolated from the rest of the application, and will not shut down when my service does. Furthermore, it will eventually persist my data to the disk, so I get a good compromise between reliability and speed.
If you designed it so that each client read from its own message queue that would be hosted in Redis, you could keep the queue independent from the service's downtime, and each worker's load apportioned when you next start the service.
Why don't you simply create two new msms queues to receive the messages for Agenta and agentb, and create a new agent that ( transactionally ) fetch the command from the main queue and dispatch the message to the proper agent queue ?
I have a LAN composed of 3 PC. Installed in PC1 is the MS SQL database. This computer will act as the server.
The PC2 and PC3 will each have a desktop application that will display the data from PC1.
My problem here is how to make each PC (PC2 and PC3) have the same copy of data.
Suppose in PC2 employee 0001 first name is updated from John to Peter and commit save. Without refreshing the application in PC3, employee 0001 will still have John for the first name.
What would be the best approach for this? My level in programming is not that good, but I'm open to all suggestions/concepts/example/etc..
Thanks.
If you want immediate update on all clients right after data changes then you need some sort of notification system either in a polled or pushed manner.
You can implement push mechanism using for example WCF with callback contract. Your client PCs would need to implement relevant callback interface and be constantly connected to server PC's WCF service. Callback call could actually carry the new data. Each client needs to filter out notifications which resulted from that client's own changes. Push mechanism is quick and efficient way.
Check this stackoverflow answer for example of WCF callback.
Pull mechanism would require a background thread on all client applications checking the server for changes. You can use a separate database table with a version counter that would get incremented each time anything changes on the server. Client applications would poll that counter, compare with latest version they have and update the data when new version is discovered. It is much less effective mechanism though as you need to do the polling frequently and get all the data each time there is a new version. You can make versioning more sophisticated and detect what exactly changed but that can get complicated quickly with multiple clients. Overall it does not scale very well. It is generally simpler than push though and for simple applications with not too much data it would be enough.
You need to tell the other machines when to update. This could be accomplished by simple messages sent over the network using UDP broadcast. Then the other PC could execute its refresh method.
well utivich...this is the same thing as a web application really. it's a common problem. usually the other clients will have stale data until the record is reloaded, or when they save maybe the server will throw an exception on stale data based on sql timestamp. however, with a desktop application you can setup a system with event notification just like a chat application where the server pushes events to subscribers and the clients will be able to update the record or whatever you need to do.
I'm trying to build a simple multithreaded tcp server. The client connects and sends data to server, the server responds and waits for data again. The problem is I need the server to listen for incoming data in separate thread and be able to send command to client any time (for example to notify about new update). As far as I understood, when ever client sends data to server, if server doesn't respond with any data, client app doesn't let me send more data, server simply doesn't receive them. If I send data ether way around, does the data need to be 'acknowledged' for tcpclient?
Here's the source for the server: http://csharp.net-informations.com/communications/files/print/csharp-multi-threaded-server-socket_print.htm
How can I make the server send command to a client in separate thread outside the "DoChat" functions loop? or do I have to handle everything in that thread? Do I have to respond to each request client sends me? Thanks!
The problem is I need the server to listen for incoming data in separate thread
No, there is an async API. You can polll a list of threads to see which ahve new data waiting, obcviously to be done froa worker thread.
As far as I understood, when ever client sends data to server, if server doesn't respond with any
data, client app doesn't let me send more data, server simply doesn't receive them.
That is a lot more crap programming than the way sockets work. Sockets are totally ok with streaming ata in sending and receiving direction att the same time.
How can I make the server send command to a client in separate thread outside the "DoChat"
functions
Wel, me diong your job costs money.
BUT: The example is retarded. As in- totally anti pattern. One thread per client? You will run into memroy problems and perforamnce problems once 1000+ clients connect. You get tons of context switches.
Second, the client is not async because it is not written so. Mayy I suggest giong to the documentation, reading up on sockts an trying to build that yourself? THEN come back with questions that show more than "i just try to copy paste".
With proper programming this is totally normal. I have a similar application in development, sending data lall the time to the client and getting commands from the client to modify the data stream. Works liek a charm.
If I send data ether way around, does the data need to be 'acknowledged' for tcpclient?
Yes and no. No, not for TCP - TCP does it'Äs wn handshake under the hoods. Yes, if your protocol decides it has to, which is a programmer level design decision. It may or may not be necesssary, depending on the content of the data. Sometimes the acknowledgement provides more information (timestamp server side, tracking numer) and is not pure ly there for "I got it".