I am new to MSMQ . We have a situation where we have many c# applications(.exe) sending data packets on a port. They can send the data packets to a port . How can I configure MSMQ to automatically read from the port and post these data packets to MSMQ. We cannot afford to loose any of these data packets and so loss of data is not acceptable at all . How can this be done efficiently. Can MSMQ be configured to accept the data packets from port 1801 . If these applications we have wrote to different ports will that be better ? Are if they wrote to one port , how can these messages be sent to the MSMQ directly with NO DATA LOSS.
We want the queue to be transactional. We dont want to use the data packets that arrive.
Please help . Any sample code which can do this task will be of great help.
Please NOTE that the applications are written in C# .NEt. Data loss and slow performance has been a problem and we are trying to solve them using MSMQ. We must see all data packets from these applications in MSMQ.
Thanks
How can I configure MSMQ to automatically read from the port
unless you are prepared to write code I don't think this is possible. So on the assumption that you don't mind a little bit of code for yourself, check out the System.Messaging.Message class. The Body property allows you to send in a byte array.
So provided you are reading from the port into some kind of byte array, putting it into a message should be a doddle.
How can this be done efficiently. Can MSMQ be configured to accept the
data packets from port 1801.
If these applications we have wrote to different ports will that be
better?
1801 is already the default port for MSMQ TCP packets, see MSDN article TCP ports, UDP ports, and RPC ports that are used by Message Queuing.
If your port numbers are indeed important, you may configure the WCF service to listen on certain ports in its address part of the service configuration.
When working with MSMQ you normally do not have to deal with any special ports, except when configuring communication between multiple networks through firewalls (MSDN Article: How To Configure a Firewall for MSMQ Access ).
If possible, I would recommend to stick with the default values.
How can these messages be sent to the MSMQ directly with NO DATA LOSS.
We want the queue to be transactional.
If you use transactional queues you secure the message automatically against data loss. More on this in Queuing in WCF and Reliable Messaging with MSMQ and .NET
Data loss and slow performance has been a problem and we are trying to
solve them using MSMQ. We must see all data packets from these
applications in MSMQ.
MSMQ itself is fast and lightweight, but as with every other tool, it all depends on what you do with it - and how. If you start sending large data packets, MSMQ might get into trouble, too. Keep in mind that MSMQ has some limitations, e.g. for message size.
Here are articles/ tutorials to get you started with WCF in combination with MSMQ:
MSDN: MSMQ and WCF: Getting Started
Message Queuing to Windows Communication Foundation
A more complete scenario with secured communation: Creating a WCF Service with MSMQ Communication and Certificate
Security (Code Project)
Related
The project is to build a messaging mechanism between a Python and C# program via ZeroMQ .
I want messages to be able to travel in/out from both ends at any time, which is NOT a basic request-reply model a.k.a. REQ/REP.
One way I can think of is to build a PUB/SUB model on two ports, i.e. two one way channels.
Is there any method to get a real duplex channel?
There are several ways to do this with ZeroMQ. I suggest using a DEALER/ROUTER socket pair:
Choose one program to be the "server", the other the "client".
The server will bind a ROUTER socket on a port.
The client will connect a DEALER socket to the server's ROUTER port.
(Note: this implies that the client must know the server's IP and port in advance.)
At this point the client can send messages to the server, but the server has no way to send to the client.
The client sends a "HELLO" message to the server.
The server will receive a message that includes the client's address and the HELLO message. Now the server can send messages to the client using the client's address.
DEALER/ROUTER is considered an "advanced" socket pair in ZeroMQ. My description here is very high level. You'll really need to read the docs to get the most out of ZeroMQ.
Oh yes, Sir!
Use the PAIR-PAIR or even the XREQ-XREP ought make it.
The best next step is to carefully read the respective Scalable Formal Communication Pattern archetypes' access-points' API documentation, so as to cross-check, that all pieces of pre-wired behavioural logic meet your Project needs and voilá, harness them in your messaging setup and tune-up the settings so as to meet you performance and latency needs.
That is this simple ( thanks to all the genuine knowhow hidden in these builtins ).
Using for years this very sort of inter-platforms integration among Python + C/MQL4 and other computing nodes, so well worth one's time to learn the powers and strengths of ZeroMQ.
HTTP transfer has been available since MSMQ 3.0, however I'm afraid MassTransit doesn't offer the feature to use HTTP protocol as transport protocol between queues.
There's a very similar question about this here, which has not been completely answered.
Does anyone know if it's possible for a client to subscribe to a bus and send/receive messages through HTTP? Here's the architecture I'm willing to implement:
I'll have 2 computers in the local network
Computer A runs a server application and MassTransit.RuntimeServices
Computer B runs a client application which sends messages to A
I want the communication between them to be done via http.
I tried to change the address in UseSubscriptionService to http instead of msmq, but it doesn't work. If I set computer's A MSMQ service to Hardened Mode, the client application running on computer B get's a timeout while trying to subscribe to mt_subscriptions.
Bus.Initialize(sbc =>
{
sbc.UseMsmq();
sbc.VerifyMsmqConfiguration();
sbc.UseMulticastSubscriptionClient();
sbc.ReceiveFrom("msmq://localhost/test_queue_client");
sbc.UseSubscriptionService("msmq://m3-dev1/mt_subscriptions"); // maybe I could use http instead of msmq, but it doesn't work
});
Any clues on that??
No, MassTransit does not support HTTP for MSMQ. You could, in theory, add your own transport that supports that. RabbitMQ is a way better transport than MSMQ in every way except if you need to enroll in distributed transactions. And RabbitMQ only requires a single port open between the boxes.
Our approach for distributed integration is to have a custom web service contract with proper security. This defines an input port. The input port actually publishes messages to mass transit buses.
On the other side, the same contract is used to deliver messages to subscribers.
By having a custom contract and http/https transport we are independent on actual message bus in the middle. And this pays of, we were using another bus for like 2 years and 2 years ago we migrated to mass transit based bus without ANY changes to clients (publishers/subscribers).
I have the same dilemma as the one who posted this topic, Real-time communication with WCF
except that my problem is not about games programming. I would like to know what's the best method to use to be able to have a real time communication in between two windows applications (server-client). I am using visual c++/c# to date and i would like to be able to display all the Feeds that are being received by my server to the client in real time.
I have started trying to use .NET remoting but in my continuous research, it appears that it will use SOAP-http, and might affect the speed of the communication. My server and client will communicate using the internet and .NET remoting does not permit the use of TCP Channel when communicating in between a firewall or the internet.
Your inputs will be greatly appreciated.
I guess it depends on your escenario, if you want "real-time" and you are willing to lose some packages in the process you are better with UDP, take a video conferencing tool for example, by the time you recover your slow packages you will have to move and display the next frame in the video or audio; that is a good example for the use of UDP. This is the reason why UDP is much faster than TCP.
If however, you are not willing to lose a single bit of your message, then TCP was made for you because if you lost a package the protocol will request it again to have your complete message as complete as possible.
Additionally it depends on the way the communication is being sustained, is the information flowing from one to many?, from many to many?, one to tone?
Take NetNamedPipeBinding for instance, this will be much faster process, but is only deployed in a single machine but accross processes. Whereas NetMsmqBinding will help you to build queues and it will be amazingly reliable and scalable for scenarios where your load will be a massive number of connections.
In the end, it all boils down to your concrete escenario and your business goals.
Hope it helps
If you are willing to do your own message parsing, you can use standard TCP sockets with the TcpClient and TcpListener classes. If your data is already a serializable object, you could serialize it into a text stream and just send it over the socket, deserializing it on the client side.
To get it to work over the internet, the server needs to have the port forwarded on your router, the client would just attach to the server's public IP. You would obviously need to add an exception in your firewall for this port as well.
The biggest problem with WCF and large data is setting up the streaming, by default WCF sends everything at once, which isn't practical for large files.
I'm quite new to C# so please bear with me. I'm writing a relatively simple client server application in C# .NET 4.0. I am using TCP protocol, TCPListener and TCPClient to be more specific. I know how does TCP protocol work in theory. But I must be 100% sure that there will be no (unhandled) errors during data transfers.
After I send data how do I know whether data was successfully received. Can I totally rely on the underlying implementation of TCP protocol? So there is no need that I confirm from the other side that data was received?
It is crucial that I truly know which data was sent&successfully received.
I know it is a dumb question, but I really want to be sure. Thank you for your time and answers.
TCP guarantees that:
Data you send will arrive in the order you send it
Data you send will be received exactly as you sent it (unmodified)
No other (spurious) data will be received
It does not guarantee that rodents will not eat your cables, the power to the building will stay on, or even that the process on the other machine you are talking to will bother to do anything when the OS tells it that some data for it has arrived.
If you need a positive acknowledgement that data was received and acted upon, you need to send the acknowledgement back manually (since TCP connections are duplex, you already have a channel to do that).
Of course all of this does is not in any way specific to .NET, Windows, or any other implementation of a network stack.
Update: I 'd like to point out specifically that, after the OS network stack accepts data for transmission, there is no way for you to know that the process at the other end has received that data. The network stack knows in most cases that the data has reached the target (through TCP ACK messages), but it does not know if the OS on the target has fed them to the process they are destined for. So sending back your own "data received and acted upon" message is the only option.
With TCP/IP alone, you cannot determine whether any data is received - you would have to layer another protocol on top.
If you can find out if some particular data was received, then TCP/IP guarantees all data before that was also received.
There is no such thing as the '.NET TCP protocol'. There is just the TCP protocol, which has been running on the Internet for about 30 years. It's reliable. That was a design goal.
An option would be to use WCF Reliable Sessions over TCP. WCF reliable sessions will mask transport failures.
If you really need reliability AND network transport, why not use Message Queues? They have transactional guarantee (about as strong as the discipline of the developers working with it, but not stronger than that!)
It's like all your traffic with database-server-type safety (and ditto performance). I think you can easily configure .NET Remoting to use an MSMQ channel.
I've personally never done that, but I've used message queuing in general.
After sending some tcp data with the blocking/non-blocking methods such as:
Socket.Send() or Socket.SendAsync()
How can I know that my data has received an ACK message?
Can .NET know if TCP data has been successfully sent?
The only way to know for sure is to implement some kind of application-level acknowledgement. The TCP level "ACK" packet is not exposed to the application level at all, so you have to use something more than that.
You make the other end respond to it.
Even if TCP has Acked it, if the receiving end terminates (for good or bad reasons) before processing the message and acting on it, you still don't know, so the only way to know is for the other end to tell you.
This information isn't available from .net's class libraries. I had the same kind of considerations when I started working on this port scanner in C#. I have made use of a .NET wrapper for libpcap (after installing the corresponding driver), the SharpPcap (http://sourceforge.net/projects/sharppcap/), in order to get this kind of information. The ACK packets are obtained through SharpPcap's interface (invoking the native libpcap interface's transparently).
My application is NScanner Port Scanner/Sweeper and you can find the code at codeplex, referencing to you my simple usage of the aforementioned library (http://nscanner.codeplex.com/).
I hope I helped.
"I'm trying to focus on how can you know when your data has been accepted by the other-side of the connection."
I think you need to be aware what type of application layer protocol you are going to implement and what impact this has on application performance.
Take HTTP as an example of a "Streaming like" protocol. A server posts a stream of data to a client. There are no more additional application layer "ACKs" and the server doesn't actually care when and how exactly his stream of data arrives. This is very efficent on high latency links.
Now compare this to SMB! Instead of streaming a file, data is partitioned into blocks. Every successfully transferred block of data is acked on the application layer. This gives you more control, however, it effectively kills the protocol on WAN networks (check out "Bandwidth Delay Product").
Taking this into consideration, you can come up with your own design for your custom protocol.
The TCP layer will keep resending the packet until it receives a successful ACK.
Send will block until this happens - SendAsync will not block, and you can continue processing other stuff while the TCP layer handles sending the packet.
I recommend using Pcap.Net.
You can easily sniff packets using this library in C# and then easily check the packet values.
You can also easily build and send packets.
If you are really certain that you need to know the packet level details of your TCP connection, then in addition to creating the TCP socket for sending, you need your application to use the winpcap API to look at the raw traffic as well. You can install a filter to only receive packets relevant to the particular IP,port combination that determines your remote side.
There are a couple of projects out there creating .NET wrappers for libpcap, for example here