I want to send commands from one application (e.g. running on mobile device) to another application (e.g. running on embedded device) which is located in a different network.
I don't want to use VPN or something like port forwarding. So after some research I found some other ways to do that, for example via a cloud messaging service like Azure Service Bus.
Sending commands/messages from the first application to the service bus is not a problem for me. But I don't really understand how two get a connection from the cloud service to the second device? I know I can also send a message from the second device to a cloud service e.g. via HTTPS. And then the cloud service can keep that connection alive. As long as the connection is alive, I can send messages to the second device.
But there are some points I can't understand:
When I have thousands of devices, isn't that a problem to keep thousands of connections alive?
How can the second device listening the connection if there are new messages? Doesn't that needing too much ressources on the embedded device?
I also read about using "long polling" techniques and web sockets. I know too little to understand what are the advantages and disadvantages of those concepts. Which technique should I use for my problem?
To be more platform agnostic, I don't want to use services like Azure IoT Hub.
Edit:
Maybe I can use a web service and implement a MQTT Broker?
I think the mentioned MQTT Broker will get you there, especially as your usecase is exactly what MQTT and it's implementations (brokers and clients) have been built for.
The simplified story is the following:
A MQTT Client running on your Application 'publishes' a MQTT message using a 'topic' (think routing key) to the MQTT broker. A MQTT client running on your Devices have a subscription for the same 'topic' on the broker. This enables the broker to route the message from the application to the devices without the requirement that they know about each others.
As far as I understand your question your concerns are the following:
can all the devices be connected at the same time (thousands of open TCP connections) and therefore receiving messages published from your first application via the broker in 'realtime'.
assuming the devices will disconnect for whatever reasons, e.g. due to network problems or for decreasing energy consumption, how would be ensured that the devices will eventually receive the messages.
how will the devices connect to the broker.
Regarding 1. MQTT brokers are built to handle (and keeping) a massive amount of TCP connections. For example VerneMQ, a MQTT broker I can talk about, as I am one of the core devs, is able to handle over a million connections on one node (with proper server configuration it's actually mainly a matter of available RAM). However we'd only recommend such a setup if the devices are mainly sleeping. Using VerneMQ you can also add more nodes to the cluster and balance the connections among all your cluster nodes.
Regarding 2. A MQTT broker typically implements an offline storage for messages that haven't been send out to a client or haven't been acknowledged by a client. This allows your device to go offline for hours and receive the messages upon reconnect.
Regarding 3. This is specific to your usecase. In the simplest case you configure a fixed IP:Port on every device, and the MQTT client running on the device uses it to connect to the broker. Depending on the ability to reconfigure the devices it makes sense to use DNS lookups, or even to provide a 'backchannel' for reconfiguration.
For standard compliant MQTT client software have a look at Eclipse Paho. For an up-to-date list of available MQTT brokers consult the list of MQTT brokers.
Related
I am running an ASP.Net Web Forms application in an Azure Cloud. Now I have a special requirement that I have no idea how to solve: the server has to accept incoming connections over the internet from Android clients and then keep that connection open (or on standby) for a long period of time (days). So after hours or even days, the server has to find it's way to the Android client and send some data to it. I can use whatever technology works for both partners, but since the internet is in between, http/https is prefered as protocol (although WCF with TCP endpoints might also work when the connection is initiated by the client). The only thing I can think of is having the clients constantly poll the server if there is a command/data available, but that solution is ugly and wastes a lot of ressources. The client should react within seconds when the server has something to do for it, so I'd have to use a polling interval of ~10s. I know that http "keep alive" exists and can be utilizied in C# WebClients, but I doubt that it will work with my requirements? Is there any other possibility to achieve this?
Based on Niraj's response, I found Google Firebase Cloud Messaging. After that, I read up on the XMPP protocol and the SignalR technology available for .Net and finally decided to go with SignalR as it's implementation is most simple and does not require involvement of a 3rd party.
For a chat application, I use Azure architecture with SignalR, with the web-role acting as SignalR server (the messages are not broadcast type but are intended for specific user/client).
I want to scale out SignalR server along with the web-roles, to handle heavy user load. Although, SignalR documentation doesn't recommend to use the pre-baked SignalR scale out methods using backplane (Redis, Service bus) for such cases when the number of messages increase as more users are connected (or in user-event driven scenario). It explicitly states: "Client-to-client (e.g., chat): In this scenario, the backplane might be a bottleneck if the number of messages scales with the number of clients; that is, if the rate of messages grows proportionally as more clients join."
Question:
Does anyone know of any custom scale-out solution for such high-frequency case, which doesn't push messages to each server instance or some other scale-out solution?
Already looked everywhere in SignalR documentation and the related videos but couldn't find anything, other than a word "filtered-bus", which was not explained what it is and how it should be used.
I figured it out myself: Basic idea is server affinity/sticky sessions.
Each instance of web-role acts as a stand-alone SignalR server. At the first connection time, I let the Azure load balancer choose any instance of web-role and save the IP address of that web-role instance with the client identifier in a map. If there is another connect request coming from the same client (e.g after page refresh) then I check the IP address of the current role instance and if it matches the entry in map then I let it proceed otherwise I disconnect the client and connects it to the correct instance of web-role.
Each instance of worker-role also acts as SignalR .net client and connects to all the available SignalR servers (all instances of web-role). Before sending a message to the SignalR server (web-role), I look up in the map to determine the correct SignalR server instance (depending on the intended JS recipient).
Benefits:
There is no need of back-plane technology (and hence no delays in message delivery).
Each web-role instance care about the client connected to it and each message doesn't have to be duplicated on every SignalR server. Hence it can scale pretty good.
Easy to implement.
HTTP transfer has been available since MSMQ 3.0, however I'm afraid MassTransit doesn't offer the feature to use HTTP protocol as transport protocol between queues.
There's a very similar question about this here, which has not been completely answered.
Does anyone know if it's possible for a client to subscribe to a bus and send/receive messages through HTTP? Here's the architecture I'm willing to implement:
I'll have 2 computers in the local network
Computer A runs a server application and MassTransit.RuntimeServices
Computer B runs a client application which sends messages to A
I want the communication between them to be done via http.
I tried to change the address in UseSubscriptionService to http instead of msmq, but it doesn't work. If I set computer's A MSMQ service to Hardened Mode, the client application running on computer B get's a timeout while trying to subscribe to mt_subscriptions.
Bus.Initialize(sbc =>
{
sbc.UseMsmq();
sbc.VerifyMsmqConfiguration();
sbc.UseMulticastSubscriptionClient();
sbc.ReceiveFrom("msmq://localhost/test_queue_client");
sbc.UseSubscriptionService("msmq://m3-dev1/mt_subscriptions"); // maybe I could use http instead of msmq, but it doesn't work
});
Any clues on that??
No, MassTransit does not support HTTP for MSMQ. You could, in theory, add your own transport that supports that. RabbitMQ is a way better transport than MSMQ in every way except if you need to enroll in distributed transactions. And RabbitMQ only requires a single port open between the boxes.
Our approach for distributed integration is to have a custom web service contract with proper security. This defines an input port. The input port actually publishes messages to mass transit buses.
On the other side, the same contract is used to deliver messages to subscribers.
By having a custom contract and http/https transport we are independent on actual message bus in the middle. And this pays of, we were using another bus for like 2 years and 2 years ago we migrated to mass transit based bus without ANY changes to clients (publishers/subscribers).
I wrote messenger in c# with sockets, but i have little problem with ports. To clear comunication I have to open port on router which i use in my messenger. How to resolve this problem? Is method to automatic open default closed ports?
Thanks.
There are a couple things you can do.
The first is to change the programming of your application so that it uses the regular http port (80) for communication. This would allow your app to make outbound calls pretty much anywhere.
Alternatively you could use a high port number from 49152 through 65535. ( See Assigning TCP/IP Ports for In-House Application Use ).
However, depending on where you are deploying your application it is highly likely that all of those ports are blocked via firewalls; and neither will solve your problem
Most messenger type applications can't go direct due to firewall issues. For example, even if you use port 80, its likely the client machines have that port blocked for incoming TCP requests.
Instead they typically connect to a known public server. When one client wishes to connect to a different one, the server will route the message between the two clients. A very simplified look at this is: Client A sends a communication request to the server for Client B. Client B polls the server for messages, sees one and shows it on the desktop.
There are ways to keep the TCP connection alive between the clients and server in order to speed up communications; but that's the basics.
There are even ways for clients to directly talk to each other, when they determine that certain ports are open for communication or that proxy servers aren't going to interfere with the traffic. However, that's a little more advanced than a simple "answer" here can provide.
I'm developing a multiple client / multiple server program in C#, and before I got down to the nitty gritty, I was wondering if anyone has ever worked on a similar project and might be able to share their tips / ideas for implementation.
The servers will sit on many PCs, and listen for incoming connections from clients (Or should the Servers broadcast, and the clients listen?).
When a client starts, it should populate a list of potential server IP addresses automatically.
When a server closes, the client should remove that server from it's list.
When a new server starts, the clients should be notified and have it added to their list.
A server may also act as a client, and should be able to see itself, as well as all other servers.
A message sent from a client to the server, that affects the server, should broadcast the change to all connected clients.
Should my server be a Windows Service? What advantages/disadvantages does that present?
Any ideas on how I might go about getting started on this? I've been looking into UDP Multicast, and LAN Scans. I'm using C# and .NET 4.0
EDIT: Found this: http://code.google.com/p/lidgren-network-gen3/ Does anyone have any experience with it and can recommend/not recommend it?
I would suggest NetPeerTcpBinding WCF communications to create a Peer Mesh. Clients and Servers would all join a mesh using a Peer resolver. You can use PNRP or create a custom peer resolver (.Net actually provides you with an implementation called CustomPeerResolverService). See Peer To Peer Networking documentation.
Also you can implement a Discovery service using DiscoveryProxy. With a discovery service, services can announce their endpoints. The discovery service can then service find requests (see FindCriteria) to return endpoints that match the requests. This is referred to as Managed Discovery. Another mode is Ad Hoc Discovery. Each service will announce their endpoints via UDP and discovery clients will probe the network for these endpoints.
I have actually implemented a Managed Discovery service in combination with Peer 2 Peer WCF networking to provide a redundant mesh of discovery services that all share published service endpoints via P2P. Using Managed Discovery I have found performs far better as Ad Hoc Discovery using UDP probing is slower and has some limitations crossing some network boundaries while Managed Discovery leverages a centralized repository of announced service endpoints.
Either/both technologies I think can lead to your solution.
So is this effectively a peer to peer style network (almost like bittorrent), where all servers are clients, but not all clients are servers.
and the requirements are every client should hold a list of all other servers (which are, in turn, clients).
The problem lies in getting the server IPs to the clients in the first place. You can use a master server that has a fixed DNS to act as a kind of tracker, which all of the servers check in to, and the clients check periodically.
Another option (or an additional method) is to use a peer exchange style system, where each of the clients and servers use UDP broadcast packets over a local network to discover each other and then transfer the servers they know of, kind of like a routing protocol. However if the PCs are spread out over a non local network such as the internet, there's little chance that they will ever discover each other on their own, making this method only useful when used in conjunction with other methods of finding servers. Also, you will probably have to deal with router UPnP to allow clients to connect to each other through each others router NAT, so this method is probably too complex for the gains you get. (However, if you're just on a LAN, this is all you need!)
A third option (and again, this sounds a lot like torrent technology), is to use Distributed Hash Tables to store information about the IPs of your servers in the cloud, without having to rely on a central master server.
I have had a shot at a project like this before (a pure P2P, server-less messaging system), but could never get it to work. Without a huge amount of peers, or a master server to track all of the other servers, it is very difficult to reliably retrieve the IPs of all the servers.