We want to use Google pub/sub to consume messages. In rabbitMQ, whenever a message published, we were getting it and processing it. Our process operation takes 3-4 hours and because of that our consumers are windows services.
We dont want to use pub/sub pull because we dont want to poll. But Pub/sub push publishing to a web endpoint. Because of our long running process, we cannot use web app or web api. Is there any chance to consume pub/sub messages like in rabbitmq without requesting always and consuming when there is a message only.
Thanks
The Google Pub Sub technology does not requires that one continually explicitly poll the PubSub environment. If one is using the client libraries one can configure a callback function within the client application that is invoked when a new message is published to the topic against which the subscription has been taken.
Related
It seems to me that Azure Web PubSub is a combination of both Event Grid and Event Hub using web sockets, instead of HTTP. However, I'm not 100% sure and could not find any articles that have a direct comparison between the 3 services.
To answer your question let me explain all the three services one after another.
Azure Event Grid
Azure Event Grid is a complete event routing service actively running on top of Azure Service Fabric. It issues events from various services like Azure Storage Blobs to different handlers like Azure Functions. It is event driven publish-subscribe model ( reactive programming).
Azure Event Grid can connect to any application that you create, and the Events generated by the application can be pulled and published to different other destinations.
Azure Event Hub
Azure Event Hub is a data ingestion service that streams a huge count of messages from any source to provide an immediate response to business challenges. It streams millions of events per second from any source to build dynamic data pipelines and immediately respond to business challenges. Think it as multiple source big data streaming pipeline (telemetry data).
The difference between them is that Event Hubs are accepting only endpoints for the ingestion of data and they don’t provide a mechanism for sending data back to publishers. On the other hand, Event Grid sends HTTP requests to notify events that happen in publishers.
Azure Web PubSub
PubSub is the short form for Publish Subscribe. If you have ever used Azure SignalR or WebSockets before to send things between publisher and subscriber, then that is what Web PubSub also does. So Azure Web PubSub is a managed service for handling real-time communication with your application.
When you are using WebSockets, you have these long running connections between clients and app server but it gets tricky to scale them as you are handling long-running requests. This problem is solved by Web PubSub service, which comes in the middle of your clients and app server. Now your clients can talk to this server and you can do http in between Azure Web PubSub service and the App Server. So your app server remains http only, but all the hard WebSocket logic is handled by the service for you.
The Azure Web PubSub service is build on the same core fundamental platform as SignalR but the main difference between these two is Azure Web PubSub is purely serverless and in SingnalR you need to use SignalR client but with Web PubSub you can use any client and any language.
Lastly, I would suggest you to read these Choose between Azure messaging services and Azure Web PubSub documentation for more information.
I am trying to create C# desktop application that can receive Google Cloud Platform Pub/Sub message. I can't get it to work with Push approach(https://cloud.google.com/pubsub/docs/subscriber#push_pull). What I understand from documentation - when message is published Pub/Sub pushes HTTP request to the subscriber to an endpoint. Then endpoint sends back HTTP success status code.
This is how I create my Subscriber:
TopicName topicName = new TopicName("test-project-123", "myTopicName");
SubscriberServiceApiClient subscriberService = await SubscriberServiceApiClient.CreateAsync();
SubscriptionName subscriptionName = new SubscriptionName("test-project-123", "mySubscriberName");
subscriberService.CreateSubscription(subscriptionName, topicName, pushConfig: null, ackDeadlineSeconds: 60);
Here, I assume, pushConfig is where I would specify endpoint to which Pub/Sub should send request to which my program in return should send success status code. What I don't understand is how my desktop application will know that request is pushed to this endpoint and how exactly does it have to respond to it? I believe the answer should be simple, but web development is a totally new thing for me so this solution is something I find very difficult to figure out. Any ideas or code samples would be highly appreciated
To use Push, you would need to set up a service that hosts a URL. It's up to you to figure out how to connect your service to your desktop application. I think the easiest way to set up a service to accept Cloud Pub/Sub Pushes is to use Cloud Run, a GCP service for hosting services: https://cloud.google.com/run/docs/tutorials/pubsub
Another option is to use Cloud Functions to set up a Pub/Sub trigger: https://cloud.google.com/functions/docs/calling/pubsub
or Firebase Functions:
https://firebase.google.com/docs/functions/pubsub-events
In general, Cloud Pub/Sub might not be the right tool for delivering notifications to your desktop application. For that, you might want to consider Firebase Cloud Messaging: https://firebase.google.com/docs/cloud-messaging/
It is possible for you to use FCM end-to-end, or you could bridge messages from Cloud Pub/Sub into FCM.
I'm developing ASP.NET Web API services and placing these onto an Azure Service Bus queue to be processed. The Web API services are hosted on Azure.
I need to implement an application that listens for these messages and processes them when they are received.
I'd like this to be hosted on Azure but not sure of the best way to approach this.
Can you implement such a listener service and host it on Azure?
What is the best way to approach implementing such an application / service?
There are several things you can do.
You could use ASB's OnMessage API which allows you to register your callback and handle incoming messages with concurrency and auto-completion.
On Azure you have several options: Cloud Services (worker roles), Azure Web Jobs, Azure Functions (if your processing is fast, otherwise I'd not recommend it), Service Fabric (might be a bit of an overkill if system is small), and plain VMs if needs to be.
Warning about functions - if you do intense work, Functions are not ideal as you'll pay for time/memory you execute.
A couple options for workers that listen to a queue are:
Functions
Web Jobs
You can see an example of using a Function here: https://learn.microsoft.com/en-us/azure/azure-functions/functions-create-an-event-processing-function.
An example of using Web Jobs is here: https://learn.microsoft.com/en-us/azure/app-service-web/websites-dotnet-webjobs-sdk-service-bus.
Both allow you to create background jobs that consume messages from a queue. Both support Storage and Service Bus queues. The main difference is that Web Jobs require an App Service Plan with some amount of instances, while Functions can run on a Dynamic plan, which scales completely automatically.
You should note that Functions are not meant for really long-running jobs (more than 5-15 minutes), though neither are Web Jobs.
Why not trying to run a linux process (daemon) in docker.
I have the following message transport scenarios
Client -> Calls SignalR -> Calls NServiceBus -> Process Message internally -> Calls NServiceBus Gateway service with Result -> Calls SignalR Hub -> Updates the client with result.
In choosing whether to use SignalR vs. long polling, I need to know if SignalR is scaleable. So in doing my homework I came across SignalR on Azure Service Bus. The setup is done on the Global.asax application start.
Ultimately I need to be able to do this, from inside an NServiceBus handler:
var context = GlobalHost.ConnectionManager.GetHubContext<MyHub>();
context.Clients.Group(group).addMessage(message);
The question is if context will be jacked up, because I'm (potentially) calling it from another machine than the one the client was connected to?
Also what is the sharding schema that the SignalR implementation uses to seed the topics? I know I can configure it to use N-number of topics, but how is it actually determining which message goes to which topics and if it's relevant from an external caller PoV.
You should be able to use GlobalHost.ConnectionManager.GetHubContext in any application where you have registered ServiceBusMessageBus as your IMessageBus via SignalR's GlobalHost.DepenencyResolver. This is done for you if you call GlobalHost.DepenencyResolver.UseServiceBus(...) in the application.
If you do this, a message will be published to Azure Service Bus for each call to addMessage or any other hub method on the IHubContext returned from GetHubContext. If there are subscribed clients connected to other nodes in the web farm, the other nodes will pick up the message from Service Bus and relay it to the subscribed clients.
Which topic a message goes to should not be relevant from the PoV of an external caller. You can use multiple topics to improve throughput, but for most use cases one should be sufficient.
If you choose to use more than one topic, you can think about the topic a message goes to as being essentially random. The only thing that is guaranteed is that messages from the same sender will go to the same topic. This allows SignalR to keep messages from the same sender in order.
Caveat emptor: SignalR has not yet had an official release supporting scale out. The 1.1 version will be the first release to support scale out officially.
I'm trying to create a feedback system which all messages get posted to then published back to the correct subsystem. We are using queues quiet heavily and i want to make the subscriber code as clean as possible. I want to switch based off the message id i get into the feedback system and publish to its specific subscriber. i don't want to make a service for each subscriber to listen for messages.. i was thinking i could set up a queue for each subscriber and trigger to invoke a com+ component.. but i'm looking for a more modern way..
I was looking into NServiceBus but it seems i'd need to make a service/executable/webservice for each listening system ( its a little less work to make a C# dll and invoke a method) and i'm not sure if NServiceBus can handle dynamic endpoints based off a preloaded config ( loaded from a db ). WCF is also a choice.. it can handle dynamic endpoints for sure..
what do you think is the best solution for the lease amount of code/ scalable for new systems to subscribe?
Thanks
In case you are ok with online solutions you could take a look at the latest .NET Services SDK for Windows Azure which has queue service bus http://www.microsoft.com/azure/netservices.mspx It relies on WCF messages and supports routing etc. Some blog posts about this here http://vasters.com/clemensv/default.aspx
Another framework you could try is MassTransit http://code.google.com/p/masstransit/
It seems you're looking for a service host, rather than a message broker. If so, Microsoft's recommended way is to host your WCF services in IIS. They can still use MSMQ as transport, but the services themselves will be managed by IIS. IIS has evolved significantly since its early days as HTTP server, now it's closer to an application server, with its choice of transports (TCP, MSMQ, HTTP), pooling, activation, lifetime policies etc.
Although I find WCF+MSMQ+IIS somewhat overcomplicated this is the price you pay to play on the Microsoft field.
For nice and simple message broker, you can use Active MQ instead of MSMQ, it will give you message brokering as well as pub/sub. It's quite easy to work with in .NET, check this link out: http://activemq.apache.org/nms/