HDInsight Storm Event Hub Spout only receiving 1024 messages - c#

I have an HDInsight Storm Cluster (Default 60 Core setup)
I have used the template project and am using the latest jar dependency (0.9.
4)
Yesterday I was running the application and passed 20,000,000 messages through my event hub, without issue.
After adding a new bolt (I now have a different bolt receiving from the EventHub) I am now only receiving 1024 messages (exactly) every time.
If I delete my topology and redeploy it, it used to process all the messages in the EventHub, now it is only doing 1024.
I've deleted the Service bus namespace in azure, I've recreated it, scaled it up to 10 MU's and still only getting 1024 messages processed at a time.
All I can think is that this is potentially something to do with the receiver credits?
Looking at the source for the Java spout on github you can see this line:
+eventhubs.checkpoint.interval = 10
+eventhubs.receiver.credits = 1024
and from MSDN the explanation of the EventHubCredits is:
The value for eventhub.receiver.credits determines how many events are
batched before releasing them to the Storm pipeline
It's as if I am only ever receiving one batch. I am currently submitting 1631 messages, more than one batch, but not enough for two batches.
[EDIT]
This seems to be directly related to the "StormConfig.setMaxSpoutPending" setting. I had previously set it to 100,000 and this has been happening since. After removing the block of code (below) that configured this, and redeploying, it still happened. But when I add the code back in, and set the MaxSpoutPending to 500 I now only get a batch of 500 and no more...
//global config:
var topologyConfig = new StormConfig();
topologyConfig.setMaxSpoutPending(500);
topologyConfig.setNumWorkers(partitionCount);
topologyBuilder.SetTopologyConfig(topologyConfig);
[EDIT2]
I rebuilt the whole cluster in azure, and redeployed the solution without any of the StormConfig (I deleted the HDFS blob store as well, so it was a fresh install. I didn't delete the table storage, but I don't think that is used for the Storm Server) .. and I am back to 1024 (not the previous 500) messages, but no more.

Check that your bolts are returning acks. The Event Hub Spout expects acks, and if it doesn't get them it will stop receiving.

Related

Getting Error on Kendo UI Folder Upload - ERR_HTTP2_PROTOCOL_ERROR

I am using Telerik Kendo File Upload for uploading folder.
In Production environment, few users are complaining issue with Folder Upload, during upload few files get errored out, using Developer tool in the console tab it logs "ERR_HTTP2_PROTOCOL_ERROR" error as attached for the failed files.
When i am trying i am not getting this error and all folders are getting uploaded properly. I asked user to share the files for which they are facing error and when i tried it uploaded successfully. When user tried again uploading same files which errored out it got succeeded today which were failing yesterday but sill there are files which is giving the same error.
I went through a post where it say the problem could be due to use of HTTP/2 and when they switched to HTTP /1.1 it worked fine. We are also using HTTP/2 but we don't have option of going back to HTTP/1.1. Link below :
https://www.telerik.com/forums/problems-with-multi-file-upload-and-http-2
Any suggestions ?
This is because on your clients machine http/2 is not enabled thus the error prompts.
If you look in your local machine you will see that under your server, you have Https protocol enabled and a valid certificate.
Your clients either lack a valid certificate on the server or are using the site through Http protocol.
you can learn more here:
Http/2 explanation
SETTINGS_MAX_CONCURRENT_STREAMS (0x3):
Indicates the maximum number of concurrent streams that the sender will allow. This limit is directional: it applies to the number of streams that the sender permits the receiver to create. Initially, there is no limit to this value. It is recommended that this value be no smaller than 100, so as to not unnecessarily limit parallelism.
A value of 0 for SETTINGS_MAX_CONCURRENT_STREAMS SHOULD NOT be treated as special by endpoints. A zero value does prevent the creation of new streams; however, this can also happen for any limit that is exhausted with active streams. Servers SHOULD only set a zero value for short durations; if a server does not wish to accept requests, closing the connection is more appropriate.
Resolution : : Add “Http2MaxConcurrentClientStreams” under HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\HTTP\Parameters
In Registry and restart server.
Set this value to 100 or >100

Inaccessible logs: Security. The source was not found, but some or all event logs could not be searched

I am assuming it has something to do with my app pool/IIS but why is this error only being thrown while in Production?
To Start- I have read every related question about this topic on SO.
I cannot reproduce this error in my Dev and/or Test environments. I have sent hundreds of POST requests using postman(DEV), I have also used my iOS application pointing to my Test Environment and sent thousands of requests and I am still in the same boat.
About my Project: I am using WebAPI and my application is running on iOS10 written in swift. I only receive this error occasionally and cannot reproduce it on command. I am wondering if anyone else has run into this issue and if so, what were the steps you took to take care of this problem?
Error #1
The source was not found, but some or all event logs could not be searched.
To create the source, you need permission to read all event logs to make
sure that the new source name is unique. Inaccessible logs: Security.
Note: This error is not causing me to lose any data when I receive it, all records are still being committed to my database. So it is not a huge issue since I only get the error 1 out of about 100 times a new record is submitted. But curiosity is killing me and wanted to see if anyone had any suggestions.

ProcessReceivedMessage Error in MqttDotNet

I am building a .NET IoT project that requires MQTT for communication. For the broker I use GnatMQ and for clients I use MqttDotNet (for mobile compatibility). The client library builds, connects, and sends messages fine but I get an error whenever the client’s PublishArrivedDelegate is triggered (i.e. message received event).
The error occurs on retained messages as well as standard received messages. The MqttDotNet error log is here.
Console output:
It seems the error is captured in the QoSManager.cs on line 91:
else if (mess.QualityOfService == QoS.OnceAndOnceOnly)
{
_responses.Add(mess.MessageID, new MqttPubrelMessage(mess.MessageID));
}
NOTE: I am using the raw libraries (as they are) without added code.
Has anyone tried these libraries and can maybe confirm that it worked for them without issues? Until then, I guess it's debugging till the extreme.
Update 1: The error only persists on subscribe with QoS level 2.
Update 2: The error can be prevented by adding a handle to prevent duplicates from being added to the hashtable, as pointed out by #hardillb . But this does not SOLVE the actual issue here.
The issue still persists on publish and subscribe with QOS 2. The problem is that the onClientPublishedArrived is being triggered exactly 3 times whenever a message gets received and exactly 2 times when a message get received as retained. NOTE that when I test this with HiveMQ the issue is gone. The problem only persists when using the GnatMQ broker.
Looks like you are trying to add duplicate keys to a Hashtable, this will be being caused by mess.MessageID returning the same value for multiple messages.
The C# implementation looks like it throws an exception rather replacing the original value with the new value.
If you just need a unique key for the Hashtable you could use something like the timestamp

Converting Microsoft EWS StreamingNotification Example to a service

I've been working to try and convert Microsoft's EWS Streaming Notification Example to a service
( MS source http://www.microsoft.com/en-us/download/details.aspx?id=27154).
I tested it as a console app. I then used a generic service template and got it to the point it would compile, install, and start. It stops after about 10 seconds with the ubiquitous "the service on local computer started and then stopped."
So I went back in and upgraded to C# 2013 express and used NLog to put a bunch of log trace commands to so I could see where it was when it exited.
The last place I can find it is in the example code, SynchronizationChanges function,
public static void SynchronizeChanges(FolderId folderId)
{
logger.Trace("Entering SynchronizeChanges");
bool moreChangesAvailable;
do
{
logger.Trace("Synchronizing changes...");
//Console.WriteLine("Synchronizing changes...");
// Get all changes since the last call. The synchronization cookie is stored in the
// _SynchronizationState field.
// Only the the ids are requested. Additional properties should be fetched via GetItem
//calls.
logger.Trace("Getting changes into var changes.");
var changes = _ExchangeService.SyncFolderItems(folderId, PropertySet.IdOnly, null, 512,
SyncFolderItemsScope.NormalItems,
_SynchronizationState);
// Update the synchronization cookie
logger.Trace("Updating _SynchronizationState");
the log file shows the trace message ""Getting changes into var changes." but not the "Updating _SynchronizationState" message.
so it never gets past var changes = _ExchangeService.SyncFolderItems
I cannot for the life figure out why its just exiting. There are many examples of EWS streaming notifications. I have 3 that compile and run just fine but nobody as far as I can tell has posted an example of it done as a service.
If you don't see the "Updating..." message it's likely the sync threw an exception. Wrap it in a try/catch.
OK, so now that I see the error, this looks like your garden-variety permissions problem. When you ran this as a console app, you likely presented the default credentials to Exchange, which were for your login ID. For a Windows service, if you're running the service with one of the built-in accounts (e.g. Local System), your default credentials will not have access to Exchange.
To rectify, either (1) run the service under the account you did the console app with, or (2) add those credentials to the Exchange Service object.

Push Notifications with PushSharp - the basics

I need to push notifications to tens of thousands of iOS devices that my app installed. I'm trying to do it with PushSharp, but I'm missing some fundamental concepts here. At first I tried to actually run this in a Windows service, but couldn't get it work - getting null reference errors coming from _push.QueueNotification() call. Then I did exactly what the documented sample code did and it worked:
PushService _push = new PushService();
_push.Events.OnNotificationSendFailure += new ChannelEvents.NotificationSendFailureDelegate(Events_OnNotificationSendFailure);
_push.Events.OnNotificationSent += new ChannelEvents.NotificationSentDelegate(Events_OnNotificationSent);
var cert = File.ReadAllBytes(HttpContext.Current.Server.MapPath("..pathtokeyfile.p12"));
_push.StartApplePushService(new ApplePushChannelSettings(false, cert, "certpwd"));
AppleNotification notification = NotificationFactory.Apple()
.ForDeviceToken(deviceToken)
.WithAlert(message)
.WithSound("default")
.WithBadge(badge);
_push.QueueNotification(notification);
_push.StopAllServices(true);
Issue #1:
This works perfectly and I see the notification pop up on the iPhone. However, since it's called a Push Service, I assumed it would behave like a service - meaning, I instantiate it and call _push.StartApplePushService() within a Windows service perhaps. And I thought to actually queue up my notifications, I could do this on the front-end (admin app, let's say):
PushService push = new PushService();
AppleNotification notification = NotificationFactory.Apple()
.ForDeviceToken(deviceToken)
.WithAlert(message)
.WithSound("default")
.WithBadge(badge);
push.QueueNotification(notification);
Obviously (and like I already said), it didn't work - the last line kept throwing a null reference exception.
I'm having trouble finding any other kind of documentation that would show how to set this up in a service/client manner (and not just call everything at once). Is it possible or am I missing the point of how PushSharp should be utilized?
Issue #2:
Also, I can't seem to find a way to target many device tokens at once, without looping through them and queuing up notifications one at a time. Is that the only way or am I missing something here as well?
Thanks in advance.
#baramuse explained it all, if you wish to see a service "processor" you can browse through my solution on https://github.com/vmandic/DevUG-PushSharp where I've implemented the workflow you seek for, i.e. a win service, win processor or even a web api ad hoc processor using the same core processor.
From what I've read and how I'm using it, the 'Service' keyword may have mislead you...
It is a service in a way that you configure it once and start it.
From this point, it will wait for you to push new notifications inside its queue system and it will raise events as soon as something happens (delivery report, delivery error...). It is asynchronous and you can push (=queue) 10000 notifications and wait for the results to come back later using the event handlers.
But still it's a regular object instance you will have to create and access as a regular one. It doesn't expose any "outside listener" (http/tcp/ipc connection for example), you will have to build that.
In my project I created a small selfhosted webservice (relying on ServiceStack) that takes care about the configuration and instance lifetime while only exposing the SendNotification function.
And about the Issue #2, there indeed isn't any "batch queue" but as the queue function returns straight away (enqueue and push later) it's just a matter of a looping into your device tokens list...
public void QueueNotification(Notification notification)
{
if (this.cancelTokenSource.IsCancellationRequested)
{
Events.RaiseChannelException(new ObjectDisposedException("Service", "Service has already been signaled to stop"), this.Platform, notification);
return;
}
notification.EnqueuedTimestamp = DateTime.UtcNow;
queuedNotifications.Enqueue(notification);
}

Categories