Azure Cache ErrorCode<ERRCA0017>:SubStatus<ES0006> - c#

Im trying to access MS Azure Cache service and pretty often I get this error:
Microsoft.ApplicationServer.Caching.DataCacheException: ErrorCode<ERRCA0017>:SubStatus<ES0006>:There is a temporary failure. Please retry later. (One or more specified cache servers are unavailable, which could be caused by busy network or servers. For on-premises cache clusters, also verify the following conditions. Ensure that security permission has been granted for this client account, and check that the AppFabric Caching Service is allowed through the firewall on all cache hosts. Also the MaxBufferSize on the server must be greater than or equal to the serialized object size sent from the client.). Additional Information : The client was trying to communicate with the server: net.tcp://remoto.cache.windows.net: line 23233.
at Microsoft.ApplicationServer.Caching.DataCache.ThrowException(ErrStatus errStatus, Guid trackingId, Exception responseException, Byte[][] payload, EndpointID destination)
at Microsoft.ApplicationServer.Caching.SocketClientProtocol.Get(String key, ref DataCacheItemVersion version, ref TimeSpan timeout, ref ErrStatus err, String region, IMonitoringListener listener)
at Microsoft.ApplicationServer.Caching.DataCache.InternalGet(String key, ref DataCacheItemVersion version, String region, IMonitoringListener listener)
at Microsoft.ApplicationServer.Caching.DataCache.<>c__DisplayClass53.<Get>b__52()
at Infrastructure.Azure.Cache.AzureCacheServiceClient.<>c__DisplayClass6`1.<Get>b__5() in AzureCacheServiceClient.cs: line 88
at Microsoft.Practices.EnterpriseLibrary.TransientFaultHandling.RetryPolicy.ExecuteAction(Func`1 func)
It happens when Im trying to access some object in cache like this:
public T Get<T>(string key)
{
retryPolicy.ExecuteAction(() =>(T) (_cache.Get(key)));
}
Here is my initialization code:
var cacheFactory = new DataCacheFactory();
_cache = cacheFactory.GetDefaultCache();
var retryStrategy = new FixedInterval(15, TimeSpan.FromSeconds(2));
_retryPolicy = new RetryPolicy<CustomCacheTransientErrorDetectionStrategy>(retryStrategy);
And app.config:
<configSections>
<section name="dataCacheClients" type="Microsoft.ApplicationServer.Caching.DataCacheClientsSection, Microsoft.ApplicationServer.Caching.Core" allowLocation="true" allowDefinition="Everywhere" />
<section name="cacheDiagnostics" type="Microsoft.ApplicationServer.Caching.AzureCommon.DiagnosticsConfigurationSection, Microsoft.ApplicationServer.Caching.AzureCommon" allowLocation="true" allowDefinition="Everywhere" />
</configSections>
<dataCacheClients>
<dataCacheClient channelOpenTimeout="1000" requestTimeout="45000" name="default">
<autoDiscover isEnabled="true" identifier="[some.host.name]" />
<securityProperties mode="Message" sslEnabled="true">
<messageSecurity authorizationInfo="***" />
</securityProperties>
</dataCacheClient>
</dataCacheClients>
It happend at least three times already while the Azure Health Status (https://azure.microsoft.com/en-us/status/) said everything was fine at that time.
As the exception message says - there are some 'temporary' failures on MS side still maybe I'm doing smth wrong in my code?

Do you get the error a few times, and then things start working again? If so, this is expected behavior and your application should have a policy where you retry some number of times before falling back to the persistent data store.
If you get the error consistently (more than a few seconds), it's likely the DataCacheFactory has gotten into a bad state. You can either restart your client process, or else refresh the DataCacheFactory as described in this blog post.

Related

SignalR - Decryption key specified has invalid hex characters

I am using SignalR 2.2.0 and MVC 4. When I load my page in Chrome, I receive an error in the console:
http://localhost:8180/signalr/negotiate?clientProtocol=1.5&connectionData=%5B%7B%22name%22%3A%22chathub%22%7D%5D&_=1467627883530 500 (Internal Server Error)
Upon further inspection using Fiddler I can view the stacktrace:
[ConfigurationErrorsException]: Decryption key specified has invalid hex characters. (C:\Users\Foo\web.config line 66)
at System.Web.Security.Cryptography.MachineKeyMasterKeyProvider.GenerateCryptographicKey(String configAttributeName, String configAttributeValue, Int32 autogenKeyOffset, Int32 autogenKeyCount, String errorResourceString)
at System.Web.Security.Cryptography.MachineKeyMasterKeyProvider.GetEncryptionKey()
at System.Web.Security.Cryptography.Purpose.GetDerivedEncryptionKey(IMasterKeyProvider masterKeyProvider, KeyDerivationFunction keyDerivationFunction)
at System.Web.Security.Cryptography.AspNetCryptoServiceProvider.GetNetFXCryptoService(Purpose purpose, CryptoServiceOptions options)
at System.Web.Security.Cryptography.AspNetCryptoServiceProvider.GetCryptoService(Purpose purpose, CryptoServiceOptions options)
at System.Web.Security.MachineKey.Protect(ICryptoServiceProvider cryptoServiceProvider, Byte[] userData, String[] purposes)
at System.Web.Security.MachineKey.Protect(Byte[] userData, String[] purposes)
at Microsoft.Owin.Host.SystemWeb.DataProtection.MachineKeyDataProtector.Protect(Byte[] userData)
at Microsoft.Owin.Security.DataProtection.AppBuilderExtensions.CallDataProtectionProvider.CallDataProtection.Protect(Byte[] userData)
at Microsoft.AspNet.SignalR.Infrastructure.DataProtectionProviderProtectedData.Protect(String data, String purpose)
at Microsoft.AspNet.SignalR.PersistentConnection.ProcessNegotiationRequest(HostContext context)
at Microsoft.AspNet.SignalR.PersistentConnection.ProcessRequest(HostContext context)
at Microsoft.AspNet.SignalR.Hubs.HubDispatcher.ProcessRequest(HostContext context)
at Microsoft.AspNet.SignalR.PersistentConnection.ProcessRequest(IDictionary`2 environment)
at Microsoft.AspNet.SignalR.Owin.Middleware.HubDispatcherMiddleware.Invoke(IOwinContext context)
at Microsoft.Owin.Infrastructure.OwinMiddlewareTransition.Invoke(IDictionary`2 environment)
at Microsoft.Owin.Mapping.MapMiddleware.<Invoke>d__0.MoveNext()
From what I've gathered, SignalR is using the machine key to perform some encryption/decryption.
This is what the machine key looks like:
<machineKey decryptionKey="5E329ED7DE2786FCD906E717C97A09BE26B3564BBE0A2B28,IsolateApps" validationKey="A64F709FB47B282CB8331205BC423D63450B4FFC92FE06EC1E00B3AE8B5B1F529A5CD1064F66C08545FC13B013F84153598B29213D5494DD5EC348B537A51DAE,IsolateApps"/>
I'm guessing it's seeing the "IsolateApps" in the decryptionKey/validationKey and throwing an exception ("...invalid hex characters").
If I remove the "IsolateApps" from the keys, I can't log in anymore since I'm using FormsAuthentication which in turn makes use of the keys to encrypt/decrypt. So removing the "IsolateApps" does not seem to be a solution.
PS. The code I'm using is from this tutorial.
I have already looked at this answer but it does not solve the problem. I have also tried to add compatibilityMode="Framework20SP1" to the machine key but that does not work either.
Is there way to fix this issue without removing the "IsolateApps" value from the keys in the web config?
I have fixed the issue. I decided to remove the ",IsolateApps" attribute from the machine key which stopped giving this exception:
Decryption key specified has invalid hex characters
Removing this attribute meant I could not log in anymore (since the machine key has technically changed) which forced me to reset my account password. Once I reset the password I could continue with logging in.
Once I was able to log in I was receiving this error:
ws://xxxxx/signalr/connect?transport=webSockets&clientProtocol=1.5&connectionToken=xxxxx'
failed: Connection closed before receiving a handshake response.
This exception was only being thrown in Chrome.
SignalR fell back onto server sent events because the web socket handshakes weren't successful. After many hours of debugging I solved the issue by adding this to the web config:
<system.web>
<compilation debug="true" targetFramework="4.5" />
<httpRuntime targetFramework="4.5" />
<pages controlRenderingCompatibilityVersion="4.0" />
</system.web>
Once that was added, SignalR was functioning as excepted.

Why can opening a Couchbase bucket be very slow?

I'm trying to use Couchbase from .NET with the official SDK. I'm hosting the Couchbase cluster in Amazon EC2.
The machine I'm trying to connect from is hosted in Microsoft Azure.
For some reason, opening a bucket is extremely slow, this afternoon it took ~3 seconds to open a bucket, but now for some reason it's more than 10 seconds.
The memory and CPU utilization of the Couchbase servers are very low.
My configuration:
<couchbaseClients>
<couchbase useSsl="false" operationLifespan="1000">
<servers>
<!-- Ip addresses obscured... -->
<add uri="http://1.1.1.1:8091/pools"></add>
<add uri="http://1.1.1.2:8091/pools"></add>
</servers>
</couchbase>
</couchbaseClients>
The code I'm trying:
var cluster = new Cluster("couchbaseClients/couchbase");
using (var bucket = cluster.OpenBucket("bucketname")) // This is taking 10-50 seconds.
{
var obj = new TestClass { };
// This is fast
var result = bucket.Insert(new Document<TestClass> { Content = obj, Expiry = 300000, Id = Guid.NewGuid().ToString() });
}
using (var bucket = cluster.OpenBucket("bucketname")) // This is taking 10-50 seconds.
{
var obj = new TestClass { };
// This is fast
var result = bucket.Insert(new Document<TestClass> { Content = obj, Expiry = 300000, Id = Guid.NewGuid().ToString() });
}
using (var bucket = cluster.OpenBucket("bucketname")) // This is taking 10-50 seconds.
{
var obj = new TestClass { };
// This is fast
var result = bucket.Insert(new Document<TestClass> { Content = obj, Expiry = 300000, Id = Guid.NewGuid().ToString() });
}
I tried to optimize by using ClusterHelper.GetBucket() instead of cluster.OpenBucket(), but the first GetBucket() is still very slow (everything else is fast in that case).
I experimented with other calls on the bucket (Get, Contains, etc.), and everything is fast, only opening the bucket itself is slow.
How can I troubleshoot what the problem is?
UPDATE: I set up logging based on #jeffrymorris' suggestion, and the I see the following error messages at the point where we spend a lot of time:
DEBUG Couchbase.Configuration.Server.Providers.ConfigProviderBase - Bootstrapping with 127.0.0.1:11210
INFO Couchbase.IO.ConnectionPool... Node 127.0.0.1:11210 failed to initialize, reason: System.Net.Sockets.SocketException (0x80004005): A connection attempt failed because the connected party did not properly respond after a period of time...
And after this there are three other exceptions about trying to connect to 127.0.0.1:11210, and there are seconds passed between those messages, so this is causing the slowness.
Why is the client trying to connect to localhost? I don't have that anywhere in my configuration.
I debugged the code of couchbase-net-client, and this is happening when I call OpenBucket() in CarrierPublicationProvider.GetConfig(). It gets the bucket configuration object, and for some reason it contains not one of the servers I configured in the web.config, but localhost:8091 instead. I'm still trying to figure out why this is happening.
(This is probably not happening on my dev machine, because there I have a local Couchbase server installation. If I stop that, then it gets slow there as well, and I'm seeing the same Bootstrapping with 127.0.0.1:11210 error messages in the logs.)
Figuring out the problem took some debugging of the Couchbase client code, and looking at the logs (thanks for the tip, #jeffrymorris!).
It turns out that if you don't have any preconfigured bucket, then the client will initially try to connect to localhost:8091 (event if that's not in your servers section), which can take seconds until it realizes there is no server there. This doesn't happen if you have any bucket configured.
So I had to change my configuration from this:
<couchbaseClients>
<couchbase useSsl="false" operationLifespan="1000">
<servers>
<!-- Ip addresses obscured... -->
<add uri="http://1.1.1.1:8091/pools"></add>
<add uri="http://1.1.1.2:8091/pools"></add>
</servers>
</couchbase>
</couchbaseClients>
To this:
<couchbaseClients>
<couchbase useSsl="false" operationLifespan="1000">
<servers>
<!-- Ip addresses obscured... -->
<add uri="http://1.1.1.1:8091/pools"></add>
<add uri="http://1.1.1.2:8091/pools"></add>
</servers>
<buckets>
<add name="default" useSsl="false" operationLifespan="1000">
</add>
</buckets>
</couchbase>
</couchbaseClients>
Note that I'm not actually going to use a bucket called default, but having the configuration there sill prevent the client from trying to connect to localhost. (This is happening in the method ConfigProviderBase.GetOrCreateConfiguration.)
Update: The issue has been fixed here: https://github.com/couchbase/couchbase-net-client/commit/020093b422a78728dd49d75d9fe9f1e00d01a0f2

Use Couchbase hosted in Amazon from a client hosted in Azure, what can cause bad performance?

I'm trying to use a Couchbase cluster hosted in Amazon EC2. The client I'm trying to use it from is hosted in Microsoft Azure.
The performance is terrible, in ~10% of the time opening a bucket takes a lot of time.
This is my configuration:
<couchbaseClients>
<couchbase useSsl="false" operationLifespan="1000">
<servers>
<!-- Ip addresses obscured... -->
<add uri="http://1.1.1.1:8091/pools"></add>
<add uri="http://1.1.1.2:8091/pools"></add>
</servers>
<buckets>
<add name="default" useSsl="false" operationLifespan="1000">
</add>
</buckets>
</couchbase>
</couchbaseClients>
This is the code I'm testing with:
var cluster = new Cluster("couchbaseClients/couchbase");
using (var bucket = cluster.OpenBucket("bucketname")) // This sometimes takes 3-50 seconds.
{
var obj = new TestClass { };
// This is fast
var result = bucket.Insert(new Document<TestClass> { Content = obj, Expiry = 300000, Id = Guid.NewGuid().ToString() });
}
Opening the Couchbase bucket sometimes (not always) takes a lot of time, anywhere between 3-50 seconds. It happens often enough that it makes it completely unusable.
When it happens, I can see the following error message in the Couchbase logs:
2015-12-10 14:18:57,644 [1] DEBUG Couchbase.Configuration.Server.Providers.ConfigProviderBase - Bootstrapping with 1.1.1.2:11210
2015-12-10 14:19:07,660 [1] INFO Couchbase.IO.ConnectionPool`1[[Couchbase.IO.Connection, Couchbase.NetClient, Version=2.2.2.0, Culture=neutral, PublicKeyToken=05e9c6b5a9ec94c2]] - Node 1.1.1.2:11210 failed to initialize, reason: System.Net.Sockets.SocketException (0x80004005): A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond
at Couchbase.IO.DefaultConnectionFactory.<GetGeneric>b__0[T](IConnectionPool`1 p, IByteConverter c, BufferAllocator b)
at Couchbase.IO.ConnectionPool`1.Initialize()
Note that 10 seconds are passed there. (I obscured the IP-addresses.)
What can cause this problem and how could I troubleshoot?
This seems to be Azure-specific, I could not reproduce this on my local dev machine or on a machine hosted in the Google cloud. However, it is consistently happening on two different Azure VMs.

Connecting to IBM MQ over SSL via .net client

I am trying to connect to a MQ server queue via a .NET client. I need to use the certificate for secured communication. Here is the code that I have:
MQEnvironment.SSLKeyRepository = "*SYSTEM";
MQEnvironment.ConnectionName = connectionName;
MQEnvironment.Channel = channelName;
MQEnvironment.properties.Add(MQC.TRANSPORT_PROPERTY, MQC.TRANSPORT_MQSERIES_MANAGED);
MQEnvironment.SSLCipherSpec = "TLS_RSA_WITH_AES_256_CBC_SHA";
queueManager = new MQQueueManager(queueManagerName, channelName, connectionName);
queue = queueManager.AccessQueue(SendQueueName,MQC.MQOO_OUTPUT + MQC.MQOO_FAIL_IF_QUIESCING);
queueMessage = new MQMessage();
queueMessage.WriteString(message);
queueMessage.Format = MQC.MQFMT_STRING;
queue.Put(queueMessage, new MQPutMessageOptions());
Every time I try to put the message on the queue, I get this error message
Reason Code: 2059
MQexp.Message: MQRC_Q_MGR_NOT_AVAILABLE
I have checked my variables for the queue manager name, queue name etc and they are correct.
I was also able to connect to a different queue without SSL, I believe that my code is not furnishing enough information to establish a successful connection.
Any help on this would be appreciated.
Thanks,
Kunal
I had the same problem and error message. After enabling tracing I was able to isolate the problem.
I always wondered, how the client is selecting the correct client certificate from the store. The trace output revealed following:
000001B2 15:53:46.828145 20776.10 Created an instance of SSLStreams
000001B3 15:53:46.828145 20776.10 Setting current certificate store as 'Computer'
000001B4 15:53:46.828145 20776.10 Created store object to access certificates
000001B5 15:53:46.834145 20776.10 Opened store
000001B6 15:53:46.834145 20776.10 Accessing certificate - ibmwebspheremqmyusername
000001B7 15:53:46.835145 20776.10 TLS12 supported - True
000001B8 15:53:46.837145 20776.10 Setting SslProtol as Tls
000001B9 15:53:46.837145 20776.10 Starting SSL Authentication
In my case, I had to set the friendly name of the client certificate to ibmwebspheremqmyusername (replace "myusername" with your userid) and set the label in the code aswell:
properties.Add(MQC.MQCA_CERT_LABEL, "ibmwebspheremqmyusername");
To enable tracing, add following to your app.config/web.config where the path points to a location that contains a file named mqtrace.config:
<appSettings>
<add key="MQTRACECONFIGFILEPATH" value="C:\MQTRACECONFIG" />
</appSettings>
Sample content of mqtrace.config (specified directories must exist in advance):
<?xml version="1.0" encoding="utf-8"?>
<traceSettings>
<MQTRACELEVEL>2</MQTRACELEVEL>
<MQTRACEPATH>C:\MQTRACEPATH</MQTRACEPATH>
<MQERRORPATH>C:\MQERRORLOGPATH</MQERRORPATH>
</traceSettings>
Here are some links for more detail:
Tracing:
https://www.ibm.com/support/knowledgecenter/SSFKSJ_8.0.0/com.ibm.mq.dev.doc/q123550_.htm
Why label:
http://www-01.ibm.com/support/docview.wss?uid=swg21245474

WCF custom message security

I need to use polling technique to notify clients about changes in server-side. So I tried to use DuplexHttpBinding (http://code.msdn.microsoft.com/duplexhttp). I works fine with non-secured messages, but I need to use message-level security in my project (UsernameForCertificate). Ok, I decided to add SymmetricSecurityBindingElement to binding collection:
var securityElement = SecurityBindingElement.CreateUserNameForCertificateBindingElement();
collection.Add(securityElement);
And then problem happened. If we use message-level security all messages include security-headers with message signature, like this:
<o:Security s:mustUnderstand="1" xmlns:o="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd" xmlns:s="http://www.w3.org/2003/05/soap-envelope">
....
</o:Security>
And custom polling messages which are send by custom request channel has no security headers, so exception occurs while sending this message through the channel with message-level security:
System.ServiceModel.Security.MessageSecurityException, System.ServiceModel, Version=3.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089
No signature message parts were specified for messages with the 'http://samples.microsoft.com/duplexhttp/pollingAction' action.
Please, advice workaround how to add proper security headers to my custom polling messages before sending them inside custom request channel. You can download source code by the link posted before and simply try to use it with UsernameForCertificate security to reproduce the issue.
Thank you.
After couple of days & deep investigation I found the solution. It seems that we should modify ChannelProtectionRequirements when creation custom Channel Facroty & Channel Listener to add encryption & signature parts to our custom messages. Here is the sample:
private static void ApplyChannelProtectionRequirements(BindingContext context)
{
var cpr = context.BindingParameters.Find<ChannelProtectionRequirements>();
if (cpr != null)
{
XmlQualifiedName qName = new XmlQualifiedName("customHeader", "namespace");
MessagePartSpecification part = new MessagePartSpecification(qName);
cpr.IncomingEncryptionParts.AddParts(part, "incomingAction");
cpr.IncomingSignatureParts.AddParts(part, "incomingAction");
cpr.OutgoingEncryptionParts.AddParts(part, "outgoingAction");
cpr.OutgoingSignatureParts.AddParts(part, "outgoingAction");
}
}

Categories