We Keep Coding

Identity Server 4 WindowsCryptographicException: Keyset does not exist

I get these these two exceptions in Identity Server 4 during a regular login to get a token.
It's on a load balanced setup with 2 servers, so I've done the following:
They both get the same PFX for encrypting tokens. It works just fine on a single server. They have the password for the PFX file, and can seemingly encrypt a token correct correctly.
Sometimes the flow works well, but it always fails on giving CORS access on the connect/token endpoint. I can see that the CORS Middleware plays a role here, but I don't understand why the signing are credentials during the verification of CORS access. I've added protection here, and during startup, I verify that the private key for both the protectionCert and the signing credentials exist. Both certs are password-protected files included with the deploy just to keep it simple(i.e. not running into access issues on private key).
Why is it all of a sudden having trouble getting access to the private key during the CORS operation though? That part I do not understand. All of this works on my test environment, which runs on a single machine, so I'm 99% sure it has to do with this setup being load balanced. Also wondering, if I can save myself a headache by enabling sticky sessions in the load balancer.
In my DataProtection store there's only a single key saved. Not sure if that's normal. I would expect the application to generate more keys.
services.AddDataProtection()
.SetApplicationName("MyApp")
.PersistKeysToDbContext<DataProtectionKeyContext>()
.ProtectKeysWithCertificate(protectionCert);
var cert = new X509Certificate2("signing.pfx",
configuration.GetValue<string>("AppSettings:SigningCredentials:FilePassword"));
builder.AddSigningCredential(cert);
Unhandled exception: "Keyset does not exist" ";"Internal.Cryptography.CryptoThrowHelper+WindowsCryptographicException: Keyset does not exist
at System.Security.Cryptography.CngKey.Open(String keyName, CngProvider provider, CngKeyOpenOptions openOptions)
at Internal.Cryptography.Pal.CertificatePal.GetPrivateKey[T](Func`2 createCsp, Func`2 createCng)
at Internal.Cryptography.Pal.CertificateExtensionsCommon.GetPrivateKey[T](X509Certificate2 certificate, Predicate`1 matchesConstraints)
at Microsoft.IdentityModel.Tokens.X509SecurityKey.get_PrivateKey()
at Microsoft.IdentityModel.Tokens.X509SecurityKey.get_PrivateKeyStatus()
at Microsoft.IdentityModel.Tokens.AsymmetricSignatureProvider..ctor(SecurityKey key, String algorithm, Boolean willCreateSignatures)
at Microsoft.IdentityModel.Tokens.CryptoProviderFactory.CreateSignatureProvider(SecurityKey key, String algorithm, Boolean willCreateSignatures)
at Microsoft.IdentityModel.JsonWebTokens.JwtTokenUtilities.CreateEncodedSignature(String input, SigningCredentials signingCredentials)
at System.IdentityModel.Tokens.Jwt.JwtSecurityTokenHandler.WriteToken(SecurityToken token)
at IdentityServer4.Services.DefaultTokenCreationService.CreateJwtAsync(JwtSecurityToken jwt)
at IdentityServer4.Services.DefaultTokenCreationService.CreateTokenAsync(Token token)
at IdentityServer4.Services.DefaultTokenService.CreateSecurityTokenAsync(Token token)
at IdentityServer4.ResponseHandling.TokenResponseGenerator.CreateAccessTokenAsync(ValidatedTokenRequest request)
at IdentityServer4.ResponseHandling.TokenResponseGenerator.ProcessAuthorizationCodeRequestAsync(TokenRequestValidationResult request)
at IdentityServer4.ResponseHandling.TokenResponseGenerator.ProcessAsync(TokenRequestValidationResult request)
at IdentityServer4.Endpoints.TokenEndpoint.ProcessTokenRequestAsync(HttpContext context)
at IdentityServer4.Endpoints.TokenEndpoint.ProcessAsync(HttpContext context)
at IdentityServer4.Hosting.IdentityServerMiddleware.Invoke(HttpContext context, IEndpointRouter router, IUserSession session, IEventService events)
Connection ID ""17365880169046365174"", Request ID ""800253f9-0001-f100-b63f-84710c7967bb"": An unhandled exception was thrown by the application.";
"Internal.Cryptography.CryptoThrowHelper+WindowsCryptographicException: Keyset does not exist
at System.Security.Cryptography.CngKey.Open(String keyName, CngProvider provider, CngKeyOpenOptions openOptions)
at Internal.Cryptography.Pal.CertificatePal.GetPrivateKey[T](Func`2 createCsp, Func`2 createCng)
at Internal.Cryptography.Pal.CertificateExtensionsCommon.GetPrivateKey[T](X509Certificate2 certificate, Predicate`1 matchesConstraints)
at Microsoft.IdentityModel.Tokens.X509SecurityKey.get_PrivateKey()
at Microsoft.IdentityModel.Tokens.X509SecurityKey.get_PrivateKeyStatus()
at Microsoft.IdentityModel.Tokens.AsymmetricSignatureProvider..ctor(SecurityKey key, String algorithm, Boolean willCreateSignatures)
at Microsoft.IdentityModel.Tokens.CryptoProviderFactory.CreateSignatureProvider(SecurityKey key, String algorithm, Boolean willCreateSignatures)
at Microsoft.IdentityModel.JsonWebTokens.JwtTokenUtilities.CreateEncodedSignature(String input, SigningCredentials signingCredentials)
at System.IdentityModel.Tokens.Jwt.JwtSecurityTokenHandler.WriteToken(SecurityToken token)
at IdentityServer4.Services.DefaultTokenCreationService.CreateJwtAsync(JwtSecurityToken jwt)
at IdentityServer4.Services.DefaultTokenCreationService.CreateTokenAsync(Token token)
at IdentityServer4.Services.DefaultTokenService.CreateSecurityTokenAsync(Token token)
at IdentityServer4.ResponseHandling.TokenResponseGenerator.CreateAccessTokenAsync(ValidatedTokenRequest request)
at IdentityServer4.ResponseHandling.TokenResponseGenerator.ProcessAuthorizationCodeRequestAsync(TokenRequestValidationResult request)
at IdentityServer4.ResponseHandling.TokenResponseGenerator.ProcessAsync(TokenRequestValidationResult request)
at IdentityServer4.Endpoints.TokenEndpoint.ProcessTokenRequestAsync(HttpContext context)
at IdentityServer4.Endpoints.TokenEndpoint.ProcessAsync(HttpContext context)
at IdentityServer4.Hosting.IdentityServerMiddleware.Invoke(HttpContext context, IEndpointRouter router, IUserSession session, IEventService events)
at IdentityServer4.Hosting.IdentityServerMiddleware.Invoke(HttpContext context, IEndpointRouter router, IUserSession session, IEventService events)
at IdentityServer4.Hosting.MutualTlsEndpointMiddleware.Invoke(HttpContext context, IAuthenticationSchemeProvider schemes)
at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Cors.Infrastructure.CorsMiddleware.<Invoke>g__InvokeCoreAwaited|15_0(HttpContext context, Task`1 policyTask)
at IdentityServer4.Hosting.BaseUrlMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Session.SessionMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Session.SessionMiddleware.Invoke(HttpContext context)
at NewRelic.Providers.Wrapper.AspNetCore.WrapPipelineMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Server.IIS.Core.IISHttpContextOfT`1.ProcessRequestAsync()

I think I solved it. I read that it could have to do with lack of access. I guess on of my tasks in my CD setup isn't working right, because I tried running an APP CMD command to set user profile to loaded as true for my app pool. Supposedly, if this flag is false, the application will by default try to store the private key as the current user, but since no user is loaded, this doesn't work. So I tried with ephemeral keyset instead(in-memory) to avoid access issues and now it works. Not sure what the consequences are of using an in-memory keyset, if the keys are appropriate etc.

Siemens OPC UA and .NET C# client can not connect to server?

I tryed to connect to OPC UA server using the client provided on this page: https://support.industry.siemens.com/cs/document/42014088/programming-an-opc-ua-net-client-with-c%23-for-the-simatic-net-opc-ua-server?dti=0&lc=en-US . Connection to OPC UA server using the Siemens OPC Scount v10 works fine. When connecting to the OPC UA server using client provided in the article, I get this message:
Could not open UA TCP request channel.
Stack trace of the exception is this:
Server stack trace:
at Opc.Ua.Bindings.UaTcpRequestChannel.OnEndOpen(IAsyncResult result)
at Opc.Ua.Bindings.UaTcpRequestChannel.OnOpen(TimeSpan timeout)
at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannel.OnOpen(TimeSpan timeout)
at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannel.CallOpenOnce.System.ServiceModel.Channels.ServiceChannel.ICallOnce.Call(ServiceChannel channel, TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannel.CallOnceManager.CallOnce(TimeSpan timeout, CallOnceManager cascade)
at System.ServiceModel.Channels.ServiceChannel.EnsureOpened(TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs)
at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)
at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)
Exception rethrown at [0]:
at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
at Opc.Ua.ISessionChannel.CreateSession(CreateSessionMessage request)
at Opc.Ua.SessionChannel.CreateSession(CreateSessionMessage request)
at Opc.Ua.SessionClient.CreateSession(RequestHeader requestHeader, ApplicationDescription clientDescription, String serverUri, String endpointUrl, String sessionName, Byte[] clientNonce, Byte[] clientCertificate, Double requestedSessionTimeout, UInt32 maxResponseMessageSize, NodeId& sessionId, NodeId& authenticationToken, Double& revisedSessionTimeout, Byte[]& serverNonce, Byte[]& serverCertificate, EndpointDescriptionCollection& serverEndpoints, SignedSoftwareCertificateCollection& serverSoftwareCertificates, SignatureData& serverSignature, UInt32& maxRequestMessageSize)
at Opc.Ua.Client.Session.Open(String sessionName, UInt32 sessionTimeout, IUserIdentity identity, IList`1 preferredLocales)
at Opc.Ua.Client.Session.Open(String sessionName, IUserIdentity identity)
Any help would be appreciated. OPC UA server runs on Siemens Simatic HMI TP700 Comfort. Configuration on the OPC UA server is default.
After andrewcullen tip, we are getting the below log in the tracelog.txt file and error when catching the exception that says
An unexpected error occurred while connecting to the server.
PID:4196 ************************* Logging started at 02/03/2016 07:41:34 *************************
4196 - 07:41:38.742 GetEndpoints Called. RequestHandle=1, PendingRequestCount=1
4196 - 07:41:38.992 SECURE CHANNEL CREATED [TcpClientChannel UA-TCP 1.00.238.1] [ID=12752] Connected To: opc.tcp://xxx.xxx.xxx.xxx:4870/
4196 - 07:41:39.008 TCPCLIENTCHANNEL SOCKET CONNECTED: 00000698, ChannelId=12752
4196 - 07:41:39.008 SECURE CHANNEL CREATED [Opc.Ua.ChannelBase WCF Client 1.00.238.1] [ID=] Connected To: opc.tcp://xxx.xxx.xxx.xxx:4870/
4196 - 07:41:39.101 GetEndpoints Completed. RequestHandle=1, PendingRequestCount=0
4196 - 07:41:39.132 TCPCLIENTCHANNEL SOCKET CLOSED: 00000698, ChannelId=12752
4196 - 07:41:44.230 Writing rejected certificate to directory:
4196 - 07:41:59.694 CreateSession Called. RequestHandle=1, PendingRequestCount=1
4196 - 07:42:13.672 TCPCLIENTCHANNEL SOCKET CLOSED: 000007C0, ChannelId=0
4196 - 07:42:13.750 CreateSession Completed. RequestHandle=1, PendingRequestCount=0
I got the answer from the Siemens official support:
The application was not tested with Comfort Panel. The code e.g. contains Block Read and Block Write which is not supported from the Panel Server.
So this application will not work.

This Siemens UaClient uses a library 'ClientAPI' which extends the OPC Foundation's Opc.Ua.Core and Opc.Ua.Client. ClientAPI has a lot of nice Helper functions to simplify connecting and subscribing. However, I see in the code for Connect(string Url) that it is using the original WCF-style channel. And your stack trace is showing the WCF types are throwing an exception that is hard to diagnose. I would change two things:
First configure tracing to write to a file. In ClientAPI, find Helpers.CreateClientConfiguration() and add
// add trace config before calling validate
configuration.TraceConfiguration = new TraceConfiguration {
OutputFilePath="tracelog.txt",
DeleteOnLoad = true,
TraceMasks = Utils.TraceMasks.All };
configuration.Validate(ApplicationType.Client);
Second, upgrade the channel type used to connect. In ClientAPI, find Server.Connect(string url) and modify the middle as shown:
// Initialize the channel which will be created with the server.
// SessionChannel channel = SessionChannel.Create(
// configuration,
// endpointDescription,
// endpointConfiguration,
// bindingFactory,
// clientCertificate,
// null);
ITransportChannel channel = WcfChannelBase.CreateUaBinaryChannel(
configuration,
endpointDescription,
endpointConfiguration,
clientCertificate,
configuration.CreateMessageContext());
// Wrap the channel with the session object.
// This call will fail if the server does not trust the client certificate.
// m_Session = new Session(channel, configuration, endpoint);
m_Session = new Session(channel, configuration, endpoint, clientCertificate);
Edit 2/4:
From the tracelog you might find certificate errors. When creating a new session, the client and server both provide and validate each others certificate. By default, UaClient is retrieving it's cert from the windows store LocalMachine\My (aka Personal). The api generates this cert during it's first run, (which requires the first run as administrator) ( to see this cert, run 'certlm.msc').
On the server machine, the server will validate the client's cert, by checking if it matches the certs in its 'TrustedPeerList'. Servers usually use a directory to store the trusted certs. If the client cert is not trusted, the server will copy the client's cert to a 'RejectedCertificates' directory. You are required to copy the cert you find in 'RejectedCertificates' to the trusted cert directory.
Back on the client machine, the client will validate the server's cert. This client uses a windows store for validation 'LocalMachine\My' (aka Personal).
Instead of using a 'Rejected' directory, the client registers an event handler that opens a message box, asking if you wish to accept the server's cert. If you choose to accept, the client sets the eventArg e.Accept = true; To suppress the message box, the server's cert should be imported into the client's 'LocalMachine\My' (aka Personal) using the tool 'certlm.msc'.

Try to ping the server with DNS name. If the server is not accessible the Hosts file in C:\Windows\System32\drivers\etc... must be edited. Open the notepad as administrator, then open the Hosts file and enter the mapping of IP Address to the host name as follows:
xxx.xxx.xxx.xxx host name

WCF SslStreamSecurity DNS Identity Check failing for just 4.6 framework

I am working on developing a new binding for a Wcf service that is hosted in IIS, I thought I got everything working, but it turns out that the client only works when it is targetting .Net framework 4.5, if I change it to target 4.6 then I get the following error when I try to open a connection:
System.ServiceModel.Security.MessageSecurityException occurred
HResult=-2146233087
Message=The Identity check failed for the outgoing message. The remote endpoint did not provide a domain name system (DNS) claim and therefore did not satisfied DNS identity 'xxx.domain.local'. This may be caused by lack of DNS or CN name in the remote endpoint X.509 certificate's distinguished name.
Source=System.ServiceModel
StackTrace:
at System.ServiceModel.Security.IdentityVerifier.EnsureIdentity(EndpointAddress serviceReference, AuthorizationContext authorizationContext, String errorString)
If I do nothing other than change the target framework in my test code back to 4.5, then it works fine. This makes me think that it could be a bug in .Net 4.6, I know there were Wcf ssl changes made in 4.6
With first chance exceptions turned on I see the following exception that is raised internally in System.ServiceModel
System.ArgumentNullException occurred
HResult=-2147467261
Message=Value cannot be null.
Parameter name: value
ParamName=value
Source=mscorlib
StackTrace:
at System.Enum.TryParseEnum(Type enumType, String value, Boolean ignoreCase, EnumResult& parseResult)
InnerException:
System.ServiceModel.dll!System.ServiceModel.Security.IssuanceTokenProviderBase<System.ServiceModel.Security.Tokens.IssuedSecurityTokenProvider.FederatedTokenProviderState>.DoNegotiation(System.TimeSpan timeout) Unknown System.ServiceModel.dll!System.ServiceModel.Security.IssuanceTokenProviderBase<System.ServiceModel.Security.Tokens.IssuedSecurityTokenProvider.FederatedTokenProviderState>.GetTokenCore(System.TimeSpan timeout) Unknown
System.IdentityModel.dll!System.IdentityModel.Selectors.SecurityTokenProvider.GetToken(System.TimeSpan timeout) Unknown
System.ServiceModel.dll!System.ServiceModel.Security.Tokens.IssuedSecurityTokenProvider.GetTokenCore(System.TimeSpan timeout) Unknown
System.IdentityModel.dll!System.IdentityModel.Selectors.SecurityTokenProvider.GetToken(System.TimeSpan timeout) Unknown
System.ServiceModel.dll!System.ServiceModel.Security.SecurityProtocol.TryGetSupportingTokens(System.ServiceModel.Security.SecurityProtocolFactory factory, System.ServiceModel.EndpointAddress target, System.Uri via, System.ServiceModel.Channels.Message message, System.TimeSpan timeout, bool isBlockingCall, out System.Collections.Generic.IList<System.ServiceModel.Security.SupportingTokenSpecification> supportingTokens) Unknown
System.ServiceModel.dll!System.ServiceModel.Security.TransportSecurityProtocol.SecureOutgoingMessageAtInitiator(ref System.ServiceModel.Channels.Message message, string actor, System.TimeSpan timeout) Unknown
System.ServiceModel.dll!System.ServiceModel.Security.TransportSecurityProtocol.SecureOutgoingMessage(ref System.ServiceModel.Channels.Message message, System.TimeSpan timeout) Unknown
System.ServiceModel.dll!System.ServiceModel.Security.SecurityProtocol.SecureOutgoingMessage(ref System.ServiceModel.Channels.Message message, System.TimeSpan timeout, System.ServiceModel.Security.SecurityProtocolCorrelationState correlationState) Unknown
System.ServiceModel.dll!System.ServiceModel.Channels.SecurityChannelFactory<System.ServiceModel.Channels.IRequestChannel>.SecurityRequestChannel.Request(System.ServiceModel.Channels.Message message, System.TimeSpan timeout) Unknown
System.ServiceModel.dll!System.ServiceModel.Channels.TransactionRequestChannelGeneric<System.ServiceModel.Channels.IRequestChannel>.Request(System.ServiceModel.Channels.Message message, System.TimeSpan timeout) Unknown
System.ServiceModel.dll!System.ServiceModel.Dispatcher.RequestChannelBinder.Request(System.ServiceModel.Channels.Message message, System.TimeSpan timeout) Unknown
System.ServiceModel.dll!System.ServiceModel.Channels.ServiceChannel.Call(string action, bool oneway, System.ServiceModel.Dispatcher.ProxyOperationRuntime operation, object[] ins, object[] outs, System.TimeSpan timeout) Unknown
System.ServiceModel.dll!System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(System.Runtime.Remoting.Messaging.IMethodCallMessage methodCall, System.ServiceModel.Dispatcher.ProxyOperationRuntime operation) Unknown
System.ServiceModel.dll!System.ServiceModel.Channels.ServiceChannelProxy.Invoke(System.Runtime.Remoting.Messaging.IMessage message) Unknown
mscorlib.dll!System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(ref System.Runtime.Remoting.Proxies.MessageData msgData, int type) Unknown
The wcf service being communicated to is targeting 4.6, and as far as I can tell I am specifying the dns identity, which does exist as a CN= in the cert subject. The binding is a custom binding so that I can do federated net.tcp, the client creates everything in code and I don't use the Add Service Reference feature in visual studio, the client code that is creating the binding:
var binding = new CustomBinding(new BindingElement[] {
new TransactionFlowBindingElement(),
security,
new SslStreamSecurityBindingElement(),
new BinaryMessageEncodingBindingElement() {
ReaderQuotas = { MaxDepth = maxReceivedSizeBytes, MaxStringContentLength = maxReceivedSizeBytes, MaxArrayLength = maxReceivedSizeBytes, MaxBytesPerRead = maxReceivedSizeBytes, MaxNameTableCharCount = maxReceivedSizeBytes },
},
new TcpTransportBindingElement {
TransferMode = TransferMode.StreamedResponse,
MaxReceivedMessageSize = maxReceivedSizeBytes,
},
}) {
SendTimeout = sendTimeout,
};
var channelFactory = new ChannelFactory<T>(binding, new EndpointAddress(new Uri(url), EndpointIdentity.CreateDnsIdentity("xxx.domain.local"), new AddressHeader[0]));
Could this be a bug in the 4.6 framework causing different behavior? Would the next steps only be trying to step through and debug framework code to try and find why 4.6 is behaving differently?
EDIT -
I created a small sample project that demonstrates the error, the repro steps are:
(Using VS 2015) Open the WcfSelfHostedServer solution
Add the IdentityFail.pfx cert to your Local Computer, Personal store using mmc
Run the WcfSelfHostedServer project (likely clicking firewall yes allow port 30000)
Open the WcfClient solution
Right click on project > properties, note that it is targetting 4.6.1
Run the project, it will throw the exception described above
Now switch the client to target 4.5.2, it will run fine with no errors
Update -
I found the following that appear related:
https://support.microsoft.com/en-us/kb/3069494
https://msdn.microsoft.com/en-us/library/mt298998(v=vs.110).aspx
But specifying Tls12 at the server and client didn't fix the issue, and even adding the DontEnableSchUseStrongCrypto=true flag didn't affect the DNS Identity Check error even though it got around the Enum.Parse internal error that was being thrown from this line

I needed to look at Retargetting Changes in the .NET Framework 4.6.1, as certificate validation logic changed in that release. (change in behavior for X509CertificateClaimSet.FindClaims that was causing my issue)
The fix is editing my app.config to add:
<runtime>
<AppContextSwitchOverrides value="Switch.System.IdentityModel.DisableMultipleDNSEntriesInSANCertificate=true" />
</runtime>
You can see the changed code on referencesource, and naturally makecert.exe doesn't appear to support generating certificates with "Subject Alternative Name" fields

You can fix in code adding one line.
like this.
AppContext.SetSwitch("Switch.System.IdentityModel.DisableMultipleDNSEntriesInSANCertificate",true);

Installing .net 4.7 on the server solved the problem for me.

Brandon.
It appears that if the flag is 'false' AND a cert does NOT contain SAN entries, we don't add the dns entry.

Frequent connection aborted exceptions when using CRM 2013 Organization Service

I have a problem running a WCF service that connects to the CRM: It frequently produces CommunicationObjectAbortedExceptions which leave me to wonder if I am doing something wrong. These execptions started occurring after many people started using it, on the test system it worked without problems.
But let's start at the beginning: I wrote two WCF Services that connect to the Microsoft CRM2013 Organization Service using my own library to execute queries on the CRM. These services are regularly called from the CRM which is used by roughly 100-200 people on a daily basis.
This works basically fine, but I frequently get a couple of exceptions which look like the following (see bottom of the Post for the full stacktrace):
System.ServiceModel.CommunicationObjectAbortedException: The HTTP request to 'http://crm/MyOrganization/XRMServices/2011/Organization.svc' was aborted. This may be due to the local channel being closed while the request was still in progress. If this behavior is not desired, then update your code so that it does not close the channel while request operations are still in progress.
By frequently I mean around 100 times a day, most often a couple of those exceptions are thrown every 5-30 minutes in batches of 3-6 exceptions. I have no idea why this is happening. I initialize the connection to the CRM Organization Service using the following class from my library in both services:
public class CrmManager : IDisposable
{
private static CrmConnection s_connection;
public static CrmConnection Connection
{
get
{
if (s_connection == null)
{
s_connection = new CrmConnection("CrmTvTest");
}
return s_connection;
}
}
public static IOrganizationService ServiceProxy
{
get { return s_serviceProxy ?? (s_serviceProxy = new CachedOrganizationService(Connection)); }
}
As can be seen, I connect to the Organization Service once per WCF service, using the CrmConnection to handle the connection details, which is stored in a static variable (acting as a singleton, since establishing the Connection is expensive and should not be done too often to my understanding). It is then passed to the CachedOrganizationService, which is static for the same reasons. The WCF service uses the default instance management (PerSession AFAIK), meaning there is probably 1 connection and organization service per user.
My Connection String looks like this (removed any sensible data, of course):
<connectionStrings>
<add name="CrmTvTest" connectionString="Url=http://crm/MyOrganization; Username=user; Password=pw;"/>
I then use the connection with CrmServiceContext objects to execute queries using this method from my CrmManager class. Which is, of course, always called within a using-statement:
using (CrmServiceContext context = new CrmServiceContext(CrmManager.ServiceProxy))
{
// do some stuff...
}
How can I prevent these exceptions from constantly occuring? I get the feeling this has to do with the Security Tokens used by the CRM connection expiring, but this shouldn't be a problem when I use the CrmConnection class. It should refresh them automatically.
Any advice would be very welcome, since I am pondering this issue for a while now.
UPDATE 1
I switched to using the Developer Extensions and using the CrmConnector class, to no avail (I updated the code above). I also tried passing the CrmConnection class directly to the CrmServiceContext:
using (CrmServiceContext context = new CrmServiceContext(CrmManager.Connection))
which led to the same problems as in this Stackoverflow Question, without using a load-balancer (we initially did, but disabled load-balancing to eliminate the possibility of it causing the problems.
Full Stacktrace:
---> System.Net.WebException: The request was aborted: The request was canceled.
at System.Net.HttpWebRequest.GetResponse()
at System.ServiceModel.Channels.HttpChannelFactory`1.HttpRequestChannel.HttpChannelRequest.WaitForReply(TimeSpan timeout)
--- End of inner exception stack trace ---
Server stack trace:
at System.ServiceModel.Channels.HttpChannelUtilities.ProcessGetResponseWebException(WebException webException, HttpWebRequest request, HttpAbortReason abortReason)
at System.ServiceModel.Channels.HttpChannelFactory1.HttpRequestChannel.HttpChannelRequest.WaitForReply(TimeSpan timeout)
at System.ServiceModel.Channels.RequestChannel.Request(Message message, TimeSpan timeout)
at System.ServiceModel.Channels.SecurityChannelFactory1.SecurityRequestChannel.Request(Message message, TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)
at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)
Exception rethrown at [0]:
at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
at Microsoft.Xrm.Sdk.IOrganizationService.Execute(OrganizationRequest request)
at Microsoft.Xrm.Sdk.Client.OrganizationServiceProxy.ExecuteCore(OrganizationRequest request)
at Microsoft.Xrm.Sdk.Client.OrganizationServiceContext.Execute(OrganizationRequest request)
at Microsoft.Xrm.Sdk.Linq.QueryProvider.RetrieveEntityCollection(OrganizationRequest request, NavigationSource source)
at Microsoft.Xrm.Sdk.Linq.QueryProvider.Execute(QueryExpression qe, Boolean throwIfSequenceIsEmpty, Boolean throwIfSequenceNotSingle, Projection projection, NavigationSource source, List1 linkLookups, String& pagingCookie, Boolean& moreRecords)
at Microsoft.Xrm.Sdk.Linq.QueryProvider.Execute[TElement](QueryExpression qe, Boolean throwIfSequenceIsEmpty, Boolean throwIfSequenceNotSingle, Projection projection, NavigationSource source, List1 linkLookups)
at Microsoft.Xrm.Sdk.Linq.QueryProvider.Execute[TElement](Expression expression)
at Microsoft.Xrm.Sdk.Linq.QueryProvider.System.Linq.IQueryProvider.Execute[TResult](Expression expression)
at System.Linq.Queryable.SingleOrDefault[TSource](IQueryable`1 source)
at CrmConnector.Entities.Contact.Get(Guid p_id, Boolean p_includeRelatedEntities) in j:\IntDev\Libraries\CrmConnector\Entities\Contact.cs:line 63
at CrmExtensionService.CrmExtension.GetPersonalizedEmailSignature(String p_contactId, String p_systemUserId) in j:\IntDev\Services\CrmExtensionService\CrmExtension.svc.cs:line 460

We had same issue and load balancer was the culprit. Now we resolved this error with load balancer activated by making few configurations in the load balancer. We enabled sticky session with least connection algorithm at the load balancer. So if this is not enabled while load balanced then authenticated connection from one server's request gets routed to different servers even though same session and fails. Once enabling the sticky session (session persistance to be client ip) requests goes to same server (in this case returning connection and not a new connection) it works well.

So, after fiddling around for about two months, we found the issue: The load balancing of the CRM FrontEnd was the culprit. I assumed this was disabled, too, with disabling the load balancing of our CRM Service, but it wasn't. Our CRM Service periodically established a connection with the Organization Service of Server 1, then got switched to Server 2 mid-operation and these exceptions occurred.
We're still trying to figure out how to get this to work with load balancing activated, but for the time being we keep it disabled to prevent these errors from popping up.
There is a similar case here on StackOverflow: Sporadic exceptions calling a web service that is load balanced. We are currently using a webHttpBinding and a quickly attempted to switch over to a basicHttpBinding but didn't get it to work (but as I said, this was just a quick attempt).

Which IE add-on is crashing my application

Case: I have a a websites which authenticates to a second SSO website using federated authentication.
Go to the main website
Redirect to the SSO authentication site -> and you fill in the correct credentials
go back to main with authentication cookies -> crash
Extra info from the error logs:
It's always in an IE browser (FF and Chrome work perfectly)
I can't reproduce the problem myself: tested on every operating system and browser version with Browserstack nor can my colluegues.
When contacting end-users who have the problem. They consistently have it in IE. When asked to use another browser it works.
My current guess is that it's due to some IE-add-on. But I do not know which one and this is a bit of a guess. The fact that it happens on end-users computers means I have no control over those computers and I'm unable to go look on their screen and check out which plug-ins they have.
Are their ways to check this? and find out which add-on's are installed on an end-users pc?
The error is this one:
The token '>' was expected but found 't'. Line 1, position 1572.
Stacktrace:
at System.Xml.XmlExceptionHelper.ThrowXmlException(XmlDictionaryReader
reader, String res, String arg1, String arg2, String arg3) at
System.Xml.XmlExceptionHelper.ThrowTokenExpected(XmlDictionaryReader
reader, String expected, Char found) at
System.Xml.XmlUTF8TextReader.ReadStartElement() at
System.Xml.XmlUTF8TextReader.Read() at
System.Xml.XmlDictionaryWriter.WriteNode(XmlDictionaryReader reader,
Boolean defattr) at System.Xml.XmlDictionaryWriter.WriteNode(XmlReader
reader, Boolean defattr) at
Microsoft.IdentityModel.Protocols.WSTrust.WSTrustSerializationHelper.ReadInnerXml(XmlReader
reader, Boolean onStartElement) at
Microsoft.IdentityModel.Protocols.WSTrust.WSTrustSerializationHelper.ReadRSTRXml(XmlReader
reader, RequestSecurityTokenResponse rstr, WSTrustSerializationContext
context, WSTrustConstantsAdapter trustConstants) at
Microsoft.IdentityModel.Protocols.WSTrust.WSTrust13ResponseSerializer.ReadXmlElement(XmlReader
reader, RequestSecurityTokenResponse rstr, WSTrustSerializationContext
context) at
Microsoft.IdentityModel.Protocols.WSTrust.WSTrustSerializationHelper.CreateResponse(XmlReader
reader, WSTrustSerializationContext context, WSTrustResponseSerializer
responseSerializer, WSTrustConstantsAdapter trustConstants) at
Microsoft.IdentityModel.Protocols.WSTrust.WSTrust13ResponseSerializer.ReadXml(XmlReader
reader, WSTrustSerializationContext context) at
Microsoft.IdentityModel.Protocols.WSFederation.WSFederationSerializer.CreateResponse(WSFederationMessage
message, WSTrustSerializationContext context) at
Microsoft.IdentityModel.Web.WSFederationAuthenticationModule.GetXmlTokenFromMessage(SignInResponseMessage
message, WSFederationSerializer federationSerializer) at
Microsoft.IdentityModel.Web.WSFederationAuthenticationModule.GetXmlTokenFromMessage(SignInResponseMessage
message) at
Microsoft.IdentityModel.Web.WSFederationAuthenticationModule.GetXmlTokenFromMessage(SignInResponseMessage
message, WSFederationSerializer federationSerializer) at
Microsoft.IdentityModel.Web.WSFederationAuthenticationModule.GetSecurityToken(SignInResponseMessage
message) at
Microsoft.IdentityModel.Web.WSFederationAuthenticationModule.GetSecurityToken(HttpRequest
request) at
Microsoft.IdentityModel.Web.WSFederationAuthenticationModule.SignInWithResponseMessage(HttpRequest
request) at
Microsoft.IdentityModel.Web.WSFederationAuthenticationModule.OnAuthenticateRequest(Object
sender, EventArgs args) at
System.Web.HttpApplication.SyncEventExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
at System.Web.HttpApplication.ExecuteStep(IExecutionStep step,
Boolean& completedSynchronously)