Related
I have an API with a single endpoint that retrieves documents from a CosmosDB collection. The endpoint works fine on common scenarios. However, when I execute stress tests on the API, to measure how it responds under heavy load, I experience outages on the site (site starts to respond requests with 502 - bad gateway).
Searching on Application Insights, I notice OutOfMemory exceptions raising while executing the sentence to retrieve the documents from the CosmosDB collection. The method that I'm using to read the documents is ReadNextAsync and the logs points this line specifically.
We read and tested the best practices and tips that the Cosmos DB documentation mentions to discard a bad usage of the SDK from our side, but even trying with different configurations (MaxItemCount, MaxBufferedItems, MaxConcurrency), the issue persisted.
After executing several tests, I noticed that if I limit the amount of documents to retrieve from the collection (e.g. using a TOP 40 clause), the exceptions or site outages don’t show. Instead, all requests are processed successfully with 200 status code. As I haven’t had these kind of issues on our Full FWK API, which has the exactly same logic as the .NET Core API described here, I'm wondering whether I could be doing a bad usage of the .NET Core SDK.
In order to share more context, I detailed below the general specifications and also the details on how I configured CosmosDB, along with the implementation to retrieve documents. Additionally, I included logs and a related stack trace found on Application Insights exceptions table.
General Specifications
API .NET CORE 2.2
Microsoft.Azure.Cosmos 3.5.0
Cosmos DB specifications
CosmosDB client connection
Connection mode: Direct
Application Region: West US
Default values for the rest
CosmosDB target collection
~600 documents
~30K size each document
PartitionKey -> id (one logical partition per document)
Write region -> West US
Read regions -> West Us, West Europe, Southeast Asia and Brazil South
Stress scenario details
Execute 400 request per second looking for retrieving up to 200 documents per request.
Document retrieving implementation
var feed = container.GetItemLinqQueryable<T>(false, null, queryRequestOptions).Where(predicate).ToFeedIterator();
var batches = new List<FeedResponse<T>>();
while (feed.HasMoreResults)
{
var batch = await feed.ReadNextAsync();
batches.Add(batch);
}
Application Insights exception stack trace
Response status code does not indicate success: 500 Substatus: 0 Reason: (System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
at System.Collections.Generic.List`1.set_Capacity(Int32 value)
at System.Collections.Generic.List`1.EnsureCapacity(Int32 min)
at System.Collections.Generic.List`1.AddWithResize(T item)
at Microsoft.Azure.Cosmos.Json.JsonNavigator.JsonTextNavigator.Parser.ParseObjectNode(IJsonReader jsonTextReader)
at Microsoft.Azure.Cosmos.Json.JsonNavigator.JsonTextNavigator.Parser.ParseNode(IJsonReader jsonTextReader)
at Microsoft.Azure.Cosmos.Json.JsonNavigator.JsonTextNavigator.Parser.ParsePropertyNode(IJsonReader jsonTextReader)
at Microsoft.Azure.Cosmos.Json.JsonNavigator.JsonTextNavigator.Parser.ParseObjectNode(IJsonReader jsonTextReader)
at Microsoft.Azure.Cosmos.Json.JsonNavigator.JsonTextNavigator.Parser.ParseNode(IJsonReader jsonTextReader)
at Microsoft.Azure.Cosmos.Json.JsonNavigator.JsonTextNavigator.Parser.ParseArrayNode(IJsonReader jsonTextReader)
at Microsoft.Azure.Cosmos.Json.JsonNavigator.JsonTextNavigator.Parser.ParseNode(IJsonReader jsonTextReader)
at Microsoft.Azure.Cosmos.Json.JsonNavigator.JsonTextNavigator.Parser.ParsePropertyNode(IJsonReader jsonTextReader)
at Microsoft.Azure.Cosmos.Json.JsonNavigator.JsonTextNavigator.Parser.ParseObjectNode(IJsonReader jsonTextReader)
at Microsoft.Azure.Cosmos.Json.JsonNavigator.JsonTextNavigator.Parser.ParseNode(IJsonReader jsonTextReader)
at Microsoft.Azure.Cosmos.Json.JsonNavigator.JsonTextNavigator.Parser.Parse(IJsonReader jsonTextReader)
at Microsoft.Azure.Cosmos.Json.JsonNavigator.JsonTextNavigator..ctor(ReadOnlyMemory`1 buffer, Boolean skipValidation)
at Microsoft.Azure.Cosmos.Json.JsonNavigator.Create(ReadOnlyMemory`1 buffer, JsonStringDictionary jsonStringDictionary, Boolean skipValidation)
at Microsoft.Azure.Cosmos.CosmosElements.CosmosElementSerializer.ToCosmosElements(MemoryStream memoryStream, ResourceType resourceType, CosmosSerializationFormatOptions cosmosSerializationOptions)
at Microsoft.Azure.Cosmos.CosmosQueryClientCore.GetCosmosElementResponse(QueryRequestOptions requestOptions, ResourceType resourceType, ResponseMessage cosmosResponseMessage, PartitionKeyRangeIdentity partitionKeyRangeIdentity, SchedulingStopwatch schedulingStopwatch)
at Microsoft.Azure.Cosmos.CosmosQueryClientCore.ExecuteItemQueryAsync[RequestOptionType](Uri resourceUri, ResourceType resourceType, OperationType operationType, RequestOptionType requestOptions, SqlQuerySpec sqlQuerySpec, String continuationToken, PartitionKeyRangeIdentity partitionKeyRange, Boolean isContinuationExpected, Int32 pageSize, SchedulingStopwatch schedulingStopwatch, CancellationToken cancellationToken)
at Microsoft.Azure.Cosmos.Query.ItemProducer.BufferMoreDocumentsAsync(CancellationToken token)
at Microsoft.Azure.Cosmos.Query.ItemProducer.BufferMoreIfEmptyAsync(CancellationToken token)
at Microsoft.Azure.Cosmos.Query.ItemProducer.TryMoveNextPageAsync(CancellationToken cancellationToken)
at Microsoft.Azure.Cosmos.Query.ItemProducerTree.TryMoveNextPageImplementationAsync(CancellationToken cancellationToken)
at Microsoft.Azure.Cosmos.Query.ItemProducerTree.ExecuteWithSplitProofingAsync(Func`2 function, Boolean functionNeedsBeReexecuted, CancellationToken cancellationToken)
at Microsoft.Azure.Cosmos.Query.ItemProducerTree.TryMoveNextPageAsync(CancellationToken cancellationToken)
at Microsoft.Azure.Cosmos.Query.CosmosParallelItemQueryExecutionContext.DrainAsync(Int32 maxElements, CancellationToken cancellationToken)
at Microsoft.Azure.Cosmos.Query.Core.ExecutionContext.PipelinedDocumentQueryExecutionContext.ExecuteNextAsync(CancellationToken token)
at Microsoft.Azure.Cosmos.Query.Core.ExecutionContext.LazyCosmosQueryExecutionContext.ExecuteNextAsync(CancellationToken cancellationToken)
at Microsoft.Azure.Cosmos.Query.Core.ExecutionContext.CosmosQueryExecutionContextWithNameCacheStaleRetry.ExecuteNextAsync(CancellationToken cancellationToken)
at Microsoft.Azure.Cosmos.Query.Core.ExecutionContext.CatchAllCosmosQueryExecutionContext.ExecuteNextAsync(CancellationToken cancellationToken)).
{"assembly":"Microsoft.Azure.Cosmos.Client, Version=3.5.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35","method":"Microsoft.Azure.Cosmos.ResponseMessage.EnsureSuccessStatusCode","level":0,"line":0}
{"assembly":"Microsoft.Azure.Cosmos.Client, Version=3.5.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35","method":"Microsoft.Azure.Cosmos.CosmosResponseFactory.CreateQueryFeedResponseHelper","level":1,"line":0}
{"assembly":"Microsoft.Azure.Cosmos.Client, Version=3.5.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35","method":"Microsoft.Azure.Cosmos.CosmosResponseFactory.CreateQueryFeedResponse","level":2,"line":0}
{"assembly":"Microsoft.Azure.Cosmos.Client, Version=3.5.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35","method":"Microsoft.Azure.Cosmos.FeedIteratorCore`1+<ReadNextAsync>d__5.MoveNext","level":3,"line":0}
I've not used cosmosdb so not sure if this really relevant but accroding to the azure documentation each request is limited 4MB per request.
Am I correct in thinking in the example code you have given above there is no filtering? Meaning all 600 documents (~30k each) are returned?
You might have more success trying to split this into multipl requests
I installed rabbitmq service on the server and on my system.
I want to use RPC pattern:
var factory = new ConnectionFactory() {
HostName = "158.2.14.42",
Port = Protocols.DefaultProtocol.DefaultPort,
UserName = "Administrator",
Password = "#server#",
VirtualHost = "/"
ContinuationTimeout = new TimeSpan(10, 0, 0, 0)
};
connection = factory.CreateConnection();
I have an error on creating connection with this message:
None of the specified endpoints were reachable
When I use it on localhost instance of the server it works, but when I create the connection from local to that server,it returned the error.
It not work with local ip and username and password of the my local computer.
Can anyone help me?
Thank you all.
As this :
https://stackoverflow.com/questions/4987438/rabbitmq-c-sharp-connection-trouble-when-using-a-username-and-password
After I installed RabbitMQ, I enabled management tools at the server and on my local computer with this:
rabbitmq-plugins enable rabbitmq_management
Then I restarted RabbitMQ service from services.msc
I could see the rabbitmq management at http://localhost:15672
I loginned to rabbit management with user:guest and pass:guest
I added my favorite user pass with administrator access, so it worked.
I was also facing the same issue and later realized I have to open both ports i.e. 15672 and 5672.
The below command works for me in the docker container model.
docker run -it --rm --name mymq -p 5672:5672 -p 15672:15672 rabbitmq:3-management
Code snippet:
var factory = new RabbitMQ.Client.ConnectionFactory
{
Uri = new Uri("amqp://guest:guest#localhost:5672/")
};
or
var factory = new ConnectionFactory() { HostName = "localhost" };
Do not use guest. Create your own account and password, and in http://localhost:15672/#/users , ensure "can access virtual hosts " is "/"
var factory = new ConnectionFactory() {
HostName = "192.168.1.121",
Port = 5672,
UserName = "fancky",
Password = "123456"
};
In our case it was an assembly binding failure of System.Threading.Tasks.Extensions, our exception logger was not logging the InnerException
Exception information:
Exception type: FileLoadException
Exception message: Could not load file or assembly 'System.Threading.Channels, Version=4.0.2.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51' or one of its dependencies. The located assembly's manifest definition does not match the assembly reference. (Exception from HRESULT: 0x80131040)
at RabbitMQ.Client.Framing.Impl.AutorecoveringConnection..ctor(ConnectionFactory factory, String clientProvidedName)
at RabbitMQ.Client.ConnectionFactory.CreateConnection(IEndpointResolver endpointResolver, String clientProvidedName) in /_/projects/RabbitMQ.Client/client/api/ConnectionFactory.cs:line 494
Very poor misleading error message.
it means that the client can't reach the server 158.2.14.42 and default vhost /.
Maybe a firewall configuration
By default as you say RabbitMQ will listen on 5672 but this can be changed, if you have a look at your config (on the MQ server) you should find a section:
rabbit.tcp_listeners
which will detail the port being used. Check it's what you think it is.
Also, are you using IPv4 or IPv6? you may need to have additional config to support both.
Have a read of this:
https://www.rabbitmq.com/networking.html
1) Open RabbitMQ Command Promp
2) Change path to "C:\Program Files\RabbitMQ Server\rabbitmq_server-3.6.10\sbin" where rabbitmq_server installed.
3) Run following commamd :
rabbitmq-plugins enable rabbitmq_management
If Firewall has blocked it, then popup display. Allow Firewall for this.
now you can access in browser.
I've been experiencing this very issue, and in the end it was just about having provided the URI as lowercase, while the name of the machine was uppercase.
As it took a couple of hours to figure it out, I thought it might help someone else.
I created a small application using .netcore and rabbitmq and created the image in docker for both of them using below docker-compose code -
version: '3'
services:
rabbitmq:
container_name: rabbitmq
hostname: "rabbitmq"
image: rabbitmq:3-management
ports:
- "5672:5672"
- "15672:15672"
volumes:
- rabbitmq:/rabbitmq
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:15672"]
interval: 30s
timeout: 10s
retries: 5
isp_hub:
depends_on:
- rabbitmq
build:
context: .
dockerfile: Dockerfile
ports:
- "9090:80"
volumes:
rabbitmq:
after running the cmd docker-compose up the image is created and container is started and i am using Jmeter client to hit the container .net code is working but while trying to pass the data in queue getting below exception -
"log":"info: Microsoft.Hosting.Lifetime[0]\r\n","stream":"stdout","time":"2022-04-04T08:25:22.6109672Z"}
{"log":" Application is shutting down...\r\n","stream":"stdout","time":"2022-04-04T08:25:22.6109672Z"}
{"log":"info: Microsoft.Hosting.Lifetime[0]\r\n","stream":"stdout","time":"2022-04-04T08:25:38.4488984Z"}
{"log":" Now listening on: http://[::]:80\r\n","stream":"stdout","time":"2022-04-04T08:25:38.4488984Z"}
{"log":"info: Microsoft.Hosting.Lifetime[0]\r\n","stream":"stdout","time":"2022-04-04T08:25:38.4498991Z"}
{"log":" Application started. Press Ctrl+C to shut down.\r\n","stream":"stdout","time":"2022-04-04T08:25:38.4498991Z"}
{"log":"info: Microsoft.Hosting.Lifetime[0]\r\n","stream":"stdout","time":"2022-04-04T08:25:38.4498991Z"}
{"log":" Hosting environment: Production\r\n","stream":"stdout","time":"2022-04-04T08:25:38.4498991Z"}
{"log":"info: Microsoft.Hosting.Lifetime[0]\r\n","stream":"stdout","time":"2022-04-04T08:25:38.4498991Z"}
{"log":" Content root path: C:\app\r\n","stream":"stdout","time":"2022-04-04T08:25:38.4498991Z"}
{"log":"fail: Microsoft.AspNetCore.SignalR.Internal.DefaultHubDispatcher[8]\r\n","stream":"stdout","time":"2022-04-04T08:26:55.7889752Z"}
{"log":" Failed to invoke hub method 'SendToMessageBroker'.\r\n","stream":"stdout","time":"2022-04-04T08:26:55.7889752Z"}
{"log":" RabbitMQ.Client.Exceptions.BrokerUnreachableException: None of the specified endpoints were reachable\r\n","stream":"stdout","time":"2022-04-04T08:26:55.7889752Z"}
{"log":" ---\u003e System.AggregateException: One or more errors occurred. (Connection failed)\r\n","stream":"stdout","time":"2022-04-04T08:26:55.7889752Z"}
{"log":" ---\u003e RabbitMQ.Client.Exceptions.ConnectFailureException: Connection failed\r\n","stream":"stdout","time":"2022-04-04T08:26:55.7889752Z"}
{"log":" ---\u003e System.Net.Sockets.SocketException (10061): No connection could be made because the target machine actively refused it.\r\n","stream":"stdout","time":"2022-04-04T08:26:55.7889752Z"}
{"log":" at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)\r\n","stream":"stdout","time":"2022-04-04T08:26:55.7889752Z"}
{"log":" at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)\r\n","stream":"stdout","time":"2022-04-04T08:26:55.7889752Z"}
{"log":" at System.Threading.Tasks.ValueTask.ValueTaskSourceAsTask.\u003c\u003ec.\u003c.cctor\u003eb__4_0(Object state)\r\n","stream":"stdout","time":"2022-04-04T08:26:55.7889752Z"}
{"log":" --- End of stack trace from previous location ---\r\n","stream":"stdout","time":"2022-04-04T08:26:55.7889752Z"}
{"log":" at RabbitMQ.Client.Impl.TcpClientAdapter.ConnectAsync(String host, Int32 port)\r\n","stream":"stdout","time":"2022-04-04T08:26:55.7889752Z"}
{"log":" at RabbitMQ.Client.Impl.TaskExtensions.TimeoutAfter(Task task, TimeSpan timeout)\r\n","stream":"stdout","time":"2022-04-04T08:26:55.7889752Z"}
{"log":" at RabbitMQ.Client.Impl.SocketFrameHandler.ConnectOrFail(ITcpClient socket, AmqpTcpEndpoint endpoint, TimeSpan timeout)\r\n","stream":"stdout","time":"2022-04-04T08:26:55.7889752Z"}
{"log":" --- End of inner exception stack trace ---\r\n","stream":"stdout","time":"2022-04-04T08:26:55.7889752Z"}
{"log":" at RabbitMQ.Client.Impl.SocketFrameHandler.ConnectOrFail(ITcpClient socket, AmqpTcpEndpoint endpoint, TimeSpan timeout)\r\n","stream":"stdout","time":"2022-04-04T08:26:55.7889752Z"}
{"log":" at RabbitMQ.Client.Impl.SocketFrameHandler.ConnectUsingAddressFamily(AmqpTcpEndpoint endpoint, Func2 socketFactory, TimeSpan timeout, AddressFamily family)\r\n","stream":"stdout","time":"2022-04-04T08:26:55.7889752Z"} {"log":" at RabbitMQ.Client.Impl.SocketFrameHandler.ConnectUsingIPv4(AmqpTcpEndpoint endpoint, Func2 socketFactory, TimeSpan timeout)\r\n","stream":"stdout","time":"2022-04-04T08:26:55.7889752Z"}
{"log":" at RabbitMQ.Client.Impl.SocketFrameHandler..ctor(AmqpTcpEndpoint endpoint, Func2 socketFactory, TimeSpan connectionTimeout, TimeSpan readTimeout, TimeSpan writeTimeout)\r\n","stream":"stdout","time":"2022-04-04T08:26:55.7889752Z"} {"log":" at RabbitMQ.Client.Framing.Impl.IProtocolExtensions.CreateFrameHandler(IProtocol protocol, AmqpTcpEndpoint endpoint, Func2 socketFactory, TimeSpan connectionTimeout, TimeSpan readTimeout, TimeSpan writeTimeout)\r\n","stream":"stdout","time":"2022-04-04T08:26:55.7889752Z"}
{"log":" at RabbitMQ.Client.ConnectionFactory.CreateFrameHandler(AmqpTcpEndpoint endpoint)\r\n","stream":"stdout","time":"2022-04-04T08:26:55.7889752Z"}
{"log":" at RabbitMQ.Client.EndpointResolverExtensions.SelectOne[T](IEndpointResolver resolver, Func2 selector)\r\n","stream":"stdout","time":"2022-04-04T08:26:55.7889752Z"} {"log":" --- End of inner exception stack trace ---\r\n","stream":"stdout","time":"2022-04-04T08:26:55.7889752Z"} {"log":" at RabbitMQ.Client.EndpointResolverExtensions.SelectOne[T](IEndpointResolver resolver, Func2 selector)\r\n","stream":"stdout","time":"2022-04-04T08:26:55.7889752Z"}
{"log":" at RabbitMQ.Client.Framing.Impl.AutorecoveringConnection.Init(IEndpointResolver endpoints)\r\n","stream":"stdout","time":"2022-04-04T08:26:55.7889752Z"}
{"log":" at RabbitMQ.Client.ConnectionFactory.CreateConnection(IEndpointResolver endpointResolver, String clientProvidedName)\r\n","stream":"stdout","time":"2022-04-04T08:26:55.7889752Z"}
{"log":" --- End of inner exception stack trace ---\r\n","stream":"stdout","time":"2022-04-04T08:26:55.7889752Z"}
{"log":" at RabbitMQ.Client.ConnectionFactory.CreateConnection(IEndpointResolver endpointResolver, String clientProvidedName)\r\n","stream":"stdout","time":"2022-04-04T08:26:55.7889752Z"}
{"log":" at RabbitMQ.Client.ConnectionFactory.CreateConnection(String clientProvidedName)\r\n","stream":"stdout","time":"2022-04-04T08:26:55.7889752Z"}
{"log":" at RabbitMQ.Client.ConnectionFactory.CreateConnection()\r\n","stream":"stdout","time":"2022-04-04T08:26:55.7889752Z"}
{"log":" at ISP_Hub.HubConfig.CenterHub.SendToMessageBroker(String requestData) in C:\src\ISP_Hub\HubConfig\CenterHub.cs:line 38\r\n","stream":"stdout","time":"2022-04-04T08:26:55.7889752Z"}
{"log":" at Microsoft.Extensions.Internal.ObjectMethodExecutor.\u003c\u003ec__DisplayClass33_0.\u003cWrapVoidMethod\u003eb__0(Object target, Object[] parameters)\r\n","stream":"stdout","time":"2022-04-04T08:26:55.7889752Z"}
{"log":" at Microsoft.AspNetCore.SignalR.Internal.DefaultHubDispatcher1.ExecuteMethod(ObjectMethodExecutor methodExecutor, Hub hub, Object[] arguments)\r\n","stream":"stdout","time":"2022-04-04T08:26:55.7889752Z"} {"log":" at Microsoft.AspNetCore.SignalR.Internal.DefaultHubDispatcher1.\u003c\u003ec__DisplayClass16_0.\u003c\u003cInvoke\u003eg__ExecuteInvocation|0\u003ed.MoveNext()\r\n","stream":"stdout","time":"2022-04-04T08:26:55.7889752Z"}
rabbitmq connection code -
public void SendToMessageBroker(string requestData)
{
var trace = JsonConvert.DeserializeObject(requestData);
// string rabbitMqUrl = "127.0.0.1";
// var factory = new ConnectionFactory() { HostName = "172.24.17.225", Port = 5672 };
//var factory = new ConnectionFactory
//{
// HostName = rabbitMqUrl,
// UserName = "guest",
// Password = "guest",
// Port = AmqpTcpEndpoint.UseDefaultPort,
// VirtualHost = "/",
// RequestedHeartbeat = new TimeSpan(60),
// Ssl = { ServerName = rabbitMqUrl, Enabled = false }
//};
var factory = new RabbitMQ.Client.ConnectionFactory
{
Uri = new Uri("amqp://guest:guest#localhost:5672/")
};
using var conn = factory.CreateConnection();
using var channel = conn.CreateModel();
RabbitMQPublisher.Publish(channel, trace);
Clients.All.SendAsync("test");
channel.Close();
conn.Close();
Debug.WriteLine("Show Message: " + requestData);
}
i am new to docker . please help me to understand the issue and how to resolve it
Don't know if it helps you but it happen after update RabbitMQ.Client from 5.1.2 to 5.2.0 for .net 4.6.1.
If you rise target framework to 4.8 it goes through.
Misleading error. The local version works ok but the remote server didn't. Turned out I did not supply the credentials i.e. username and password which are mandatory for remote server. Local server happily connects without them.
I tryed to connect to OPC UA server using the client provided on this page: https://support.industry.siemens.com/cs/document/42014088/programming-an-opc-ua-net-client-with-c%23-for-the-simatic-net-opc-ua-server?dti=0&lc=en-US . Connection to OPC UA server using the Siemens OPC Scount v10 works fine. When connecting to the OPC UA server using client provided in the article, I get this message:
Could not open UA TCP request channel.
Stack trace of the exception is this:
Server stack trace:
at Opc.Ua.Bindings.UaTcpRequestChannel.OnEndOpen(IAsyncResult result)
at Opc.Ua.Bindings.UaTcpRequestChannel.OnOpen(TimeSpan timeout)
at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannel.OnOpen(TimeSpan timeout)
at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannel.CallOpenOnce.System.ServiceModel.Channels.ServiceChannel.ICallOnce.Call(ServiceChannel channel, TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannel.CallOnceManager.CallOnce(TimeSpan timeout, CallOnceManager cascade)
at System.ServiceModel.Channels.ServiceChannel.EnsureOpened(TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs)
at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)
at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)
Exception rethrown at [0]:
at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
at Opc.Ua.ISessionChannel.CreateSession(CreateSessionMessage request)
at Opc.Ua.SessionChannel.CreateSession(CreateSessionMessage request)
at Opc.Ua.SessionClient.CreateSession(RequestHeader requestHeader, ApplicationDescription clientDescription, String serverUri, String endpointUrl, String sessionName, Byte[] clientNonce, Byte[] clientCertificate, Double requestedSessionTimeout, UInt32 maxResponseMessageSize, NodeId& sessionId, NodeId& authenticationToken, Double& revisedSessionTimeout, Byte[]& serverNonce, Byte[]& serverCertificate, EndpointDescriptionCollection& serverEndpoints, SignedSoftwareCertificateCollection& serverSoftwareCertificates, SignatureData& serverSignature, UInt32& maxRequestMessageSize)
at Opc.Ua.Client.Session.Open(String sessionName, UInt32 sessionTimeout, IUserIdentity identity, IList`1 preferredLocales)
at Opc.Ua.Client.Session.Open(String sessionName, IUserIdentity identity)
Any help would be appreciated. OPC UA server runs on Siemens Simatic HMI TP700 Comfort. Configuration on the OPC UA server is default.
After andrewcullen tip, we are getting the below log in the tracelog.txt file and error when catching the exception that says
An unexpected error occurred while connecting to the server.
PID:4196 ************************* Logging started at 02/03/2016 07:41:34 *************************
4196 - 07:41:38.742 GetEndpoints Called. RequestHandle=1, PendingRequestCount=1
4196 - 07:41:38.992 SECURE CHANNEL CREATED [TcpClientChannel UA-TCP 1.00.238.1] [ID=12752] Connected To: opc.tcp://xxx.xxx.xxx.xxx:4870/
4196 - 07:41:39.008 TCPCLIENTCHANNEL SOCKET CONNECTED: 00000698, ChannelId=12752
4196 - 07:41:39.008 SECURE CHANNEL CREATED [Opc.Ua.ChannelBase WCF Client 1.00.238.1] [ID=] Connected To: opc.tcp://xxx.xxx.xxx.xxx:4870/
4196 - 07:41:39.101 GetEndpoints Completed. RequestHandle=1, PendingRequestCount=0
4196 - 07:41:39.132 TCPCLIENTCHANNEL SOCKET CLOSED: 00000698, ChannelId=12752
4196 - 07:41:44.230 Writing rejected certificate to directory:
4196 - 07:41:59.694 CreateSession Called. RequestHandle=1, PendingRequestCount=1
4196 - 07:42:13.672 TCPCLIENTCHANNEL SOCKET CLOSED: 000007C0, ChannelId=0
4196 - 07:42:13.750 CreateSession Completed. RequestHandle=1, PendingRequestCount=0
I got the answer from the Siemens official support:
The application was not tested with Comfort Panel. The code e.g. contains Block Read and Block Write which is not supported from the Panel Server.
So this application will not work.
This Siemens UaClient uses a library 'ClientAPI' which extends the OPC Foundation's Opc.Ua.Core and Opc.Ua.Client. ClientAPI has a lot of nice Helper functions to simplify connecting and subscribing. However, I see in the code for Connect(string Url) that it is using the original WCF-style channel. And your stack trace is showing the WCF types are throwing an exception that is hard to diagnose. I would change two things:
First configure tracing to write to a file. In ClientAPI, find Helpers.CreateClientConfiguration() and add
// add trace config before calling validate
configuration.TraceConfiguration = new TraceConfiguration {
OutputFilePath="tracelog.txt",
DeleteOnLoad = true,
TraceMasks = Utils.TraceMasks.All };
configuration.Validate(ApplicationType.Client);
Second, upgrade the channel type used to connect. In ClientAPI, find Server.Connect(string url) and modify the middle as shown:
// Initialize the channel which will be created with the server.
// SessionChannel channel = SessionChannel.Create(
// configuration,
// endpointDescription,
// endpointConfiguration,
// bindingFactory,
// clientCertificate,
// null);
ITransportChannel channel = WcfChannelBase.CreateUaBinaryChannel(
configuration,
endpointDescription,
endpointConfiguration,
clientCertificate,
configuration.CreateMessageContext());
// Wrap the channel with the session object.
// This call will fail if the server does not trust the client certificate.
// m_Session = new Session(channel, configuration, endpoint);
m_Session = new Session(channel, configuration, endpoint, clientCertificate);
Edit 2/4:
From the tracelog you might find certificate errors. When creating a new session, the client and server both provide and validate each others certificate. By default, UaClient is retrieving it's cert from the windows store LocalMachine\My (aka Personal). The api generates this cert during it's first run, (which requires the first run as administrator) ( to see this cert, run 'certlm.msc').
On the server machine, the server will validate the client's cert, by checking if it matches the certs in its 'TrustedPeerList'. Servers usually use a directory to store the trusted certs. If the client cert is not trusted, the server will copy the client's cert to a 'RejectedCertificates' directory. You are required to copy the cert you find in 'RejectedCertificates' to the trusted cert directory.
Back on the client machine, the client will validate the server's cert. This client uses a windows store for validation 'LocalMachine\My' (aka Personal).
Instead of using a 'Rejected' directory, the client registers an event handler that opens a message box, asking if you wish to accept the server's cert. If you choose to accept, the client sets the eventArg e.Accept = true; To suppress the message box, the server's cert should be imported into the client's 'LocalMachine\My' (aka Personal) using the tool 'certlm.msc'.
Try to ping the server with DNS name. If the server is not accessible the Hosts file in C:\Windows\System32\drivers\etc... must be edited. Open the notepad as administrator, then open the Hosts file and enter the mapping of IP Address to the host name as follows:
xxx.xxx.xxx.xxx host name
I have a problem running a WCF service that connects to the CRM: It frequently produces CommunicationObjectAbortedExceptions which leave me to wonder if I am doing something wrong. These execptions started occurring after many people started using it, on the test system it worked without problems.
But let's start at the beginning: I wrote two WCF Services that connect to the Microsoft CRM2013 Organization Service using my own library to execute queries on the CRM. These services are regularly called from the CRM which is used by roughly 100-200 people on a daily basis.
This works basically fine, but I frequently get a couple of exceptions which look like the following (see bottom of the Post for the full stacktrace):
System.ServiceModel.CommunicationObjectAbortedException: The HTTP request to 'http://crm/MyOrganization/XRMServices/2011/Organization.svc' was aborted. This may be due to the local channel being closed while the request was still in progress. If this behavior is not desired, then update your code so that it does not close the channel while request operations are still in progress.
By frequently I mean around 100 times a day, most often a couple of those exceptions are thrown every 5-30 minutes in batches of 3-6 exceptions. I have no idea why this is happening. I initialize the connection to the CRM Organization Service using the following class from my library in both services:
public class CrmManager : IDisposable
{
private static CrmConnection s_connection;
public static CrmConnection Connection
{
get
{
if (s_connection == null)
{
s_connection = new CrmConnection("CrmTvTest");
}
return s_connection;
}
}
public static IOrganizationService ServiceProxy
{
get { return s_serviceProxy ?? (s_serviceProxy = new CachedOrganizationService(Connection)); }
}
As can be seen, I connect to the Organization Service once per WCF service, using the CrmConnection to handle the connection details, which is stored in a static variable (acting as a singleton, since establishing the Connection is expensive and should not be done too often to my understanding). It is then passed to the CachedOrganizationService, which is static for the same reasons. The WCF service uses the default instance management (PerSession AFAIK), meaning there is probably 1 connection and organization service per user.
My Connection String looks like this (removed any sensible data, of course):
<connectionStrings>
<add name="CrmTvTest" connectionString="Url=http://crm/MyOrganization; Username=user; Password=pw;"/>
I then use the connection with CrmServiceContext objects to execute queries using this method from my CrmManager class. Which is, of course, always called within a using-statement:
using (CrmServiceContext context = new CrmServiceContext(CrmManager.ServiceProxy))
{
// do some stuff...
}
How can I prevent these exceptions from constantly occuring? I get the feeling this has to do with the Security Tokens used by the CRM connection expiring, but this shouldn't be a problem when I use the CrmConnection class. It should refresh them automatically.
Any advice would be very welcome, since I am pondering this issue for a while now.
UPDATE 1
I switched to using the Developer Extensions and using the CrmConnector class, to no avail (I updated the code above). I also tried passing the CrmConnection class directly to the CrmServiceContext:
using (CrmServiceContext context = new CrmServiceContext(CrmManager.Connection))
which led to the same problems as in this Stackoverflow Question, without using a load-balancer (we initially did, but disabled load-balancing to eliminate the possibility of it causing the problems.
Full Stacktrace:
---> System.Net.WebException: The request was aborted: The request was canceled.
at System.Net.HttpWebRequest.GetResponse()
at System.ServiceModel.Channels.HttpChannelFactory`1.HttpRequestChannel.HttpChannelRequest.WaitForReply(TimeSpan timeout)
--- End of inner exception stack trace ---
Server stack trace:
at System.ServiceModel.Channels.HttpChannelUtilities.ProcessGetResponseWebException(WebException webException, HttpWebRequest request, HttpAbortReason abortReason)
at System.ServiceModel.Channels.HttpChannelFactory1.HttpRequestChannel.HttpChannelRequest.WaitForReply(TimeSpan timeout)
at System.ServiceModel.Channels.RequestChannel.Request(Message message, TimeSpan timeout)
at System.ServiceModel.Channels.SecurityChannelFactory1.SecurityRequestChannel.Request(Message message, TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)
at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)
Exception rethrown at [0]:
at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
at Microsoft.Xrm.Sdk.IOrganizationService.Execute(OrganizationRequest request)
at Microsoft.Xrm.Sdk.Client.OrganizationServiceProxy.ExecuteCore(OrganizationRequest request)
at Microsoft.Xrm.Sdk.Client.OrganizationServiceContext.Execute(OrganizationRequest request)
at Microsoft.Xrm.Sdk.Linq.QueryProvider.RetrieveEntityCollection(OrganizationRequest request, NavigationSource source)
at Microsoft.Xrm.Sdk.Linq.QueryProvider.Execute(QueryExpression qe, Boolean throwIfSequenceIsEmpty, Boolean throwIfSequenceNotSingle, Projection projection, NavigationSource source, List1 linkLookups, String& pagingCookie, Boolean& moreRecords)
at Microsoft.Xrm.Sdk.Linq.QueryProvider.Execute[TElement](QueryExpression qe, Boolean throwIfSequenceIsEmpty, Boolean throwIfSequenceNotSingle, Projection projection, NavigationSource source, List1 linkLookups)
at Microsoft.Xrm.Sdk.Linq.QueryProvider.Execute[TElement](Expression expression)
at Microsoft.Xrm.Sdk.Linq.QueryProvider.System.Linq.IQueryProvider.Execute[TResult](Expression expression)
at System.Linq.Queryable.SingleOrDefault[TSource](IQueryable`1 source)
at CrmConnector.Entities.Contact.Get(Guid p_id, Boolean p_includeRelatedEntities) in j:\IntDev\Libraries\CrmConnector\Entities\Contact.cs:line 63
at CrmExtensionService.CrmExtension.GetPersonalizedEmailSignature(String p_contactId, String p_systemUserId) in j:\IntDev\Services\CrmExtensionService\CrmExtension.svc.cs:line 460
We had same issue and load balancer was the culprit. Now we resolved this error with load balancer activated by making few configurations in the load balancer. We enabled sticky session with least connection algorithm at the load balancer. So if this is not enabled while load balanced then authenticated connection from one server's request gets routed to different servers even though same session and fails. Once enabling the sticky session (session persistance to be client ip) requests goes to same server (in this case returning connection and not a new connection) it works well.
So, after fiddling around for about two months, we found the issue: The load balancing of the CRM FrontEnd was the culprit. I assumed this was disabled, too, with disabling the load balancing of our CRM Service, but it wasn't. Our CRM Service periodically established a connection with the Organization Service of Server 1, then got switched to Server 2 mid-operation and these exceptions occurred.
We're still trying to figure out how to get this to work with load balancing activated, but for the time being we keep it disabled to prevent these errors from popping up.
There is a similar case here on StackOverflow: Sporadic exceptions calling a web service that is load balanced. We are currently using a webHttpBinding and a quickly attempted to switch over to a basicHttpBinding but didn't get it to work (but as I said, this was just a quick attempt).
We have a particular SQL Server which we need to access from a thick (.Net 4.0 WPF) client, and the only credentials available to us for that connection is a 'service' account which is effectively an Active Directory account with permission on the SQL Server.
I am using Entity Framework, and ObjectContext, throughout the project, so am continuing with it here. After looking around, I have implemented an impersonation routine based on LogonUserEx, and DuplicateTokenEx, which allows me, via Dependency Injection, to write the following:
using (container.Resolve<Impersonate>())
using (var context = container.Resolve<MyObjectContext>())
{
context.Connection.Open();
//Do some work with the data as the service account.
context.Connection.Close();
}
The constructor of the Impersonate class above calls LogonUserEx and so on. I am explicitly opening and closing the connection as part of a Unit Of Work pattern, which shouldn't be relevant.
Now, via debugging I have found that the token is successfully retrieved for the service account and the user is 'impersonated'. However, as soon as I try to instantiate the ObjectContext I get the following error:
System.TypeInitializationException: The type initializer for 'EntityBid' threw a
n exception. ---> System.IO.FileLoadException: Could not load file or assembly '
System.Data.Entity.dll' or one of its dependencies. Either a required impersonat
ion level was not provided, or the provided impersonation level is invalid. (Exc
eption from HRESULT: 0x80070542)
at System.Runtime.InteropServices.Marshal.GetHINSTANCE(RuntimeModule m)
at System.Runtime.InteropServices.Marshal.GetHINSTANCE(Module m)
at EntityBid.initEntryPoint()
at EntityBid.internalInitialize()
at EntityBid..cctor()
--- End of inner exception stack trace ---
at EntityBid.Trace(String fmtPrintfW, String a1)
at System.Data.EntityUtil.ProviderExceptionWithMessage(String message, Except
ion inner)
at System.Data.EntityClient.EntityConnection.OpenStoreConnectionIf(Boolean op
enCondition, DbConnection storeConnectionToOpen, DbConnection originalConnection
, String exceptionCode, String attemptedOperation, Boolean& closeStoreConnection
OnFailure)
at System.Data.EntityClient.EntityConnection.Open()
at Core.Security.TestHarness._2.Class1..ctor()
Consequently, it would appear that the account I am impersonating no longer has access or sufficient privelage to load the DLL's from the GAC. The Fusion log is not giving any additional information.
I am not sure how to solve this. I wonder if I am not currently retrieving a token with sufficient privelage. Note that I am providing these paramteres to LogonUserEx: LOGON32_LOGON_NETWORK_CLEARTEXT and LOGON32_PROVIDER_DEFAULT.
Finally, note that this process works absolutely fine on a machine with administrative privelages for the logged on user. It breaks when I run it on a user with a 'normal' account, subject to the usual corporate GPO!
EDIT: Just to include the 'important' part of the impersonation. Notice that I have now swapped to LOGON32_LOGON_UNLOCK, as it is working better. Apologies for the slightly iffy formatting:
if (LogonUserEx(dUser, dDomain, dPassword, LOGON32_LOGON_UNLOCK,
LOGON32_PROVIDER_DEFAULT, out token, IntPtr.Zero, IntPtr.Zero, IntPtr.Zero, IntPtr.Zero))
{
if (DuplicateTokenEx(token, MAXIMUM_ALLOWED, ref sa, SECURITY_IMPERSONATION_LEVEL.SecurityDelegation, TOKEN_TYPE.TokenPrimary, out tokenDuplicate))
{
m_ImpersonatedUser = new WindowsIdentity(token);
_windowsImpersonationContext = m_ImpersonatedUser.Impersonate();
Any help greatly appreciated.
Nick.