I'm trying to use Couchbase from .NET with the official SDK. I'm hosting the Couchbase cluster in Amazon EC2.
The machine I'm trying to connect from is hosted in Microsoft Azure.
For some reason, opening a bucket is extremely slow, this afternoon it took ~3 seconds to open a bucket, but now for some reason it's more than 10 seconds.
The memory and CPU utilization of the Couchbase servers are very low.
My configuration:
<couchbaseClients>
<couchbase useSsl="false" operationLifespan="1000">
<servers>
<!-- Ip addresses obscured... -->
<add uri="http://1.1.1.1:8091/pools"></add>
<add uri="http://1.1.1.2:8091/pools"></add>
</servers>
</couchbase>
</couchbaseClients>
The code I'm trying:
var cluster = new Cluster("couchbaseClients/couchbase");
using (var bucket = cluster.OpenBucket("bucketname")) // This is taking 10-50 seconds.
{
var obj = new TestClass { };
// This is fast
var result = bucket.Insert(new Document<TestClass> { Content = obj, Expiry = 300000, Id = Guid.NewGuid().ToString() });
}
using (var bucket = cluster.OpenBucket("bucketname")) // This is taking 10-50 seconds.
{
var obj = new TestClass { };
// This is fast
var result = bucket.Insert(new Document<TestClass> { Content = obj, Expiry = 300000, Id = Guid.NewGuid().ToString() });
}
using (var bucket = cluster.OpenBucket("bucketname")) // This is taking 10-50 seconds.
{
var obj = new TestClass { };
// This is fast
var result = bucket.Insert(new Document<TestClass> { Content = obj, Expiry = 300000, Id = Guid.NewGuid().ToString() });
}
I tried to optimize by using ClusterHelper.GetBucket() instead of cluster.OpenBucket(), but the first GetBucket() is still very slow (everything else is fast in that case).
I experimented with other calls on the bucket (Get, Contains, etc.), and everything is fast, only opening the bucket itself is slow.
How can I troubleshoot what the problem is?
UPDATE: I set up logging based on #jeffrymorris' suggestion, and the I see the following error messages at the point where we spend a lot of time:
DEBUG Couchbase.Configuration.Server.Providers.ConfigProviderBase - Bootstrapping with 127.0.0.1:11210
INFO Couchbase.IO.ConnectionPool... Node 127.0.0.1:11210 failed to initialize, reason: System.Net.Sockets.SocketException (0x80004005): A connection attempt failed because the connected party did not properly respond after a period of time...
And after this there are three other exceptions about trying to connect to 127.0.0.1:11210, and there are seconds passed between those messages, so this is causing the slowness.
Why is the client trying to connect to localhost? I don't have that anywhere in my configuration.
I debugged the code of couchbase-net-client, and this is happening when I call OpenBucket() in CarrierPublicationProvider.GetConfig(). It gets the bucket configuration object, and for some reason it contains not one of the servers I configured in the web.config, but localhost:8091 instead. I'm still trying to figure out why this is happening.
(This is probably not happening on my dev machine, because there I have a local Couchbase server installation. If I stop that, then it gets slow there as well, and I'm seeing the same Bootstrapping with 127.0.0.1:11210 error messages in the logs.)
Figuring out the problem took some debugging of the Couchbase client code, and looking at the logs (thanks for the tip, #jeffrymorris!).
It turns out that if you don't have any preconfigured bucket, then the client will initially try to connect to localhost:8091 (event if that's not in your servers section), which can take seconds until it realizes there is no server there. This doesn't happen if you have any bucket configured.
So I had to change my configuration from this:
<couchbaseClients>
<couchbase useSsl="false" operationLifespan="1000">
<servers>
<!-- Ip addresses obscured... -->
<add uri="http://1.1.1.1:8091/pools"></add>
<add uri="http://1.1.1.2:8091/pools"></add>
</servers>
</couchbase>
</couchbaseClients>
To this:
<couchbaseClients>
<couchbase useSsl="false" operationLifespan="1000">
<servers>
<!-- Ip addresses obscured... -->
<add uri="http://1.1.1.1:8091/pools"></add>
<add uri="http://1.1.1.2:8091/pools"></add>
</servers>
<buckets>
<add name="default" useSsl="false" operationLifespan="1000">
</add>
</buckets>
</couchbase>
</couchbaseClients>
Note that I'm not actually going to use a bucket called default, but having the configuration there sill prevent the client from trying to connect to localhost. (This is happening in the method ConfigProviderBase.GetOrCreateConfiguration.)
Update: The issue has been fixed here: https://github.com/couchbase/couchbase-net-client/commit/020093b422a78728dd49d75d9fe9f1e00d01a0f2
Related
I have a application, where the user can upload xml files. Everything under 25mb is no problem, but when i try to upload my test file (117 mb), I get an error when the application is hosted.
Since I prepared the application like described in 1000 other posts, its working locally up to 2gb and also when hosted. But after upload i get "HTTP Error 503.0 - Service Unavailable".
When i log in again, the file is there but the error is inconvenienced.
web.config:
<security>
<requestFiltering>
<requestLimits maxAllowedContentLength="1073741824" />
</requestFiltering>
</security>
upload function:
[HttpPost]
[RequestFormLimits(MultipartBodyLengthLimit = 1073741824)]
public IActionResult Upload(IFormFile file){
if(file == null) return RedirectToAction("OnixIndex");
string completePath = app_resources_path + file.FileName;
using(FileStream fs = System.IO.File.Create(completePath))
{
file.CopyTo(fs);
fs.Flush();
}
startup.cs
services.Configure<FormOptions>(options =>
{
options.ValueLengthLimit = int.MaxValue;
options.MultipartBodyLengthLimit = int.MaxValue;
options.MultipartHeadersLengthLimit = int.MaxValue;
});
Should i use a different way to upload the file? I tryed a stream, but without improvement. I there a different technology or nuget bib?
could you please list system specs (Ram,Storage Type,Cpu Cores, .net version),
this error is mostly related to system specs not being able to handle the 1 gb file and you need to scale your system as mentioned in the following thread.
503 Server Unavailable after user uploads many files. Is this an httpRuntime setting?
When I publish my API project to Azure here is the error what i'm getting.
Is there any way to resolve this issue and here is the code and the problem comes when this symbol ":" is init.
Here are some more details.
It's a web API Project
Version 4.6
Locally it's running without any issues but when it comes to release automation I should be able to change the value of endpointurl , key, and TTL manually so that I need to modify .csfg and .csdef file environment to environment. When i do that .csdef does not support that colon mark ":" so the build fails.
Expecting : Build is successful so that the log works as expected.
Actual : Build fails and log is not working as expected.
As you mentioned that ServiceDefinition:NamedElementNameString doesn't allow ':' (colon) in name. But we could add it with Azure friendly name. Then we could get it with following code. I also do a demo on my side, it works as expected.
var endpoint = RoleEnvironment.GetConfigurationSettingValue("endpointUrl");
var authorizationKey = RoleEnvironment.GetConfigurationSettingValue("authorizationKey");
var logger = new LoggerConfiguration()
.WriteTo.Console() //if no writeto.console there is no document in documentdb
.WriteTo.AzureDocumentDB(endpoint, authorizationKey)
.CreateLogger();
logger.Information("Tom Test");//log demo
About .csdef configuration please refer to the screenshot.
We could get more information from Configure Azure cloud service roles with Visual Studio
Check from Azure Portal:
Related serilog sdk
It seems this is not supported.
ServiceDefinition:NamedElementNameString doesn't allow ':' (colon) in name
So that I implemented custom configuration values and extract the values at run-time when the logger is initialized.
Here is the implementation.
.cscfg, .csdef, and web.config contains
<add key="LogEndpointUrl" value="xxxxxx/" />
<add key="LogAuthorizationKey" value="xxxxxxxxxxxxxxx=" />
<add key="LogTTL" value="1" />
When initializing got the values as follows from web.config
var endpoint = Common.Configuration.GetSetting(Constants.AppSettings.LogEndpointUrl);
var authorizationKey = Common.Configuration.GetSetting(Constants.AppSettings.LogAuthorizationKey);
int ttl = (int)Convert.ToInt64((Common.Configuration.GetSetting(Constants.AppSettings.LogTTL)));
And then
Log.Logger = new LoggerConfiguration().ReadFrom.AppSettings().WriteTo.AzureDocumentDB(endpoint, authorizationKey,timeToLive: ttl).CreateLogger();
// Used to debug serilog itself and confirm it is writing entries to document db
Serilog.Debugging.SelfLog.Enable(Console.Out);
var errorOrInformation = new Dictionary<string, string>();
errorOrInformation.Add(Constants.LoggingProperties.PartitionKey, logMetadata.PartitionKey);
errorOrInformation.Add(Constants.LoggingProperties.RowKey, logMetadata.RowKey);
//Add as many items as you want
Log.Verbose("Log Information Message {Information}", errorOrInformation);
// Also good idea to force flush of log entries before the app ends
Log.CloseAndFlush();
I'm migrating over from Couchbase Server 4.x to Couchbase Server 5.x. I understand that there is a new user-based authentication. But it's not clear to me how I should be using ClusterHelper since there is no longer a password for the bucket.
Previously:
// at app startup
ClusterHelper.Initialize(new ClientDefinition { ... });
// later in DAL
var cluster = ClusterHelper.GetBucket("bucketname", "bucketpassword");
But now, using 5.x, there's no longer a bucket password. Where do I enter the user credentials?
There is a new overload to ClusterHelper.Initialize with an IAuthenticator parameter.
So now, for example:
// at app startup
ClusterHelper.Initialize(new ClientDefinition { ... }, new PasswordAuthenticator("username", "password"));
// then later in the your data access:
var cluster = ClusterHelper.GetBucket("bucketname");
Here's the overload in the ClusterHelper.cs
If you were using Web.config previously, you may also have to change that. The password is no longer on the <bucket /> node, it's now on the <couchbase /> node. See this answer in the forums.
I'm trying to use a Couchbase cluster hosted in Amazon EC2. The client I'm trying to use it from is hosted in Microsoft Azure.
The performance is terrible, in ~10% of the time opening a bucket takes a lot of time.
This is my configuration:
<couchbaseClients>
<couchbase useSsl="false" operationLifespan="1000">
<servers>
<!-- Ip addresses obscured... -->
<add uri="http://1.1.1.1:8091/pools"></add>
<add uri="http://1.1.1.2:8091/pools"></add>
</servers>
<buckets>
<add name="default" useSsl="false" operationLifespan="1000">
</add>
</buckets>
</couchbase>
</couchbaseClients>
This is the code I'm testing with:
var cluster = new Cluster("couchbaseClients/couchbase");
using (var bucket = cluster.OpenBucket("bucketname")) // This sometimes takes 3-50 seconds.
{
var obj = new TestClass { };
// This is fast
var result = bucket.Insert(new Document<TestClass> { Content = obj, Expiry = 300000, Id = Guid.NewGuid().ToString() });
}
Opening the Couchbase bucket sometimes (not always) takes a lot of time, anywhere between 3-50 seconds. It happens often enough that it makes it completely unusable.
When it happens, I can see the following error message in the Couchbase logs:
2015-12-10 14:18:57,644 [1] DEBUG Couchbase.Configuration.Server.Providers.ConfigProviderBase - Bootstrapping with 1.1.1.2:11210
2015-12-10 14:19:07,660 [1] INFO Couchbase.IO.ConnectionPool`1[[Couchbase.IO.Connection, Couchbase.NetClient, Version=2.2.2.0, Culture=neutral, PublicKeyToken=05e9c6b5a9ec94c2]] - Node 1.1.1.2:11210 failed to initialize, reason: System.Net.Sockets.SocketException (0x80004005): A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond
at Couchbase.IO.DefaultConnectionFactory.<GetGeneric>b__0[T](IConnectionPool`1 p, IByteConverter c, BufferAllocator b)
at Couchbase.IO.ConnectionPool`1.Initialize()
Note that 10 seconds are passed there. (I obscured the IP-addresses.)
What can cause this problem and how could I troubleshoot?
This seems to be Azure-specific, I could not reproduce this on my local dev machine or on a machine hosted in the Google cloud. However, it is consistently happening on two different Azure VMs.
Im trying to access MS Azure Cache service and pretty often I get this error:
Microsoft.ApplicationServer.Caching.DataCacheException: ErrorCode<ERRCA0017>:SubStatus<ES0006>:There is a temporary failure. Please retry later. (One or more specified cache servers are unavailable, which could be caused by busy network or servers. For on-premises cache clusters, also verify the following conditions. Ensure that security permission has been granted for this client account, and check that the AppFabric Caching Service is allowed through the firewall on all cache hosts. Also the MaxBufferSize on the server must be greater than or equal to the serialized object size sent from the client.). Additional Information : The client was trying to communicate with the server: net.tcp://remoto.cache.windows.net: line 23233.
at Microsoft.ApplicationServer.Caching.DataCache.ThrowException(ErrStatus errStatus, Guid trackingId, Exception responseException, Byte[][] payload, EndpointID destination)
at Microsoft.ApplicationServer.Caching.SocketClientProtocol.Get(String key, ref DataCacheItemVersion version, ref TimeSpan timeout, ref ErrStatus err, String region, IMonitoringListener listener)
at Microsoft.ApplicationServer.Caching.DataCache.InternalGet(String key, ref DataCacheItemVersion version, String region, IMonitoringListener listener)
at Microsoft.ApplicationServer.Caching.DataCache.<>c__DisplayClass53.<Get>b__52()
at Infrastructure.Azure.Cache.AzureCacheServiceClient.<>c__DisplayClass6`1.<Get>b__5() in AzureCacheServiceClient.cs: line 88
at Microsoft.Practices.EnterpriseLibrary.TransientFaultHandling.RetryPolicy.ExecuteAction(Func`1 func)
It happens when Im trying to access some object in cache like this:
public T Get<T>(string key)
{
retryPolicy.ExecuteAction(() =>(T) (_cache.Get(key)));
}
Here is my initialization code:
var cacheFactory = new DataCacheFactory();
_cache = cacheFactory.GetDefaultCache();
var retryStrategy = new FixedInterval(15, TimeSpan.FromSeconds(2));
_retryPolicy = new RetryPolicy<CustomCacheTransientErrorDetectionStrategy>(retryStrategy);
And app.config:
<configSections>
<section name="dataCacheClients" type="Microsoft.ApplicationServer.Caching.DataCacheClientsSection, Microsoft.ApplicationServer.Caching.Core" allowLocation="true" allowDefinition="Everywhere" />
<section name="cacheDiagnostics" type="Microsoft.ApplicationServer.Caching.AzureCommon.DiagnosticsConfigurationSection, Microsoft.ApplicationServer.Caching.AzureCommon" allowLocation="true" allowDefinition="Everywhere" />
</configSections>
<dataCacheClients>
<dataCacheClient channelOpenTimeout="1000" requestTimeout="45000" name="default">
<autoDiscover isEnabled="true" identifier="[some.host.name]" />
<securityProperties mode="Message" sslEnabled="true">
<messageSecurity authorizationInfo="***" />
</securityProperties>
</dataCacheClient>
</dataCacheClients>
It happend at least three times already while the Azure Health Status (https://azure.microsoft.com/en-us/status/) said everything was fine at that time.
As the exception message says - there are some 'temporary' failures on MS side still maybe I'm doing smth wrong in my code?
Do you get the error a few times, and then things start working again? If so, this is expected behavior and your application should have a policy where you retry some number of times before falling back to the persistent data store.
If you get the error consistently (more than a few seconds), it's likely the DataCacheFactory has gotten into a bad state. You can either restart your client process, or else refresh the DataCacheFactory as described in this blog post.