We Keep Coding

Dapr Resource Bindings HTTP GET from external service

I tried to create a Dapr component like the following:
apiVersion: dapr.io/v1alpha1
kind: Component
metadata:
name: nominatim
namespace: default
spec:
type: bindings.http
version: v1
metadata:
- name: url
value: https://nominatim.openstreetmap.org
- name: method
value: GET
- name: UserAgent
value: Other
Then in my service I Invoked this binging like this:
var getAddressUrl = $"{UriString}/reverse?format=jsonv2&lat={latitude}&lon={longitude}&zoom=18&addressdetails=1";
var invokeBindingAsync = _daprClient.InvokeBindingAsync<object, AddressLocationDto>(
DaprComponentsSettings.HttpNominatim,
DaprComponentsSettings.CreateBindingOperation,
null,
new Dictionary<string, string>
{
["url"] = getAddressUrl
});
var address = await invokeBindingAsync;
return address;
Unfortunately, I receive the following error:
Status(StatusCode="Internal", Detail="Error starting gRPC call. HttpRequestException: Connection refused (127.0.0.1:50001) SocketException: Connection refused", DebugException="System.Net.Http.HttpRequestException: Connection refused (127.0.0.1:50001)
---> System.Net.Sockets.SocketException (111): Connection refused
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
at System.Net.Sockets.Socket.<ConnectAsync>g__WaitForConnectWithCancellation|283_0(AwaitableSocketAsyncEventArgs saea, ValueTask connectTask, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.DefaultConnectAsync(SocketsHttpConnectionContext context, CancellationToken cancellationToken)
at System.Net.Http.ConnectHelper.ConnectAsync(Func`3 callback, DnsEndPoint endPoint, HttpRequestMessage requestMessage, CancellationToken cancellationToken)
--- End of inner exception stack trace ---
at System.Net.Http.ConnectHelper.ConnectAsync(Func`3 callback, DnsEndPoint endPoint, HttpRequestMessage requestMessage, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.GetHttp2ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at Grpc.Net.Client.Internal.GrpcCall`2.RunCall(HttpRequestMessage request, Nullable`1 timeout)")
I do not know how to call HTTP get like this. Please help.

Elasticsearch.Net 7 & Nest 7 - Invalid bulk update request

I'm trying to create a .Net Core (netcoreapp3.1) application to send my data from Azure Eventhub to ES 7.
I'm using the following packages:
ElasticSearch.Net 7.8.1
Nest 7.8.1
The data that I retrieve from Eventhub are 2 types that inherit from the IElasticBaseEntity.
I want to bulk update a list of these objects, which can be the object with all the information or the object to update one field of an already indexed person.
The field to match / search in ES is the id field.
To simplify my example I will use these dummy classes:
public interface IElasticBaseEntity
{
string Id { get; set; }
DateTime ServerTimestamp { get; set; }
}
The person class is the one with all the information
public abstract class Person: IElasticBaseEntity
{
public string Id { get; set; }
public string Firstname {get; set;}
public string Name {get; set;}
public decimal? Score { get; set; }
}
The Score class is the update that I want to do on the indexed Person, based on the Id
public abstract class Score : IElasticBaseEntity
{
public string Id {get; set;}
public decimal? Score { get; set; }
}
I use this method to build the connection to ES
public static IElasticClient CreateInstance(ElasticsearchConfig config)
{
IConnectionPool pool;
var connection = new HttpConnection();
pool = new SniffingConnectionPool(new[] { new Uri(config.Url) }) { SniffedOnStartup = true };
var connectionSettings = new ConnectionSettings(pool, connection);
connectionSettings.BasicAuthentication(config.Username, config.Password);
connectionSettings.ServerCertificateValidationCallback(ServerCertificateValidationCallback);
connectionSettings.RequestTimeout(TimeSpan.FromMinutes(2)).EnableHttpCompression();
var client = new ElasticClient(connectionSettings);
return client;
}
So I started with the bulk command on ElasticClient.
In the past I was able to add the objects using the descriptor.Index but of course, I want an update an not an insert/create of everything.
So I've come up with this, but for some reason I keep on receiving an error in Visual Studio 2019 on "Invalid /_bulk request" without any other information.
IEnumerable<IElasticBaseEntity> list = RetrievedData();
var descriptor = new BulkDescriptor();
foreach (var eachDoc in list)
{
var doc = eachDoc;
descriptor.Update<IElasticBaseEntity>(i => i
.Id(doc.Id)
.Doc(doc)
.DocAsUpsert(true));
}
var response = await _client.BulkAsync(descriptor);
// Try to debug & see what was send
if (response.ApiCall.RequestBodyInBytes != null)
{
var jsonOutput = System.Text.Encoding.UTF8.GetString(response.ApiCall.RequestBodyInBytes);
}
The error that I receive is the following (retrieved from the response.DebugInformation):
Invalid NEST response built from a unsuccessful () low level call on POST: /persons-20200717/_bulk
Invalid Bulk items:
Audit trail of this API call:
[1] PingFailure: Node: https://myconnectiontoES:9243/ Exception: PipelineException Took: 00:00:01.2859155
[2] SniffOnFail: Took: 00:00:02.1577638
[3] SniffFailure: Node: https://myconnectiontoES:9243/ Exception: PipelineException Took: 00:00:02.0840985
OriginalException: Elasticsearch.Net.ElasticsearchClientException: Failed sniffing cluster state.. >Call: unknown resource
---> Elasticsearch.Net.PipelineException: Failed sniffing cluster state.
---> Elasticsearch.Net.PipelineException: An error occurred trying to read the response from the >specified node.
at Elasticsearch.Net.RequestPipeline.SniffAsync(CancellationToken cancellationToken)
--- End of inner exception stack trace ---
at Elasticsearch.Net.RequestPipeline.SniffAsync(CancellationToken cancellationToken)
at Elasticsearch.Net.RequestPipeline.SniffOnConnectionFailureAsync(CancellationToken >cancellationToken)
at Elasticsearch.Net.Transport1.PingAsync(IRequestPipeline pipeline, Node node, CancellationToken >cancellationToken) at Elasticsearch.Net.Transport1.RequestAsync[TResponse](HttpMethod method, String path, >CancellationToken cancellationToken, PostData data, IRequestParameters requestParameters)
--- End of inner exception stack trace ---
Audit exception in step 1 PingFailure:
Elasticsearch.Net.PipelineException: An error occurred trying to read the response from the specified >node.
at Elasticsearch.Net.RequestPipeline.PingAsync(Node node, CancellationToken cancellationToken)
Audit exception in step 3 SniffFailure:
Elasticsearch.Net.PipelineException: An error occurred trying to read the response from the specified >node.
at Elasticsearch.Net.RequestPipeline.SniffAsync(CancellationToken cancellationToken)
Request:
<Request stream not captured or already read to completion by serializer. Set DisableDirectStreaming() >on ConnectionSettings to force it to be set on the response.>
Response:
So I looks like there is something wrong with my /_bulk request
What I've tried:
Catch the /_bulk request with fiddler > the request is invalid & not send
Try to set connectionSettings.DisableDirectStreaming(true); to print out the request > this is always null
So if anyone could point out the mistake I made in building the the /_bulk request or could point me in a direction to debug this & retrieve more information, I would be thankfull.
Currently I'm going around in circles, re-reading the documentation, google-ing, but without any result.
Thanks

The failure is occurring when attempting to read the response to sniffing the cluster. It looks like sniffing has occurred because pinging the cluster has failed though, so it's worth checking that you can make a HEAD request to the base address of the Elasticsearch cluster using the credentials the client is configured to use.
Based on the port 9243 being used in the connection to Elasticsearch, I suspect the Elasticsearch cluster is running in Elastic Cloud. SniffingConnectionPool must not be used with an Elasticsearch cluster running in Elastic Cloud, as the addresses returned in the sniff response for where to reach Elasticsearch nodes are internal addresses which will be unreachable from the client. Instead, the CloudConnectionPool should be used, which has some convenience methods to use when creating an instance of ElasticClient
var client = new ElasticClient(
"<cloud id retrieved from the Elastic Cloud web console>",
new BasicAuthenticationCredentials(config.Username, config.Password));
CloudConnectionPool will use http compression by default, but if you need more control over other configuration like request timeout, you can use
var pool = new CloudConnectionPool(
"<cloud id retrieved from the Elastic Cloud web console>",
new BasicAuthenticationCredentials(config.Username, config.Password));
var settings = new ConnectionSettings(pool)
.RequestTimeout(TimeSpan.FromMinutes(2));
var client = new ElasticClient(settings);

After going around in circles, I started creating test to check the basic ES command.
Based on the example I found for ElasticSearch.Net, I was unable to succesfully run any of the examples.
So I checked the ES Index by trying it with a curl command.
curl -H "Content-Type: application/json" -u username:password -XPOST "https://elasticsearch_url:port/indexname/stats/1" -d "{\"application\": \"test\"}"
This command failed, with the message that I'm not allowed to contact ES.
So we've checked the ES user and appearently there was an error in the role that was assigned to this user (in ElasticSearch).
Once we fixed this, I was able to run my code & execute the partial update.
IEnumerable<IElasticBaseEntity> list = RetrievedData();
var descriptor = new BulkDescriptor();
foreach (var eachDoc in list)
{
var doc = eachDoc;
descriptor.Update<IElasticBaseEntity>(i => i
.Id(doc.Id)
.Doc(doc)
.DocAsUpsert(true));
}
var response = await _client.BulkAsync(descriptor);
The issue I have now, is that this runs for about 1 hour and then I get a SnifFailure.
The connection to ES is retried but mostly ends in a "MaxTimeOutReached"
The RequestTimeOut is set to 2 minutes
The Retries & timeout is not set, so the defaults are used (-> this is retrying until the RequestTimeout of 2 minutes is reached)
As you can see in the log below: I'm able to send docs to ES and then at some point I receive this error. Mostly after running my application for 1h.
2020-08-05 11:22:01.5553| INFO| 001 ES_indexName processed documents 08/05/2020 11:22:01 1 in 202.3803 ms
2020-08-05 11:29:28.9633| INFO| 001 ES_indexName processed documents 08/05/2020 11:29:28 1 in 179.7982 ms
2020-08-05 11:40:08.4666| INFO| 001 ES_indexName processed documents 08/05/2020 11:40:07 1 in 291.5695 ms
2020-08-05 11:47:26.2924|ERROR| failed Invalid NEST response built from a unsuccessful () low level call on POST: /ES_indexName/_bulk
# Invalid Bulk items:
# Audit trail of this API call:
- [1] PingFailure: Node: https://ES_node1:port/ Exception: PipelineException Took: 00:01:40.0193513
- [2] SniffOnFail: Took: 00:05:00.0132241
- [3] SniffFailure: Node: https://ES_node2:port/ Exception: PipelineException Took: 00:01:40.0036955
- [4] SniffFailure: Node: https://ES_node3:port/ Exception: PipelineException Took: 00:01:40.0019296
- [5] SniffFailure: Node: https://ES_node1:port/ Exception: PipelineException Took: 00:01:40.0005639
- [6] MaxTimeoutReached:
# OriginalException: Elasticsearch.Net.ElasticsearchClientException: Maximum timeout reached while retrying request. Call: unknown resource
---> Elasticsearch.Net.PipelineException: Failed sniffing cluster state.
---> System.AggregateException: One or more errors occurred. (An error occurred trying to write the request data to the specified node.) (An error occurred trying to write the request data to the specified node.) (An error occurred trying to write the request data to the specified node.)
---> Elasticsearch.Net.PipelineException: An error occurred trying to write the request data to the specified node.
---> System.Threading.Tasks.TaskCanceledException: The operation was canceled.
at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean allowHttp2, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.GetHttpConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.DecompressionHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpClient.FinishSendAsyncUnbuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
at Elasticsearch.Net.HttpConnection.RequestAsync[TResponse](RequestData requestData, CancellationToken cancellationToken)
--- End of inner exception stack trace ---
at Elasticsearch.Net.RequestPipeline.SniffAsync(CancellationToken cancellationToken)
--- End of inner exception stack trace ---
---> (Inner Exception #1) Elasticsearch.Net.PipelineException: An error occurred trying to write the request data to the specified node.
---> System.Threading.Tasks.TaskCanceledException: The operation was canceled.
at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean allowHttp2, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.GetHttpConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.DecompressionHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpClient.FinishSendAsyncUnbuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
at Elasticsearch.Net.HttpConnection.RequestAsync[TResponse](RequestData requestData, CancellationToken cancellationToken)
--- End of inner exception stack trace ---
at Elasticsearch.Net.RequestPipeline.SniffAsync(CancellationToken cancellationToken)<---
---> (Inner Exception #2) Elasticsearch.Net.PipelineException: An error occurred trying to write the request data to the specified node.
---> System.Threading.Tasks.TaskCanceledException: The operation was canceled.
at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean allowHttp2, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.GetHttpConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.DecompressionHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpClient.FinishSendAsyncUnbuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
at Elasticsearch.Net.HttpConnection.RequestAsync[TResponse](RequestData requestData, CancellationToken cancellationToken)
--- End of inner exception stack trace ---
at Elasticsearch.Net.RequestPipeline.SniffAsync(CancellationToken cancellationToken)<---
--- End of inner exception stack trace ---
at Elasticsearch.Net.RequestPipeline.SniffAsync(CancellationToken cancellationToken)
at Elasticsearch.Net.RequestPipeline.SniffOnConnectionFailureAsync(CancellationToken cancellationToken)
at Elasticsearch.Net.Transport`1.PingAsync(IRequestPipeline pipeline, Node node, CancellationToken cancellationToken)
at Elasticsearch.Net.Transport`1.RequestAsync[TResponse](HttpMethod method, String path, CancellationToken cancellationToken, PostData data, IRequestParameters requestParameters)
--- End of inner exception stack trace ---
# Audit exception in step 1 PingFailure:
Elasticsearch.Net.PipelineException: An error occurred trying to write the request data to the specified node.
---> System.Threading.Tasks.TaskCanceledException: The operation was canceled.
at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean allowHttp2, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.GetHttpConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.DecompressionHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpClient.FinishSendAsyncUnbuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
at Elasticsearch.Net.HttpConnection.RequestAsync[TResponse](RequestData requestData, CancellationToken cancellationToken)
--- End of inner exception stack trace ---
at Elasticsearch.Net.RequestPipeline.PingAsync(Node node, CancellationToken cancellationToken)
# Audit exception in step 3 SniffFailure:
Elasticsearch.Net.PipelineException: An error occurred trying to write the request data to the specified node.
---> System.Threading.Tasks.TaskCanceledException: The operation was canceled.
at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean allowHttp2, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.GetHttpConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.DecompressionHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpClient.FinishSendAsyncUnbuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
at Elasticsearch.Net.HttpConnection.RequestAsync[TResponse](RequestData requestData, CancellationToken cancellationToken)
--- End of inner exception stack trace ---
at Elasticsearch.Net.RequestPipeline.SniffAsync(CancellationToken cancellationToken)
# Audit exception in step 4 SniffFailure:
Elasticsearch.Net.PipelineException: An error occurred trying to write the request data to the specified node.
---> System.Threading.Tasks.TaskCanceledException: The operation was canceled.
at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean allowHttp2, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.GetHttpConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.DecompressionHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpClient.FinishSendAsyncUnbuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
at Elasticsearch.Net.HttpConnection.RequestAsync[TResponse](RequestData requestData, CancellationToken cancellationToken)
--- End of inner exception stack trace ---
at Elasticsearch.Net.RequestPipeline.SniffAsync(CancellationToken cancellationToken)
# Audit exception in step 5 SniffFailure:
Elasticsearch.Net.PipelineException: An error occurred trying to write the request data to the specified node.
---> System.Threading.Tasks.TaskCanceledException: The operation was canceled.
at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean allowHttp2, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.GetHttpConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.DecompressionHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpClient.FinishSendAsyncUnbuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
at Elasticsearch.Net.HttpConnection.RequestAsync[TResponse](RequestData requestData, CancellationToken cancellationToken)
--- End of inner exception stack trace ---
at Elasticsearch.Net.RequestPipeline.SniffAsync(CancellationToken cancellationToken)
# Request:
<Request stream not captured or already read to completion by serializer. Set DisableDirectStreaming() on ConnectionSettings to force it to be set on the response.>
# Response:
<Response stream not captured or already read to completion by serializer. Set DisableDirectStreaming() on ConnectionSettings to force it to be set on the response.>
400039.9975 ms MUST RETRY```

C# Azure Blob Storage upload on slow internet connection fail

I am using the Azure Blob storage client library v12 for .NET i used this guide to setup https://learn.microsoft.com/en-us/azure/storage/blobs/storage-quickstart-blobs-dotnet
It is all working fine on internet connections with fast upload speeds but once I try run this program on a site with upload speed on 1Mbps it doesnt work i have now been able to replicate this issue but using software on my computer to restrict the upload speed in my dev environment when i set the upload speed to 1Mbps it fails to upload the file after about 10 minutes of trying
The file is about 15Mb of json
15Mb file should upload in about 2 - 3 minutes on 1Mbps i would think
This is the code i am trying to use as i said when tested on 30Mbps upload connection it all working fine
public static async Task UploadData(string json)
{
BlobClientOptions MyBlobClientOptions = new BlobClientOptions
{
Transport = new HttpClientTransport(
new HttpClient { Timeout = TimeSpan.FromSeconds(600) })
};
//Setup Azure BlobServiceClient
string datetime = DateTimeOffset.Now.ToString("yyyyMMddHHmmss");
string AzureConnectionString = _iconfiguration.GetConnectionString("AzureStorage");
BlobServiceClient blobServiceClient = new BlobServiceClient(AzureConnectionString,MyBlobClientOptions);
string localFilePath = $"{TempDirPath}data{datetime}.json";
// Write text to the file
await File.WriteAllTextAsync(localFilePath, json);
string localFileName = $"UPLOAD_SPEED_TEST/data{datetime}.json";
// Get a reference to a blob
BlobContainerClient containerClient = blobServiceClient.GetBlobContainerClient("data");
BlobClient blobClient = containerClient.GetBlobClient(localFileName);
_logger.LogInformation("Uploading to Blob storage as blob:\n\t {0}\n", blobClient.Uri);
try
{
//Test this Option to see if it would fix upload issue on slow connections
StorageTransferOptions transferOptions = new StorageTransferOptions();
transferOptions.MaximumConcurrency = 2;
// Open the file and upload its data
using FileStream uploadFileStream = File.OpenRead(localFilePath);
uploadFileStream.Position = 0;
// Upload file to Azure Blob Storage
blobClient.Upload(uploadFileStream, transferOptions: transferOptions);
_logger.LogInformation("UPLOAD COMPLETE");
uploadFileStream.Close();
}
catch (Exception ex)
{
//Error upload Failed
_logger.LogInformation("UPLOAD ERROR" + ex.ToString());
}
}
Error Message
System.AggregateException: Retry failed after 6 tries. (The operation was canceled.) (The operation was canceled.) (The operation was canceled.) (The operation was canceled.) (The operation was canceled.) (The operation was canceled.)
---> System.Threading.Tasks.TaskCanceledException: The operation was canceled.
---> System.Net.Http.HttpRequestException: Error while copying content to a stream.
---> System.IO.IOException: Unable to read data from the transport connection: The I/O operation has been aborted because of either a thread exit or an application request..
---> System.Net.Sockets.SocketException (995): The I/O operation has been aborted because of either a thread exit or an application request.
--- End of inner exception stack trace ---
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
at System.Net.Security.SslStream.<WriteSingleChunk>g__CompleteAsync|210_1[TWriteAdapter](ValueTask writeTask, Byte[] bufferToReturn)
at System.Net.Security.SslStream.WriteAsyncChunked[TWriteAdapter](TWriteAdapter writeAdapter, ReadOnlyMemory`1 buffer)
at System.Net.Security.SslStream.WriteAsyncInternal[TWriteAdapter](TWriteAdapter writeAdapter, ReadOnlyMemory`1 buffer)
at System.Net.Http.HttpConnection.WriteAsync(ReadOnlyMemory`1 source)
at System.IO.Stream.CopyToAsyncInternal(Stream destination, Int32 bufferSize, CancellationToken cancellationToken)
at Azure.Core.RequestContent.StreamContent.WriteToAsync(Stream stream, CancellationToken cancellation)
at Azure.Core.Pipeline.HttpClientTransport.PipelineRequest.PipelineContentAdapter.SerializeToStreamAsync(Stream stream, TransportContext context)
at System.Net.Http.HttpContent.CopyToAsyncCore(ValueTask copyTask)
--- End of inner exception stack trace ---
at System.Net.Http.HttpContent.CopyToAsyncCore(ValueTask copyTask)
at System.Net.Http.HttpConnection.SendRequestContentAsync(HttpRequestMessage request, HttpContentWriteStream stream, CancellationToken cancellationToken)
at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
--- End of inner exception stack trace ---
at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithNtConnectionAuthAsync(HttpConnection connection, HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpClient.FinishSendAsyncUnbuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
at Azure.Core.Pipeline.HttpClientTransport.ProcessAsync(HttpMessage message)
at Azure.Core.Pipeline.HttpClientTransport.Process(HttpMessage message)
at Azure.Core.Pipeline.HttpPipelineTransportPolicy.Process(HttpMessage message, ReadOnlyMemory`1 pipeline)
at Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(HttpMessage message, ReadOnlyMemory`1 pipeline)
at Azure.Core.Pipeline.RequestActivityPolicy.ProcessNextAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean isAsync)
at Azure.Core.Pipeline.RequestActivityPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean isAsync)
at Azure.Core.Pipeline.TaskExtensions.EnsureCompleted(ValueTask task)
at Azure.Core.Pipeline.RequestActivityPolicy.Process(HttpMessage message, ReadOnlyMemory`1 pipeline)
at Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(HttpMessage message, ReadOnlyMemory`1 pipeline)
at Azure.Core.Pipeline.ResponseBodyPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
at Azure.Core.Pipeline.TaskExtensions.EnsureCompleted(ValueTask task)
at Azure.Core.Pipeline.ResponseBodyPolicy.Process(HttpMessage message, ReadOnlyMemory`1 pipeline)
at Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(HttpMessage message, ReadOnlyMemory`1 pipeline)
at Azure.Core.Pipeline.LoggingPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
at Azure.Core.Pipeline.TaskExtensions.EnsureCompleted(ValueTask task)
at Azure.Core.Pipeline.LoggingPolicy.Process(HttpMessage message, ReadOnlyMemory`1 pipeline)
at Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(HttpMessage message, ReadOnlyMemory`1 pipeline)
at Azure.Core.Pipeline.HttpPipelineSynchronousPolicy.Process(HttpMessage message, ReadOnlyMemory`1 pipeline)
at Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(HttpMessage message, ReadOnlyMemory`1 pipeline)
at Azure.Core.Pipeline.HttpPipelineSynchronousPolicy.Process(HttpMessage message, ReadOnlyMemory`1 pipeline)
at Azure.Core.Pipeline.RetryPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
--- End of inner exception stack trace ---
at Azure.Core.Pipeline.RetryPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
at Azure.Core.Pipeline.TaskExtensions.EnsureCompleted(ValueTask task)
at Azure.Core.Pipeline.RetryPolicy.Process(HttpMessage message, ReadOnlyMemory`1 pipeline)
at Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(HttpMessage message, ReadOnlyMemory`1 pipeline)
at Azure.Core.Pipeline.HttpPipelineSynchronousPolicy.Process(HttpMessage message, ReadOnlyMemory`1 pipeline)
at Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(HttpMessage message, ReadOnlyMemory`1 pipeline)
at Azure.Core.Pipeline.HttpPipelineSynchronousPolicy.Process(HttpMessage message, ReadOnlyMemory`1 pipeline)
at Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(HttpMessage message, ReadOnlyMemory`1 pipeline)
at Azure.Core.Pipeline.HttpPipelineSynchronousPolicy.Process(HttpMessage message, ReadOnlyMemory`1 pipeline)
at Azure.Core.Pipeline.HttpPipeline.Send(HttpMessage message, CancellationToken cancellationToken)
at Azure.Storage.Blobs.BlobRestClient.BlockBlob.UploadAsync(ClientDiagnostics clientDiagnostics, HttpPipeline pipeline, Uri resourceUri, Stream body, Int64 contentLength, String version, Nullable`1 timeout, Byte[] transactionalContentHash, String blobContentType, String blobContentEncoding, String blobContentLanguage, Byte[] blobContentHash, String blobCacheControl, IDictionary`2 metadata, String leaseId, String blobContentDisposition, String encryptionKey, String encryptionKeySha256, Nullable`1 encryptionAlgorithm, String encryptionScope, Nullable`1 tier, Nullable`1 ifModifiedSince, Nullable`1 ifUnmodifiedSince, Nullable`1 ifMatch, Nullable`1 ifNoneMatch, String requestId, Boolean async, String operationName, CancellationToken cancellationToken)
at Azure.Storage.Blobs.Specialized.BlockBlobClient.UploadInternal(Stream content, BlobHttpHeaders blobHttpHeaders, IDictionary`2 metadata, BlobRequestConditions conditions, Nullable`1 accessTier, IProgress`1 progressHandler, String operationName, Boolean async, CancellationToken cancellationToken)
at Azure.Core.Pipeline.TaskExtensions.EnsureCompleted[T](Task`1 task)
at Azure.Storage.Blobs.PartitionedUploader.Upload(Stream content, BlobHttpHeaders blobHttpHeaders, IDictionary`2 metadata, BlobRequestConditions conditions, IProgress`1 progressHandler, Nullable`1 accessTier, CancellationToken cancellationToken)
at Azure.Storage.Blobs.BlobClient.StagedUploadAsync(Stream content, BlobHttpHeaders blobHttpHeaders, IDictionary`2 metadata, BlobRequestConditions conditions, IProgress`1 progressHandler, Nullable`1 accessTier, StorageTransferOptions transferOptions, Boolean async, CancellationToken cancellationToken)
at Azure.Core.Pipeline.TaskExtensions.EnsureCompleted[T](Task`1 task)
at Azure.Storage.Blobs.BlobClient.Upload(Stream content, BlobHttpHeaders httpHeaders, IDictionary`2 metadata, BlobRequestConditions conditions, IProgress`1 progressHandler, Nullable`1 accessTier, StorageTransferOptions transferOptions, CancellationToken cancellationToken)
at LMGAgent.AzureStorage.UploadBasketData(String json) in C:\Users\anewman\source\repos\agent\LMGAgent\AzureStorage.cs:line 68
---> (Inner Exception #1) System.Threading.Tasks.TaskCanceledException: The operation was canceled.
---> System.Net.Http.HttpRequestException: Error while copying content to a stream.
---> System.IO.IOException: Unable to read data from the transport connection: The I/O operation has been aborted because of either a thread exit or an application request..
---> System.Net.Sockets.SocketException (995): The I/O operation has been aborted because of either a thread exit or an application request.
--- End of inner exception stack trace ---
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
at System.Net.Security.SslStream.<WriteSingleChunk>g__CompleteAsync|210_1[TWriteAdapter](ValueTask writeTask, Byte[] bufferToReturn)
at System.Net.Security.SslStream.WriteAsyncChunked[TWriteAdapter](TWriteAdapter writeAdapter, ReadOnlyMemory`1 buffer)
at System.Net.Security.SslStream.WriteAsyncInternal[TWriteAdapter](TWriteAdapter writeAdapter, ReadOnlyMemory`1 buffer)
at System.Net.Http.HttpConnection.WriteAsync(ReadOnlyMemory`1 source)
at System.IO.Stream.CopyToAsyncInternal(Stream destination, Int32 bufferSize, CancellationToken cancellationToken)
at Azure.Core.RequestContent.StreamContent.WriteToAsync(Stream stream, CancellationToken cancellation)
at Azure.Core.Pipeline.HttpClientTransport.PipelineRequest.PipelineContentAdapter.SerializeToStreamAsync(Stream stream, TransportContext context)
at System.Net.Http.HttpContent.CopyToAsyncCore(ValueTask copyTask)
--- End of inner exception stack trace ---
at System.Net.Http.HttpContent.CopyToAsyncCore(ValueTask copyTask)
at System.Net.Http.HttpConnection.SendRequestContentAsync(HttpRequestMessage request, HttpContentWriteStream stream, CancellationToken cancellationToken)
at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
--- End of inner exception stack trace ---
at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithNtConnectionAuthAsync(HttpConnection connection, HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpClient.FinishSendAsyncUnbuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
at Azure.Core.Pipeline.HttpClientTransport.ProcessAsync(HttpMessage message)
at Azure.Core.Pipeline.HttpClientTransport.Process(HttpMessage message)
at Azure.Core.Pipeline.HttpPipelineTransportPolicy.Process(HttpMessage message, ReadOnlyMemory`1 pipeline)
at Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(HttpMessage message, ReadOnlyMemory`1 pipeline)
at Azure.Core.Pipeline.RequestActivityPolicy.ProcessNextAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean isAsync)
at Azure.Core.Pipeline.RequestActivityPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean isAsync)
at Azure.Core.Pipeline.TaskExtensions.EnsureCompleted(ValueTask task)
at Azure.Core.Pipeline.RequestActivityPolicy.Process(HttpMessage message, ReadOnlyMemory`1 pipeline)
at Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(HttpMessage message, ReadOnlyMemory`1 pipeline)
at Azure.Core.Pipeline.ResponseBodyPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
at Azure.Core.Pipeline.TaskExtensions.EnsureCompleted(ValueTask task)
at Azure.Core.Pipeline.ResponseBodyPolicy.Process(HttpMessage message, ReadOnlyMemory`1 pipeline)
at Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(HttpMessage message, ReadOnlyMemory`1 pipeline)
at Azure.Core.Pipeline.LoggingPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
at Azure.Core.Pipeline.TaskExtensions.EnsureCompleted(ValueTask task)
at Azure.Core.Pipeline.LoggingPolicy.Process(HttpMessage message, ReadOnlyMemory`1 pipeline)
at Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(HttpMessage message, ReadOnlyMemory`1 pipeline)
at Azure.Core.Pipeline.HttpPipelineSynchronousPolicy.Process(HttpMessage message, ReadOnlyMemory`1 pipeline)
at Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(HttpMessage message, ReadOnlyMemory`1 pipeline)
at Azure.Core.Pipeline.HttpPipelineSynchronousPolicy.Process(HttpMessage message, ReadOnlyMemory`1 pipeline)
at Azure.Core.Pipeline.RetryPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)<---
---> (Inner Exception #2) System.Threading.Tasks.TaskCanceledException: The operation was canceled.
---> System.Net.Http.HttpRequestException: Error while copying content to a stream.
---> System.IO.IOException: Unable to read data from the transport connection: The I/O operation has been aborted because of either a thread exit or an application request..
---> System.Net.Sockets.SocketException (995): The I/O operation has been aborted because of either a thread exit or an application request.
--- End of inner exception stack trace ---
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
at System.Net.Security.SslStream.<WriteSingleChunk>g__CompleteAsync|210_1[TWriteAdapter](ValueTask writeTask, Byte[] bufferToReturn)
at System.Net.Security.SslStream.WriteAsyncChunked[TWriteAdapter](TWriteAdapter writeAdapter, ReadOnlyMemory`1 buffer)
at System.Net.Security.SslStream.WriteAsyncInternal[TWriteAdapter](TWriteAdapter writeAdapter, ReadOnlyMemory`1 buffer)
at System.Net.Http.HttpConnection.WriteAsync(ReadOnlyMemory`1 source)
at System.IO.Stream.CopyToAsyncInternal(Stream destination, Int32 bufferSize, CancellationToken cancellationToken)
at Azure.Core.RequestContent.StreamContent.WriteToAsync(Stream stream, CancellationToken cancellation)
at Azure.Core.Pipeline.HttpClientTransport.PipelineRequest.PipelineContentAdapter.SerializeToStreamAsync(Stream stream, TransportContext context)
at System.Net.Http.HttpContent.CopyToAsyncCore(ValueTask copyTask)
--- End of inner exception stack trace ---
at System.Net.Http.HttpContent.CopyToAsyncCore(ValueTask copyTask)
at System.Net.Http.HttpConnection.SendRequestContentAsync(HttpRequestMessage request, HttpContentWriteStream stream, CancellationToken cancellationToken)
at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
--- End of inner exception stack trace ---
at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithNtConnectionAuthAsync(HttpConnection connection, HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpClient.FinishSendAsyncUnbuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
at Azure.Core.Pipeline.HttpClientTransport.ProcessAsync(HttpMessage message)
at Azure.Core.Pipeline.HttpClientTransport.Process(HttpMessage message)
at Azure.Core.Pipeline.HttpPipelineTransportPolicy.Process(HttpMessage message, ReadOnlyMemory`1 pipeline)
at Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(HttpMessage message, ReadOnlyMemory`1 pipeline)
at Azure.Core.Pipeline.RequestActivityPolicy.ProcessNextAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean isAsync)
at Azure.Core.Pipeline.RequestActivityPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean isAsync)
at Azure.Core.Pipeline.TaskExtensions.EnsureCompleted(ValueTask task)
at Azure.Core.Pipeline.RequestActivityPolicy.Process(HttpMessage message, ReadOnlyMemory`1 pipeline)
at Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(HttpMessage message, ReadOnlyMemory`1 pipeline)
at Azure.Core.Pipeline.ResponseBodyPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
at Azure.Core.Pipeline.TaskExtensions.EnsureCompleted(ValueTask task)
at Azure.Core.Pipeline.ResponseBodyPolicy.Process(HttpMessage message, ReadOnlyMemory`1 pipeline)
at Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(HttpMessage message, ReadOnlyMemory`1 pipeline)
at Azure.Core.Pipeline.LoggingPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
at Azure.Core.Pipeline.TaskExtensions.EnsureCompleted(ValueTask task)
at Azure.Core.Pipeline.LoggingPolicy.Process(HttpMessage message, ReadOnlyMemory`1 pipeline)
at Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(HttpMessage message, ReadOnlyMemory`1 pipeline)
at Azure.Core.Pipeline.HttpPipelineSynchronousPolicy.Process(HttpMessage message, ReadOnlyMemory`1 pipeline)
at Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(HttpMessage message, ReadOnlyMemory`1 pipeline)
at Azure.Core.Pipeline.HttpPipelineSynchronousPolicy.Process(HttpMessage message, ReadOnlyMemory`1 pipeline)
at Azure.Core.Pipeline.RetryPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)<---
---> (Inner Exception #3) System.Threading.Tasks.TaskCanceledException: The operation was canceled.
---> System.Net.Http.HttpRequestException: Error while copying content to a stream.
---> System.IO.IOException: Unable to read data from the transport connection: The I/O operation has been aborted because of either a thread exit or an application request..
---> System.Net.Sockets.SocketException (995): The I/O operation has been aborted because of either a thread exit or an application request.
--- End of inner exception stack trace ---
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
at System.Net.Security.SslStream.<WriteSingleChunk>g__CompleteAsync|210_1[TWriteAdapter](ValueTask writeTask, Byte[] bufferToReturn)
at System.Net.Security.SslStream.WriteAsyncChunked[TWriteAdapter](TWriteAdapter writeAdapter, ReadOnlyMemory`1 buffer)
at System.Net.Security.SslStream.WriteAsyncInternal[TWriteAdapter](TWriteAdapter writeAdapter, ReadOnlyMemory`1 buffer)
at System.Net.Http.HttpConnection.WriteAsync(ReadOnlyMemory`1 source)
at System.IO.Stream.CopyToAsyncInternal(Stream destination, Int32 bufferSize, CancellationToken cancellationToken)
at Azure.Core.RequestContent.StreamContent.WriteToAsync(Stream stream, CancellationToken cancellation)
at Azure.Core.Pipeline.HttpClientTransport.PipelineRequest.PipelineContentAdapter.SerializeToStreamAsync(Stream stream, TransportContext context)
at System.Net.Http.HttpContent.CopyToAsyncCore(ValueTask copyTask)
--- End of inner exception stack trace ---
at System.Net.Http.HttpContent.CopyToAsyncCore(ValueTask copyTask)
at System.Net.Http.HttpConnection.SendRequestContentAsync(HttpRequestMessage request, HttpContentWriteStream stream, CancellationToken cancellationToken)
at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)

I was getting this same issue exception when trying to upload a 60MB file over a 4Mbps connection.:
System.AggregateException: Retry failed after 6 tries.
I did the following to get it working for me:
Set the Retry timeout as well as the Transport timeout (thanks to this GitHub post for that idea):
var blobClientOptions = new BlobClientOptions
{
Transport = new HttpClientTransport(new HttpClient { Timeout = Timeout.InfiniteTimeSpan }),
Retry = { NetworkTimeout = Timeout.InfiniteTimeSpan }
};
See this answer for more on setting the Retry options.
I also overwrote the cancellation token in the hope of stopping the code calling my function cancelling it:
await blobClient.UploadAsync(
content: uploadFileStream, httpHeaders:
cancellationToken: CancellationToken.None);

As Gaurav said, you need to split the file in blocks and read those blocks and upload them using StageBlock method. Once all blocks are uploaded, you call CommitBlockList method to commit those blocks and create the blob.
Please refer to this SO thread.

How to deploy aspnetcore app to ubuntu on digital ocean

After publishing my asp.net core 2.1 app to Digital Ocean on ubuntu server if i enter the server Ip in other to open the app in a browser I get this error:
Service Unavailable
The server is temporarily unable to service your request due to maintenance downtime or capacity problems. Please try again later.
Apache/version (Ubuntu) Server at Server_IP Port 80
From this Line:
Microsoft.AspNetCore.Identity.UserManager`1.FindByNameAsync(String userName)
at App_name.Data.ApplicationDbContext.CreateAdminAccount(IServiceProvider serviceProvider, IConfiguration configuration) in Data\ApplicationDbContext.cs:line
When i lunch the app locally i check if no user exist then CreateAdminAccount creates an admin user.
Am sure the app cannot make any migrations before the app lunches in other to create the admin user.
I want to know how to apply the migration to the database when the app launches.
If I check the application logs I get this error too:
at System.Net.Sockets.Socket.BeginConnectEx(EndPoint remoteEP, Boolean flowContext, AsyncCallback callback, Object state)
at System.Net.Sockets.Socket.UnsafeBeginConnect(EndPoint remoteEP, AsyncCallback callback, Object state, Boolean flowContext)
at System.Net.Sockets.Socket.BeginConnect(EndPoint remoteEP, AsyncCallback callback, Object state)
at System.Threading.Tasks.TaskFactory1.FromAsyncImpl[TArg1](Func4 beginMethod, Func2 endFunction, Action1 endAction, TArg1 arg1, Object state, TaskCreationOptions creationOptions)
at Npgsql.NpgsqlConnector.ConnectAsync(NpgsqlTimeout timeout, CancellationToken cancellationToken)
at Npgsql.NpgsqlConnector.RawOpen(NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
at Npgsql.NpgsqlConnector.Open(NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
at Npgsql.ConnectorPool.AllocateLong(NpgsqlConnection conn, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
at Npgsql.NpgsqlConnection.<>cDisplayClass32_0.<<Open>gOpenLong|0>d.MoveNext()
— End of stack trace from previous location where exception was thrown —
at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.OpenDbConnectionAsync(Boolean errorsExpected, CancellationToken cancellationToken)
at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.OpenAsync(CancellationToken cancellationToken, Boolean errorsExpected)
at Microsoft.EntityFrameworkCore.Query.Internal.AsyncQueryingEnumerable1.AsyncEnumerator.BufferlessMoveNext(DbContext _, Boolean buffer, CancellationToken cancellationToken)
at Npgsql.EntityFrameworkCore.PostgreSQL.Storage.Internal.NpgsqlExecutionStrategy.ExecuteAsync[TState,TResult](TState state, Func4 operation, Func4 verifySucceeded, CancellationToken cancellationToken)
at Microsoft.EntityFrameworkCore.Query.Internal.AsyncQueryingEnumerable1.AsyncEnumerator.MoveNext(CancellationToken cancellationToken)
at System.Linq.AsyncEnumerable.FirstOrDefault[TSource](IAsyncEnumerable1 source, CancellationToken cancellationToken)
at Microsoft.EntityFrameworkCore.Query.Internal.AsyncLinqOperatorProvider.TaskResultAsyncEnumerable1.Enumerator.MoveNext(CancellationToken cancellationToken)
at System.Linq.AsyncEnumerable.SelectEnumerableAsyncIterator2.MoveNextCore(CancellationToken cancellationToken)
at System.Linq.AsyncEnumerable.AsyncIterator1.MoveNext(CancellationToken cancellationToken)
at Microsoft.EntityFrameworkCore.Query.Internal.AsyncLinqOperatorProvider.ExceptionInterceptor1.EnumeratorExceptionInterceptor.MoveNext(CancellationToken cancellationToken)
at Microsoft.EntityFrameworkCore.Query.Internal.QueryCompiler.ExecuteSingletonAsyncQuery[TResult](QueryContext queryContext, Func2 compiledQuery, IDiagnosticsLogger1 logger, Type contextType)
at Microsoft.AspNetCore.Identity.UserManager1.FindByNameAsync(String userName)
at Appname.Data.ApplicationDbContext.CreateAdminAccount(IServiceProvider serviceProvider, IConfiguration configuration) in Data\ApplicationDbContext.cs:line 94
— End of inner exception stack trace —
at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
at Presscode.Startup.Configure(IApplicationBuilder app)
— End of stack trace from previous location where exception was thrown —
at Microsoft.AspNetCore.Hosting.ConventionBasedStartup.Configure(IApplicationBuilder app)
at Microsoft.AspNetCore.Hosting.Internal.AutoRequestServicesStartupFilter.<>cDisplayClass0_0.<Configure>b0(IApplicationBuilder builder)
at Microsoft.AspNetCore.Hosting.Internal.WebHost.BuildApplication()
—> (Inner Exception #0) System.Net.Sockets.SocketException (111): Connection refused
at System.Net.Sockets.Socket.BeginConnectEx(EndPoint remoteEP, Boolean flowContext, AsyncCallback callback, Object state)
at System.Net.Sockets.Socket.UnsafeBeginConnect(EndPoint remoteEP, AsyncCallback callback, Object state, Boolean flowContext)
at System.Net.Sockets.Socket.BeginConnect(EndPoint remoteEP, AsyncCallback callback, Object state)
at System.Threading.Tasks.TaskFactory1.FromAsyncImpl[TArg1](Func4 beginMethod, Func2 endFunction, Action1 endAction, TArg1 arg1, Object state, TaskCreationOptions creationOptions)
at Npgsql.NpgsqlConnector.ConnectAsync(NpgsqlTimeout timeout, CancellationToken cancellationToken)
at Npgsql.NpgsqlConnector.RawOpen(NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
at Npgsql.NpgsqlConnector.Open(NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
at Npgsql.ConnectorPool.AllocateLong(NpgsqlConnection conn, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
at Npgsql.NpgsqlConnection.<>cDisplayClass32_0.<<Open>gOpenLong|0>d.MoveNext()
— End of stack trace from previous location where exception was thrown —
at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.OpenDbConnectionAsync(Boolean errorsExpected, CancellationToken cancellationToken)
at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.OpenAsync(CancellationToken cancellationToken, Boolean errorsExpected)
at Microsoft.EntityFrameworkCore.Query.Internal.AsyncQueryingEnumerable1.AsyncEnumerator.BufferlessMoveNext(DbContext _, Boolean buffer, CancellationToken cancellationToken)
at Npgsql.EntityFrameworkCore.PostgreSQL.Storage.Internal.NpgsqlExecutionStrategy.ExecuteAsync[TState,TResult](TState state, Func4 operation, Func4 verifySucceeded, CancellationToken cancellationToken)
at Microsoft.EntityFrameworkCore.Query.Internal.AsyncQueryingEnumerable1.AsyncEnumerator.MoveNext(CancellationToken cancellationToken)
at System.Linq.AsyncEnumerable.FirstOrDefault[TSource](IAsyncEnumerable1 source, CancellationToken cancellationToken)
at Microsoft.EntityFrameworkCore.Query.Internal.AsyncLinqOperatorProvider.TaskResultAsyncEnumerable1.Enumerator.MoveNext(CancellationToken cancellationToken)
at System.Linq.AsyncEnumerable.SelectEnumerableAsyncIterator2.MoveNextCore(CancellationToken cancellationToken)
at System.Linq.AsyncEnumerable.AsyncIterator1.MoveNext(CancellationToken cancellationToken)
at Microsoft.EntityFrameworkCore.Query.Internal.AsyncLinqOperatorProvider.ExceptionInterceptor1.EnumeratorExceptionInterceptor.MoveNext(CancellationToken cancellationToken)
at Microsoft.EntityFrameworkCore.Query.Internal.QueryCompiler.ExecuteSingletonAsyncQuery[TResult](QueryContext queryContext, Func2 compiledQuery, IDiagnosticsLogger1 logger, Type contextType)
at Microsoft.AspNetCore.Identity.UserManager1.FindByNameAsync(String userName)
at AppName.Data.ApplicationDbContext.CreateAdminAccount(IServiceProvider serviceProvider, IConfiguration configuration) in Data\ApplicationDbContext.cs:line 94<—```

"The response ended prematurely" when connecting to insecure gRPC channel

I'm trying to establish a connection to an insecure gRPC server. I'm using gRPC for communication between two processes inside of a Docker container, that's why I don't need any encryption or strong authentication.
The server behaves as expected and I can do calls using grpcurl like that:
grpcurl -plaintext localhost:42652 SomeService.DoSomething
Now I'm trying to call the same RPC method from a .Net Core application:
// Registration of the DI service
services.AddGrpcClient<DaemonService.DaemonServiceClient>(options => {
// Enable support for unencrypted HTTP2
AppContext.SetSwitch("System.Net.Http.SocketsHttpHandler.Http2UnencryptedSupport", true);
options.Address = new Uri("http://localhost:42652");
// Just a test, doesn't change anything.
options.ChannelOptionsActions.Add(channelOptions => channelOptions.Credentials = ChannelCredentials.Insecure);
});
// Call
var reply = _someServiceClient.DoSomething(new Request());
But the call in the last line results in an exception:
fail: Grpc.Net.Client.Internal.GrpcCall[6]
Error starting gRPC call.
System.Net.Http.HttpRequestException: An error occurred while sending the request.
---> System.IO.IOException: The response ended prematurely.
at System.Net.Http.HttpConnection.FillAsync()
at System.Net.Http.HttpConnection.ReadNextResponseHeaderLineAsync(Boolean foldedHeadersAllowed)
at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
--- End of inner exception stack trace ---
at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithNtConnectionAuthAsync(HttpConnection connection, HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.DiagnosticsHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at Microsoft.Extensions.Http.Logging.LoggingHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at Microsoft.Extensions.Http.Logging.LoggingScopeHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpClient.FinishSendAsyncUnbuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
at Grpc.Net.Client.Internal.GrpcCall`2.SendAsync(HttpRequestMessage request)
fail: Grpc.Net.Client.Internal.GrpcCall[3]
Call failed with gRPC error status. Status code: 'Cancelled', Message: 'Error starting gRPC call.'.
fail: SomeNamespace.Session.Program[0]
An error occured.
System.Net.Http.HttpRequestException: An error occurred while sending the request.
---> System.IO.IOException: The response ended prematurely.
at System.Net.Http.HttpConnection.FillAsync()
at System.Net.Http.HttpConnection.ReadNextResponseHeaderLineAsync(Boolean foldedHeadersAllowed)
at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
--- End of inner exception stack trace ---
at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithNtConnectionAuthAsync(HttpConnection connection, HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.DiagnosticsHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at Microsoft.Extensions.Http.Logging.LoggingHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at Microsoft.Extensions.Http.Logging.LoggingScopeHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpClient.FinishSendAsyncUnbuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
at Grpc.Net.Client.Internal.GrpcCall`2.SendAsync(HttpRequestMessage request)
at Grpc.Net.Client.Internal.GrpcCall`2.GetResponseAsync()
at Grpc.Net.Client.Internal.HttpClientCallInvoker.BlockingUnaryCall[TRequest,TResponse](Method`2 method, String host, CallOptions options, TRequest request)
at Grpc.Core.Interceptors.InterceptingCallInvoker.<BlockingUnaryCall>b__3_0[TRequest,TResponse](TRequest req, ClientInterceptorContext`2 ctx)
at Grpc.Core.ClientBase.ClientBaseConfiguration.ClientBaseConfigurationInterceptor.BlockingUnaryCall[TRequest,TResponse](TRequest request, ClientInterceptorContext`2 context, BlockingUnaryCallContinuation`2 continuation)
at Grpc.Core.Interceptors.InterceptingCallInvoker.BlockingUnaryCall[TRequest,TResponse](Method`2 method, String host, CallOptions options, TRequest request)
at SomeNamespace.RpcServices.DaemonService.DaemonServiceClient.GetVncContainerEnvironment(EmptyRequest request, CallOptions options) in /src/SomeNamespace.RpcServices/obj/Release/netcoreapp3.0/Vnc-container-daemonGrpc.cs:line 98
at SomeNamespace.RpcServices.DaemonService.DaemonServiceClient.GetVncContainerEnvironment(EmptyRequest request, Metadata headers, Nullable`1 deadline, CancellationToken cancellationToken) in /src/SomeNamespace.RpcServices/obj/Release/netcoreapp3.0/Vnc-container-daemonGrpc.cs:line 94
at SomeNamespace.Rpc.RpcEventListener.StartListening() in /src/SomeNamespace/Rpc/RpcEventListener.cs:line 32
Do you have an idea what I've to do different? I cannot find much documentation about establishing insecure connections like that.

I found the fix on my own:
It works when I move the AppContext.SetSwitch above the AddGrpcClient.
// Enable support for unencrypted HTTP2
AppContext.SetSwitch("System.Net.Http.SocketsHttpHandler.Http2UnencryptedSupport", true);
// Registration of the DI service
services.AddGrpcClient<DaemonService.DaemonServiceClient>(options => {
options.Address = new Uri("http://localhost:42652");
options.ChannelOptionsActions.Add(channelOptions => channelOptions.Credentials = ChannelCredentials.Insecure);
});