We Keep Coding

Making multiple http calls from Azure Function App using RestSharp

I have the following solution which works, but I'm not sure I'm using all the components correctly. This is a simplified example, but what I'm doing is, every 10 minutes I check a database for account updates, build a list of them, and then use RestSharp to make the updates through an API, using 10 parallel processes.
One thing I might be doing wrong is handling the responses. When I look at this more sophisticated example, instead of using a List, they're using a List of IReadOnlyCollections.
using System;
using System.Collections.Concurrent;
using System.Collections.Generic;
using System.Data.SqlClient;
using System.Linq;
using System.Threading;
using System.Threading.Tasks;
using Microsoft.Azure.WebJobs;
using Microsoft.Azure.WebJobs.Host;
using Microsoft.Extensions.Logging;
using Newtonsoft.Json;
using Newtonsoft.Json.Linq;
using RestSharp;
namespace AccountUpdate_Test
{
public class UpdateAccount
{
private static RestClient client = new RestClient("https://this.api.com/");
[FunctionName("UpdateAccount")]
public async Task Run([TimerTrigger("0 */10 * * * *")]TimerInfo myTimer, ILogger log)
{
var startTime = DateTime.Now;
log.LogInformation($"C# Timer trigger function executed at: {startTime}");
var updates = new List<AccountUpdate>();
var semaphoreSlim = new SemaphoreSlim(
initialCount: 10,
maxCount: 10);
//var responses = new ConcurrentBag<IReadOnlyCollection<RestResponse>>();
var responses = new List<RestResponse>();
using (SqlConnection connection = new SqlConnection(Environment.GetEnvironmentVariable("SqlConnectionString")))
{
string sql = "select data from table";
SqlCommand command = new SqlCommand(sql, connection);
connection.Open();
using (SqlDataReader reader = command.ExecuteReader())
{
while (reader.Read())
{
updates.Add(new AccountUpdate(reader.GetString(0), reader.GetString(3), reader.GetString(4)));
}
}
}
var tasks = updates.Select(async update =>
{
await semaphoreSlim.WaitAsync();
try
{
var response = await httpRequest(update, log);
responses.Add(response);
}
finally
{
semaphoreSlim.Release();
}
});
await Task.WhenAll(tasks);
Console.WriteLine(responses.Count);
}
private async Task<RestResponse> httpRequest(AccountUpdate update, ILogger log)
{
var request = new RestRequest($"test?api_key=abc123", Method.Post);
RestResponse restResponse = null;
int count = 3;
while (count > 0)
{
try
{
request.AddStringBody(update.GetCrowdTwistJson(), DataFormat.Json);
restResponse = await client.ExecutePutAsync(request);
string httpStatus = ((int)restResponse.StatusCode).ToString();
JObject jObject = JObject.Parse(restResponse.Content);
switch (httpStatus.Substring(0, 1))
{
// Success, http 200, log and complete
case "2":
count = 0;
insertSent("C", httpStatus, $"Success: ID {jObject["id"]}", log);
break;
// Network or infrastructure error, likely temporary, retry after wait
case "5":
// wait 5, 10, 15 seconds then give up
await Task.Delay((4 - count) * 5);
count--;
if (count == 0)
{
insertSent("E", httpStatus, restResponse.Content, log);
}
log.LogInformation($"Retry, {count} left.");
break;
// Likely a business/data issue. Log and and complete.
case "4":
count = 0;
insertSent("E", httpStatus, restResponse.Content, log);
break;
// Whatever other schenanigans
default:
count = 0;
insertSent("E", httpStatus, restResponse.Content, log);
break;
}
}
catch (Exception ex)
{
insertSent("E", "General", ex.ToString(), log);
}
}
return restResponse;
}
}
}

How to determine whether PostgreSQL running in a Docker container is done initializing with Marten?

I am basically writing a little bit of a program to benchmark the insertion performance of PostgreSQL over a given table growth, and I would like to make sure that when I am using Marten to insert data, the database is fully ready to accept insertions.
I am using Docker.DotNet to spawn a new container running the latest PostgreSQL image but even if the container is in a running state it is sometimes not the case for the Postgre running inside that container.
Of course, I could have added a Thread. Sleep but this is a bit random so I would rather like something deterministic to figure out when the database is ready to accept insertions?
public static class Program
{
public static async Task Main(params string[] args)
{
const string imageName = "postgres:latest";
const string containerName = "MyProgreSQL";
var client = new DockerClientConfiguration(Docker.DefaultLocalApiUri).CreateClient();
var containers = await client.Containers.SearchByNameAsync(containerName);
var container = containers.SingleOrDefault();
if (container != null)
{
await client.Containers.StopAndRemoveContainerAsync(container);
}
var createdContainer = await client.Containers.RunContainerAsync(new CreateContainerParameters
{
Image = imageName,
HostConfig = new HostConfig
{
PortBindings = new Dictionary<string, IList<PortBinding>>
{
{"5432/tcp", new List<PortBinding>
{
new PortBinding
{
HostPort = "5432"
}
}}
},
PublishAllPorts = true
},
Env = new List<string>
{
"POSTGRES_PASSWORD=root",
"POSTGRES_USER=root"
},
Name = containerName
});
var containerState = string.Empty;
while (containerState != "running")
{
containers = await client.Containers.SearchByNameAsync(containerName);
container = containers.Single();
containerState = container.State;
}
var store = DocumentStore.For("host=localhost;database=postgres;password=root;username=root");
var stopwatch = new Stopwatch();
using (var session = store.LightweightSession())
{
var orders = OrderHelpers.FakeOrders(10000);
session.StoreObjects(orders);
stopwatch.Start();
await session.SaveChangesAsync();
var elapsed = stopwatch.Elapsed;
// Whatever else needs to be done...
}
}
}
FYI, the if I am running the program above without pausing before the line await session.SaveChangesAsync(); I am running than into the following exception:
Unhandled Exception: Npgsql.NpgsqlException: Exception while reading from stream ---> System.IO.EndOfStreamException: Attempted to read past the end of the streams.
at Npgsql.NpgsqlReadBuffer.<>c__DisplayClass31_0.<<Ensure>g__EnsureLong|0>d.MoveNext() in C:\projects\npgsql\src\Npgsql\NpgsqlReadBuffer.cs:line 154
--- End of inner exception stack trace ---
at Npgsql.NpgsqlReadBuffer.<>c__DisplayClass31_0.<<Ensure>g__EnsureLong|0>d.MoveNext() in C:\projects\npgsql\src\Npgsql\NpgsqlReadBuffer.cs:line 175
--- End of stack trace from previous location where exception was thrown ---
at Npgsql.NpgsqlConnector.<>c__DisplayClass161_0.<<ReadMessage>g__ReadMessageLong|0>d.MoveNext() in C:\projects\npgsql\src\Npgsql\NpgsqlConnector.cs:l
ine 955
--- End of stack trace from previous location where exception was thrown ---
at Npgsql.NpgsqlConnector.Authenticate(String username, NpgsqlTimeout timeout, Boolean async) in C:\projects\npgsql\src\Npgsql\NpgsqlConnector.Auth.cs
:line 26
at Npgsql.NpgsqlConnector.Open(NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken) in C:\projects\npgsql\src\Npgsql\NpgsqlConne
ctor.cs:line 425
at Npgsql.ConnectorPool.AllocateLong(NpgsqlConnection conn, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken) in C:\projects\
npgsql\src\Npgsql\ConnectorPool.cs:line 246
at Npgsql.NpgsqlConnection.<>c__DisplayClass32_0.<<Open>g__OpenLong|0>d.MoveNext() in C:\projects\npgsql\src\Npgsql\NpgsqlConnection.cs:line 300
--- End of stack trace from previous location where exception was thrown ---
at Npgsql.NpgsqlConnection.Open() in C:\projects\npgsql\src\Npgsql\NpgsqlConnection.cs:line 153
at Marten.Storage.Tenant.generateOrUpdateFeature(Type featureType, IFeatureSchema feature)
at Marten.Storage.Tenant.ensureStorageExists(IList`1 types, Type featureType)
at Marten.Storage.Tenant.ensureStorageExists(IList`1 types, Type featureType)
at Marten.Storage.Tenant.StorageFor(Type documentType)
at Marten.DocumentSession.Store[T](T[] entities)
at Baseline.GenericEnumerableExtensions.Each[T](IEnumerable`1 values, Action`1 eachAction)
at Marten.DocumentSession.StoreObjects(IEnumerable`1 documents)
at Benchmark.Program.Main(String[] args) in C:\Users\eperret\Desktop\Benchmark\Benchmark\Program.cs:line 117
at Benchmark.Program.<Main>(String[] args)
[EDIT]
I accepted an answer but due to a bug about health parameters equivalence in the Docker.DotNet I could not leverage the solution given in the answer (I still think that a proper translation of that docker command in the .NET client, if actually possible) would be the best solution. In the meanwhile this is how I solved my problem, I basically poll the command that was expected to run in the health check aside until the result is ok:
Program.cs, the actual meat of the code:
using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.IO;
using System.Linq;
using System.Threading.Tasks;
using Benchmark.DockerClient;
using Benchmark.Domain;
using Benchmark.Utils;
using Docker.DotNet;
using Docker.DotNet.Models;
using Marten;
using Microsoft.Extensions.Configuration;
namespace Benchmark
{
public static class Program
{
public static async Task Main(params string[] args)
{
var configurationBuilder = new ConfigurationBuilder()
.SetBasePath(Directory.GetCurrentDirectory())
.AddJsonFile("appsettings.json");
var configuration = new Configuration();
configurationBuilder.Build().Bind(configuration);
var client = new DockerClientConfiguration(DockerClient.Docker.DefaultLocalApiUri).CreateClient();
var containers = await client.Containers.SearchByNameAsync(configuration.ContainerName);
var container = containers.SingleOrDefault();
if (container != null)
{
await client.Containers.StopAndRemoveContainerAsync(container.ID);
}
var createdContainer = await client.Containers.RunContainerAsync(new CreateContainerParameters
{
Image = configuration.ImageName,
HostConfig = new HostConfig
{
PortBindings = new Dictionary<string, IList<PortBinding>>
{
{$#"{configuration.ContainerPort}/{configuration.ContainerPortProtocol}", new List<PortBinding>
{
new PortBinding
{
HostPort = configuration.HostPort
}
}}
},
PublishAllPorts = true
},
Env = new List<string>
{
$"POSTGRES_USER={configuration.Username}",
$"POSTGRES_PASSWORD={configuration.Password}"
},
Name = configuration.ContainerName
});
var isContainerReady = false;
while (!isContainerReady)
{
var result = await client.Containers.RunCommandInContainerAsync(createdContainer.ID, "pg_isready -U postgres");
if (result.stdout.TrimEnd('\n') == $"/var/run/postgresql:{configuration.ContainerPort} - accepting connections")
{
isContainerReady = true;
}
}
var store = DocumentStore.For($"host=localhost;" +
$"database={configuration.DatabaseName};" +
$"username={configuration.Username};" +
$"password={configuration.Password}");
// Whatever else needs to be done...
}
}
Extensions being defined in ContainerOperationsExtensions.cs:
using System;
using System.Collections.Generic;
using System.Threading;
using System.Threading.Tasks;
using Docker.DotNet;
using Docker.DotNet.Models;
namespace Benchmark.DockerClient
{
public static class ContainerOperationsExtensions
{
public static async Task<IList<ContainerListResponse>> SearchByNameAsync(this IContainerOperations source, string name, bool all = true)
{
return await source.ListContainersAsync(new ContainersListParameters
{
All = all,
Filters = new Dictionary<string, IDictionary<string, bool>>
{
{"name", new Dictionary<string, bool>
{
{name, true}
}
}
}
});
}
public static async Task StopAndRemoveContainerAsync(this IContainerOperations source, string containerId)
{
await source.StopContainerAsync(containerId, new ContainerStopParameters());
await source.RemoveContainerAsync(containerId, new ContainerRemoveParameters());
}
public static async Task<CreateContainerResponse> RunContainerAsync(this IContainerOperations source, CreateContainerParameters parameters)
{
var createdContainer = await source.CreateContainerAsync(parameters);
await source.StartContainerAsync(createdContainer.ID, new ContainerStartParameters());
return createdContainer;
}
public static async Task<(string stdout, string stderr)> RunCommandInContainerAsync(this IContainerOperations source, string containerId, string command)
{
var commandTokens = command.Split(" ", StringSplitOptions.RemoveEmptyEntries);
var createdExec = await source.ExecCreateContainerAsync(containerId, new ContainerExecCreateParameters
{
AttachStderr = true,
AttachStdout = true,
Cmd = commandTokens
});
var multiplexedStream = await source.StartAndAttachContainerExecAsync(createdExec.ID, false);
return await multiplexedStream.ReadOutputToEndAsync(CancellationToken.None);
}
}
}
Docker.cs to get the local docker api uri:
using System;
using System.Runtime.InteropServices;
namespace Benchmark.DockerClient
{
public static class Docker
{
static Docker()
{
DefaultLocalApiUri = RuntimeInformation.IsOSPlatform(OSPlatform.Windows)
? new Uri("npipe://./pipe/docker_engine")
: new Uri("unix:/var/run/docker.sock");
}
public static Uri DefaultLocalApiUri { get; }
}
}

I suggest you to use a custom healtcheck to check if the database is ready to accept connections.
I am not familiar with the .NET client of Docker, but the following docker run command shows what you should try :
docker run --name postgres \
--health-cmd='pg_isready -U postgres' \
--health-interval='10s' \
--health-timeout='5s' \
--health-start-period='10s' \
postgres:latest
Time parameters should be updated accordingly to your needs.
With this healtcheck configured, your application must wait for the container to be in state "healthy" before trying to connect to the database. The status "healthy", in that particular case, means that the command pg_isready -U postgres have succeeded (so the database is ready to accept connections).
The status of your container can be retrieved with :
docker inspect --format "{{json .State.Health.Status }}" postgres

Azure Function Runs Locally But Not On Azure

My function is running locally but when I publish it to Azure it is erroring.
The error is
Value cannot be null. Parameter name: format
Googling this seems to suggest the input to the function is wrong but I am posting the exact same JSON that allows it to run locally.
I am lost to how I fix this. Any ideas?
Code below
using System;
using System.Configuration;
using System.Net;
using System.Net.Http;
using System.Security.Cryptography;
using System.ServiceModel.Description;
using System.Text;
using System.Threading.Tasks;
using Microsoft.Azure.WebJobs;
using Microsoft.Azure.WebJobs.Extensions.Http;
using Microsoft.Azure.WebJobs.Host;
using Microsoft.Xrm.Sdk;
using Microsoft.Xrm.Sdk.Client;
using Microsoft.Xrm.Sdk.Query;
using Newtonsoft.Json;
using Microsoft.Extensions.Logging;
namespace MyFunction
{
public static class Login
{
[FunctionName("Login")]
public static async Task<HttpResponseMessage> Run([HttpTrigger(AuthorizationLevel.Function, "get", "post", Route = null)]HttpRequestMessage req, ILogger log)
{
Boolean websiteEnabled = false;
Guid contactId = new Guid();
log.LogInformation("C# HTTP trigger function processed a request.");
dynamic data = await req.Content.ReadAsAsync<object>();
string username = data?.username;
string password = data?.password;
string passwordHash = "";
User user = new User();
OrganizationServiceProxy _serviceProxy;
IOrganizationService _service;
ClientCredentials clientCredentials = new ClientCredentials();
clientCredentials.UserName.UserName = ConfigurationManager.AppSettings["OrganisationUsername"];
clientCredentials.UserName.Password = ConfigurationManager.AppSettings["OrganisationPassword"];
Uri organisationUri = new Uri(String.Format(ConfigurationManager.AppSettings["OrganisationURL"]));
Uri realm = new Uri(String.Format(ConfigurationManager.AppSettings["OrganisationURL"]));
using (_serviceProxy = new OrganizationServiceProxy(organisationUri, realm, clientCredentials, null))
{
_serviceProxy.EnableProxyTypes();
_service = (IOrganizationService)_serviceProxy;
QueryByAttribute querybyattribute = new QueryByAttribute("contact");
querybyattribute.ColumnSet = new ColumnSet("cbob_websitepassword","cbob_websiteenabled","contactid","fullname", "parentcustomerid");
querybyattribute.Attributes.AddRange("emailaddress1");
querybyattribute.Values.AddRange(username);
EntityCollection retrieved = _service.RetrieveMultiple(querybyattribute);
if(retrieved.Entities.Count == 1)
{
passwordHash = retrieved.Entities[0].GetAttributeValue<String>("cbob_websitepassword");
websiteEnabled = retrieved.Entities[0].GetAttributeValue<Boolean>("cbob_websiteenabled");
contactId = retrieved.Entities[0].GetAttributeValue<Guid>("contactid");
user.Account = retrieved.Entities[0].GetAttributeValue<EntityReference>("parentcustomerid").Name.ToString();
user.Email = username;
user.LoggedInUser = retrieved.Entities[0].GetAttributeValue<String>("fullname");
user.AccountID = retrieved.Entities[0].GetAttributeValue<EntityReference>("parentcustomerid").Id.ToString();
user.BookingID = retrieved.Entities[0].Id.ToString();
} else
{
return req.CreateResponse(HttpStatusCode.BadRequest, "Not allowed");
}
}
Boolean hash = bCryptHash(passwordHash, contactId.ToString() + "-" + password);
Console.WriteLine(hash);
if (!websiteEnabled)
{
return req.CreateResponse(HttpStatusCode.BadRequest, "Not allowed");
}
if (hash)
{
string output = JsonConvert.SerializeObject(user).ToString();
return req.CreateResponse(HttpStatusCode.OK, output);
} else
{
return req.CreateResponse(HttpStatusCode.BadRequest, "Not allowed");
}
}
public static Boolean bCryptHash(string hash, string submitted)
{
Boolean hashPassword = BCrypt.Net.BCrypt.Verify(submitted,hash);
return hashPassword;
}
public static String sha256_hash(string value)
{
StringBuilder Sb = new StringBuilder();
using (var hash = SHA256.Create())
{
Encoding enc = Encoding.UTF8;
Byte[] result = hash.ComputeHash(enc.GetBytes(value));
foreach (Byte b in result)
Sb.Append(b.ToString("x2"));
}
return Sb.ToString();
}
}
}

Uri organisationUri = new Uri(String.Format(ConfigurationManager.AppSettings["OrganisationURL"]));
Uri realm = new Uri(String.Format(ConfigurationManager.AppSettings["OrganisationURL"]));
My guess is that one or both of those lines may be the problem. You are using String.Format here where the first parameter is the format parameter. The AppSettings you are providing for that parameter seem to be unavailable. Make sure you have those configuration values available when you deploy your function.
Additionally: If you don't provide any objects to the String.Format that get inserted in the String, why are you using it at all?

Make sure you have added those local app settings (i.e OrganisationUsername and so on in local.settings.json file) to Application settings. Find it in Azure portal, Platform features> Application settings. When we publish Function project to Azure, it's by design that content in local.settings.json is not published because it's designed for local dev.
When we publish Functions with VS, there's a friendly dialog to update Application settings.

C# Error In My Code Source (using System.Net.Http)

Langage : C# /
Codded Using : Visual Studio /
Using The System.Net.Http.dll
Hello , Please help Me I have 4 error in my code source project created in C# here is all error :
(I am a beginner) but if you can post the code cleaned fixed I thank you very much
Error 1
Error 1 (Code)
Error 2
Error 2 (Code)
Error 3
Error 3 (Code)
Error 4
Error 4 (Code)
using System;
using System.ComponentModel;
using System.Drawing;
using System.Net;
using System.Net.Http;
using System.Runtime.CompilerServices;
using System.Text;
using System.Windows.Forms;
namespace CheckerProject
{
public partial class Checker
{
public Checker()
{
InitializeComponent();
}
private async void Check()
{
string text = this.textBox1.Text;
using (HttpClientHandler httpClientHandler = new HttpClientHandler
{
AutomaticDecompression = (DecompressionMethods.GZip | DecompressionMethods.Deflate)
})
{
using (HttpClient httpClient = new HttpClient(httpClientHandler))
{
TaskAwaiter<HttpResponseMessage> taskAwaiter = httpClient.PostAsync("https:\\API.com", new StringContent("{\"onlineId\":\"" + text + "\",\"reserveIfAvailable\":false}".ToString(), Encoding.UTF8, "application/json")).GetAwaiter();
if (!taskAwaiter.IsCompleted)
{
await taskAwaiter;
TaskAwaiter<HttpResponseMessage> taskAwaiter2;
taskAwaiter = taskAwaiter2;
taskAwaiter2 = default(TaskAwaiter<HttpResponseMessage>);
}
HttpResponseMessage result = taskAwaiter.GetResult();
taskAwaiter = default(TaskAwaiter<HttpResponseMessage>);
HttpResponseMessage httpResponseMessage = result;
HttpResponseMessage httpResponseMessage2 = httpResponseMessage;
httpResponseMessage = null;
TaskAwaiter<string> taskAwaiter3 = httpResponseMessage2.Content.ReadAsStringAsync().GetAwaiter();
if (!taskAwaiter3.IsCompleted)
{
await taskAwaiter3;
TaskAwaiter<string> taskAwaiter4;
taskAwaiter3 = taskAwaiter4;
taskAwaiter4 = default(TaskAwaiter<string>);
}
string result2 = taskAwaiter3.GetResult();
taskAwaiter3 = default(TaskAwaiter<string>);
string text2 = result2;
string text3 = text2;
text2 = null;
if (httpResponseMessage2.StatusCode.ToString() == "429")
{
//Function
}
if (httpResponseMessage2.StatusCode != HttpStatusCode.BadRequest)
{
if (httpResponseMessage2.StatusCode == HttpStatusCode.Created)
{
//Function
}
else
{
//Function
}
}
else
{
//Function
if (text3.Contains("Online id already exists"))
{
//Function
}
if (text3.Contains("Improper"))
{
//Function
}
}
httpResponseMessage2 = null;
text3 = null;
}
HttpClient httpClient = null;
}
HttpClientHandler httpClientHandler = null;
}
}
}

I will answer the first two errors (they are the same problem).
You should remove the following 2 lines:
HttpClient httpClient = null;
HttpClientHandler httpClientHandler = null;
What you do here is you declare 2 NEW variables and assign the value 'null' to both of them.
What you meant to do is propably to assign 'null' to the existing variables. However that's not needed, since they are declared inside a 'using' block, which will automatically call the 'Dispose' method.

Web API - Get progress when uploading to Azure storage

The task i want to accomplish is to create a Web API service in order to upload a file to Azure storage. At the same time, i would like to have a progress indicator that reflects the actual upload progress. After some research and studying i found out two important things:
First is that i have to split the file manually into chunks, and upload them asynchronously using the PutBlockAsync method from Microsoft.WindowsAzure.Storage.dll.
Second, is that i have to receive the file in my Web API service in Streamed mode and not in Buffered mode.
So until now i have the following implementation:
UploadController.cs
using System.Configuration;
using System.Net;
using System.Net.Http;
using System.Threading.Tasks;
using System.Web.Http;
using Microsoft.WindowsAzure.Storage;
using Microsoft.WindowsAzure.Storage.Blob;
using WebApiFileUploadToAzureStorage.Infrastructure;
using WebApiFileUploadToAzureStorage.Models;
namespace WebApiFileUploadToAzureStorage.Controllers
{
public class UploadController : ApiController
{
[HttpPost]
public async Task<HttpResponseMessage> UploadFile()
{
if (!Request.Content.IsMimeMultipartContent("form-data"))
{
return Request.CreateResponse(HttpStatusCode.UnsupportedMediaType,
new UploadStatus(null, false, "No form data found on request.", string.Empty, string.Empty));
}
var streamProvider = new MultipartAzureBlobStorageProvider(GetAzureStorageContainer());
var result = await Request.Content.ReadAsMultipartAsync(streamProvider);
if (result.FileData.Count < 1)
{
return Request.CreateResponse(HttpStatusCode.BadRequest,
new UploadStatus(null, false, "No files were uploaded.", string.Empty, string.Empty));
}
return Request.CreateResponse(HttpStatusCode.OK);
}
private static CloudBlobContainer GetAzureStorageContainer()
{
var storageConnectionString = ConfigurationManager.AppSettings["AzureBlobStorageConnectionString"];
var storageAccount = CloudStorageAccount.Parse(storageConnectionString);
var blobClient = storageAccount.CreateCloudBlobClient();
blobClient.DefaultRequestOptions.SingleBlobUploadThresholdInBytes = 1024 * 1024;
var container = blobClient.GetContainerReference("photos");
if (container.Exists())
{
return container;
}
container.Create();
container.SetPermissions(new BlobContainerPermissions
{
PublicAccess = BlobContainerPublicAccessType.Container
});
return container;
}
}
}
MultipartAzureBlobStorageProvider.cs
using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.IO;
using System.Linq;
using System.Net.Http;
using System.Text;
using System.Threading;
using System.Threading.Tasks;
using Microsoft.WindowsAzure.Storage.Blob;
namespace WebApiFileUploadToAzureStorage.Infrastructure
{
public class MultipartAzureBlobStorageProvider : MultipartFormDataStreamProvider
{
private readonly CloudBlobContainer _blobContainer;
public MultipartAzureBlobStorageProvider(CloudBlobContainer blobContainer) : base(Path.GetTempPath())
{
_blobContainer = blobContainer;
}
public override Task ExecutePostProcessingAsync()
{
const int blockSize = 256 * 1024;
var fileData = FileData.First();
var fileName = Path.GetFileName(fileData.Headers.ContentDisposition.FileName.Trim('"'));
var blob = _blobContainer.GetBlockBlobReference(fileName);
var bytesToUpload = (new FileInfo(fileData.LocalFileName)).Length;
var fileSize = bytesToUpload;
blob.Properties.ContentType = fileData.Headers.ContentType.MediaType;
blob.StreamWriteSizeInBytes = blockSize;
if (bytesToUpload < blockSize)
{
var cancellationToken = new CancellationToken();
using (var fileStream = new FileStream(fileData.LocalFileName, FileMode.Open, FileAccess.ReadWrite))
{
var upload = blob.UploadFromStreamAsync(fileStream, cancellationToken);
Debug.WriteLine($"Status {upload.Status}.");
upload.ContinueWith(task =>
{
Debug.WriteLine($"Status {task.Status}.");
Debug.WriteLine("Upload is over successfully.");
}, TaskContinuationOptions.OnlyOnRanToCompletion);
upload.ContinueWith(task =>
{
Debug.WriteLine($"Status {task.Status}.");
if (task.Exception != null)
{
Debug.WriteLine("Task could not be completed." + task.Exception.InnerException);
}
}, TaskContinuationOptions.OnlyOnFaulted);
upload.Wait(cancellationToken);
}
}
else
{
var blockIds = new List<string>();
var index = 1;
long startPosition = 0;
long bytesUploaded = 0;
do
{
var bytesToRead = Math.Min(blockSize, bytesToUpload);
var blobContents = new byte[bytesToRead];
using (var fileStream = new FileStream(fileData.LocalFileName, FileMode.Open))
{
fileStream.Position = startPosition;
fileStream.Read(blobContents, 0, (int)bytesToRead);
}
var manualResetEvent = new ManualResetEvent(false);
var blockId = Convert.ToBase64String(Encoding.UTF8.GetBytes(index.ToString("d6")));
Debug.WriteLine($"Now uploading block # {index.ToString("d6")}");
blockIds.Add(blockId);
var upload = blob.PutBlockAsync(blockId, new MemoryStream(blobContents), null);
upload.ContinueWith(task =>
{
bytesUploaded += bytesToRead;
bytesToUpload -= bytesToRead;
startPosition += bytesToRead;
index++;
var percentComplete = (double)bytesUploaded / fileSize;
Debug.WriteLine($"Percent complete: {percentComplete.ToString("P")}");
manualResetEvent.Set();
});
manualResetEvent.WaitOne();
} while (bytesToUpload > 0);
Debug.WriteLine("Now committing block list.");
var putBlockList = blob.PutBlockListAsync(blockIds);
putBlockList.ContinueWith(task =>
{
Debug.WriteLine("Blob uploaded completely.");
});
putBlockList.Wait();
}
File.Delete(fileData.LocalFileName);
return base.ExecutePostProcessingAsync();
}
}
}
I also enabled Streamed mode as this blog post suggests. This approach works great, in terms that the file is uploaded successfully to Azure storage. Then, when i make a call to this service making use of XMLHttpRequest (and subscribing to the progress event) i see the indicator moving to 100% very quickly. If a 5MB file needs around 1 minute to upload, my indicator moves to the end in just 1 second. So probably the problem resides in the way that the server informs the client about the upload progress. Any thoughts about this? Thank you.
================================ Update 1 ===================================
That is the JavaScript code i use to call the service
function uploadFile(file, index, uploadCompleted) {
var authData = localStorageService.get("authorizationData");
var xhr = new XMLHttpRequest();
xhr.upload.addEventListener("progress", function (event) {
fileUploadPercent = Math.floor((event.loaded / event.total) * 100);
console.log(fileUploadPercent + " %");
});
xhr.onreadystatechange = function (event) {
if (event.target.readyState === event.target.DONE) {
if (event.target.status !== 200) {
} else {
var parsedResponse = JSON.parse(event.target.response);
uploadCompleted(parsedResponse);
}
}
};
xhr.open("post", uploadFileServiceUrl, true);
xhr.setRequestHeader("Authorization", "Bearer " + authData.token);
var data = new FormData();
data.append("file-" + index, file);
xhr.send(data);
}

your progress indicator might be moving rapidly fast, might be because of
public async Task<HttpResponseMessage> UploadFile()
i have encountered this before, when creating an api of async type, im not even sure if it can be awaited, it will just of course just finish your api call on the background, reason your progress indicator instantly finish, because of the async method (fire and forget). the api will immediately give you a response, but will actually finish on the server background (if not awaited).
please kindly try making it just
public HttpResponseMessage UploadFile()
and also try these ones
var result = Request.Content.ReadAsMultipartAsync(streamProvider).Result;
var upload = blob.UploadFromStreamAsync(fileStream, cancellationToken).Result;
OR
var upload = await blob.UploadFromStreamAsync(fileStream, cancellationToken);
hope it helps.

Other way to acomplish what you want (I don't understand how the XMLHttpRequest's progress event works) is using the ProgressMessageHandler to get the upload progress in the request. Then, in order to notify the client, you could use some cache to store the progress, and from the client request the current state in other endpoint, or use SignalR to send the progress from the server to the client
Something like:
//WebApiConfigRegister
var progress = new ProgressMessageHandler();
progress.HttpSendProgress += HttpSendProgress;
config.MessageHandlers.Add(progress);
//End WebApiConfig Register
private static void HttpSendProgress(object sender, HttpProgressEventArgs e)
{
var request = sender as HttpRequestMessage;
//todo: check if request is not null
//Get an Id from the client or something like this to identify the request
var id = request.RequestUri.Query[0];
var perc = e.ProgressPercentage;
var b = e.TotalBytes;
var bt = e.BytesTransferred;
Cache.InsertOrUpdate(id, perc);
}
You can check more documentation on this MSDN blog post (Scroll down to "Progress Notifications" section)
Also, you could calculate the progress based on the data chunks, store the progress in a cache, and notify in the same way as above. Something like this solution