I have the following solution which works, but I'm not sure I'm using all the components correctly. This is a simplified example, but what I'm doing is, every 10 minutes I check a database for account updates, build a list of them, and then use RestSharp to make the updates through an API, using 10 parallel processes.
One thing I might be doing wrong is handling the responses. When I look at this more sophisticated example, instead of using a List, they're using a List of IReadOnlyCollections.
using System;
using System.Collections.Concurrent;
using System.Collections.Generic;
using System.Data.SqlClient;
using System.Linq;
using System.Threading;
using System.Threading.Tasks;
using Microsoft.Azure.WebJobs;
using Microsoft.Azure.WebJobs.Host;
using Microsoft.Extensions.Logging;
using Newtonsoft.Json;
using Newtonsoft.Json.Linq;
using RestSharp;
namespace AccountUpdate_Test
{
public class UpdateAccount
{
private static RestClient client = new RestClient("https://this.api.com/");
[FunctionName("UpdateAccount")]
public async Task Run([TimerTrigger("0 */10 * * * *")]TimerInfo myTimer, ILogger log)
{
var startTime = DateTime.Now;
log.LogInformation($"C# Timer trigger function executed at: {startTime}");
var updates = new List<AccountUpdate>();
var semaphoreSlim = new SemaphoreSlim(
initialCount: 10,
maxCount: 10);
//var responses = new ConcurrentBag<IReadOnlyCollection<RestResponse>>();
var responses = new List<RestResponse>();
using (SqlConnection connection = new SqlConnection(Environment.GetEnvironmentVariable("SqlConnectionString")))
{
string sql = "select data from table";
SqlCommand command = new SqlCommand(sql, connection);
connection.Open();
using (SqlDataReader reader = command.ExecuteReader())
{
while (reader.Read())
{
updates.Add(new AccountUpdate(reader.GetString(0), reader.GetString(3), reader.GetString(4)));
}
}
}
var tasks = updates.Select(async update =>
{
await semaphoreSlim.WaitAsync();
try
{
var response = await httpRequest(update, log);
responses.Add(response);
}
finally
{
semaphoreSlim.Release();
}
});
await Task.WhenAll(tasks);
Console.WriteLine(responses.Count);
}
private async Task<RestResponse> httpRequest(AccountUpdate update, ILogger log)
{
var request = new RestRequest($"test?api_key=abc123", Method.Post);
RestResponse restResponse = null;
int count = 3;
while (count > 0)
{
try
{
request.AddStringBody(update.GetCrowdTwistJson(), DataFormat.Json);
restResponse = await client.ExecutePutAsync(request);
string httpStatus = ((int)restResponse.StatusCode).ToString();
JObject jObject = JObject.Parse(restResponse.Content);
switch (httpStatus.Substring(0, 1))
{
// Success, http 200, log and complete
case "2":
count = 0;
insertSent("C", httpStatus, $"Success: ID {jObject["id"]}", log);
break;
// Network or infrastructure error, likely temporary, retry after wait
case "5":
// wait 5, 10, 15 seconds then give up
await Task.Delay((4 - count) * 5);
count--;
if (count == 0)
{
insertSent("E", httpStatus, restResponse.Content, log);
}
log.LogInformation($"Retry, {count} left.");
break;
// Likely a business/data issue. Log and and complete.
case "4":
count = 0;
insertSent("E", httpStatus, restResponse.Content, log);
break;
// Whatever other schenanigans
default:
count = 0;
insertSent("E", httpStatus, restResponse.Content, log);
break;
}
}
catch (Exception ex)
{
insertSent("E", "General", ex.ToString(), log);
}
}
return restResponse;
}
}
}
Related
I have a relatively simple Azure Function running under the consumption plan with a timeout setting of 4 four minutes.
When I execute through the portal it gets executed multiple times with runs overlapping each other, even though I had executed it just once.
For example:
Execution 1
00:36:52.917
00:41:25.750
Execution 2
00:40:41.793
00:41:25.700
Execution 3
00:44:27.430
00:48:30.857
There's no logic as to the multiple and overlapping execution times.
Any idea what is going on?
using System;
using System.Threading.Tasks;
using System.Net.Http;
using Microsoft.AspNetCore.Http;
using Microsoft.AspNetCore.Mvc;
using Microsoft.Azure.WebJobs;
using Microsoft.Azure.WebJobs.Extensions.Http;
using Microsoft.Extensions.Logging;
using System.Collections.Generic;
using Newtonsoft.Json;
using System.IO;
using Microsoft.Data.SqlClient;
namespace FMPApiCall
{
public static class FMPTestAPI
{
#region Private Data Members
private static readonly HttpClient Client = new HttpClient();
#endregion
[FunctionName("FMPTestAPI")]
public static async Task<IActionResult> Run(
[HttpTrigger(AuthorizationLevel.Function, "get", "post", Route = null)]
HttpRequest req, ILogger log)
{
log.LogInformation("FMPTestAPI triggered - BEGIN");
List<string> symbolsList = FMPCallAPI.CommonFunctions.GetSymbolsFromDB(); //gets as list of IDs from the database
int counter = 0;
using SqlConnection dbconnection = new SqlConnection(Environment.GetEnvironmentVariable("SqlServerConnectionString"));
{
dbconnection.Open();
foreach (string symbol in symbolsList)
{
counter++;
var apiRequest =
$"https://myapi.com/api/v3/dataset1/{symbol}?period=quarter&limit=4&apikey=123";
var response = await Client.GetAsync(apiRequest);
var symbolsData = await response.Content.ReadAsStringAsync();
var sqlStr = $"INSERT INTO [dbo].[_azure] (symbol,runid, response,lastupdated) VALUES ( '{symbol}' , {counter} ,'{response}', getutcdate() )";
using (SqlCommand cmd = new SqlCommand(sqlStr, dbconnection))
{
var rows = cmd.ExecuteNonQuery();
}
}
dbconnection.Close();
}
log.LogInformation("FMPTestAPI triggered - END");
string name = req.Query["name"];
string requestBody = await new StreamReader(req.Body).ReadToEndAsync();
dynamic data = JsonConvert.DeserializeObject(requestBody);
name = name ?? data?.name;
return name != null
? (ActionResult)new OkObjectResult(new { Hello = name })
: new BadRequestObjectResult("Please pass a name on the query string or in the request body");
}
}
}
eee
i am new to C# and I would like to make a bot mixing Luis services and customvision.
I would like the user to be able to send an image or text to the bot.
I am starting from the microsoft bot framework "core bot"* and the imageprocessing bot** (*https://github.com/microsoft/BotBuilder-Samples/tree/master/samples/csharp_dotnetcore/13.core-bot, **https://github.com/mandardhikari/ImageProcessingBot )
for the moment, i do have this code which is mix between the two samples
#region References
using System;
using System.Collections.Generic;
using System.Text;
using System.Text.RegularExpressions;
using System.Linq;
using System.Threading;
using System.Threading.Tasks;
using Microsoft.Bot.Builder;
using Microsoft.Bot.Builder.Dialogs;
using Microsoft.Bot.Schema;
using Newtonsoft.Json;
using Microsoft.Extensions.Configuration;
using System.Net.Http;
using System.IO;
using Microsoft.Azure.CognitiveServices.Vision.ComputerVision.Models;
using Microsoft.Extensions.Logging;
using Microsoft.Recognizers.Text.DataTypes.TimexExpression;
#endregion
namespace ImageProcessingBot
{
public class ImageProcessingBot : IBot
{
private readonly FlightBookingRecognizer _luisRecognizer;
protected readonly ILogger Logger;
private readonly ImageProcessingBotAccessors _accessors;
private readonly IConfiguration _configuration;
private readonly DialogSet _dialogs;
public ImageProcessingBot(ImageProcessingBotAccessors accessors, IConfiguration configuration)
{
_accessors = accessors ?? throw new ArgumentNullException(nameof(accessors));
_configuration = configuration ?? throw new ArgumentNullException(nameof(configuration));
_dialogs = new DialogSet(_accessors.ConversationDialogState);
}
public async Task OnTurnAsync(ITurnContext turnContext, CancellationToken cancellationToken = default(CancellationToken))
{
Activity reply = null;
HeroCard card ;
StringBuilder sb;
switch(turnContext.Activity.Type)
{
case ActivityTypes.ConversationUpdate:
foreach(var member in turnContext.Activity.MembersAdded)
{
if(member.Id != turnContext.Activity.Recipient.Id)
{
reply = await CreateReplyAsync(turnContext, "Welcome. Please select and operation");
await turnContext.SendActivityAsync(reply, cancellationToken:cancellationToken);
}
}
break;
case ActivityTypes.Message:
int attachmentCount = turnContext.Activity.Attachments != null ? turnContext.Activity.Attachments.Count() : 0;
var command = !string.IsNullOrEmpty(turnContext.Activity.Text) ? turnContext.Activity.Text : await _accessors.CommandState.GetAsync(turnContext, () => string.Empty, cancellationToken);
command = command.ToLowerInvariant();
if(attachmentCount == 0)
{
var luisResult = await _luisRecognizer.RecognizeAsync<FlightBooking>(turnContext, cancellationToken);
switch (luisResult.TopIntent().intent)
{
case FlightBooking.Intent.Weather:
// We haven't implemented the GetWeatherDialog so we just display a TODO message.
var getWeatherMessageText = "TODO: get weather flow here";
var getWeatherMessage = MessageFactory.Text(getWeatherMessageText, getWeatherMessageText, InputHints.IgnoringInput);
await turnContext.SendActivityAsync(getWeatherMessage, cancellationToken);
var attachments = new List<Attachment>();
var reply = MessageFactory.Attachment(attachments);
reply.Attachments.Add(Cards.CreateAdaptiveCardAttachment());
default:
// Catch all for unhandled intents
var didntUnderstandMessageText = $"Sorry, I didn't get that. Please try asking in a different way (intent was {luisResult.TopIntent().intent})";
var didntUnderstandMessage = MessageFactory.Text(didntUnderstandMessageText, didntUnderstandMessageText, InputHints.IgnoringInput);
await turnContext.SendActivityAsync(didntUnderstandMessage, cancellationToken);
break;
}
}
else
{
HttpClient client = new HttpClient();
Attachment attachment = turnContext.Activity.Attachments[0];
...// then it stays as in https://github.com/mandardhikari/ImageProcessingBot/blob/master/ImageProcessingBot/ImageProcessingBot/ImageProcessingBot.cs
I am getting these logs.
1>ImageProcessingBot.cs(78,41,78,46): error CS0136: A local or parameter named 'reply' cannot be declared in this scope because that name is used in an enclosing local scope to define a local or parameter
1>ImageProcessingBot.cs(72,33,72,67): error CS0163: Control cannot fall through from one case label ('case FlightBooking.Intent.Weather:') to another
I am new to c#, so any insights on the above would be greatly appreciated!
This is essentially repeating what #stuartd says above in the comments.
The method MessageFactory.Attachemnt returns an IMessageActivity so probably best to use a second variable rather than your Activity reply to avoid casting
case FlightBooking.Intent.Weather:
// We haven't implemented the GetWeatherDialog so we just display a TODO message.
var getWeatherMessageText = "TODO: get weather flow here";
var getWeatherMessage = MessageFactory.Text(getWeatherMessageText, getWeatherMessageText, InputHints.IgnoringInput);
await turnContext.SendActivityAsync(getWeatherMessage, cancellationToken);
var attachments = new List<Attachment>();
meesageReply = MessageFactory.Attachment(attachments); //new variable
messageReply.Attachments.Add(Cards.CreateAdaptiveCardAttachment());
break;
I'm trying to get the Bing Speech API to work in C# via WebSockets. I've looked through the implementation in Javascript here and have been following the protocol instructions here, but I've come up against a complete brick wall. I can't use the existing C# service because I'm running in a Linux container, so I need to use an implementation on .net Core. Annoyingly, the existing service is closed-source!
I can connect to the web socket successfully, but I can't ever get the server to respond to my connection. I'm expecting to receive a turn.start text message from the server, but I get booted off the server as soon as I've sent a few bytes of an audio file. I know the audio file is in the right format because I've got it directly from the C# service sample here.
I feel like I’ve exhausted the options here. The only thing I can think of now is that I’m not sending the audio chunks correctly. Currently, I’m just sending the audio file in consecutive 4096 bytes. I know the first audio message contains the RIFF header which is only 36 bytes, and then I'm just sending this along with the next (4096-36) bytes.
Here is my code in full. You should just be able to run it as a .net core or .net framework console application, and will need an audio file and an API key.
using Newtonsoft.Json;
using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Net.Http;
using System.Net.WebSockets;
using System.Text;
using System.Threading;
using System.Threading.Tasks;
namespace ConsoleApp3
{
class Program
{
static void Main(string[] args)
{
Task.Run(async () =>
{
var bingService = new BingSpeechToTextService();
var audioFilePath = #"FILEPATH GOES HERE";
var authenticationKey = #"BING AUTHENTICATION KEY GOES HERE";
await bingService.RegisterJob(audioFilePath, authenticationKey);
}).Wait();
}
}
public class BingSpeechToTextService
{
/* #region Private Static Methods */
private static async Task Receiving(ClientWebSocket client)
{
var buffer = new byte[128];
while (true)
{
var result = await client.ReceiveAsync(new ArraySegment<byte>(buffer), CancellationToken.None);
var res = Encoding.UTF8.GetString(buffer, 0, result.Count);
if (result.MessageType == WebSocketMessageType.Text)
{
Console.WriteLine(Encoding.UTF8.GetString(buffer, 0, result.Count));
}
else if (result.MessageType == WebSocketMessageType.Close)
{
Console.WriteLine($"Closing ... reason {client.CloseStatusDescription}");
var description = client.CloseStatusDescription;
//await client.CloseOutputAsync(WebSocketCloseStatus.NormalClosure, "", CancellationToken.None);
break;
}
else
{
Console.WriteLine("Other result");
}
}
}
/* #endregion Private Static Methods */
/* #region Public Static Methods */
public static UInt16 ReverseBytes(UInt16 value)
{
return (UInt16)((value & 0xFFU) << 8 | (value & 0xFF00U) >> 8);
}
/* #endregion Public Static Methods */
/* #region Interface: 'Unscrypt.Bing.SpeechToText.Client.Api.IBingSpeechToTextJobService' Methods */
public async Task<int?> RegisterJob(string audioFilePath, string authenticationKeyStr)
{
var authenticationKey = new BingSocketAuthentication(authenticationKeyStr);
var token = authenticationKey.GetAccessToken();
/* #region Connect web socket */
var cws = new ClientWebSocket();
var connectionId = Guid.NewGuid().ToString("N");
var lang = "en-US";
cws.Options.SetRequestHeader("X-ConnectionId", connectionId);
cws.Options.SetRequestHeader("Authorization", "Bearer " + token);
Console.WriteLine("Connecting to web socket.");
var url = $"wss://speech.platform.bing.com/speech/recognition/interactive/cognitiveservices/v1?format=simple&language={lang}";
await cws.ConnectAsync(new Uri(url), new CancellationToken());
Console.WriteLine("Connected.");
/* #endregion*/
/* #region Receiving */
var receiving = Receiving(cws);
/* #endregion*/
/* #region Sending */
var sending = Task.Run(async () =>
{
/* #region Send speech.config */
dynamic speechConfig =
new
{
context = new
{
system = new
{
version = "1.0.00000"
},
os = new
{
platform = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36",
name = "Browser",
version = ""
},
device = new
{
manufacturer = "SpeechSample",
model = "SpeechSample",
version = "1.0.00000"
}
}
};
var requestId = Guid.NewGuid().ToString("N");
var speechConfigJson = JsonConvert.SerializeObject(speechConfig, Formatting.None);
StringBuilder outputBuilder = new StringBuilder();
outputBuilder.Append("path:speech.config\r\n"); //Should this be \r\n
outputBuilder.Append($"x-timestamp:{DateTime.UtcNow.ToString("yyyy-MM-ddTHH:mm:ss.fffK")}\r\n");
outputBuilder.Append($"content-type:application/json\r\n");
outputBuilder.Append("\r\n\r\n");
outputBuilder.Append(speechConfigJson);
var strh = outputBuilder.ToString();
var encoded = Encoding.UTF8.GetBytes(outputBuilder.ToString());
var buffer = new ArraySegment<byte>(encoded, 0, encoded.Length);
if (cws.State != WebSocketState.Open) return;
Console.WriteLine("Sending speech.config");
await cws.SendAsync(buffer, WebSocketMessageType.Text, true, new CancellationToken());
Console.WriteLine("Sent.");
/* #endregion*/
/* #region Send audio parts. */
var fileInfo = new FileInfo(audioFilePath);
var streamReader = fileInfo.OpenRead();
for (int cursor = 0; cursor < fileInfo.Length; cursor++)
{
outputBuilder.Clear();
outputBuilder.Append("path:audio\r\n");
outputBuilder.Append($"x-requestid:{requestId}\r\n");
outputBuilder.Append($"x-timestamp:{DateTime.UtcNow.ToString("yyyy-MM-ddTHH:mm:ss.fffK")}\r\n");
outputBuilder.Append($"content-type:audio/x-wav");
var headerBytes = Encoding.ASCII.GetBytes(outputBuilder.ToString());
var headerbuffer = new ArraySegment<byte>(headerBytes, 0, headerBytes.Length);
var str = "0x" + (headerBytes.Length).ToString("X");
var headerHeadBytes = BitConverter.GetBytes((UInt16)headerBytes.Length);
var isBigEndian = !BitConverter.IsLittleEndian;
var headerHead = !isBigEndian ? new byte[] { headerHeadBytes[1], headerHeadBytes[0] } : new byte[] { headerHeadBytes[0], headerHeadBytes[1] };
//Audio should be pcm 16kHz, 16bps mono
var byteLen = 8192 - headerBytes.Length - 2;
var fbuff = new byte[byteLen];
streamReader.Read(fbuff, 0, byteLen);
var arr = headerHead.Concat(headerBytes).Concat(fbuff).ToArray();
var arrSeg = new ArraySegment<byte>(arr, 0, arr.Length);
Console.WriteLine($"Sending data from {cursor}");
if (cws.State != WebSocketState.Open) return;
cursor += byteLen;
var end = cursor >= fileInfo.Length;
await cws.SendAsync(arrSeg, WebSocketMessageType.Binary, true, new CancellationToken());
Console.WriteLine("Data sent");
var dt = Encoding.ASCII.GetString(arr);
}
await cws.SendAsync(new ArraySegment<byte>(), WebSocketMessageType.Binary, true, new CancellationToken());
streamReader.Dispose();
/* #endregion*/
{
var startWait = DateTime.UtcNow;
while ((DateTime.UtcNow - startWait).TotalSeconds < 30)
{
await Task.Delay(1);
}
if (cws.State != WebSocketState.Open) return;
}
});
/* #endregion*/
/* #region Wait for tasks to complete */
await Task.WhenAll(sending, receiving);
if (sending.IsFaulted)
{
var err = sending.Exception;
throw err;
}
if (receiving.IsFaulted)
{
var err = receiving.Exception;
throw err;
}
/* #endregion*/
return null;
}
/* #endregion Interface: 'Unscrypt.Bing.SpeechToText.Client.Api.IBingSpeechToTextJobService' Methods */
public class BingSocketAuthentication
{
public static readonly string FetchTokenUri = "https://api.cognitive.microsoft.com/sts/v1.0";
private string subscriptionKey;
private string token;
private Timer accessTokenRenewer;
//Access token expires every 10 minutes. Renew it every 9 minutes.
private const int RefreshTokenDuration = 9;
public BingSocketAuthentication(string subscriptionKey)
{
this.subscriptionKey = subscriptionKey;
this.token = FetchToken(FetchTokenUri, subscriptionKey).Result;
// renew the token on set duration.
accessTokenRenewer = new Timer(new TimerCallback(OnTokenExpiredCallback),
this,
TimeSpan.FromMinutes(RefreshTokenDuration),
TimeSpan.FromMilliseconds(-1));
}
public string GetAccessToken()
{
return this.token;
}
private void RenewAccessToken()
{
this.token = FetchToken(FetchTokenUri, this.subscriptionKey).Result;
Console.WriteLine("Renewed token.");
}
private void OnTokenExpiredCallback(object stateInfo)
{
try
{
RenewAccessToken();
}
catch (Exception ex)
{
Console.WriteLine(string.Format("Failed renewing access token. Details: {0}", ex.Message));
}
finally
{
try
{
accessTokenRenewer.Change(TimeSpan.FromMinutes(RefreshTokenDuration), TimeSpan.FromMilliseconds(-1));
}
catch (Exception ex)
{
Console.WriteLine(string.Format("Failed to reschedule the timer to renew access token. Details: {0}", ex.Message));
}
}
}
private async Task<string> FetchToken(string fetchUri, string subscriptionKey)
{
using (var client = new HttpClient())
{
client.DefaultRequestHeaders.Add("Ocp-Apim-Subscription-Key", subscriptionKey);
UriBuilder uriBuilder = new UriBuilder(fetchUri);
uriBuilder.Path += "/issueToken";
var result = await client.PostAsync(uriBuilder.Uri.AbsoluteUri, null);
Console.WriteLine("Token Uri: {0}", uriBuilder.Uri.AbsoluteUri);
return await result.Content.ReadAsStringAsync();
}
}
}
}
}
I knew it was going to be simple.
After a frustrating few hours of coding, I've found the problem. I've been forgetting to send a request id along with the speech.config call.
How do you delete the dead letters in an Azure Service Bus queue?
I can process the messages in the queue ...
var queueClient = QueueClient.CreateFromConnectionString(sbConnectionString, queueName);
while (queueClient.Peek() != null)
{
var brokeredMessage = queueClient.Receive();
brokeredMessage.Complete();
}
but can't see anyway to handle the dead letter messages
The trick is to get the deadletter path for the queue which you can get by using QueueClient.FormatDeadLetterPath(queueName).
Please try the following:
var queueClient = QueueClient.CreateFromConnectionString(sbConnectionString, QueueClient.FormatDeadLetterPath(queueName));
while (queueClient.Peek() != null)
{
var brokeredMessage = queueClient.Receive();
brokeredMessage.Complete();
}
using System;
using Microsoft.VisualStudio.TestTools.UnitTesting;
using Microsoft.Azure.ServiceBus;
using Microsoft.Azure.ServiceBus.Core;
using System.Text;
using System.Collections.Generic;
using System.Threading.Tasks;
namespace ClearDeadLetterQ
{
[TestClass]
public class UnitTest1
{
const string ServiceBusConnectionString = "Endpoint=sb://my-domain.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=youraccesskeyhereyouraccesskeyhere";
[TestMethod]
public async Task TestMethod1()
{
await this.ClearDeadLetters("my.topic.name", "MySubscriptionName/$DeadLetterQueue");
}
public async Task ClearDeadLetters(string topicName, string subscriptionName)
{
var messageReceiver = new MessageReceiver(ServiceBusConnectionString, EntityNameHelper.FormatSubscriptionPath(topicName, subscriptionName), ReceiveMode.PeekLock);
var message = await messageReceiver.ReceiveAsync();
while ((message = await messageReceiver.ReceiveAsync()) != null)
{
await messageReceiver.CompleteAsync(message.SystemProperties.LockToken);
}
await messageReceiver.CloseAsync();
}
}
}
There are some great samples available in our GitHub sample repo (https://github.com/Azure-Samples/azure-servicebus-messaging-samples). The DeadletterQueue project should show you an example of how to do this in your code:
var dead-letterReceiver = await receiverFactory.CreateMessageReceiverAsync(
QueueClient.FormatDeadLetterPath(queueName), ReceiveMode.PeekLock);
while (true)
{
var msg = await dead-letterReceiver.ReceiveAsync(TimeSpan.Zero);
if (msg != null)
{
foreach (var prop in msg.Properties)
{
Console.WriteLine("{0}={1}", prop.Key, prop.Value);
}
await msg.CompleteAsync();
}
else
{
break;
}
}
}
Hope that helps!
Open Azure into your target Bus Queue Service.
(Set this value in place of <BUS-QUEUE-NAME> on the code)
Go into the queue you want to delete.
(Set this value in place of <QUEUE-NAME> on the code)
Create a Shared Access Policy and name it: RemoveDeadLetterQueue with checkbox Manage been selected.
Copy this Primary Key onto <QUEUE-POLICY-PRIMARYKEY> in this code.
And the code is ready to run.
using Microsoft.Azure.ServiceBus.Core;
using Microsoft.Azure.ServiceBus;
string serviceBusQueue = "<BUS-QUEUE-NAME>";
string serviceBusQueueName = "<QUEUE-NAME>";
string policyName = "RemoveDeadLetterQueue";
string policyPrimaryKey = "<QUEUE-POLICY-PRIMARYKEY>";
var receiver = new MessageReceiver(
connectionString: $"Endpoint=sb://{serviceBusQueue}.servicebus.windows.net/;SharedAccessKeyName={policyName};SharedAccessKey={policyPrimaryKey}",
entityPath: $"{serviceBusQueueName}/$DeadLetterQueue",
receiveMode: ReceiveMode.ReceiveAndDelete
);
var messages = await receiver.ReceiveAsync(maxMessageCount: 1000);
while(messages != null)
{
foreach (var message in messages)
{
Console.WriteLine($"[Delete Message] ID: {message.MessageId}");
}
messages = await receiver.ReceiveAsync(maxMessageCount: 1000);
}
await receiver.CloseAsync();
The task i want to accomplish is to create a Web API service in order to upload a file to Azure storage. At the same time, i would like to have a progress indicator that reflects the actual upload progress. After some research and studying i found out two important things:
First is that i have to split the file manually into chunks, and upload them asynchronously using the PutBlockAsync method from Microsoft.WindowsAzure.Storage.dll.
Second, is that i have to receive the file in my Web API service in Streamed mode and not in Buffered mode.
So until now i have the following implementation:
UploadController.cs
using System.Configuration;
using System.Net;
using System.Net.Http;
using System.Threading.Tasks;
using System.Web.Http;
using Microsoft.WindowsAzure.Storage;
using Microsoft.WindowsAzure.Storage.Blob;
using WebApiFileUploadToAzureStorage.Infrastructure;
using WebApiFileUploadToAzureStorage.Models;
namespace WebApiFileUploadToAzureStorage.Controllers
{
public class UploadController : ApiController
{
[HttpPost]
public async Task<HttpResponseMessage> UploadFile()
{
if (!Request.Content.IsMimeMultipartContent("form-data"))
{
return Request.CreateResponse(HttpStatusCode.UnsupportedMediaType,
new UploadStatus(null, false, "No form data found on request.", string.Empty, string.Empty));
}
var streamProvider = new MultipartAzureBlobStorageProvider(GetAzureStorageContainer());
var result = await Request.Content.ReadAsMultipartAsync(streamProvider);
if (result.FileData.Count < 1)
{
return Request.CreateResponse(HttpStatusCode.BadRequest,
new UploadStatus(null, false, "No files were uploaded.", string.Empty, string.Empty));
}
return Request.CreateResponse(HttpStatusCode.OK);
}
private static CloudBlobContainer GetAzureStorageContainer()
{
var storageConnectionString = ConfigurationManager.AppSettings["AzureBlobStorageConnectionString"];
var storageAccount = CloudStorageAccount.Parse(storageConnectionString);
var blobClient = storageAccount.CreateCloudBlobClient();
blobClient.DefaultRequestOptions.SingleBlobUploadThresholdInBytes = 1024 * 1024;
var container = blobClient.GetContainerReference("photos");
if (container.Exists())
{
return container;
}
container.Create();
container.SetPermissions(new BlobContainerPermissions
{
PublicAccess = BlobContainerPublicAccessType.Container
});
return container;
}
}
}
MultipartAzureBlobStorageProvider.cs
using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.IO;
using System.Linq;
using System.Net.Http;
using System.Text;
using System.Threading;
using System.Threading.Tasks;
using Microsoft.WindowsAzure.Storage.Blob;
namespace WebApiFileUploadToAzureStorage.Infrastructure
{
public class MultipartAzureBlobStorageProvider : MultipartFormDataStreamProvider
{
private readonly CloudBlobContainer _blobContainer;
public MultipartAzureBlobStorageProvider(CloudBlobContainer blobContainer) : base(Path.GetTempPath())
{
_blobContainer = blobContainer;
}
public override Task ExecutePostProcessingAsync()
{
const int blockSize = 256 * 1024;
var fileData = FileData.First();
var fileName = Path.GetFileName(fileData.Headers.ContentDisposition.FileName.Trim('"'));
var blob = _blobContainer.GetBlockBlobReference(fileName);
var bytesToUpload = (new FileInfo(fileData.LocalFileName)).Length;
var fileSize = bytesToUpload;
blob.Properties.ContentType = fileData.Headers.ContentType.MediaType;
blob.StreamWriteSizeInBytes = blockSize;
if (bytesToUpload < blockSize)
{
var cancellationToken = new CancellationToken();
using (var fileStream = new FileStream(fileData.LocalFileName, FileMode.Open, FileAccess.ReadWrite))
{
var upload = blob.UploadFromStreamAsync(fileStream, cancellationToken);
Debug.WriteLine($"Status {upload.Status}.");
upload.ContinueWith(task =>
{
Debug.WriteLine($"Status {task.Status}.");
Debug.WriteLine("Upload is over successfully.");
}, TaskContinuationOptions.OnlyOnRanToCompletion);
upload.ContinueWith(task =>
{
Debug.WriteLine($"Status {task.Status}.");
if (task.Exception != null)
{
Debug.WriteLine("Task could not be completed." + task.Exception.InnerException);
}
}, TaskContinuationOptions.OnlyOnFaulted);
upload.Wait(cancellationToken);
}
}
else
{
var blockIds = new List<string>();
var index = 1;
long startPosition = 0;
long bytesUploaded = 0;
do
{
var bytesToRead = Math.Min(blockSize, bytesToUpload);
var blobContents = new byte[bytesToRead];
using (var fileStream = new FileStream(fileData.LocalFileName, FileMode.Open))
{
fileStream.Position = startPosition;
fileStream.Read(blobContents, 0, (int)bytesToRead);
}
var manualResetEvent = new ManualResetEvent(false);
var blockId = Convert.ToBase64String(Encoding.UTF8.GetBytes(index.ToString("d6")));
Debug.WriteLine($"Now uploading block # {index.ToString("d6")}");
blockIds.Add(blockId);
var upload = blob.PutBlockAsync(blockId, new MemoryStream(blobContents), null);
upload.ContinueWith(task =>
{
bytesUploaded += bytesToRead;
bytesToUpload -= bytesToRead;
startPosition += bytesToRead;
index++;
var percentComplete = (double)bytesUploaded / fileSize;
Debug.WriteLine($"Percent complete: {percentComplete.ToString("P")}");
manualResetEvent.Set();
});
manualResetEvent.WaitOne();
} while (bytesToUpload > 0);
Debug.WriteLine("Now committing block list.");
var putBlockList = blob.PutBlockListAsync(blockIds);
putBlockList.ContinueWith(task =>
{
Debug.WriteLine("Blob uploaded completely.");
});
putBlockList.Wait();
}
File.Delete(fileData.LocalFileName);
return base.ExecutePostProcessingAsync();
}
}
}
I also enabled Streamed mode as this blog post suggests. This approach works great, in terms that the file is uploaded successfully to Azure storage. Then, when i make a call to this service making use of XMLHttpRequest (and subscribing to the progress event) i see the indicator moving to 100% very quickly. If a 5MB file needs around 1 minute to upload, my indicator moves to the end in just 1 second. So probably the problem resides in the way that the server informs the client about the upload progress. Any thoughts about this? Thank you.
================================ Update 1 ===================================
That is the JavaScript code i use to call the service
function uploadFile(file, index, uploadCompleted) {
var authData = localStorageService.get("authorizationData");
var xhr = new XMLHttpRequest();
xhr.upload.addEventListener("progress", function (event) {
fileUploadPercent = Math.floor((event.loaded / event.total) * 100);
console.log(fileUploadPercent + " %");
});
xhr.onreadystatechange = function (event) {
if (event.target.readyState === event.target.DONE) {
if (event.target.status !== 200) {
} else {
var parsedResponse = JSON.parse(event.target.response);
uploadCompleted(parsedResponse);
}
}
};
xhr.open("post", uploadFileServiceUrl, true);
xhr.setRequestHeader("Authorization", "Bearer " + authData.token);
var data = new FormData();
data.append("file-" + index, file);
xhr.send(data);
}
your progress indicator might be moving rapidly fast, might be because of
public async Task<HttpResponseMessage> UploadFile()
i have encountered this before, when creating an api of async type, im not even sure if it can be awaited, it will just of course just finish your api call on the background, reason your progress indicator instantly finish, because of the async method (fire and forget). the api will immediately give you a response, but will actually finish on the server background (if not awaited).
please kindly try making it just
public HttpResponseMessage UploadFile()
and also try these ones
var result = Request.Content.ReadAsMultipartAsync(streamProvider).Result;
var upload = blob.UploadFromStreamAsync(fileStream, cancellationToken).Result;
OR
var upload = await blob.UploadFromStreamAsync(fileStream, cancellationToken);
hope it helps.
Other way to acomplish what you want (I don't understand how the XMLHttpRequest's progress event works) is using the ProgressMessageHandler to get the upload progress in the request. Then, in order to notify the client, you could use some cache to store the progress, and from the client request the current state in other endpoint, or use SignalR to send the progress from the server to the client
Something like:
//WebApiConfigRegister
var progress = new ProgressMessageHandler();
progress.HttpSendProgress += HttpSendProgress;
config.MessageHandlers.Add(progress);
//End WebApiConfig Register
private static void HttpSendProgress(object sender, HttpProgressEventArgs e)
{
var request = sender as HttpRequestMessage;
//todo: check if request is not null
//Get an Id from the client or something like this to identify the request
var id = request.RequestUri.Query[0];
var perc = e.ProgressPercentage;
var b = e.TotalBytes;
var bt = e.BytesTransferred;
Cache.InsertOrUpdate(id, perc);
}
You can check more documentation on this MSDN blog post (Scroll down to "Progress Notifications" section)
Also, you could calculate the progress based on the data chunks, store the progress in a cache, and notify in the same way as above. Something like this solution