We Keep Coding

Making multiple http calls from Azure Function App using RestSharp

I have the following solution which works, but I'm not sure I'm using all the components correctly. This is a simplified example, but what I'm doing is, every 10 minutes I check a database for account updates, build a list of them, and then use RestSharp to make the updates through an API, using 10 parallel processes.
One thing I might be doing wrong is handling the responses. When I look at this more sophisticated example, instead of using a List, they're using a List of IReadOnlyCollections.
using System;
using System.Collections.Concurrent;
using System.Collections.Generic;
using System.Data.SqlClient;
using System.Linq;
using System.Threading;
using System.Threading.Tasks;
using Microsoft.Azure.WebJobs;
using Microsoft.Azure.WebJobs.Host;
using Microsoft.Extensions.Logging;
using Newtonsoft.Json;
using Newtonsoft.Json.Linq;
using RestSharp;
namespace AccountUpdate_Test
{
public class UpdateAccount
{
private static RestClient client = new RestClient("https://this.api.com/");
[FunctionName("UpdateAccount")]
public async Task Run([TimerTrigger("0 */10 * * * *")]TimerInfo myTimer, ILogger log)
{
var startTime = DateTime.Now;
log.LogInformation($"C# Timer trigger function executed at: {startTime}");
var updates = new List<AccountUpdate>();
var semaphoreSlim = new SemaphoreSlim(
initialCount: 10,
maxCount: 10);
//var responses = new ConcurrentBag<IReadOnlyCollection<RestResponse>>();
var responses = new List<RestResponse>();
using (SqlConnection connection = new SqlConnection(Environment.GetEnvironmentVariable("SqlConnectionString")))
{
string sql = "select data from table";
SqlCommand command = new SqlCommand(sql, connection);
connection.Open();
using (SqlDataReader reader = command.ExecuteReader())
{
while (reader.Read())
{
updates.Add(new AccountUpdate(reader.GetString(0), reader.GetString(3), reader.GetString(4)));
}
}
}
var tasks = updates.Select(async update =>
{
await semaphoreSlim.WaitAsync();
try
{
var response = await httpRequest(update, log);
responses.Add(response);
}
finally
{
semaphoreSlim.Release();
}
});
await Task.WhenAll(tasks);
Console.WriteLine(responses.Count);
}
private async Task<RestResponse> httpRequest(AccountUpdate update, ILogger log)
{
var request = new RestRequest($"test?api_key=abc123", Method.Post);
RestResponse restResponse = null;
int count = 3;
while (count > 0)
{
try
{
request.AddStringBody(update.GetCrowdTwistJson(), DataFormat.Json);
restResponse = await client.ExecutePutAsync(request);
string httpStatus = ((int)restResponse.StatusCode).ToString();
JObject jObject = JObject.Parse(restResponse.Content);
switch (httpStatus.Substring(0, 1))
{
// Success, http 200, log and complete
case "2":
count = 0;
insertSent("C", httpStatus, $"Success: ID {jObject["id"]}", log);
break;
// Network or infrastructure error, likely temporary, retry after wait
case "5":
// wait 5, 10, 15 seconds then give up
await Task.Delay((4 - count) * 5);
count--;
if (count == 0)
{
insertSent("E", httpStatus, restResponse.Content, log);
}
log.LogInformation($"Retry, {count} left.");
break;
// Likely a business/data issue. Log and and complete.
case "4":
count = 0;
insertSent("E", httpStatus, restResponse.Content, log);
break;
// Whatever other schenanigans
default:
count = 0;
insertSent("E", httpStatus, restResponse.Content, log);
break;
}
}
catch (Exception ex)
{
insertSent("E", "General", ex.ToString(), log);
}
}
return restResponse;
}
}
}

Issue with more than one LUIS Dialog and its entities

I'm try to create a bot with multiple Luis intents, it means multiple dialogs naturally. I've installed the default CoreBot template (Downloaded from Azure). After setting intents and entities;
created second dialog which name is 'WeatherDialog',
created "WeatherDetails" which consists of getter and setter for my LUIS weather entities,
implemented some of code for accessing to entities' results to partial class which is already in the project for BookingDialog.
Then in the MainDialog, I tried to AddDialog.
MainDialog:
// Copyright (c) Microsoft Corporation. All rights reserved.
// Licensed under the MIT License.
using System;
using System.Collections.Generic;
using System.Linq;
using System.Threading;
using System.Threading.Tasks;
using Microsoft.Bot.Builder;
using Microsoft.Bot.Builder.Dialogs;
using Microsoft.Bot.Schema;
using Microsoft.Extensions.Logging;
using Microsoft.Recognizers.Text.DataTypes.TimexExpression;
using CoreBot;
namespace Microsoft.BotBuilderSamples.Dialogs
{
public class MainDialog : ComponentDialog
{
private readonly FlightBookingRecognizer _luisRecognizer;
protected readonly ILogger Logger;
// Dependency injection uses this constructor to instantiate MainDialog
public MainDialog(FlightBookingRecognizer luisRecognizer, BookingDialog bookingDialog, CoreBot.Dialogs.WeatherDialog weatherDialog, ILogger<MainDialog> logger)
: base(nameof(MainDialog))
{
_luisRecognizer = luisRecognizer;
Logger = logger;
AddDialog(new TextPrompt(nameof(TextPrompt)));
AddDialog(bookingDialog);
AddDialog(weatherDialog);
AddDialog(new WaterfallDialog(nameof(WaterfallDialog), new WaterfallStep[]
{
IntroStepAsync,
ActStepAsync,
FinalStepAsync,
}));
// The initial child Dialog to run.
InitialDialogId = nameof(WaterfallDialog);
}
private async Task<DialogTurnResult> IntroStepAsync(WaterfallStepContext stepContext, CancellationToken cancellationToken)
{
if (!_luisRecognizer.IsConfigured)
{
await stepContext.Context.SendActivityAsync(
MessageFactory.Text("NOTE: LUIS is not configured. To enable all capabilities, add 'LuisAppId', 'LuisAPIKey' and 'LuisAPIHostName' to the appsettings.json file.", inputHint: InputHints.IgnoringInput), cancellationToken);
return await stepContext.NextAsync(null, cancellationToken);
}
// Use the text provided in FinalStepAsync or the default if it is the first time.
var messageText = stepContext.Options?.ToString() ?? "How can I help you?";
var promptMessage = MessageFactory.Text(messageText, messageText, InputHints.ExpectingInput);
return await stepContext.PromptAsync(nameof(TextPrompt), new PromptOptions { Prompt = promptMessage }, cancellationToken);
}
private async Task<DialogTurnResult> ActStepAsync(WaterfallStepContext stepContext, CancellationToken cancellationToken)
{
if (!_luisRecognizer.IsConfigured)
{
// LUIS is not configured, we just run the BookingDialog path with an empty BookingDetailsInstance.
return await stepContext.BeginDialogAsync(nameof(BookingDialog), new BookingDetails(), cancellationToken);
}
// Call LUIS and gather any potential booking details. (Note the TurnContext has the response to the prompt.)
var luisResult = await _luisRecognizer.RecognizeAsync<CoreBot.WeatherModel>(stepContext.Context, cancellationToken);
switch (luisResult.TopIntent().intent)
{
case CoreBot.WeatherModel.Intent.BookFlight:
//await ShowWarningForUnsupportedCities(stepContext.Context, luisResult, cancellationToken);
Console.WriteLine("This is bookflight");
// Initialize BookingDetails with any entities we may have found in the response.
var bookingDetails = new BookingDetails()
{
// Get destination and origin from the composite entities arrays.
Destination = luisResult.ToEntities.Airport,
Origin = luisResult.FromEntities.Airport,
TravelDate = luisResult.TravelDate,
};
// Run the BookingDialog giving it whatever details we have from the LUIS call, it will fill out the remainder.
return await stepContext.BeginDialogAsync(nameof(BookingDialog), bookingDetails, cancellationToken);
case CoreBot.WeatherModel.Intent.GetWeather:
Console.WriteLine("This is getweather");
var weatherDetails = new CoreBot.WeatherDetails()
{
Location = luisResult.Location,
TravelDate = luisResult.TravelDate,
};
return await stepContext.BeginDialogAsync(nameof(CoreBot.Dialogs.WeatherDialog), weatherDetails, cancellationToken);
default:
// Catch all for unhandled intents
var didntUnderstandMessageText = $"Sorry, I didn't get that. Please try asking in a different way (intent was {luisResult.TopIntent().intent})";
var didntUnderstandMessage = MessageFactory.Text(didntUnderstandMessageText, didntUnderstandMessageText, InputHints.IgnoringInput);
await stepContext.Context.SendActivityAsync(didntUnderstandMessage, cancellationToken);
break;
}
return await stepContext.NextAsync(null, cancellationToken);
}
// Shows a warning if the requested From or To cities are recognized as entities but they are not in the Airport entity list.
// In some cases LUIS will recognize the From and To composite entities as a valid cities but the From and To Airport values
// will be empty if those entity values can't be mapped to a canonical item in the Airport.
private static async Task ShowWarningForUnsupportedCities(ITurnContext context, FlightBooking luisResult, CancellationToken cancellationToken)
{
var unsupportedCities = new List<string>();
var fromEntities = luisResult.FromEntities;
if (!string.IsNullOrEmpty(fromEntities.From) && string.IsNullOrEmpty(fromEntities.Airport))
{
unsupportedCities.Add(fromEntities.From);
}
var toEntities = luisResult.ToEntities;
if (!string.IsNullOrEmpty(toEntities.To) && string.IsNullOrEmpty(toEntities.Airport))
{
unsupportedCities.Add(toEntities.To);
}
if (unsupportedCities.Any())
{
var messageText = $"Sorry but the following airports are not supported: {string.Join(',', unsupportedCities)}";
var message = MessageFactory.Text(messageText, messageText, InputHints.IgnoringInput);
await context.SendActivityAsync(message, cancellationToken);
}
}
private async Task<DialogTurnResult> FinalStepAsync(WaterfallStepContext stepContext, CancellationToken cancellationToken)
{
// If the child dialog ("BookingDialog") was cancelled, the user failed to confirm or if the intent wasn't BookFlight
// the Result here will be null.
if (stepContext.Result is BookingDetails result)
{
// Now we have all the booking details call the booking service.
// If the call to the booking service was successful tell the user.
var timeProperty = new TimexProperty(result.TravelDate);
var travelDateMsg = timeProperty.ToNaturalLanguage(DateTime.Now);
var messageText = $"I have you booked to {result.Destination} from {result.Origin} on {travelDateMsg}";
var message = MessageFactory.Text(messageText, messageText, InputHints.IgnoringInput);
await stepContext.Context.SendActivityAsync(message, cancellationToken);
}
else if (stepContext.Result is CoreBot.WeatherDetails sonuc)
{
var timeProperty = new TimexProperty(sonuc.TravelDate);
var weatherDateMsg = timeProperty.ToNaturalLanguage(DateTime.Now);
var messageText = $"Thats your weather result for {weatherDateMsg} at {sonuc.Location}: 00 F";
var message = MessageFactory.Text(messageText, messageText, InputHints.IgnoringInput);
await stepContext.Context.SendActivityAsync(message, cancellationToken);
}
// Restart the main dialog with a different message the second time around
var promptMessage = "What else can I do for you?";
return await stepContext.ReplaceDialogAsync(InitialDialogId, promptMessage, cancellationToken);
}
}
}
And also my WeatherDialog:
using System.Threading;
using System.Threading.Tasks;
using Microsoft.Bot.Builder;
using Microsoft.Bot.Builder.Dialogs;
using Microsoft.Bot.Schema;
using Microsoft.BotBuilderSamples.Dialogs;
using Microsoft.Recognizers.Text.DataTypes.TimexExpression;
namespace CoreBot.Dialogs
{
public class WeatherDialog : CancelAndHelpDialog
{
private const string DestinationStepMsgText = "Type the location.";
public WeatherDialog()
: base(nameof(WeatherDialog))
{
AddDialog(new TextPrompt(nameof(TextPrompt)));
AddDialog(new ConfirmPrompt(nameof(ConfirmPrompt)));
AddDialog(new DateResolverDialog());
AddDialog(new WaterfallDialog(nameof(WaterfallDialog), new WaterfallStep[]
{
LocationStepAsync,
TravelDateStepAsync,
ConfirmStepAsync,
FinalStepAsync,
}));
// The initial child Dialog to run.
InitialDialogId = nameof(WaterfallDialog);
}
private async Task<DialogTurnResult> LocationStepAsync(WaterfallStepContext stepContext, CancellationToken cancellationToken)
{
var weatherDetails = (WeatherDetails)stepContext.Options;
if (weatherDetails.Location == null)
{
var promptMessage = MessageFactory.Text(DestinationStepMsgText, DestinationStepMsgText, InputHints.ExpectingInput);
return await stepContext.PromptAsync(nameof(TextPrompt), new PromptOptions { Prompt = promptMessage }, cancellationToken);
}
return await stepContext.NextAsync(weatherDetails.Location, cancellationToken);
}
private async Task<DialogTurnResult> TravelDateStepAsync(WaterfallStepContext stepContext, CancellationToken cancellationToken)
{
var weatherDetails = (WeatherDetails)stepContext.Options;
weatherDetails.Location = (string)stepContext.Result;
if (weatherDetails.TravelDate == null || IsAmbiguous(weatherDetails.TravelDate))
{
return await stepContext.BeginDialogAsync(nameof(DateResolverDialog), weatherDetails.TravelDate, cancellationToken);
}
return await stepContext.NextAsync(weatherDetails.TravelDate, cancellationToken);
}
private async Task<DialogTurnResult> ConfirmStepAsync(WaterfallStepContext stepContext, CancellationToken cancellationToken)
{
var weatherDetails = (WeatherDetails)stepContext.Options;
weatherDetails.TravelDate = (string)stepContext.Result;
var messageText = $"You want to know weather status at {weatherDetails.Location} for {weatherDetails.TravelDate}. Is this correct?";
var promptMessage = MessageFactory.Text(messageText, messageText, InputHints.ExpectingInput);
return await stepContext.PromptAsync(nameof(ConfirmPrompt), new PromptOptions { Prompt = promptMessage }, cancellationToken);
}
private async Task<DialogTurnResult> FinalStepAsync(WaterfallStepContext stepContext, CancellationToken cancellationToken)
{
if ((bool)stepContext.Result)
{
var weatherDetails = (WeatherDetails)stepContext.Options;
return await stepContext.EndDialogAsync(weatherDetails, cancellationToken);
}
return await stepContext.EndDialogAsync(null, cancellationToken);
}
private static bool IsAmbiguous(string timex)
{
var timexProperty = new TimexProperty(timex);
return !timexProperty.Types.Contains(Constants.TimexTypes.Definite);
}
}
}
After I build the code from Visual Studio then start conversation from Bot Framework Emulator.
And the following dialog goes like this:
How is the weather today?
+The bot encounted an error or bug. -> Failed
+To continue to run this bot, please fix the bot source code.
How is the weather today at Los Angeles?
-> Successful
How is the weather at Los Angeles?
+The bot encounted an error or bug. -> Failed
+To continue to run this bot, please fix the bot source code.
book me flight
-> Successful
How is the weather tomorrow at Tokyo?
-> Successful ............
Briefly, when running the bookingDialog with missing parameters (entities), bot asks the missing parameter. However, when I run the weatherDialog with missing parameters conversation overs with error.
When I look the LUIS trace in Bot Framework Emulator, intent and entity recognized successfully.
As you can see, bookingDialog works like a charm but my dialog doesn't work. Can you please help me? What am I missing?

How to determine whether PostgreSQL running in a Docker container is done initializing with Marten?

I am basically writing a little bit of a program to benchmark the insertion performance of PostgreSQL over a given table growth, and I would like to make sure that when I am using Marten to insert data, the database is fully ready to accept insertions.
I am using Docker.DotNet to spawn a new container running the latest PostgreSQL image but even if the container is in a running state it is sometimes not the case for the Postgre running inside that container.
Of course, I could have added a Thread. Sleep but this is a bit random so I would rather like something deterministic to figure out when the database is ready to accept insertions?
public static class Program
{
public static async Task Main(params string[] args)
{
const string imageName = "postgres:latest";
const string containerName = "MyProgreSQL";
var client = new DockerClientConfiguration(Docker.DefaultLocalApiUri).CreateClient();
var containers = await client.Containers.SearchByNameAsync(containerName);
var container = containers.SingleOrDefault();
if (container != null)
{
await client.Containers.StopAndRemoveContainerAsync(container);
}
var createdContainer = await client.Containers.RunContainerAsync(new CreateContainerParameters
{
Image = imageName,
HostConfig = new HostConfig
{
PortBindings = new Dictionary<string, IList<PortBinding>>
{
{"5432/tcp", new List<PortBinding>
{
new PortBinding
{
HostPort = "5432"
}
}}
},
PublishAllPorts = true
},
Env = new List<string>
{
"POSTGRES_PASSWORD=root",
"POSTGRES_USER=root"
},
Name = containerName
});
var containerState = string.Empty;
while (containerState != "running")
{
containers = await client.Containers.SearchByNameAsync(containerName);
container = containers.Single();
containerState = container.State;
}
var store = DocumentStore.For("host=localhost;database=postgres;password=root;username=root");
var stopwatch = new Stopwatch();
using (var session = store.LightweightSession())
{
var orders = OrderHelpers.FakeOrders(10000);
session.StoreObjects(orders);
stopwatch.Start();
await session.SaveChangesAsync();
var elapsed = stopwatch.Elapsed;
// Whatever else needs to be done...
}
}
}
FYI, the if I am running the program above without pausing before the line await session.SaveChangesAsync(); I am running than into the following exception:
Unhandled Exception: Npgsql.NpgsqlException: Exception while reading from stream ---> System.IO.EndOfStreamException: Attempted to read past the end of the streams.
at Npgsql.NpgsqlReadBuffer.<>c__DisplayClass31_0.<<Ensure>g__EnsureLong|0>d.MoveNext() in C:\projects\npgsql\src\Npgsql\NpgsqlReadBuffer.cs:line 154
--- End of inner exception stack trace ---
at Npgsql.NpgsqlReadBuffer.<>c__DisplayClass31_0.<<Ensure>g__EnsureLong|0>d.MoveNext() in C:\projects\npgsql\src\Npgsql\NpgsqlReadBuffer.cs:line 175
--- End of stack trace from previous location where exception was thrown ---
at Npgsql.NpgsqlConnector.<>c__DisplayClass161_0.<<ReadMessage>g__ReadMessageLong|0>d.MoveNext() in C:\projects\npgsql\src\Npgsql\NpgsqlConnector.cs:l
ine 955
--- End of stack trace from previous location where exception was thrown ---
at Npgsql.NpgsqlConnector.Authenticate(String username, NpgsqlTimeout timeout, Boolean async) in C:\projects\npgsql\src\Npgsql\NpgsqlConnector.Auth.cs
:line 26
at Npgsql.NpgsqlConnector.Open(NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken) in C:\projects\npgsql\src\Npgsql\NpgsqlConne
ctor.cs:line 425
at Npgsql.ConnectorPool.AllocateLong(NpgsqlConnection conn, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken) in C:\projects\
npgsql\src\Npgsql\ConnectorPool.cs:line 246
at Npgsql.NpgsqlConnection.<>c__DisplayClass32_0.<<Open>g__OpenLong|0>d.MoveNext() in C:\projects\npgsql\src\Npgsql\NpgsqlConnection.cs:line 300
--- End of stack trace from previous location where exception was thrown ---
at Npgsql.NpgsqlConnection.Open() in C:\projects\npgsql\src\Npgsql\NpgsqlConnection.cs:line 153
at Marten.Storage.Tenant.generateOrUpdateFeature(Type featureType, IFeatureSchema feature)
at Marten.Storage.Tenant.ensureStorageExists(IList`1 types, Type featureType)
at Marten.Storage.Tenant.ensureStorageExists(IList`1 types, Type featureType)
at Marten.Storage.Tenant.StorageFor(Type documentType)
at Marten.DocumentSession.Store[T](T[] entities)
at Baseline.GenericEnumerableExtensions.Each[T](IEnumerable`1 values, Action`1 eachAction)
at Marten.DocumentSession.StoreObjects(IEnumerable`1 documents)
at Benchmark.Program.Main(String[] args) in C:\Users\eperret\Desktop\Benchmark\Benchmark\Program.cs:line 117
at Benchmark.Program.<Main>(String[] args)
[EDIT]
I accepted an answer but due to a bug about health parameters equivalence in the Docker.DotNet I could not leverage the solution given in the answer (I still think that a proper translation of that docker command in the .NET client, if actually possible) would be the best solution. In the meanwhile this is how I solved my problem, I basically poll the command that was expected to run in the health check aside until the result is ok:
Program.cs, the actual meat of the code:
using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.IO;
using System.Linq;
using System.Threading.Tasks;
using Benchmark.DockerClient;
using Benchmark.Domain;
using Benchmark.Utils;
using Docker.DotNet;
using Docker.DotNet.Models;
using Marten;
using Microsoft.Extensions.Configuration;
namespace Benchmark
{
public static class Program
{
public static async Task Main(params string[] args)
{
var configurationBuilder = new ConfigurationBuilder()
.SetBasePath(Directory.GetCurrentDirectory())
.AddJsonFile("appsettings.json");
var configuration = new Configuration();
configurationBuilder.Build().Bind(configuration);
var client = new DockerClientConfiguration(DockerClient.Docker.DefaultLocalApiUri).CreateClient();
var containers = await client.Containers.SearchByNameAsync(configuration.ContainerName);
var container = containers.SingleOrDefault();
if (container != null)
{
await client.Containers.StopAndRemoveContainerAsync(container.ID);
}
var createdContainer = await client.Containers.RunContainerAsync(new CreateContainerParameters
{
Image = configuration.ImageName,
HostConfig = new HostConfig
{
PortBindings = new Dictionary<string, IList<PortBinding>>
{
{$#"{configuration.ContainerPort}/{configuration.ContainerPortProtocol}", new List<PortBinding>
{
new PortBinding
{
HostPort = configuration.HostPort
}
}}
},
PublishAllPorts = true
},
Env = new List<string>
{
$"POSTGRES_USER={configuration.Username}",
$"POSTGRES_PASSWORD={configuration.Password}"
},
Name = configuration.ContainerName
});
var isContainerReady = false;
while (!isContainerReady)
{
var result = await client.Containers.RunCommandInContainerAsync(createdContainer.ID, "pg_isready -U postgres");
if (result.stdout.TrimEnd('\n') == $"/var/run/postgresql:{configuration.ContainerPort} - accepting connections")
{
isContainerReady = true;
}
}
var store = DocumentStore.For($"host=localhost;" +
$"database={configuration.DatabaseName};" +
$"username={configuration.Username};" +
$"password={configuration.Password}");
// Whatever else needs to be done...
}
}
Extensions being defined in ContainerOperationsExtensions.cs:
using System;
using System.Collections.Generic;
using System.Threading;
using System.Threading.Tasks;
using Docker.DotNet;
using Docker.DotNet.Models;
namespace Benchmark.DockerClient
{
public static class ContainerOperationsExtensions
{
public static async Task<IList<ContainerListResponse>> SearchByNameAsync(this IContainerOperations source, string name, bool all = true)
{
return await source.ListContainersAsync(new ContainersListParameters
{
All = all,
Filters = new Dictionary<string, IDictionary<string, bool>>
{
{"name", new Dictionary<string, bool>
{
{name, true}
}
}
}
});
}
public static async Task StopAndRemoveContainerAsync(this IContainerOperations source, string containerId)
{
await source.StopContainerAsync(containerId, new ContainerStopParameters());
await source.RemoveContainerAsync(containerId, new ContainerRemoveParameters());
}
public static async Task<CreateContainerResponse> RunContainerAsync(this IContainerOperations source, CreateContainerParameters parameters)
{
var createdContainer = await source.CreateContainerAsync(parameters);
await source.StartContainerAsync(createdContainer.ID, new ContainerStartParameters());
return createdContainer;
}
public static async Task<(string stdout, string stderr)> RunCommandInContainerAsync(this IContainerOperations source, string containerId, string command)
{
var commandTokens = command.Split(" ", StringSplitOptions.RemoveEmptyEntries);
var createdExec = await source.ExecCreateContainerAsync(containerId, new ContainerExecCreateParameters
{
AttachStderr = true,
AttachStdout = true,
Cmd = commandTokens
});
var multiplexedStream = await source.StartAndAttachContainerExecAsync(createdExec.ID, false);
return await multiplexedStream.ReadOutputToEndAsync(CancellationToken.None);
}
}
}
Docker.cs to get the local docker api uri:
using System;
using System.Runtime.InteropServices;
namespace Benchmark.DockerClient
{
public static class Docker
{
static Docker()
{
DefaultLocalApiUri = RuntimeInformation.IsOSPlatform(OSPlatform.Windows)
? new Uri("npipe://./pipe/docker_engine")
: new Uri("unix:/var/run/docker.sock");
}
public static Uri DefaultLocalApiUri { get; }
}
}

I suggest you to use a custom healtcheck to check if the database is ready to accept connections.
I am not familiar with the .NET client of Docker, but the following docker run command shows what you should try :
docker run --name postgres \
--health-cmd='pg_isready -U postgres' \
--health-interval='10s' \
--health-timeout='5s' \
--health-start-period='10s' \
postgres:latest
Time parameters should be updated accordingly to your needs.
With this healtcheck configured, your application must wait for the container to be in state "healthy" before trying to connect to the database. The status "healthy", in that particular case, means that the command pg_isready -U postgres have succeeded (so the database is ready to accept connections).
The status of your container can be retrieved with :
docker inspect --format "{{json .State.Health.Status }}" postgres

How do you delete the dead letters in an Azure Service Bus queue?

How do you delete the dead letters in an Azure Service Bus queue?
I can process the messages in the queue ...
var queueClient = QueueClient.CreateFromConnectionString(sbConnectionString, queueName);
while (queueClient.Peek() != null)
{
var brokeredMessage = queueClient.Receive();
brokeredMessage.Complete();
}
but can't see anyway to handle the dead letter messages

The trick is to get the deadletter path for the queue which you can get by using QueueClient.FormatDeadLetterPath(queueName).
Please try the following:
var queueClient = QueueClient.CreateFromConnectionString(sbConnectionString, QueueClient.FormatDeadLetterPath(queueName));
while (queueClient.Peek() != null)
{
var brokeredMessage = queueClient.Receive();
brokeredMessage.Complete();
}

using System;
using Microsoft.VisualStudio.TestTools.UnitTesting;
using Microsoft.Azure.ServiceBus;
using Microsoft.Azure.ServiceBus.Core;
using System.Text;
using System.Collections.Generic;
using System.Threading.Tasks;
namespace ClearDeadLetterQ
{
[TestClass]
public class UnitTest1
{
const string ServiceBusConnectionString = "Endpoint=sb://my-domain.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=youraccesskeyhereyouraccesskeyhere";
[TestMethod]
public async Task TestMethod1()
{
await this.ClearDeadLetters("my.topic.name", "MySubscriptionName/$DeadLetterQueue");
}
public async Task ClearDeadLetters(string topicName, string subscriptionName)
{
var messageReceiver = new MessageReceiver(ServiceBusConnectionString, EntityNameHelper.FormatSubscriptionPath(topicName, subscriptionName), ReceiveMode.PeekLock);
var message = await messageReceiver.ReceiveAsync();
while ((message = await messageReceiver.ReceiveAsync()) != null)
{
await messageReceiver.CompleteAsync(message.SystemProperties.LockToken);
}
await messageReceiver.CloseAsync();
}
}
}

There are some great samples available in our GitHub sample repo (https://github.com/Azure-Samples/azure-servicebus-messaging-samples). The DeadletterQueue project should show you an example of how to do this in your code:
var dead-letterReceiver = await receiverFactory.CreateMessageReceiverAsync(
QueueClient.FormatDeadLetterPath(queueName), ReceiveMode.PeekLock);
while (true)
{
var msg = await dead-letterReceiver.ReceiveAsync(TimeSpan.Zero);
if (msg != null)
{
foreach (var prop in msg.Properties)
{
Console.WriteLine("{0}={1}", prop.Key, prop.Value);
}
await msg.CompleteAsync();
}
else
{
break;
}
}
}
Hope that helps!

Open Azure into your target Bus Queue Service.
(Set this value in place of <BUS-QUEUE-NAME> on the code)
Go into the queue you want to delete.
(Set this value in place of <QUEUE-NAME> on the code)
Create a Shared Access Policy and name it: RemoveDeadLetterQueue with checkbox Manage been selected.
Copy this Primary Key onto <QUEUE-POLICY-PRIMARYKEY> in this code.
And the code is ready to run.
using Microsoft.Azure.ServiceBus.Core;
using Microsoft.Azure.ServiceBus;
string serviceBusQueue = "<BUS-QUEUE-NAME>";
string serviceBusQueueName = "<QUEUE-NAME>";
string policyName = "RemoveDeadLetterQueue";
string policyPrimaryKey = "<QUEUE-POLICY-PRIMARYKEY>";
var receiver = new MessageReceiver(
connectionString: $"Endpoint=sb://{serviceBusQueue}.servicebus.windows.net/;SharedAccessKeyName={policyName};SharedAccessKey={policyPrimaryKey}",
entityPath: $"{serviceBusQueueName}/$DeadLetterQueue",
receiveMode: ReceiveMode.ReceiveAndDelete
);
var messages = await receiver.ReceiveAsync(maxMessageCount: 1000);
while(messages != null)
{
foreach (var message in messages)
{
Console.WriteLine($"[Delete Message] ID: {message.MessageId}");
}
messages = await receiver.ReceiveAsync(maxMessageCount: 1000);
}
await receiver.CloseAsync();

Web API - Get progress when uploading to Azure storage

The task i want to accomplish is to create a Web API service in order to upload a file to Azure storage. At the same time, i would like to have a progress indicator that reflects the actual upload progress. After some research and studying i found out two important things:
First is that i have to split the file manually into chunks, and upload them asynchronously using the PutBlockAsync method from Microsoft.WindowsAzure.Storage.dll.
Second, is that i have to receive the file in my Web API service in Streamed mode and not in Buffered mode.
So until now i have the following implementation:
UploadController.cs
using System.Configuration;
using System.Net;
using System.Net.Http;
using System.Threading.Tasks;
using System.Web.Http;
using Microsoft.WindowsAzure.Storage;
using Microsoft.WindowsAzure.Storage.Blob;
using WebApiFileUploadToAzureStorage.Infrastructure;
using WebApiFileUploadToAzureStorage.Models;
namespace WebApiFileUploadToAzureStorage.Controllers
{
public class UploadController : ApiController
{
[HttpPost]
public async Task<HttpResponseMessage> UploadFile()
{
if (!Request.Content.IsMimeMultipartContent("form-data"))
{
return Request.CreateResponse(HttpStatusCode.UnsupportedMediaType,
new UploadStatus(null, false, "No form data found on request.", string.Empty, string.Empty));
}
var streamProvider = new MultipartAzureBlobStorageProvider(GetAzureStorageContainer());
var result = await Request.Content.ReadAsMultipartAsync(streamProvider);
if (result.FileData.Count < 1)
{
return Request.CreateResponse(HttpStatusCode.BadRequest,
new UploadStatus(null, false, "No files were uploaded.", string.Empty, string.Empty));
}
return Request.CreateResponse(HttpStatusCode.OK);
}
private static CloudBlobContainer GetAzureStorageContainer()
{
var storageConnectionString = ConfigurationManager.AppSettings["AzureBlobStorageConnectionString"];
var storageAccount = CloudStorageAccount.Parse(storageConnectionString);
var blobClient = storageAccount.CreateCloudBlobClient();
blobClient.DefaultRequestOptions.SingleBlobUploadThresholdInBytes = 1024 * 1024;
var container = blobClient.GetContainerReference("photos");
if (container.Exists())
{
return container;
}
container.Create();
container.SetPermissions(new BlobContainerPermissions
{
PublicAccess = BlobContainerPublicAccessType.Container
});
return container;
}
}
}
MultipartAzureBlobStorageProvider.cs
using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.IO;
using System.Linq;
using System.Net.Http;
using System.Text;
using System.Threading;
using System.Threading.Tasks;
using Microsoft.WindowsAzure.Storage.Blob;
namespace WebApiFileUploadToAzureStorage.Infrastructure
{
public class MultipartAzureBlobStorageProvider : MultipartFormDataStreamProvider
{
private readonly CloudBlobContainer _blobContainer;
public MultipartAzureBlobStorageProvider(CloudBlobContainer blobContainer) : base(Path.GetTempPath())
{
_blobContainer = blobContainer;
}
public override Task ExecutePostProcessingAsync()
{
const int blockSize = 256 * 1024;
var fileData = FileData.First();
var fileName = Path.GetFileName(fileData.Headers.ContentDisposition.FileName.Trim('"'));
var blob = _blobContainer.GetBlockBlobReference(fileName);
var bytesToUpload = (new FileInfo(fileData.LocalFileName)).Length;
var fileSize = bytesToUpload;
blob.Properties.ContentType = fileData.Headers.ContentType.MediaType;
blob.StreamWriteSizeInBytes = blockSize;
if (bytesToUpload < blockSize)
{
var cancellationToken = new CancellationToken();
using (var fileStream = new FileStream(fileData.LocalFileName, FileMode.Open, FileAccess.ReadWrite))
{
var upload = blob.UploadFromStreamAsync(fileStream, cancellationToken);
Debug.WriteLine($"Status {upload.Status}.");
upload.ContinueWith(task =>
{
Debug.WriteLine($"Status {task.Status}.");
Debug.WriteLine("Upload is over successfully.");
}, TaskContinuationOptions.OnlyOnRanToCompletion);
upload.ContinueWith(task =>
{
Debug.WriteLine($"Status {task.Status}.");
if (task.Exception != null)
{
Debug.WriteLine("Task could not be completed." + task.Exception.InnerException);
}
}, TaskContinuationOptions.OnlyOnFaulted);
upload.Wait(cancellationToken);
}
}
else
{
var blockIds = new List<string>();
var index = 1;
long startPosition = 0;
long bytesUploaded = 0;
do
{
var bytesToRead = Math.Min(blockSize, bytesToUpload);
var blobContents = new byte[bytesToRead];
using (var fileStream = new FileStream(fileData.LocalFileName, FileMode.Open))
{
fileStream.Position = startPosition;
fileStream.Read(blobContents, 0, (int)bytesToRead);
}
var manualResetEvent = new ManualResetEvent(false);
var blockId = Convert.ToBase64String(Encoding.UTF8.GetBytes(index.ToString("d6")));
Debug.WriteLine($"Now uploading block # {index.ToString("d6")}");
blockIds.Add(blockId);
var upload = blob.PutBlockAsync(blockId, new MemoryStream(blobContents), null);
upload.ContinueWith(task =>
{
bytesUploaded += bytesToRead;
bytesToUpload -= bytesToRead;
startPosition += bytesToRead;
index++;
var percentComplete = (double)bytesUploaded / fileSize;
Debug.WriteLine($"Percent complete: {percentComplete.ToString("P")}");
manualResetEvent.Set();
});
manualResetEvent.WaitOne();
} while (bytesToUpload > 0);
Debug.WriteLine("Now committing block list.");
var putBlockList = blob.PutBlockListAsync(blockIds);
putBlockList.ContinueWith(task =>
{
Debug.WriteLine("Blob uploaded completely.");
});
putBlockList.Wait();
}
File.Delete(fileData.LocalFileName);
return base.ExecutePostProcessingAsync();
}
}
}
I also enabled Streamed mode as this blog post suggests. This approach works great, in terms that the file is uploaded successfully to Azure storage. Then, when i make a call to this service making use of XMLHttpRequest (and subscribing to the progress event) i see the indicator moving to 100% very quickly. If a 5MB file needs around 1 minute to upload, my indicator moves to the end in just 1 second. So probably the problem resides in the way that the server informs the client about the upload progress. Any thoughts about this? Thank you.
================================ Update 1 ===================================
That is the JavaScript code i use to call the service
function uploadFile(file, index, uploadCompleted) {
var authData = localStorageService.get("authorizationData");
var xhr = new XMLHttpRequest();
xhr.upload.addEventListener("progress", function (event) {
fileUploadPercent = Math.floor((event.loaded / event.total) * 100);
console.log(fileUploadPercent + " %");
});
xhr.onreadystatechange = function (event) {
if (event.target.readyState === event.target.DONE) {
if (event.target.status !== 200) {
} else {
var parsedResponse = JSON.parse(event.target.response);
uploadCompleted(parsedResponse);
}
}
};
xhr.open("post", uploadFileServiceUrl, true);
xhr.setRequestHeader("Authorization", "Bearer " + authData.token);
var data = new FormData();
data.append("file-" + index, file);
xhr.send(data);
}

your progress indicator might be moving rapidly fast, might be because of
public async Task<HttpResponseMessage> UploadFile()
i have encountered this before, when creating an api of async type, im not even sure if it can be awaited, it will just of course just finish your api call on the background, reason your progress indicator instantly finish, because of the async method (fire and forget). the api will immediately give you a response, but will actually finish on the server background (if not awaited).
please kindly try making it just
public HttpResponseMessage UploadFile()
and also try these ones
var result = Request.Content.ReadAsMultipartAsync(streamProvider).Result;
var upload = blob.UploadFromStreamAsync(fileStream, cancellationToken).Result;
OR
var upload = await blob.UploadFromStreamAsync(fileStream, cancellationToken);
hope it helps.

Other way to acomplish what you want (I don't understand how the XMLHttpRequest's progress event works) is using the ProgressMessageHandler to get the upload progress in the request. Then, in order to notify the client, you could use some cache to store the progress, and from the client request the current state in other endpoint, or use SignalR to send the progress from the server to the client
Something like:
//WebApiConfigRegister
var progress = new ProgressMessageHandler();
progress.HttpSendProgress += HttpSendProgress;
config.MessageHandlers.Add(progress);
//End WebApiConfig Register
private static void HttpSendProgress(object sender, HttpProgressEventArgs e)
{
var request = sender as HttpRequestMessage;
//todo: check if request is not null
//Get an Id from the client or something like this to identify the request
var id = request.RequestUri.Query[0];
var perc = e.ProgressPercentage;
var b = e.TotalBytes;
var bt = e.BytesTransferred;
Cache.InsertOrUpdate(id, perc);
}
You can check more documentation on this MSDN blog post (Scroll down to "Progress Notifications" section)
Also, you could calculate the progress based on the data chunks, store the progress in a cache, and notify in the same way as above. Something like this solution