We Keep Coding

Unable to download directory from Azure container using DataMovement library

I am using the DataMovement Nuget to download a directory from Azure using a presigned URL since I don't want account details on the client. Thus I am generating a Presigned URL from the server and sending it to the client. Using C#
Downloading a single file works, but not a directory. The presinged URL is correct.
Here is the code:
private static async Task Main(string[] args)
{
string containerName = "main";
CloudBlobContainer container = blobClient.GetContainerReference(containerName);
var desitination = Path.Combine(Environment.GetFolderPath(Environment.SpecialFolder.MyDocuments), "Temp");
var directoryName = "videos";
var url = GetClientDownloadURL(containerName, directoryName, blobClient);
Stopwatch stopWatch = Stopwatch.StartNew();
var checkpoint = await DownloadFromAzurePauseResume(url, desitination, containerName, "videos");
if (canceled)
{
var resuming = false;
Console.WriteLine("Press R to resume");
while (!resuming)
{
var keyInfo = Console.ReadKey();
if (keyInfo.Key == ConsoleKey.R)
{
resuming = true;
}
}
if (resuming)
await ResumeDownload(url, desitination, checkpoint);
}
stopWatch.Stop();
Console.WriteLine("\nTransfer operation completed in " + stopWatch.Elapsed.TotalSeconds + " seconds.");
}
public static DirectoryTransferContext GetDirectoryTransferContext(TransferCheckpoint checkpoint)
{
DirectoryTransferContext context = new DirectoryTransferContext(checkpoint);
context.ProgressHandler = new Progress<TransferStatus>((progress) =>
{
Console.Write("\rBytes transferred: {0}", progress.BytesTransferred);
});
return context;
}
public static async Task<TransferCheckpoint> DownloadFromAzurePauseResume(string directoryUri, string destination, string containerName, string folderName)
{
Console.ForegroundColor = ConsoleColor.Green;
Console.WriteLine($"dirUrl: {directoryUri}");
Console.ForegroundColor = ConsoleColor.White;
CloudBlobClient cbc = new CloudBlobClient(new Uri(directoryUri));
CloudBlobContainer container = cbc.GetContainerReference(containerName);
CloudBlobDirectory cloudBlobDir = container.GetDirectoryReference(folderName);
// track transfer progress
TransferCheckpoint? checkPoint = null;
var context = GetDirectoryTransferContext(checkPoint);
CancellationTokenSource cancellationSource = new CancellationTokenSource();
Console.WriteLine("\nTransfer started...\nPress 'c' to temporarily cancel your transfer...\n");
var downloadOption = new DownloadDirectoryOptions() { Recursive = true };
Task task;
ConsoleKeyInfo keyinfo;
try
{
//task = TransferManager.DownloadAsync(blob, destination, null, context, cancellationSource.Token);
task = TransferManager.DownloadDirectoryAsync(cloudBlobDir, destination, downloadOption, context, cancellationSource.Token);
while (!task.IsCompleted)
{
if (Console.KeyAvailable)
{
keyinfo = Console.ReadKey(true);
if (keyinfo.Key == ConsoleKey.C)
{
cancellationSource.Cancel();
canceled = true;
}
}
}
await task;
}
catch (Exception e)
{
Console.WriteLine(e.Message);
}
return context.LastCheckpoint;
}
This is the method from the server and is working correctly:
private static string GetClientDownloadURL(string containerName, string directoryName, CloudBlobClient blobClient)
{
var primaryStorageAccountURL = blobClient.BaseUri.AbsoluteUri;
// TODO : Add presigned URL
var container = blobClient.GetContainerReference(containerName);
var fullURL = $"{container.Uri.AbsoluteUri}/{directoryName}";
var accessStartTime = DateTime.Now;
fullURL += container.GetSharedAccessSignature(new SharedAccessBlobPolicy
{
Permissions = SharedAccessBlobPermissions.Read,
SharedAccessExpiryTime = accessStartTime.Add(TimeSpan.FromHours(6)),
SharedAccessStartTime = accessStartTime,
}, "PrivateAccess");
return fullURL;
}
I am testing this in a console application, but eventually want to move it to an app im building.

windows service in c# microsoft graph api getAsync() call for users goes in hung state

I am reading different mailboxes in windows service using MS graph api using threads. Service runs well for few days, but after it goes in hung state.
By observing logs found that stuck point is while calling GetAsync() method for users, after that it is not updating log file also but service is showing in running state.
After restarting the service it runs normally for few days.
public static async Task MainAsync()
{
Task t1 = ParallelThreadOne.MainAsync();
Task t2 = ParallelThreadSecond.MainAsync();
Task t3 = ParallelThreadThird.MainAsync();
await Task.WhenAll(t1, t2, t3);
}
public static async Task MainAsync()
{
try
{
EmailMaster objEmailMaster = new EmailMaster();
List<MailBoxConfiguration> lstMailBoxConfiguration = objEmailMaster.GetMailBoxInformation(1,logger);
if (lstMailBoxConfiguration != null)
{
if (lstMailBoxConfiguration.Count != 0)
{
GraphServiceClient client = GetAuthenticatedClient();
if (client != null)
{
for (int j = 0; j < lstMailBoxConfiguration.Count; j++)
{
var users = await graphClient
.Users
.Request()
.Filter("startswith(Mail,'" + lstMailBoxConfiguration[j].EmailId + "')")
.GetAsync();
if (users.Count > 0)
{
var msgs = await graphClient
.Users[users[0].Id]
.MailFolders["Inbox"].Messages
.Request().Top(500)
.GetAsync();
if (msgs.Count > 0)
{
foreach (var item in msgs)
{
//business logic goes here
}
}
else
{
logger.Info("msg.Count is zero");
}
}
else
{
logger.Info("users.Count is zero");
}
}
}
else
{
logger.Info("client is null");
}
}
else
{
logger.Info("lstMailBoxConfiguration.Count is zero from the database");
}
}
else
{
logger.Info("lstMailBoxConfiguration is null from the database");
}
logger.Info("MainAsync(1) : End of MainAsync(1)");
}
catch (Exception ex)
{
logger.Error("MainAsync(1) : Exception : " + ex.Message);
}
}
public static GraphServiceClient GetAuthenticatedClient()
{
string clientId = ConfigurationManager.AppSettings["AzureClientId"];
string password = ConfigurationManager.AppSettings["password"];
string tenantId = ConfigurationManager.AppSettings["tenantId"];
string getTokenUrl = $"https://login.microsoftonline.com/{tenantId}/oauth2/v2.0/token";
const string grantType = "client_credentials";
const string myScopes = "https://graph.microsoft.com/.default";
string postBody = $"client_id={clientId}&scope={myScopes}&client_secret={password}&grant_type={grantType}";
try
{
if (graphClient == null)
{
graphClient = new GraphServiceClient(
"https://graph.microsoft.com/v1.0",
new DelegateAuthenticationProvider(
async (requestMessage) =>
{
HttpRequestMessage httpRequestMessage = new HttpRequestMessage(HttpMethod.Post, getTokenUrl);
httpRequestMessage.Content = new StringContent(postBody, Encoding.UTF8, "application/x-www-form-urlencoded");
HttpResponseMessage httpResponseMessage = await client.SendAsync(httpRequestMessage);
string responseBody = await httpResponseMessage.Content.ReadAsStringAsync();
userToken = JObject.Parse(responseBody).GetValue("access_token").ToString();
requestMessage.Headers.Authorization = new AuthenticationHeaderValue("bearer", userToken);
}));
}
return graphClient;
}
catch (Exception ex)
{
logger.Error("Could not create a graph client: " + ex.Message);
}
finally
{
logger.Info("GetAuthenticatedClient() :inside finally!");
}
return graphClient;
}

From looking through the source code of the GraphServiceClient, it is using HTTPClient as its underlying communication provider. The HTTPClient has an issue because windows keeps TCP/IP connections open for a certain amount of time after the HTTPClient is disposed. If you call new and then dispose on the HTTPClient class fast enough for long enough it can lead to socket starvation. (note, using(var client = new HTTPClient()) calls dispose under the covers when the instance goes out of scope)
Take a look at this blog post on the issue.
https://www.aspnetmonsters.com/2016/08/2016-08-27-httpclientwrong/
You should be able to use a single instance of the GraphServiceClient as long as you are talking to the same GraphQL endpoint and fix your issues with the service hanging. If you add logging you will probably notice that the service hangs after a flurry of activity causing lots of new instances of the GraphServiceClient to be created then disposed in a short time frame and your open network connections on the server exploding causing an error that crashes one of your threads.

Return success/error response after making some graph calls

I have the following code to add users to AAD group:
public async Task AddUsersToGroup(IEnumerable<BatchRequestContent> batches)
{
try
{
foreach (var batchRequestContent in batches)
{
var response = await _graphServiceClient
.Batch
.Request()
.WithMaxRetry(10)
.PostAsync(batchRequestContent);
var responses = await response.GetResponsesAsync();
foreach (string key in responses.Keys)
{
var httpResponse = await response.GetResponseByIdAsync(key);
httpResponse.EnsureSuccessStatusCode();
}
}
}
catch (Exception ex)
{
await _log.LogMessageAsync(new LogMessage
{
Message = ex.GetBaseException().ToString(),
RunId = RunId
});
throw;
}
}
I need to update this method to return 'Ok' if all the requests returned a success response, else return 'Error' if any of the requests returned an error response. How would I do that? Please help!
private IEnumerable<BatchRequestContent> GetBatchRequest(IEnumerable<AzureADUser> users, AzureADGroup targetGroup)
{
var batches = new List<BatchRequestContent>();
int maxNoBatchItems = 20;
var batchRequestContent = new BatchRequestContent();
int requestId = 1;
foreach (var user in users)
{
JObject body = new JObject
{
["members#odata.bind"] = $"https://graph.microsoft.com/v1.0/users/{user.ObjectId}"
};
var httpRequestMessage = new HttpRequestMessage(HttpMethod.Patch, $"https://graph.microsoft.com/v1.0/groups/{targetGroup.ObjectId}");
httpRequestMessage.Content = new StringContent(body.ToString(), Encoding.UTF8, "application/json");
batchRequestContent.AddBatchRequestStep(new BatchRequestStep(requestId.ToString(), httpRequestMessage));
if (batchRequestContent.BatchRequestSteps.Count() % maxNoBatchItems == 0)
{
batches.Add(batchRequestContent);
batchRequestContent = new BatchRequestContent();
}
requestId++;
}
if (batchRequestContent.BatchRequestSteps.Count < maxNoBatchItems)
{
batches.Add(batchRequestContent);
}
return batches;
}

Calling Rest API in a loop

Let's assume I get game orders by calling a local web site's API. (A local web site selling game codes, when a user makes a purchase on the web site, I am getting orders from this web site's API.) I will do this with a .net core console app which will check the web site's API every 10 minutes. Now assume I get 5 game orders from the web site's API. Inside of this .net core console app, I will call my API in order to complete the purchase of these 5 game orders as follows. (I am calling 3rd party Rest APIs in order to get game codes.)
Can you please check my code and give me some feedback\suggestions? Is there any place to cause a problem?
static async Task Main(string[] args)
{
// Assume I am getting game orders from the web site here
//
.....
const int numberOfProducts = webSiteGameOrders;
try
{
for (var i = 0; i < numberOfProducts; i++)
{
var product = webSiteProduct;
//DEV
var request =
(HttpWebRequest)WebRequest.Create("http://test//api/purchase");
var svcCredentials =
Convert.ToBase64String(
Encoding.ASCII.GetBytes("username" + ":" + "password"));
request.Headers.Add("Authorization", "Basic " + svcCredentials);
request.Method = "POST";
request.KeepAlive = false;
request.ContentType = "application/x-www-form-urlencoded";
var content = new FormUrlEncodedContent(new[]
{
new KeyValuePair<string, string>("productCode", webSiteProduct),
new KeyValuePair<string, string>("quantity", "1"),
new KeyValuePair<string, string>("shopNo", "1"),
new KeyValuePair<string, string>("safeNo", "1"),
new KeyValuePair<string, string>("cashierNo", "1")
});
var formUrlEncodedContent = content;
var urlEncodedString = await formUrlEncodedContent.ReadAsStringAsync();
using (var streamWriter = new StreamWriter(await request.GetRequestStreamAsync()))
{
streamWriter.Write(urlEncodedString);
}
var httpResponse = (HttpWebResponse)(await request.GetResponseAsync());
var response = new HttpResponseMessage
{
StatusCode = httpResponse.StatusCode,
Content = new StreamContent(httpResponse.GetResponseStream()),
};
//Read response
var htmlResponse = await response.Content.ReadAsStringAsync();
var deserializedResult = JObject.Parse(htmlResponse);
Console.WriteLine((string)deserializedResult["coupons"]?[0]?["Serial"] + ":" + (string)deserializedResult["coupons"]?[0]?["Pin"]);
}
}
catch (Exception e)
{
Console.WriteLine(e.Message);
throw;
}
}
UPDATE:
Here is how I end up:
public class BusinessService : IHostedService
{
private IHttpClientFactory _httpClientFactory;
private readonly IHostApplicationLifetime _applicationLifetime;
private IConfiguration _configuration;
private readonly ILogger _logger;
public BusinessService(IHttpClientFactory httpClientFactory, IHostApplicationLifetime applicationLifetime, IConfiguration configuration, ILogger<BusinessService> logger)
{
_httpClientFactory = httpClientFactory;
_applicationLifetime = applicationLifetime;
_configuration = configuration;
_logger = logger;
}
public async Task StartAsync(CancellationToken cancellationToken)
{
_logger.LogCritical("Critical!");
_logger.LogInformation("Application starts: ");
await MakeRequestsToRemoteService();
//Stop Application
_applicationLifetime.StopApplication();
_logger.LogInformation("Application stops: ");
}
public async Task MakeRequestsToRemoteService()
{
try
{
HttpClient httpClient = _httpClientFactory.CreateClient("OrdersClient");
var authenticationBytes = Encoding.ASCII.GetBytes("zWedH5OUUp");
httpClient.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Basic",
Convert.ToBase64String(authenticationBytes));
httpClient.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
var response = await httpClient.GetAsync("https://api.test.com/orders?startDate=1612126800000");
if (response.IsSuccessStatusCode)
{
Root orders = await response.Content.ReadAsAsync<Root>();
foreach (var content in orders.Content)
{
Console.Write(content.CustomerId + " " + content.CustomerFirstName + " " +
content.CustomerLastName + " " + content.CustomerEmail + " " + content.TotalPrice +
" " + content.InvoiceAddress.Phone + " " + content.Lines[0].ProductCode + " " +
content.Lines[0].ProductName);
_logger.LogInformation("Order {ID} {CustomerId}, {Name} {LastName}, {Phone}, {ProductCode}, {ProductName}, {Price}, {OrderDate} ", content.Id, content.CustomerId, content.CustomerFirstName, content.CustomerLastName, content.InvoiceAddress.Phone, content.Lines[0].ProductCode, content.Lines[0].ProductName, content.TotalPrice, DateTimeOffset.FromUnixTimeMilliseconds(content.OrderDate).DateTime);
InsertData(content);
}
}
}
catch (Exception e)
{
_logger.LogError("Error: {Error} ", e.Message);
throw;
}
}
public void InsertData(Content content)
{
var connString = _configuration["ConnectionStrings:Development"];
using (var sqlConnection = new SqlConnection(connString))
{
sqlConnection.Open();
using (SqlCommand command = new SqlCommand())
{
command.Connection = sqlConnection;
string sql = #"insert into Orders (id, customerId, firstName, lastName, phone, productCode, productName, price, orderDate, status) values (#id, #customerId, #firstName, #lastName, #phone, #productCode, #productName, #price, #orderDate, #status)";
command.CommandText = sql;
try
{
command.Parameters.Add("id", SqlDbType.BigInt).Value = content.Id;
command.Parameters.Add("customerId", SqlDbType.Int).Value = content.CustomerId;
command.Parameters.Add("firstName", SqlDbType.VarChar).Value = content.CustomerFirstName;
command.Parameters.Add("lastName", SqlDbType.VarChar).Value = content.CustomerLastName;
command.Parameters.Add("phone", SqlDbType.VarChar).Value = content.InvoiceAddress.Phone;
command.Parameters.Add("productCode", SqlDbType.Int).Value = content.Lines[0].ProductCode;
command.Parameters.Add("productName", SqlDbType.VarChar).Value = content.Lines[0].ProductName;
command.Parameters.Add("price", SqlDbType.Float).Value = content.TotalPrice;
command.Parameters.Add("orderDate", SqlDbType.BigInt).Value = content.OrderDate;
command.Parameters.Add("status", SqlDbType.TinyInt).Value = 1; //Retrieved
command.ExecuteNonQuery();
}
catch (SqlException exception)
{
if (exception.Number == 2627) // Cannot insert duplicate key row in object error
{
_logger.LogInformation("Duplicate: {ID} ", content.Id);
}
else
{
_logger.LogError("Error: {Error} ", exception.Message);
throw; // Throw exception if this exception is unexpected
}
}
}
}
}
public Task StopAsync(CancellationToken cancellationToken)
{
return Task.CompletedTask;
}
}
}

asynchronous module or handler completed while an asynchronous operation was still pending

I am working on bot technology, in one of my projet i wrote below lines of code
private async void DeliveryProgressReport(IDialogContext context, Activity message)
{
MessagesController.progressdetails = SQLDatabaseService.getinprogressdetails();
var progress = MessagesController.progressdetails;
if (progress.Count > 0)
{
try
{
Activity replyToConversation = message.CreateReply("**In Progress Report Details**");
replyToConversation.Recipient = message.From;
replyToConversation.Type = "message";
replyToConversation.Attachments = new List<Attachment>();
Dictionary<string, string> progresslist = new Dictionary<string, string>();
foreach (var progressreport in progress)
{
//Invoke the machine learning model for predicting the delivery status of delivery person
//var deliveryStatus= await InvokeRequestResponseServiceOfDeliveryPersonPredictionExp1();
//await Task.Delay(TimeSpan.FromSeconds(5));
var deliveryStatus = await InvokeRequestResponseServiceOfDeliveryPersonPredictionExp(progress[0].Name, progress[0].Mobile_Number);
progresslist.Add(progressreport.Name, progressreport.Mobile_Number);
List<CardImage> cardImages = new List<CardImage>();
cardImages.Add(new CardImage(url: progressreport.Photo_Url));
ThumbnailCard tlcard = new ThumbnailCard()
{
Title = "Name:" + progressreport.Name,
Subtitle = "Call:" + progressreport.Mobile_Number,
Images = cardImages,
Text = "Staus by Using Machine Learning Prediction:" + deliveryStatus
};
Attachment plAttachment = tlcard.ToAttachment();
replyToConversation.Attachments.Add(plAttachment);
}
replyToConversation.AttachmentLayout = AttachmentLayoutTypes.List;
await context.PostAsync(replyToConversation);
} catch (Exception ex)
{
Debug.WriteLine(ex.Message);
}
}
else
{
Activity replyToConversation = message.CreateReply("**There are no in progress deliveries are found**");
await context.PostAsync(replyToConversation);
}
}
private async Task<string> InvokeRequestResponseServiceOfDeliveryPersonPredictionExp(string name, string mobile_Number)
{
string status = "";
//Func<Stream, Task> copyStreamAsync = async stream =>
//{
//await Task.Factory.StartNew(async () =>
//{
//using (stream)
//using (var sourceStream = await sourceContent.Content.ReadAsStreamAsync())
//{
// await sourceStream.CopyToAsync(stream);
//}
//var client = new HttpClient();
using (var client = new HttpClient())
{
var scoreRequest = new
{
Inputs = new Dictionary<string, StringTable>() {
{
"input1",
new StringTable()
{
ColumnNames = new string[] {"Id", "Name", "Mobile_Number", "CourierCompany_Name", "Status", "EmailId"},
Values = new string[,] { { "", name, mobile_Number, "", "","" }, { "", name, mobile_Number, "", "", "" }, }
}
},
},
GlobalParameters = new Dictionary<string, string>()
{
}
};
const string apiKey = "xxxxxxxxxxxxxxxxxxxxxxxxxxxx=="; // Replace this with the API key for the web service
client.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Bearer", apiKey);
client.BaseAddress = new Uri("My Request URL");
// WARNING: The 'await' statement below can result in a deadlock if you are calling this code from the UI thread of an ASP.Net application.
// One way to address this would be to call ConfigureAwait(false) so that the execution does not attempt to resume on the original context.
// For instance, replace code such as:
// result = await DoSomeTask()
// with the following:
// result = await DoSomeTask().ConfigureAwait(false)
//var status = await PostRequest(scoreRequest,client).ConfigureAwait(false);
HttpResponseMessage response = await client.PostAsJsonAsync("", scoreRequest);//.ConfigureAwait(false);
string correctLocation = "";
string wrongLocation = "";
string notReached = "";
string personMismatch = "";
if (response.IsSuccessStatusCode)
{
string result = await response.Content.ReadAsStringAsync();
var results = JsonConvert.DeserializeObject<RootObject>(result);
foreach (var value in results.Results.output1.value.Values)
{
status = value[8].ToString();
correctLocation = value[4].ToString();
notReached = value[5].ToString();
personMismatch = value[6].ToString();
wrongLocation = value[7].ToString();
}
Debug.WriteLine("Result: {0}", result);
return status;
}
else
{
Debug.WriteLine(string.Format("The request failed with status code: {0}", response.StatusCode));
// Print the headers - they include the requert ID and the timestamp, which are useful for debugging the failure
Debug.WriteLine(response.Headers.ToString());
string responseContent = await response.Content.ReadAsStringAsync();
Debug.WriteLine(responseContent);
return status;
}
};
// return status;
}
After executing this below line I got the exception like asynchronous module or handler completed while an asynchronous operation was still pending
await context.PostAsync(replyToConversation);
Before posting this question I had followed through this below links, but I didn't resolve it.
Async Void, ASP.Net, and Count of Outstanding Operations
Web Api + HttpClient: An asynchronous module or handler completed while an asynchronous operation was still pending
Please tell how to resolve this exception.
-Pradeep

Finally, I resolved the above exception when I am return Task instead of void in DeliveryProgressReport method. and also where ever I was called the await DeliveryProgressReport() method there also I return Task instead of void.
-Pradeep