We Keep Coding

Difference between AmazonSecurityTokenServiceClient.AssumeRoleAsync and AssumeRoleAWSCredentials?

I'm currently expanding one of our projects which downloads objects from an S3 bucket to support RoleAWSCredentials.
I've only connected to an S3 bucket by using BasicAWSCredentials before using an accessKey and a secretKey.
Both of these code snippets work and I'm trying to understand the functional differences to make sure that I am implementing this correctly.
// version 1
try
{
var credentials =
new BasicAWSCredentials(accessKey, secretKey);
var assumeRequest = new AssumeRoleRequest
{
RoleArn = roleArn,
DurationSeconds = 3600,
RoleSessionName = roleSessionsName,
ExternalId = externalId
};
var assumeRoleResult =
await new AmazonSecurityTokenServiceClient(credentials, RegionEndpoint.USEast1)
.AssumeRoleAsync(assumeRequest, cancellationToken);
var tempCredentials = new SessionAWSCredentials(
assumeRoleResult.Credentials.AccessKeyId,
assumeRoleResult.Credentials.SecretAccessKey,
assumeRoleResult.Credentials.SessionToken);
var s3Client = new AmazonS3Client(tempCredentials, RegionEndpoint.USEast1);
var s3listedObjects = await s3Client.ListObjectsAsync(BucketName, s3Directory , cancellationToken);
}
catch (Exception e)
{
Console.WriteLine(e);
throw;
}
// Version 2
try
{
var credentials =
new BasicAWSCredentials(accessKey, secretKey);
var options = new AssumeRoleAWSCredentialsOptions()
{
ExternalId = externalId,
DurationSeconds = 3600
};
var roleAwsCredentials = new AssumeRoleAWSCredentials(credentials, roleArn, roleSessionsName, options);
var amazons3 = new AmazonS3Client(roleAwsCredentials, RegionEndpoint.USEast1);
var listedObjects = await amazons3.ListObjectsAsync(BucketName, s3Directory, cancellationToken);
}
catch (Exception e)
{
Console.WriteLine(e);
throw;
}
The first one includes a session token, which I could see allow tracking batches of requests to different sessions but is there anything else significantly different between these two ways of using RoleAWSCredentials?

Performance hit when converting IEnumerable to List of Tasks

public async Task<List<usp_TeamMeetingGetAll_Result>> TeamsMeetingReportGetAll(bool activeOnly)
{
List<usp_TeamMeetingGetAll_Result> teamMeetingGetAllResults = _db
.usp_TeamMeetingGetAll(activeOnly)
.ToList();
IEnumerable<Task<usp_TeamMeetingGetAll_Result>> taskListQuery =
from teamMeetingGetAllResult in teamMeetingGetAllResults
select TeamsMeetingGet(teamMeetingGetAllResult);
List<Task<usp_TeamMeetingGetAll_Result>> taskList = taskListQuery.ToList();
usp_TeamMeetingGetAll_Result[] results = await Task.WhenAll(taskList);
return results.OrderBy(r => r.DisplayStartDate).ToList();
}
I'm trying to figure out why line 3 (the taskList assignment) takes 7-8 seconds to execute. Line 1 calls a stored proc and puts the results (26 rows, about 15 columns) in a list and takes about 1.5 seconds. Line 2 takes less than a millisecond. Line 4 takes about 4 seconds for all async tasks to complete.
Edit: Okay, thanks to the answers below I understand that:
List<Task<usp_TeamMeetingGetAll_Result>> taskList = taskListQuery.ToList();
is causing calls to TeamsMeetingGet. I should have checked that before.
TeamsMeetingGet is making a aync call to a web API, waiting for the results, populating some properties of the usp_TeamMeetingGetAll_Result object with the results, and passing it back.
public async Task<usp_TeamMeetingGetAll_Result> TeamsMeetingGet(usp_TeamMeetingGetAll_Result teamMeetingGetAllResult)
{
//OnlineMeeting onlineMeeting = await TeamsMeetingGet(teamMeetingGetAllResult.JoinWebUrl);
HttpClient client = GraphHelper.GetAuthenticatedHttpClient();
string request =
"users/xxxxxxxxxx/onlineMeetings?$filter=JoinWebUrl%20eq%20'" + HttpUtility.UrlEncode(teamMeetingGetAllResult.JoinWebUrl) + "'";
var response = await client.GetAsync(request);
var result = await response.Content.ReadAsStringAsync();
var jresult = JObject.Parse(result);
var onlineMeetings = jresult["value"].ToObject<OnlineMeeting[]>();
if (onlineMeetings[0].Id != null)
{
teamMeetingGetAllResult.NumAttendees = onlineMeetings[0].Participants.Attendees.Count();
teamMeetingGetAllResult.Subject = onlineMeetings[0].Subject;
//More assignments
}
else
{
teamMeetingGetAllResult.NumAttendees = 0;
teamMeetingGetAllResult.Subject = "";
}
return teamMeetingGetAllResult;
}
If I have 100 rows, I'd like to call TeamsMeetingGet 100 times simultaneously, instead of synchronously. Is this how I do it?
I asked for the GetAuthenticatedHttpClient code and got back:
public class GraphHelper
{
private static string _graphClientSecret = ConfigurationManager.AppSettings["graphClientSecret"];
private static string _graphClientId = ConfigurationManager.AppSettings["graphClientId"];
private static string _graphTenantId = ConfigurationManager.AppSettings["graphTenantId"];
public static HttpClient GetAuthenticatedHttpClient()
{
IConfidentialClientApplication app;
var clientId = _graphClientId;
var tenantID = _graphTenantId;
app = ConfidentialClientApplicationBuilder.Create(clientId)
.WithClientSecret(_graphClientSecret)
.WithAuthority(new Uri("https://login.microsoftonline.com/" + tenantID))
.Build();
string[] scopes = new string[] { "https://graph.microsoft.com/.default" };
AuthenticationResult result = null;
try
{
result = app.AcquireTokenForClient(scopes)
.ExecuteAsync().Result;
}
catch (MsalUiRequiredException ex)
{
// The application doesn't have sufficient permissions.
// - Did you declare enough app permissions during app creation?
// - Did the tenant admin grant permissions to the application?
throw ex;
}
catch (MsalServiceException ex) when (ex.Message.Contains("AADSTS70011"))
{
// Invalid scope. The scope has to be in the form "https://resourceurl/.default"
// Mitigation: Change the scope to be as expected.
throw ex;
}
catch (Exception ex)
{
throw ex;
}
HttpClient client = new HttpClient();
client.DefaultRequestHeaders.Authorization = new System.Net.Http.Headers.AuthenticationHeaderValue("bearer", result.AccessToken);
client.BaseAddress = new Uri("https://graph.microsoft.com/v1.0/");
return client;
}
}

Handling Parallel POST to API

I have a API Post method that takes is a string which represents a Bae64 string of bytes from a word document that the API converts to PDF. My test client sends multiple documents, each on its own task, to the API to be converted. The problem is with concurrency and writing the files. I end up with a file in use since the calls are parallel. I have tried a lot of different way to block the conversion process until a document is converted but none of it has worked. Everything works fine if it's jsut a single file being converted but as soon as it's 2 or more, the problem happens. Can anyone guide me in the correct direction to solve this issue?
API:
[HttpPost]
public async Task<SimpleResponse> Post([FromBody]string request)
{
var response = new SimpleResponse();
Task t = Task.Factory.StartNew(async () =>
{
try
{
Converter convert = new Converter();
var result = await convert.CovertDocToPDF(request, WebConfigurationManager.AppSettings["tempDocPath"], WebConfigurationManager.AppSettings["tempPdfPath"]);
response.Result = result;
response.Success = true;
}
catch (Exception ex)
{
response.Exception = ex;
response.Success = false;
response.Errors = new List<string>();
response.Errors.Add(string.Format("{0}, {1}", ex.Message, ex.InnerException?.Message ?? ""));
}
});
t.Wait();
return response;
}
Conversion code
public Task<string> CovertDocToPDF(string blob, string tempDocPath, string tempPdfPath)
{
try
{
// Convert blob back to bytes
byte[] bte = Convert.FromBase64String(blob);
// Process and return blob
return Process(bte, tempDocPath, tempPdfPath);
}
catch (Exception Ex)
{
throw Ex;
}
}
private async Task<string> Process(byte[] bytes, string tempDocPath, string tempPdfPath)
{
try
{
string rs = RandomString(16, true);
tempDocPath = tempDocPath + rs + ".docx";
tempPdfPath = tempPdfPath + rs + ".pdf";
// This is where the problem happens with concurrent calls. I added
// the try catch when the file is in use to generate a new
// filename but the error still happens.
try
{
// Create a temp file
File.WriteAllBytes(tempDocPath, bytes);
}
catch (Exception Ex)
{
rs = RandomString(16, true);
tempDocPath = tempDocPath + rs + ".docx";
tempPdfPath = tempPdfPath + rs + ".pdf";
File.WriteAllBytes(tempDocPath, bytes);
}
word.Application app = new word.Application();
word.Document doc = app.Documents.Open(tempDocPath);
doc.SaveAs2(tempPdfPath, word.WdSaveFormat.wdFormatPDF);
doc.Close();
app.Quit(); // Clean up the word instance.
// Need the bytes to return the blob
byte[] pdfFileBytes = File.ReadAllBytes(tempPdfPath);
// Delete temp files
File.Delete(tempDocPath);
File.Delete(tempPdfPath);
// return blob
return Convert.ToBase64String(pdfFileBytes);
}
catch (Exception Ex)
{
throw Ex;
}
}
Client:
public async void btnConvert_Click(object sender, EventArgs e)
{
var response = await StartConvert();
foreach (SimpleResponse sr in response)
{
if (sr.Success)
{
byte[] bte = Convert.FromBase64String(sr.Result.ToString());
string rs = RandomString(16, true);
string pdfFileName = tempPdfPath + rs + ".pdf";
if (File.Exists(pdfFileName))
{
File.Delete(pdfFileName);
}
System.IO.File.WriteAllBytes(pdfFileName, bte);
}
else
{
}
}
}
private async Task<IEnumerable<SimpleResponse>> StartConvert()
{
var tasks = new List<Task<SimpleResponse>>();
foreach (string s in docPaths)
{
byte[] bte = File.ReadAllBytes(s);
tasks.Add(ConvertDocuments(Convert.ToBase64String(bte)));
}
return (await Task.WhenAll(tasks));
}
private async Task<SimpleResponse> ConvertDocuments(string requests)
{
using (var client = new HttpClient(new HttpClientHandler() { UseDefaultCredentials = true }))
{
client.BaseAddress = new Uri(BaseApiUrl);
client.DefaultRequestHeaders.Add("Accept", "application/json");
// Add an Accept header for JSON format.
client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));//application/json
HttpRequestMessage request = new HttpRequestMessage(HttpMethod.Post, BaseApiUrl + ApiUrl);
var data = JsonConvert.SerializeObject(requests);
request.Content = new StringContent(data, Encoding.UTF8, "application/json");
HttpResponseMessage response1 = await client.PostAsync(BaseApiUrl + ApiUrl, request.Content).ConfigureAwait(false);
var response = JsonConvert.DeserializeObject<SimpleResponse>(await response1.Content.ReadAsStringAsync());
return response;
}
}
Random String Generator
public string RandomString(int size, bool lowerCase = false)
{
var builder = new StringBuilder(size);
// Unicode/ASCII Letters are divided into two blocks
// (Letters 65–90 / 97–122):
// The first group containing the uppercase letters and
// the second group containing the lowercase.
// char is a single Unicode character
char offset = lowerCase ? 'a' : 'A';
const int lettersOffset = 26; // A...Z or a..z: length = 26
for (var i = 0; i < size; i++)
{
var #char = (char)_random.Next(offset, offset + lettersOffset);
builder.Append(#char);
}
return lowerCase ? builder.ToString().ToLower() : builder.ToString();
}

First, get rid of Task.Factory.StartNew ... t.Wait() - you don't need an additional task, the root level method is async and your blocking Wait just spoils the benefits of async by blocking synchronously.
Second, like a comment suggested above, the file name random string generator is most likely to be not really random. Either do not supply anything to the seed value of your pseudo-random gen, or use something like Environment.TickCount which should be sufficient for this. Guid.NewGuid() will work too.
Another good option for temp files is Path.GetTempFileName (also generates an empty file for you): https://learn.microsoft.com/en-us/dotnet/api/system.io.path.gettempfilename?view=netstandard-2.0
[HttpPost]
public async Task<SimpleResponse> Post([FromBody]string request)
{
var response = new SimpleResponse();
try
{
...
var result = await convert.CovertDocToPDF(...);
...
}
catch (Exception ex)
{
...
}
return response;
}

Based on your code it seems that you have a "faulty" random string generator for file name (I would say _random.Next is a suspect, possibly some locking and/or "app wide" instance could fix the issue). You can use Guid.NewGuid to create random part of file name (which in theory can have collisions also but in most practical cases should be fine) or Path.GetTempFileName:
var rs = Guid.NewGuid().ToString("N");
tempDocPath = tempDocPath + rs + ".docx";
tempPdfPath = tempPdfPath + rs + ".pdf";

Cancellation Token Disposed Exception

I had this piece of code that initially worked fine. However, After adding it to a class where I store my methods that are reused, it keeps failing. The exception that is caught states that the CancellationTokenSource has been Disposed. Can someone point me in the right direction?
I have tried creating a new client and Adding CancellationToken.None to the PutAsync() method from HTTPClient Class but it still fails with the CancellationTokenSource Disposed exception.
public async void AddProduct(Product product)
{
string storeId = "";
try
{
var storeData = JObject.Parse(Connect.Json).SelectToken("store").ToString();
var stores = JsonConvert.DeserializeObject<List<Store>>(storeData);
var store = stores[0];
storeId = store.Id;
store.Products.Add(product);
ProdInfo info = new Info();
foreach(Product p in store.Products)
{
info.AddedProducts = + p.Id;
}
var content = JsonConvert.SerializeObject(info);
using (Connect.Client)
using (var response = await Connect.Client.PutAsync(_url + "/stores/" + storeId, new StringContent(content)))
{
var cont = response.Content;
string result = await cont.ReadAsStringAsync();
if ((int)response.StatusCode == 200)
{
this.JobResult = result;
//this.JobResult = "Store has been successfully updated";
}
else
{
this.JobResult = result;
//this.JobResult = "Store was not updated!";
}
}
}
catch (Exception ex)
{
//this.JobResult = "Store has not been updated due to an error.";
this.JobResult = ex.Message;
}
}

I was able to solve this by simple removing 'using(Connect.Client)' from all of my methods. As #sellotape stated, They were disposing of the HttpClient before I was able to use it again. Thank you all for your contributions.

.Net Core Async Await is not continuing Execution

I have a list of tasks, that will call a webservice restfully. I begin the execution of them when calling downloadTasksQuery.ToList(). At that point I would like to process each task as they finish, and add the Distance to the database as soon as it finishes.
When I watch this in the console. All records seem to print to the console all at the same time, and I see the outputis from this line. Console.WriteLine(" Google Distance : " + d.google_distance);
The number of times it prints to the console is the number that were in the List of Tasks to begin with.
IEnumerable<Task<Distances>> downloadTasksQuery =
(from tup in urlList select ProcessURL(tup));
List<Task<Distances>> downloadTasks = downloadTasksQuery.ToList();
while (downloadTasks.Count > 0)
{
Task<Distances> thisFinished = await Task.WhenAny(downloadTasks);
downloadTasks.Remove(thisFinished);
try
{
Distances d = await thisFinished;
if (d.has_error == false)
{
dr.Add(d);
}
else
{
dr.AddError(d);
}
Console.WriteLine(" Google Distance : " + d.google_distance);
}
catch (Exception e)
{
Console.WriteLine("Deleted task, did not save data");
}
}
Process URL
private static async Task<Distances> ProcessURL(Tuple<string, int, string> tup)
{
HttpClient client = new HttpClient();
Distances d = new Distances();
try
{
client.DefaultRequestHeaders.Accept.Clear();
client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/xml"));
//// Post is required by the web api
var stringTask = client.PostAsync("https://website.com" + tup.Item1, new StringContent("null"));
var msg2 = await stringTask.Result.Content.ReadAsStringAsync();
d = FromXml<Distances>(msg2.Trim());
}
catch (Exception e)
{
d.has_error = true;
}
d.state_student_identifier = tup.Item3;
d.student_transportation_id = tup.Item2;
return d;
}

I would suggest utilizing ContinueWith as it seems more suited for what you are trying to do:
var runningTasks = urlList.Select(tup => ProcessURL(tup)
.ContinueWith(task =>
{
var d = task.Result;
if (d.has_error)
dr.AddError(d);
else
dr.Add(d);
Console.WriteLine(" Google Distance : " + d.google_distance);
}));
await Task.WhenAll(runningTasks);