We Keep Coding

Move Async/Await JSON deserialize result to another function

I have this function that can get up to 10 items as an input list
public async Task<KeyValuePair<string, bool>[]> PayCallSendSMS(List<SmsRequest> ListSms)
{
List<Task<KeyValuePair<string, bool>>> tasks = new List<Task<KeyValuePair<string, bool>>>();
foreach (SmsRequest sms in ListSms)
{
tasks.Add(Task.Run(() => SendSMS(sms)));
}
var result = await Task.WhenAll(tasks);
return result;
}
and in this function, i await for some JSON to be downloaded and after it's done in deserialize it.
public async Task<KeyValuePair<string, bool>> SendSMS(SmsRequest sms)
{
//some code
using (WebResponse response = webRequest.GetResponse())
{
using (Stream responseStream = response.GetResponseStream())
{
StreamReader rdr = new StreamReader(responseStream, Encoding.UTF8);
string Json = await rdr.ReadToEndAsync();
deserializedJsonDictionary = (Dictionary<string, object>)jsonSerializer.DeserializeObject(Json);
}
}
//some code
return GetResult(sms.recipient);
}
public KeyValuePair<string, bool> GetResult(string recipient)
{
if (deserializedJsonDictionary[STATUS].ToString().ToLower().Equals("true"))
{
return new KeyValuePair<string, bool>(recipient, true);
}
else // deserializedJsonDictionary[STATUS] == "False"
{
return new KeyValuePair<string, bool>(recipient, false);
}
}
My problem is in the return GetResult(); part in which deserializedJsonDictionary is null(and ofc it is becuase the json havent done downloading).
but I don't know how to solve it
I tried to use ContinueWith but it doesn't work for me.
I'm willing to accept any change to my original code and/or the design of the solution

Unrelated tip: Don't abuse KeyValuePair<>, use C# 7 value-tuples instead (not least because they're much easier to read).
Using a foreach loop to build a List<Task> is fine - though it can be more succint to use .Select() instead. I use this approach in my answer.
But don't use Task.Run with the ancient WebRequest (HttpWebRequest) type. Instead use HttpClient which has full support for async IO.
Also, you should conform to the .NET naming-convention:
All methods that are async should have Async has a method-name suffix (e.g. PayCallSendSMS should be named PayCallSendSmsAsync).
Acronyms and initialisms longer than 2 characters should be in PascalCase, not CAPS, so use Sms instead of SMS.
Use camelCase, not PascalCase for parameters and locals - and List is a redundant prefix. A better name for ListSms would be smsRequests as its type is List<SmsRequest>).
Generally speaking, parameters should be declared using the least-specific type required - especially collection parameters, consider typing them as IEnumerable<T> or IReadOnlyCollection<T> instead of T[], List<T>, and so on).
You need to first check that the response from the remote server actually is a JSON response (instead of a HTML error message or XML response) and has the expected status code - otherwise you'll be trying to deserialize something that is not JSON.
Consider supporting CancellationToken too (this is not included in my answer as it adds too much visual noise).
Always use Dictionary.TryGetValue instead of blindly assuming the dictionary indexer will match.
public async Task< IReadOnlyList<(String recipient, Boolean ok)> > PayCallSendSmsAsync( IEnumerable<SmsRequest> smsRequests )
{
using( HttpClient httpClient = this.httpClientFactory.Create() )
{
var tasks = smsRequests
.Select(r => SendSmsAsync(httpClient, r))
.ToList(); // <-- The call to ToList is important as it materializes the list and triggers all of the Tasks.
(String recipient, Boolean ok)[] results = await Task.WhenAll(tasks);
return results;
}
}
private static async Task<(String recipient, Boolean ok)> SendSmsAsync(HttpClient httpClient, SmsRequest smsRequest)
{
using (HttpRequestMessage request = new HttpRequestMessage( ... ) )
using (HttpResponseMessage response = await httpClient.SendAsync(request).ConfigureAwait(false))
{
String responseType = response.Content.Headers.ContentType?.MediaType ?? "";
if (responseType != "application/json" || response.StatusCode != HttpStatusCode.OK)
{
throw new InvalidOperationException("Expected HTTP 200 JSON response but encountered an HTTP " + response.StatusCode + " " + responseType + " response instead." );
}
String jsonText = await response.Content.ReadAsStringAsync().ConfigureAwait(false);
Dictionary<String,Object> dict = JsonConvert.DeserializeObject< Dictionary<String,Object> >(jsonText);
if(
dict != null &&
dict.TryGetValue(STATUS, out Object statusValue) &&
statusValue is String statusStr &&
"true".Equals( statusStr, StringComparison.OrdinalIgnoreCase )
)
{
return ( smsRequest.Recipient, ok: true );
}
else
{
return ( smsRequest.Recipient, ok: false );
}
}
}

How to Bulk Insert in Cosmos DB with .NET Core 2.1 and Stream API

I'm trying to implement bulk insert with this CosmosDB sample. This sample is created with .NET Core 3.* and support of System.Text.Json.
When using the CreateItemAsync method, it works perfectly:
var concurrentTasks = new List<Task<ItemResponse<Notification>>>();
foreach (var entity in entities)
{
entity.Id = GenerateId(entity);
var requestOptions = new ItemRequestOptions();
requestOptions.EnableContentResponseOnWrite = false; // We don't need to get the entire body returend.
concurrentTasks.Add(Container.CreateItemAsync(entity, new PartitionKey(entity.UserId), requestOptions));
}
await Task.WhenAll(concurrentTasks);
However, I'm trying to see if I can reduce the number of RU's by streaming the data directly into CosmosDB, hoping CosmosDB doesn't charge me for deserializing JSON itself.
I'm working in .NET Core 2.1 and Newtonsoft.Json. This is my code that does not return a succesfull status code. The sub-status code in the response header is "0".
Notification[] notifications = entities.ToArray();
var itemsToInsert = new Dictionary<PartitionKey, Stream>();
foreach (var notification in notifications)
{
MemoryStream ms = new MemoryStream();
StreamWriter writer = new StreamWriter(ms);
JsonTextWriter jsonWriter = new JsonTextWriter(writer);
JsonSerializer ser = new JsonSerializer();
ser.Serialize(jsonWriter, notification);
await jsonWriter.FlushAsync();
await writer.FlushAsync();
itemsToInsert.Add(new PartitionKey(notification.UserId), ms);
}
List<Task> tasks = new List<Task>(notifications.Length);
foreach (KeyValuePair<PartitionKey, Stream> item in itemsToInsert)
{
tasks.Add(Container.CreateItemStreamAsync(item.Value, item.Key)
.ContinueWith((Task<ResponseMessage> task) =>
{
using (ResponseMessage response = task.Result)
{
if (!response.IsSuccessStatusCode)
{
Console.WriteLine($"Received {response.StatusCode} ({response.ErrorMessage}).");
}
else
{
}
}
}));
}
// Wait until all are done
await Task.WhenAll(tasks);
response.StatusCode: BadRequest
response.ErrorMessage: null
I'm assuming I don't serialize into the Stream in a correct way. Anyone got a clue?
Update
I discovered that the new System.Text.Json package also implements .NET Standard 2.0 so I installed it from NUget. Now I can copy the sample code from Github, mentioned earlier.
Notification[] notifications = entities.ToArray();
var itemsToInsert = new List<Tuple<PartitionKey, Stream>>();
foreach (var notification in notifications)
{
notification.id = $"{notification.UserId}:{Guid.NewGuid()}";
MemoryStream stream = new MemoryStream();
await JsonSerializer.SerializeAsync(stream, notification);
itemsToInsert.Add(new Tuple<PartitionKey, Stream>(new PartitionKey(notification.RoleId), stream));
}
List<Task> tasks = new List<Task>(notifications.Length);
foreach (var item in itemsToInsert)
{
tasks.Add(Container.CreateItemStreamAsync(item.Item2, item.Item1)
.ContinueWith((Task<ResponseMessage> task) =>
{
using (ResponseMessage response = task.Result)
{
if (!response.IsSuccessStatusCode)
{
Console.WriteLine($"Received {response.StatusCode} ({response.ErrorMessage}).");
}
else
{
}
}
}));
}
// Wait until all are done
await Task.WhenAll(tasks);
I double checked that BulkInsert is enabled (or else the first method also won't work). Still there is a BadRequest and a NULL for errorMessage.
I also checked that the data isn't added to the container dispite the BadRequest.

I found the problem.
I've setup my Cosmos Context with the following options:
var cosmosSerializationOptions = new CosmosSerializationOptions();
cosmosSerializationOptions.PropertyNamingPolicy = CosmosPropertyNamingPolicy.CamelCase;
CosmosClientOptions cosmosClientOptions = new CosmosClientOptions();
cosmosClientOptions.SerializerOptions = cosmosSerializationOptions;
Hence the CamelCase convention. In my first (working) code sample, I would let the CosmosDB Context deserialize to JSON. He would serialize with this CamelCase convention, so my PartionKey UserId would be serialized into userId.
However, to reduce some RU's I will use the CreateItemStreamAsync that makes me responsible for the serialization. And there was the mistake, my property was defined like:
public int UserId { get; set; }
So he would be serialized to json UserId: 1.
However, the partition key is defined as /userId. So if I add the JsonPropertyName attribute, it works:
[JsonPropertyName("userId")]
public int UserId { get; set; }
...if only an error message would tell me that.
There is about 3% RU savings on using this CreateItemStream method. However, over time this would slowly save some RU's in total I guess.

It looks like the stream is not readable. Hence the bad request.
I would make little modification to how MemoryStream is created:
foreach (var notification in notifications)
{
itemsToInsert.Add(new PartitionKey(notification.UserId), new MemoryStream(Encoding.UTF8.GetBytes(JsonConvert.SerializeObject(notification))));
}
Of course, I am using Newtonsoft.json for jsonConvert.

Correct way to use Async/Await

I've been working for a few days on a performance problem.
Before I delve deeper I want to quickly explain how the specific service work.
I have a main Service that get a request and send requests to other micro-services but the single Entry Point for the user is to Main service, I thinks is more simple to understand with this image:
After the Main service get request from API he do some logic, query the Db and then get a list, every item on the list has Id, to get enrichment about every item the main service create request to one of the micro-service.
For example John request main service, main service get from Db a list of 90 items then the main service will create 90 calls to micro service and return to John single response that include 90 items.
Now the question is only about the right way to create async call to micro service.
This how I develop this part:
GetDetailsAsync(Id, result.Items, request.SystemComponentId);
private static void GetDetailsAsync(string Id, List<MainItem> items, int systemId)
{
var getDetailsTasks = new List<Task>();
foreach (MainItem single in items)
{
getDetailsTasks.Add(SetSingleDetailsAsync(Id, single, systemId));
}
Task.WhenAll(getDetailsTasks);
}
private static async Task SetSingleDetailsAsync(string Id, MainItem single, int systemId)
{
single.ActivityExtendedDetails = await ProcessItemDetailsRequest.GetItemDetailsAsync(Id, single.TypeId,
single.ItemId, systemId);
}
public static Task<JObject> GetItemDetailsAsync(string id, short type,
string itemId, int systemId)
{
var typeList = ActivityTypeDetails.GetActivityTypes();
var url = GetActivityUrl(id, type, itemId, typeList);
if (url == null)
{
throw new Failure($"No url defined for type {type}");
}
try
{
JObject res;
using (var stream = client.GetStreamAsync(url).Result)
using (var sr = new StreamReader(stream))
using (var reader = new JsonTextReader(sr))
{
var serializer = new JsonSerializer();
res = serializer.Deserialize<JObject>(reader);
}
return Task.FromResult(res);
}
catch(Exception ex)
{
Logger.Warn(
$"The uri {url} threw exception {ex.Message}.");
//[Todo]throw exception
return null;
}
}
This code run and the result is not good enough, the CPU rises very quickly and becomes very high, I think that I has a problem on GetItemDetailsAsync func because I use client.GetStreamAsync(url).Result
when using .Result it's block until the task is completed.
So I do some minor change on GetItemDetailsAsync to try to be really async:
public static async Task<JObject> GetItemDetailsAsync(string id, short type,
string itemId, int systemId)
{
var typeList = ActivityTypeDetails.GetActivityTypes();
var url = GetActivityUrl(id, type, itemId, typeList);
if (url == null)
{
throw new Failure($"No url defined for type {type}");
}
try
{
JObject res;
using (var stream = await client.GetStreamAsync(url))
using (var sr = new StreamReader(stream))
using (var reader = new JsonTextReader(sr))
{
var serializer = new JsonSerializer();
res = serializer.Deserialize<JObject>(reader);
}
return res;
}
catch(Exception ex)
{
Logger.Warn(
$"The uri {url} threw exception {ex.Message}.");
//[Todo]throw exception
return null;
}
}
But now I get null where I supposed to get the data that come from Async function.
I try to debugging and I noticed something weird, everything happen likes as I would expect: the methods was called, request to micro-service was executed and get response but the response from the End-Point(which is found on main-service) return before the async method return from micro-service, that cause that I get null instead of my expected data.
I thinks that maybe I don't use correctly async\await and would be happy if anyone could explain how this behavior happens

Make Http Request to Api and Convert to json

My question may be trivial but I have spent almost 6hrs just trying things out.
public async Task<object> save()
{
var uri = "https://newsapi.org/v1/articles?source=talksport&apiKey=longKey";
var httpClient = new HttpClient ();
HttpResponseMessage res = await httpClient.GetAsync(uri);
var data = await res.Content.ReadAsStreamAsync();
// this is what I want to achieve like in python you can do something like this
foreach(var item in data){
Console.writeline(item.summary);
}
// end of arbitrary code
return data;
}
My problem is ,am unable to do this conversion to get the response and then accessing the json data.
In python you can do something
r = request.get(apiUrl)
data = r.json()
for item in data:
print(item.summary)
This is all I have struggle to achieve with c#, Any help to complete the code or explanation. Thanks

Try to use something like this:
Install Newtonsoft.Json package and add using Newtonsoft.Json;
using (var request = new HttpRequestMessage()) {
request.RequestUri = new Uri("https://newsapi.org/v1/articles?source=talksport&apiKey=longKey");
request.Method = HttpMethod.Get;
using (var response = await httpClient.SendAsync(request)) {
string content = await response.Content.ReadAsStringAsync();
var result = JsonConvert.DeserializeObject<IList<dynamic>>(content);
foreach(var item in result){
Console.writeline(item.summary);
}
}
}
From comment
Then i get this
"{\"vouchers\":[\"UN9NKK\",\"FYMFVS\",\"WV5AX7\",\"M2TJJ8\",‌​\"FBB9AL\",\"MBW8Z4\‌​"]}"
You can create a new class
public class MyResponse {
public IEnumerable<string> Vouchers {get;set; }
}
then
var response = JsonConvert.DeserializeObject<MyResponse>(content);
foreach(var item in response.Vouchers){
Console.WriteLine(item);
}

If you don't mind a small library dependency, Flurl (disclaimer: I'm the author) gets you Python's simplicity in C#:
var data = await apiUrl.GetJsonAsync();
In this case, data is a C# dynamic type, which means you can access all the JSON object's properties by name without defining a corresponding C# class, much like an untyped language. If you do want to declare a class and get compile-time type checking, that works with Flurl too:
var data = await apiUrl.GetJsonAsync<MyClass>();
Now data is an instance of MyClass instead of a dynamic.
Get Flurl.Http on Nuget, and reference it with using Flurl.Http;.

Using using in a constructor

I am using a create method for a constructor of a converter.
public void loadData()
{
byte [] data = new byte [] {......}; // some byte data in here
var converter = GetDataConverter(data);
}
Now inside the GetDataConverter I need to create a memorystream from the binary data (the converter is 3rd party and takes a stream)
If I write the GetDataConverter like this I get an error telling me I didnt' dispose which I understand. I created a MemoryStream and I need to dispose.
public MyDataConverter GetDataConverter(byte [] data)
{
return new MyDataConverter(new MemoryStream(data));
}
So my solution would be this:
public MyDataConverter GetDataConverter(byte [] data)
{
using(var ms = new MemoryStream(data))
{
return new MyDataConverter(ms);
}
}
The question is, is my solution correct? Should I be using a 'using' here? isn't the 'using' going to destroy my memory stream once it's out of scope so the converter will have nothing to work on?
I need an answer AND an explanation please, I'm a bit vague on the whole 'using' thing here.
Thanks

If you have no access to the code of ´MyDataConverter´ and the type doesn't implements ´IDisposable´ you can do:
public void loadData()
{
byte[] data = new byte[] { 0 }; // some byte data in here
using (var stream = new MemoryStream(data))
{
var converter = new MyDataConverter(stream);
// do work here...
}
}
If you have to use this many times and want to reuse your code you can do something like this:
public void loadData()
{
byte[] data = new byte[] { 0 }; // some byte data in here
UsingConverter(data, x =>
{
// do work here...
});
}
void UsingConverter(byte[] data, Action<MyDataConverter> action)
{
using (var stream = new MemoryStream(data))
{
var converter = new MyDataConverter(stream);
action(converter);
}
}

It really depends on the implementation of MyDataConverter. If the MemoryStream is only used inside the constructor to retrieve some data from it, then your solution with using is OK.
If, OTOH, MyDataConverter keeps a reference to the MemoryStream to access it later, you must not dispose it here.