My objective is to stream the data from the Database to the Client that consumes the api. The reason for that it the size of the data (MB).
I have the following code.
public async IAsyncEnumerable<ProductAvailability> GetProductsAsync()
{
await using var connection = new SqlConnection(connectionStrings.HpcDatabase);
foreach (var product in connection.Query<ProductAvailability>(query, buffered: false))
{
yield return product;
}
}
[HttpGet("products-availability2")]
public async IAsyncEnumerable<ProductAvailability> Get()
{
await foreach (var pro in productRepository.GetProductsAsync())
{
yield return pro;
}
}
Yet everytime I call the api the memory usage keeps increasing.
image 1
image 2
I'm doing something wrong here? Am I looking at the wrong metric?
Thank you
Related
In my Asp.Net Core 3.1 project I am using AmazonEC2Client for getting info about AWS instances.
I implemented helper method for getting instance list.Method looks like:
public static async Task<List<string>> AwsList(string awsAccessKeyId, string
awsSecretAccessKey)
{
AmazonEC2Client client = new AmazonEC2Client(awsAccessKeyId,awsSecretAccessKey,
RegionEndpoint.EUWest1);
bool done = false;
var instanceIds = new List<string>();
DescribeInstancesRequest request = new DescribeInstancesRequest();
while (!done)
{
DescribeInstancesResponse response = await
client.DescribeInstancesAsync(request);
foreach ( Reservation reservation in response.Reservations)
{
foreach (Instance instance in reservation.Instances)
{
instanceIds.Add(instance.InstanceType);
}
}
request.NextToken= response.NextToken;
if (response.NextToken == null)
{
done = true;
}
}
return instanceIds;
}
Json result is:
[
"t3a.xlarge",
"t2.medium",
"t2.medium",
"t2.micro",
"t3a.xlarge",
"t2.medium",
"t3a.xlarge",
"t3a.xlarge",
"t3a.xlarge"
]
I dont know ram and cpu info inside instance type or not, no experience with aws.
I would like to get cpu and ram info according to instance type.
Later I would like to create method which is accepting string instanceType and according to this get ram and cpu.
For ex: GetRam("t2.micro") -> 2gb
Instead of using DescribeInstanceRequests need to be use DescribeInstanceTypesRequest and appropriate response as well.
foreach ( var instanceType in response.InstanceTypes.Where(x => x.InstanceType == name))
{
instanceIds.Add(instanceType.MemoryInfo.SizeInMiB); // ram
instanceIds.Add(instanceType.VCpuInfo.DefaultVCpus); //cpu
}
I've been working for a few days on a performance problem.
Before I delve deeper I want to quickly explain how the specific service work.
I have a main Service that get a request and send requests to other micro-services but the single Entry Point for the user is to Main service, I thinks is more simple to understand with this image:
After the Main service get request from API he do some logic, query the Db and then get a list, every item on the list has Id, to get enrichment about every item the main service create request to one of the micro-service.
For example John request main service, main service get from Db a list of 90 items then the main service will create 90 calls to micro service and return to John single response that include 90 items.
Now the question is only about the right way to create async call to micro service.
This how I develop this part:
GetDetailsAsync(Id, result.Items, request.SystemComponentId);
private static void GetDetailsAsync(string Id, List<MainItem> items, int systemId)
{
var getDetailsTasks = new List<Task>();
foreach (MainItem single in items)
{
getDetailsTasks.Add(SetSingleDetailsAsync(Id, single, systemId));
}
Task.WhenAll(getDetailsTasks);
}
private static async Task SetSingleDetailsAsync(string Id, MainItem single, int systemId)
{
single.ActivityExtendedDetails = await ProcessItemDetailsRequest.GetItemDetailsAsync(Id, single.TypeId,
single.ItemId, systemId);
}
public static Task<JObject> GetItemDetailsAsync(string id, short type,
string itemId, int systemId)
{
var typeList = ActivityTypeDetails.GetActivityTypes();
var url = GetActivityUrl(id, type, itemId, typeList);
if (url == null)
{
throw new Failure($"No url defined for type {type}");
}
try
{
JObject res;
using (var stream = client.GetStreamAsync(url).Result)
using (var sr = new StreamReader(stream))
using (var reader = new JsonTextReader(sr))
{
var serializer = new JsonSerializer();
res = serializer.Deserialize<JObject>(reader);
}
return Task.FromResult(res);
}
catch(Exception ex)
{
Logger.Warn(
$"The uri {url} threw exception {ex.Message}.");
//[Todo]throw exception
return null;
}
}
This code run and the result is not good enough, the CPU rises very quickly and becomes very high, I think that I has a problem on GetItemDetailsAsync func because I use client.GetStreamAsync(url).Result
when using .Result it's block until the task is completed.
So I do some minor change on GetItemDetailsAsync to try to be really async:
public static async Task<JObject> GetItemDetailsAsync(string id, short type,
string itemId, int systemId)
{
var typeList = ActivityTypeDetails.GetActivityTypes();
var url = GetActivityUrl(id, type, itemId, typeList);
if (url == null)
{
throw new Failure($"No url defined for type {type}");
}
try
{
JObject res;
using (var stream = await client.GetStreamAsync(url))
using (var sr = new StreamReader(stream))
using (var reader = new JsonTextReader(sr))
{
var serializer = new JsonSerializer();
res = serializer.Deserialize<JObject>(reader);
}
return res;
}
catch(Exception ex)
{
Logger.Warn(
$"The uri {url} threw exception {ex.Message}.");
//[Todo]throw exception
return null;
}
}
But now I get null where I supposed to get the data that come from Async function.
I try to debugging and I noticed something weird, everything happen likes as I would expect: the methods was called, request to micro-service was executed and get response but the response from the End-Point(which is found on main-service) return before the async method return from micro-service, that cause that I get null instead of my expected data.
I thinks that maybe I don't use correctly async\await and would be happy if anyone could explain how this behavior happens
Referring to this sample code on GitHub:
https://github.com/DblV/StreamingWebApi/blob/master/StreamingService/StreamingService/Controllers/StreamingController.cs
I want to stream content stored in a database, which my query returns as a sequence of blobs (essentially one file split into "blocks"). Due to the potential size of the complete response, I want to stream it, and I am following the above example as follows:
public class FileController : ApiController
{
[HttpGet]
public HttpResponseMessage Get(string id, [FromUri] string contentType)
{
var message = new HttpResponseMessage(HttpStatusCode.OK);
message.Content = new PushStreamContent((stream, content, context) =>
{
GetFileContent(stream, int.Parse(id));
Thread.Sleep(1000);
stream.Close();
});
message.Content.Headers.ContentType = new MediaTypeHeaderValue(contentType);
return message;
}
private void GetFileContent(Stream stream, int id)
{
var result = Query(reader => reader.GetStream(0), id);
foreach (var b in result)
{
b.CopyToAsync(stream);
stream.Flush();
}
}
private IEnumerable<Stream> Query(Func<DbDataReader,Stream> func, int id)
{
var command = // Not shown - SELECT command creating the result set
var reader = command.ExecuteReader();
if (reader.HasRows)
{
while (reader.Read())
{
yield return func(reader);
}
}
reader.Close();
}
Note the use and placement of Thread.Sleep. When I test this in the browser, the content downloads to a file, but without the Sleep, it hangs at the point of completion; with the Sleep, it completes properly, and the resulting download is perfect.
My question: what is the Sleep doing that averts the hang condition? My suspicion is that this is more of a work-around than a proper solution; if so, what should I be doing instead?
Found that it is a mistake on my part. I was incorrectly using CopyToAsync where I should be using CopyTo. Correcting this mistake, it works just fine.
Is it possible to modify Table Controllers in Azure Mobile Services .Net backend to handle multi-insertion per http request?
After coming back online it takes 2+ minutes for my app to sync its data. over 70% of the 2 minute is wasted over the network handshaking overhead.
I had to do something similar. My app creates around 10,000 new rows every time a user creates a new project so I made a custom controller in my Mobile Service to accept this. After the 10,000 entities are inserted I pull all of them back down to the local sync database.
I first created a custom controller.
public class BatchInsertController : ApiController
{
DbContext context;
protected override void Initialize(HttpControllerContext controllerContext)
{
base.Initialize(controllerContext);
this.context = new DbContext();
}
[HttpPost]
public async Task<bool> BatchInsert(List<Entity> entities)
{
try
{
this.context.Entities.AddRange(entities);
await this.context.SaveChangesAsync();
return true;
}
catch (System.Exception ex)
{
Trace.WriteLine("Error: " + ex);
return false;
}
}
}
Then I would call this custom controller method from my client code.
var entities = new List<Entity>();
// Add a bunch of entities to the list...
foreach (List<Entity> chunkedEntities in entities.ChunksOf(1000))
{
var response = await _client.InvokeApiAsync<List<Entity>, bool>("batchinsert", chunkedEntities);
}
I would have over 10,000 records at a time so I created an extension method to chunk the list and send 1,000 records at a time.
public static IEnumerable<IList<T>> ChunksOf<T>(this IEnumerable<T> sequence, int size)
{
List<T> chunk = new List<T>(size);
foreach (T element in sequence)
{
chunk.Add(element);
if (chunk.Count == size)
{
yield return chunk;
chunk = new List<T>(size);
}
}
if (chunk.Any())
{
yield return chunk;
}
}
After that I just did a PullAsync() on my local db. I came up with this after reading through this article.
I have a class "Image" with three properties: Url, Id, Content.
I have a list of 10 such images.
This is a silverlight app.
I want to create a method:
IObservable<Image> DownloadImages(List<Image> imagesToDownload)
{
//start downloading all images in imagesToDownload
//OnImageDownloaded:
image.Content = webResponse.Content
yield image
}
This method starts downloading all 10 images in parallel.
Then, when each downloads completes, it sets the Image.Content to the WebResponse.Content of that download.
The result should be an IObservable stream with each downloaded image.
I'm a beginner in RX, and I think what I want can be achieved with ForkJoin, but that's in an experimental release of reactive extensions dll which I don't want to use.
Also I really don't like download counting on callbacks to detect that all images have been downloaded and then call onCompleted().
Doesn't seem to be in the Rx spirit to me.
Also I post what solution I've coded so far, though I don't like my solution because its long/ugly and uses counters.
return Observable.Create((IObserver<Attachment> observer) =>
{
int downloadCount = attachmentsToBeDownloaded.Count;
foreach (var attachment in attachmentsToBeDownloaded)
{
Action<Attachment> action = attachmentDDD =>
this.BeginDownloadAttachment2(attachment).Subscribe(imageDownloadWebResponse =>
{
try
{
using (Stream stream = imageDownloadWebResponse.GetResponseStream())
{
attachment.FileContent = stream.ReadToEnd();
}
observer.OnNext(attachmentDDD);
lock (downloadCountLocker)
{
downloadCount--;
if (downloadCount == 0)
{
observer.OnCompleted();
}
}
} catch (Exception ex)
{
observer.OnError(ex);
}
});
action.Invoke(attachment);
}
return () => { }; //do nothing when subscriber disposes subscription
});
}
Ok, I did manage it to make it work in the end based on Jim's answer.
var obs = from image in attachmentsToBeDownloaded.ToObservable()
from webResponse in this.BeginDownloadAttachment2(image).ObserveOn(Scheduler.ThreadPool)
from responseStream in Observable.Using(webResponse.GetResponseStream, Observable.Return)
let newImage = setAttachmentValue(image, responseStream.ReadToEnd())
select newImage;
where setAttachmentValue just takes does `image.Content = bytes; return image;
BeginDownloadAttachment2 code:
private IObservable<WebResponse> BeginDownloadAttachment2(Attachment attachment)
{
Uri requestUri = new Uri(this.DownloadLinkBaseUrl + attachment.Id.ToString();
WebRequest imageDownloadWebRequest = HttpWebRequest.Create(requestUri);
IObservable<WebResponse> imageDownloadObservable = Observable.FromAsyncPattern<WebResponse>(imageDownloadWebRequest.BeginGetResponse, imageDownloadWebRequest.EndGetResponse)();
return imageDownloadObservable;
}
How about we simplify this a bit. Take your image list and convert it to an observable. Next, consider using the Observable.FromAsyncPattern to manage the service requests. Finally use SelectMany to coordinate the request with the response. I'm making some assumptions on how you are getting the file streams here. Essentially if you can pass in the BeginInvoke/EndInvoke delegates into FromAsyncPattern for your service request you are good.
var svcObs = Observable.FromAsyncPattern<Stream>(this.BeginDownloadAttachment2, This.EndDownloadAttchment2);
var obs = from image in imagesToDownload.ToObservable()
from responseStream in svcObs(image)
.ObserveOnDispatcher()
.Do(response => image.FileContent = response.ReadToEnd())
select image;
return obs;