Referring to this sample code on GitHub:
https://github.com/DblV/StreamingWebApi/blob/master/StreamingService/StreamingService/Controllers/StreamingController.cs
I want to stream content stored in a database, which my query returns as a sequence of blobs (essentially one file split into "blocks"). Due to the potential size of the complete response, I want to stream it, and I am following the above example as follows:
public class FileController : ApiController
{
[HttpGet]
public HttpResponseMessage Get(string id, [FromUri] string contentType)
{
var message = new HttpResponseMessage(HttpStatusCode.OK);
message.Content = new PushStreamContent((stream, content, context) =>
{
GetFileContent(stream, int.Parse(id));
Thread.Sleep(1000);
stream.Close();
});
message.Content.Headers.ContentType = new MediaTypeHeaderValue(contentType);
return message;
}
private void GetFileContent(Stream stream, int id)
{
var result = Query(reader => reader.GetStream(0), id);
foreach (var b in result)
{
b.CopyToAsync(stream);
stream.Flush();
}
}
private IEnumerable<Stream> Query(Func<DbDataReader,Stream> func, int id)
{
var command = // Not shown - SELECT command creating the result set
var reader = command.ExecuteReader();
if (reader.HasRows)
{
while (reader.Read())
{
yield return func(reader);
}
}
reader.Close();
}
Note the use and placement of Thread.Sleep. When I test this in the browser, the content downloads to a file, but without the Sleep, it hangs at the point of completion; with the Sleep, it completes properly, and the resulting download is perfect.
My question: what is the Sleep doing that averts the hang condition? My suspicion is that this is more of a work-around than a proper solution; if so, what should I be doing instead?
Found that it is a mistake on my part. I was incorrectly using CopyToAsync where I should be using CopyTo. Correcting this mistake, it works just fine.
Related
This may not be 100% an EPPlus issue, but since it is Blazor WASM it appears I cannot get the file path to a static image in the wwwroot/images folder. I can get the url and paste it into a browser and that works, even adding that same path to the src attribute of an img works, neither of those helps me.
FYI "background" in this context means a watermark.
It appears that the EPPlus dev team only wants a drive path the file (ex. C:\SomeFolder\SomeFile.png), and I am not seeing how to get that within Blazor WASM. I can get the bytes of the file in c# and even a stream, but no direct path.
My code is the following:
using (var package = new ExcelPackage(fileName))
{
var sheet = package.Workbook.Worksheets.Add(exportModel.OSCode);
sheet.BackgroundImage.SetFromFile("https://localhost:44303/images/Draft.png");
...
}
This returns an exception:
Unhandled exception rendering component: Can't find file /https:/localhost:44303/images/Draft.png
Noticing that leading / I even tried:
sheet.BackgroundImage.SetFromFile("images/Draft.png");
Which returned the same error:
Unhandled exception rendering component: Can't find file /images/Draft.png
So, I am perhaps needing 1 of 2 possible answers:
A way to get a local drive path to the file so the .SetFromFile method is not going to error.
To have a way to set that BackgroundImage property with a byte array or stream of the image. There is this property BackgroundImage.Image but it is readonly.
Thanks to a slap in the face from #Panagiotis-Kanavos I wound up taking the processing out of the client and moving it to the server. With that, I was able to use Static Files to add the watermark with relatively little pain.
In case anyone may need the full solution (which I always find helpful) here it is:
Here is the code within the button click on the Blazor component or page:
private async Task GenerateFile(bool isFinal)
{
...
var fileStream = await excelExportService.ProgramMap(exportModel);
var fileName = "SomeFileName.xlsx";
using var streamRef = new DotNetStreamReference(stream: fileStream);
await jsRuntime.InvokeVoidAsync("downloadFileFromStream", fileName, streamRef);
}
That calls a client-side service that really just passes control over to the server:
public class ExcelExportService : IExcelExportService
{
private const string baseUri = "api/excel-export";
private readonly IHttpService httpService;
public ExcelExportService(IHttpService httpService)
{
this.httpService = httpService;
}
public async Task<Stream> ProgramMap(ProgramMapExportModel exportModel)
{
return await httpService.PostAsJsonForStreamAsync<ProgramMapExportModel>($"{baseUri}/program-map", exportModel);
}
}
Here is the server-side controller that catches the call from the client:
[Route("api/excel-export")]
[ApiController]
public class ExcelExportController : ControllerBase
{
private readonly ExcelExportService excelExportService;
public ExcelExportController(ExcelExportService excelExportService)
{
this.excelExportService = excelExportService;
}
[HttpPost]
[Route("program-map")]
public async Task<Stream> ProgramMap([FromBody] ProgramMapExportModel exportModel)
{
return await excelExportService.ProgramMap(exportModel);
}
}
And that in-turn calls the server-side service where the magic happens:
public async Task<Stream> ProgramMap(ProgramMapExportModel exportModel)
{
var result = new MemoryStream();
ExcelPackage.LicenseContext = LicenseContext.Commercial;
var fileName = #$"Gets Overwritten";
using (var package = new ExcelPackage(fileName))
{
var sheet = package.Workbook.Worksheets.Add(exportModel.OSCode);
if (!exportModel.IsFinal)
{
var pathToDraftImage = #$"{Directory.GetCurrentDirectory()}\StaticFiles\Images\Draft.png";
sheet.BackgroundImage.SetFromFile(pathToDraftImage);
}
...
sheet.Cells.AutoFitColumns();
package.SaveAs(result);
}
result.Position = 0; // Without this, data does not get written
return result;
}
For some reason, this next method was not needed when doing this on the client-side but now that it is back here, I had to add a method that returned a stream specifically and used the ReadAsStreamAsync instead of ReadAsJsonAsync:
public async Task<Stream> PostAsJsonForStreamAsync<TValue>(string requestUri, TValue value, CancellationToken cancellationToken = default)
{
Stream result = default;
var responseMessage = await httpClient.PostAsJsonAsync(requestUri, value, cancellationToken);
try
{
result = await responseMessage.Content.ReadAsStreamAsync(cancellationToken: cancellationToken);
}
catch (HttpRequestException e)
{
...
}
return result;
}
Lastly, in order for it to give the end-user a download link, this was used (taken from the Microsoft Docs):
window.downloadFileFromStream = async (fileName, contentStreamReference) => {
const arrayBuffer = await contentStreamReference.arrayBuffer();
const blob = new Blob([arrayBuffer]);
const url = URL.createObjectURL(blob);
const anchorElement = document.createElement("a");
anchorElement.href = url;
anchorElement.download = fileName ?? "";
anchorElement.click();
anchorElement.remove();
URL.revokeObjectURL(url);
}
I've been working for a few days on a performance problem.
Before I delve deeper I want to quickly explain how the specific service work.
I have a main Service that get a request and send requests to other micro-services but the single Entry Point for the user is to Main service, I thinks is more simple to understand with this image:
After the Main service get request from API he do some logic, query the Db and then get a list, every item on the list has Id, to get enrichment about every item the main service create request to one of the micro-service.
For example John request main service, main service get from Db a list of 90 items then the main service will create 90 calls to micro service and return to John single response that include 90 items.
Now the question is only about the right way to create async call to micro service.
This how I develop this part:
GetDetailsAsync(Id, result.Items, request.SystemComponentId);
private static void GetDetailsAsync(string Id, List<MainItem> items, int systemId)
{
var getDetailsTasks = new List<Task>();
foreach (MainItem single in items)
{
getDetailsTasks.Add(SetSingleDetailsAsync(Id, single, systemId));
}
Task.WhenAll(getDetailsTasks);
}
private static async Task SetSingleDetailsAsync(string Id, MainItem single, int systemId)
{
single.ActivityExtendedDetails = await ProcessItemDetailsRequest.GetItemDetailsAsync(Id, single.TypeId,
single.ItemId, systemId);
}
public static Task<JObject> GetItemDetailsAsync(string id, short type,
string itemId, int systemId)
{
var typeList = ActivityTypeDetails.GetActivityTypes();
var url = GetActivityUrl(id, type, itemId, typeList);
if (url == null)
{
throw new Failure($"No url defined for type {type}");
}
try
{
JObject res;
using (var stream = client.GetStreamAsync(url).Result)
using (var sr = new StreamReader(stream))
using (var reader = new JsonTextReader(sr))
{
var serializer = new JsonSerializer();
res = serializer.Deserialize<JObject>(reader);
}
return Task.FromResult(res);
}
catch(Exception ex)
{
Logger.Warn(
$"The uri {url} threw exception {ex.Message}.");
//[Todo]throw exception
return null;
}
}
This code run and the result is not good enough, the CPU rises very quickly and becomes very high, I think that I has a problem on GetItemDetailsAsync func because I use client.GetStreamAsync(url).Result
when using .Result it's block until the task is completed.
So I do some minor change on GetItemDetailsAsync to try to be really async:
public static async Task<JObject> GetItemDetailsAsync(string id, short type,
string itemId, int systemId)
{
var typeList = ActivityTypeDetails.GetActivityTypes();
var url = GetActivityUrl(id, type, itemId, typeList);
if (url == null)
{
throw new Failure($"No url defined for type {type}");
}
try
{
JObject res;
using (var stream = await client.GetStreamAsync(url))
using (var sr = new StreamReader(stream))
using (var reader = new JsonTextReader(sr))
{
var serializer = new JsonSerializer();
res = serializer.Deserialize<JObject>(reader);
}
return res;
}
catch(Exception ex)
{
Logger.Warn(
$"The uri {url} threw exception {ex.Message}.");
//[Todo]throw exception
return null;
}
}
But now I get null where I supposed to get the data that come from Async function.
I try to debugging and I noticed something weird, everything happen likes as I would expect: the methods was called, request to micro-service was executed and get response but the response from the End-Point(which is found on main-service) return before the async method return from micro-service, that cause that I get null instead of my expected data.
I thinks that maybe I don't use correctly async\await and would be happy if anyone could explain how this behavior happens
I don't know if it's a bug but i'm unable to get raw request on server side.
Consider following controller method:
[AllowAnonymous]
[Route("api/sayHello")]
[HttpPost]
public string SayHello([FromBody] string userName)
{
return $"Hello, {userName}.";
}
I call it via cUrl:
curl -X POST 'https://localhost:809/api/sayHello' --insecure -d "=userName"
It works fine.
Now I'm trying to add some logging. I add a global filter which is doing following:
public async Task LogFilterAction(HttpActionContext context)
{
if (context == null)
throw new ArgumentNullException(nameof(context));
LogFilterAction(context.ActionDescriptor?.ControllerDescriptor?.ControllerType,
context.ActionDescriptor?.ActionName,
context.Request?.RequestUri,
await GetDataAsString(context.Request?.Content),
context.Response?.StatusCode
);
}
private static async Task<string> GetDataAsString(HttpContent content)
{
if (content == null)
return null;
var contentBytes = await content.ReadAsByteArrayAsync();
return Encoding.UTF8.GetString(contentBytes);
}
But here is the problem: for unknown reason reason contentBytes are always an empty array. I see that it's actual length is 9 (it's length of =userName string)
Or Even
As you can see, ASP.Net has successfully request arguments, however, it doesn't return it's in raw manner. Stream has position=0, contentConsumed=false, and everything else is just fine. But i can't read data passed to the controller.
What's wrong here?
ASP.NET Web API reads the content only once, so at the time that you access the content stream, it has already been read and the stream is positioned at its end. Another attempt to read the content will return nothing.
However, in a small sample I was able to reset the stream and read it again:
private async Task<string> GetDataAsString(HttpContent content)
{
if (content == null)
return null;
using (var str = await content.ReadAsStreamAsync())
{
if (str.CanSeek)
str.Seek(0, System.IO.SeekOrigin.Begin);
using (var rdr = new StreamReader(str))
{
return rdr.ReadToEnd();
}
}
}
However, in order to avoid side effects, you might want to consider to use the ActionArguments property of the ActionContext. You can use this property to retrieve the values that are handed to the action for logging. This does not interfere with the internal plumbing of ASP.NET Web API.
I suspect I have a deadlock issue, but it's an odd one that I can't rationalize. I have an API that needs to verify a few things in order to process the call. As part of the business logic, I might have to make more of those same calls as well. In this case, if a particular piece of data associated with an entity is not found, we attempt to use a backup (if one is configured), which requires checking other entities. Eventually, the code will hang.
Let's just dive into the code (comments highlight the calls in question).
API Controller:
public async Task<HttpResponseMessage> Get(int entityID, string content, bool? useBackUp = true)
{
//Some look-ups here, no issues at all
//This works, but it's this method that has an issue later in the process.
SystemEntity entityObj =
await BusinessLayer.GetSystemEntityAsync(SystemEntityID);
if (entityObj == null)
{
return new HttpResponseMessage
{
StatusCode = System.Net.HttpStatusCode.BadRequest,
Content = new StringContent("Entity is unavailable.")
};
}
string text = BusinessLayer.GetContentTextAsync(entityID
new List<string> {contentName}, useBackUp).Result.FirstOrDefault().Value;
if (text == null)
{
return new HttpResponseMessage {StatusCode = System.Net.HttpStatusCode.NoContent};
}
return new HttpResponseMessage
{
StatusCode = System.Net.HttpStatusCode.OK,
Content = new StringContent(text)
};
}
Business Layer:
public async Task<Dictionary<string, string>> GetContentTextAsync(int systemEntityID, List<string> contentNames, bool useBackUp)
{
Dictionary<string, string> records = new Dictionary<string, string>();
//We iterate for caching purposes
foreach (string name in contentNames)
{
string nameCopy = name;
string record = Cache.GetData(
string.Format("{0}_{1}_{2}", CONTENT, systemEntityID, name), () =>
DataLayer.GetCotnent(systemEntityID, nameCopy));
if (record == null && useBackUp)
{
List<int> entityIDs = new List<int> {systemEntityID};
int currentEntityID = systemEntityID;
//Here's that method again. This call seems to work.
SystemEntity currentEntity = await GetSystemEntityAsync(systemEntityID);
if (currentEntity != null && currentEntity.BackUpID.HasValue)
{
currentEntityID = (int) currentEntity.BackUpID;
}
while (!entityIDs.Contains(currentEntityID))
{
int id = currentEntityID;
record = Cache.GetData(
string.Format("{0}_{1}_{2}", CONTENT, systemEntityID, name), () =>
DataLayer.GetCotnent(id, nameCopy));
if (record != null) break;
entityIDs.Add(currentEntityID);
//This call seems to cause the deadlock
currentEntity = await GetSystemEntityAsync(currentEntityID);
if (currentEntity != null && currentEntity.BackUpID.HasValue)
{
currentEntityID = (int) currentEntity.UseBackupID;
}
}
}
if (record != null)
{
records.Add(name, record);
}
}
return records;
}
public async Task<SystemEntity> GetSystemEntityAsync(int systemEntityID)
{
SystemEntity systemEntity = await DataLayer.GetSystemEntity(
scc => scc.SystemEntityID == systemEntityID);
return systemEntity;
}
Data Layer:
public async Task<SystemEntity> GetSystemEntity(Expression<Func<SystemEntity, bool>> whereExpression)
{
using (EntityContext dbContext = createDbInstance())
{
//This is the last line that the debugger in VS 2013 brings me to. Stepping into this returns to whatever called the API method, waiting endlessly.
return await
dbContext.SystemEntity.Include(sc => sc.OtherEntity).Where(whereExpression).FirstOrDefaultAsync();
}
}
To recap: I call GetSystemEntityAsync three times. The first two times, it completes successfully. The third time, it hangs. If I comment out the first two calls so they don't run at all, the third one still hangs. If I remove the await and use just a normal FirstOrDefault in the return statement of the data layer method, then everything completes just fine.
Note: I have to keep the GetSystemEntityAsync method asynchronous. I cannot alter it to be synchronous.
What are the possible sources of the deadlock I'm encountering? I'm out of ideas on how to solve it.
Which one of these async calls is not like the other?
This one, I suspect:
string text = BusinessLayer.GetContentTextAsync(entityID
new List<string> {contentName}, useBackUp).Result.FirstOrDefault().Value;
Try changing it to this:
string text = (await BusinessLayer.GetContentTextAsync(entityID
new List<string> {contentName}, useBackUp)).FirstOrDefault().Value;
The possible source of the deadlock is described by Stephen Cleary in his "Don't Block on Async Code" blog post.
I have a class "Image" with three properties: Url, Id, Content.
I have a list of 10 such images.
This is a silverlight app.
I want to create a method:
IObservable<Image> DownloadImages(List<Image> imagesToDownload)
{
//start downloading all images in imagesToDownload
//OnImageDownloaded:
image.Content = webResponse.Content
yield image
}
This method starts downloading all 10 images in parallel.
Then, when each downloads completes, it sets the Image.Content to the WebResponse.Content of that download.
The result should be an IObservable stream with each downloaded image.
I'm a beginner in RX, and I think what I want can be achieved with ForkJoin, but that's in an experimental release of reactive extensions dll which I don't want to use.
Also I really don't like download counting on callbacks to detect that all images have been downloaded and then call onCompleted().
Doesn't seem to be in the Rx spirit to me.
Also I post what solution I've coded so far, though I don't like my solution because its long/ugly and uses counters.
return Observable.Create((IObserver<Attachment> observer) =>
{
int downloadCount = attachmentsToBeDownloaded.Count;
foreach (var attachment in attachmentsToBeDownloaded)
{
Action<Attachment> action = attachmentDDD =>
this.BeginDownloadAttachment2(attachment).Subscribe(imageDownloadWebResponse =>
{
try
{
using (Stream stream = imageDownloadWebResponse.GetResponseStream())
{
attachment.FileContent = stream.ReadToEnd();
}
observer.OnNext(attachmentDDD);
lock (downloadCountLocker)
{
downloadCount--;
if (downloadCount == 0)
{
observer.OnCompleted();
}
}
} catch (Exception ex)
{
observer.OnError(ex);
}
});
action.Invoke(attachment);
}
return () => { }; //do nothing when subscriber disposes subscription
});
}
Ok, I did manage it to make it work in the end based on Jim's answer.
var obs = from image in attachmentsToBeDownloaded.ToObservable()
from webResponse in this.BeginDownloadAttachment2(image).ObserveOn(Scheduler.ThreadPool)
from responseStream in Observable.Using(webResponse.GetResponseStream, Observable.Return)
let newImage = setAttachmentValue(image, responseStream.ReadToEnd())
select newImage;
where setAttachmentValue just takes does `image.Content = bytes; return image;
BeginDownloadAttachment2 code:
private IObservable<WebResponse> BeginDownloadAttachment2(Attachment attachment)
{
Uri requestUri = new Uri(this.DownloadLinkBaseUrl + attachment.Id.ToString();
WebRequest imageDownloadWebRequest = HttpWebRequest.Create(requestUri);
IObservable<WebResponse> imageDownloadObservable = Observable.FromAsyncPattern<WebResponse>(imageDownloadWebRequest.BeginGetResponse, imageDownloadWebRequest.EndGetResponse)();
return imageDownloadObservable;
}
How about we simplify this a bit. Take your image list and convert it to an observable. Next, consider using the Observable.FromAsyncPattern to manage the service requests. Finally use SelectMany to coordinate the request with the response. I'm making some assumptions on how you are getting the file streams here. Essentially if you can pass in the BeginInvoke/EndInvoke delegates into FromAsyncPattern for your service request you are good.
var svcObs = Observable.FromAsyncPattern<Stream>(this.BeginDownloadAttachment2, This.EndDownloadAttchment2);
var obs = from image in imagesToDownload.ToObservable()
from responseStream in svcObs(image)
.ObserveOnDispatcher()
.Do(response => image.FileContent = response.ReadToEnd())
select image;
return obs;