Parallel HttpWebRequests with Reactive Extensions - c#

I have a class "Image" with three properties: Url, Id, Content.
I have a list of 10 such images.
This is a silverlight app.
I want to create a method:
IObservable<Image> DownloadImages(List<Image> imagesToDownload)
{
//start downloading all images in imagesToDownload
//OnImageDownloaded:
image.Content = webResponse.Content
yield image
}
This method starts downloading all 10 images in parallel.
Then, when each downloads completes, it sets the Image.Content to the WebResponse.Content of that download.
The result should be an IObservable stream with each downloaded image.
I'm a beginner in RX, and I think what I want can be achieved with ForkJoin, but that's in an experimental release of reactive extensions dll which I don't want to use.
Also I really don't like download counting on callbacks to detect that all images have been downloaded and then call onCompleted().
Doesn't seem to be in the Rx spirit to me.
Also I post what solution I've coded so far, though I don't like my solution because its long/ugly and uses counters.
return Observable.Create((IObserver<Attachment> observer) =>
{
int downloadCount = attachmentsToBeDownloaded.Count;
foreach (var attachment in attachmentsToBeDownloaded)
{
Action<Attachment> action = attachmentDDD =>
this.BeginDownloadAttachment2(attachment).Subscribe(imageDownloadWebResponse =>
{
try
{
using (Stream stream = imageDownloadWebResponse.GetResponseStream())
{
attachment.FileContent = stream.ReadToEnd();
}
observer.OnNext(attachmentDDD);
lock (downloadCountLocker)
{
downloadCount--;
if (downloadCount == 0)
{
observer.OnCompleted();
}
}
} catch (Exception ex)
{
observer.OnError(ex);
}
});
action.Invoke(attachment);
}
return () => { }; //do nothing when subscriber disposes subscription
});
}
Ok, I did manage it to make it work in the end based on Jim's answer.
var obs = from image in attachmentsToBeDownloaded.ToObservable()
from webResponse in this.BeginDownloadAttachment2(image).ObserveOn(Scheduler.ThreadPool)
from responseStream in Observable.Using(webResponse.GetResponseStream, Observable.Return)
let newImage = setAttachmentValue(image, responseStream.ReadToEnd())
select newImage;
where setAttachmentValue just takes does `image.Content = bytes; return image;
BeginDownloadAttachment2 code:
private IObservable<WebResponse> BeginDownloadAttachment2(Attachment attachment)
{
Uri requestUri = new Uri(this.DownloadLinkBaseUrl + attachment.Id.ToString();
WebRequest imageDownloadWebRequest = HttpWebRequest.Create(requestUri);
IObservable<WebResponse> imageDownloadObservable = Observable.FromAsyncPattern<WebResponse>(imageDownloadWebRequest.BeginGetResponse, imageDownloadWebRequest.EndGetResponse)();
return imageDownloadObservable;
}

How about we simplify this a bit. Take your image list and convert it to an observable. Next, consider using the Observable.FromAsyncPattern to manage the service requests. Finally use SelectMany to coordinate the request with the response. I'm making some assumptions on how you are getting the file streams here. Essentially if you can pass in the BeginInvoke/EndInvoke delegates into FromAsyncPattern for your service request you are good.
var svcObs = Observable.FromAsyncPattern<Stream>(this.BeginDownloadAttachment2, This.EndDownloadAttchment2);
var obs = from image in imagesToDownload.ToObservable()
from responseStream in svcObs(image)
.ObserveOnDispatcher()
.Do(response => image.FileContent = response.ReadToEnd())
select image;
return obs;

Related

Add BackgroundImage with EPPlus only allows path but cannot get path in Blazor WASM

This may not be 100% an EPPlus issue, but since it is Blazor WASM it appears I cannot get the file path to a static image in the wwwroot/images folder. I can get the url and paste it into a browser and that works, even adding that same path to the src attribute of an img works, neither of those helps me.
FYI "background" in this context means a watermark.
It appears that the EPPlus dev team only wants a drive path the file (ex. C:\SomeFolder\SomeFile.png), and I am not seeing how to get that within Blazor WASM. I can get the bytes of the file in c# and even a stream, but no direct path.
My code is the following:
using (var package = new ExcelPackage(fileName))
{
var sheet = package.Workbook.Worksheets.Add(exportModel.OSCode);
sheet.BackgroundImage.SetFromFile("https://localhost:44303/images/Draft.png");
...
}
This returns an exception:
Unhandled exception rendering component: Can't find file /https:/localhost:44303/images/Draft.png
Noticing that leading / I even tried:
sheet.BackgroundImage.SetFromFile("images/Draft.png");
Which returned the same error:
Unhandled exception rendering component: Can't find file /images/Draft.png
So, I am perhaps needing 1 of 2 possible answers:
A way to get a local drive path to the file so the .SetFromFile method is not going to error.
To have a way to set that BackgroundImage property with a byte array or stream of the image. There is this property BackgroundImage.Image but it is readonly.
Thanks to a slap in the face from #Panagiotis-Kanavos I wound up taking the processing out of the client and moving it to the server. With that, I was able to use Static Files to add the watermark with relatively little pain.
In case anyone may need the full solution (which I always find helpful) here it is:
Here is the code within the button click on the Blazor component or page:
private async Task GenerateFile(bool isFinal)
{
...
var fileStream = await excelExportService.ProgramMap(exportModel);
var fileName = "SomeFileName.xlsx";
using var streamRef = new DotNetStreamReference(stream: fileStream);
await jsRuntime.InvokeVoidAsync("downloadFileFromStream", fileName, streamRef);
}
That calls a client-side service that really just passes control over to the server:
public class ExcelExportService : IExcelExportService
{
private const string baseUri = "api/excel-export";
private readonly IHttpService httpService;
public ExcelExportService(IHttpService httpService)
{
this.httpService = httpService;
}
public async Task<Stream> ProgramMap(ProgramMapExportModel exportModel)
{
return await httpService.PostAsJsonForStreamAsync<ProgramMapExportModel>($"{baseUri}/program-map", exportModel);
}
}
Here is the server-side controller that catches the call from the client:
[Route("api/excel-export")]
[ApiController]
public class ExcelExportController : ControllerBase
{
private readonly ExcelExportService excelExportService;
public ExcelExportController(ExcelExportService excelExportService)
{
this.excelExportService = excelExportService;
}
[HttpPost]
[Route("program-map")]
public async Task<Stream> ProgramMap([FromBody] ProgramMapExportModel exportModel)
{
return await excelExportService.ProgramMap(exportModel);
}
}
And that in-turn calls the server-side service where the magic happens:
public async Task<Stream> ProgramMap(ProgramMapExportModel exportModel)
{
var result = new MemoryStream();
ExcelPackage.LicenseContext = LicenseContext.Commercial;
var fileName = #$"Gets Overwritten";
using (var package = new ExcelPackage(fileName))
{
var sheet = package.Workbook.Worksheets.Add(exportModel.OSCode);
if (!exportModel.IsFinal)
{
var pathToDraftImage = #$"{Directory.GetCurrentDirectory()}\StaticFiles\Images\Draft.png";
sheet.BackgroundImage.SetFromFile(pathToDraftImage);
}
...
sheet.Cells.AutoFitColumns();
package.SaveAs(result);
}
result.Position = 0; // Without this, data does not get written
return result;
}
For some reason, this next method was not needed when doing this on the client-side but now that it is back here, I had to add a method that returned a stream specifically and used the ReadAsStreamAsync instead of ReadAsJsonAsync:
public async Task<Stream> PostAsJsonForStreamAsync<TValue>(string requestUri, TValue value, CancellationToken cancellationToken = default)
{
Stream result = default;
var responseMessage = await httpClient.PostAsJsonAsync(requestUri, value, cancellationToken);
try
{
result = await responseMessage.Content.ReadAsStreamAsync(cancellationToken: cancellationToken);
}
catch (HttpRequestException e)
{
...
}
return result;
}
Lastly, in order for it to give the end-user a download link, this was used (taken from the Microsoft Docs):
window.downloadFileFromStream = async (fileName, contentStreamReference) => {
const arrayBuffer = await contentStreamReference.arrayBuffer();
const blob = new Blob([arrayBuffer]);
const url = URL.createObjectURL(blob);
const anchorElement = document.createElement("a");
anchorElement.href = url;
anchorElement.download = fileName ?? "";
anchorElement.click();
anchorElement.remove();
URL.revokeObjectURL(url);
}

Listen for only an image in discord.net

I have an issue in which i'm looking for our discord bot to only look for images received and ignore any text typed.
From the guides I have read, I have yet to come across any that hasn't required a command.
I have tried to use a command with no command within the string, however it doesn't build as it doesn't contain a parameter.
Does anyone have any ideas how I can just listen for an image only?
Below is an example of my code.
private async Task _client_MessageReceived(SocketMessage arg)
{
var message = arg as SocketUserMessage;
var context = new SocketCommandContext(_client, message);
if (message.Author.IsBot) return;
int argPos = 0;
if (message.HasStringPrefix("!", ref argPos) || message.Attachments.Count > 0)
{
var result = await _commands.ExecuteAsync(context, argPos, _services);
if (!result.IsSuccess) Console.WriteLine(result.ErrorReason);
}
else
await message.DeleteAsync();
}
[Command("")]
public async Task Photo()
{
var attachments = Context.Message.Attachments;
WebClient myWebClient = new WebClient();
string file = attachments.ElementAt(0).Filename;
string url = attachments.ElementAt(0).Url;
myWebClient.DownloadFile(url, #"mydirect");
_ = Task.Run(async () =>
{
AWS.AWS.Get_kv_map(#"mydirect");
});
}
my sugestion is to check if message.Attachments != 0 ==> do your thing, with that you can check if it has any Attachments is on it and after that you can check if its ending on an .jpg or .png or so.
example:
if(message.Attachments.Count != 0){
var image attachements = message.Attachments.Where(x =>
x.Filename.EndsWith(".jpg") || x.Filename.EndsWith(".png") ||
x.Filename.EndsWith(".gif")); // or what you want as "image"
if(image.Any()){
// do your stuff from your method Photo() here or just call here your method your decision
}else{
// ignore or whatever you want to do it with it
}
I hope it helped and good luck on your project :D
var image = message.Attachments.Where(x => x.Filename.EndsWith("*.png") ...);
This code will be one of the great solution but If you want to check that's real image then.
First, download image.
Second check magic number.
https://en.wikipedia.org/wiki/List_of_file_signatures
This wikipedia link has list of magic number, for example, MZ means PE file.
If byte of file not starts with MZ, windows will deny of execution.

Why is this Sleep needed when using PushStreamContent?

Referring to this sample code on GitHub:
https://github.com/DblV/StreamingWebApi/blob/master/StreamingService/StreamingService/Controllers/StreamingController.cs
I want to stream content stored in a database, which my query returns as a sequence of blobs (essentially one file split into "blocks"). Due to the potential size of the complete response, I want to stream it, and I am following the above example as follows:
public class FileController : ApiController
{
[HttpGet]
public HttpResponseMessage Get(string id, [FromUri] string contentType)
{
var message = new HttpResponseMessage(HttpStatusCode.OK);
message.Content = new PushStreamContent((stream, content, context) =>
{
GetFileContent(stream, int.Parse(id));
Thread.Sleep(1000);
stream.Close();
});
message.Content.Headers.ContentType = new MediaTypeHeaderValue(contentType);
return message;
}
private void GetFileContent(Stream stream, int id)
{
var result = Query(reader => reader.GetStream(0), id);
foreach (var b in result)
{
b.CopyToAsync(stream);
stream.Flush();
}
}
private IEnumerable<Stream> Query(Func<DbDataReader,Stream> func, int id)
{
var command = // Not shown - SELECT command creating the result set
var reader = command.ExecuteReader();
if (reader.HasRows)
{
while (reader.Read())
{
yield return func(reader);
}
}
reader.Close();
}
Note the use and placement of Thread.Sleep. When I test this in the browser, the content downloads to a file, but without the Sleep, it hangs at the point of completion; with the Sleep, it completes properly, and the resulting download is perfect.
My question: what is the Sleep doing that averts the hang condition? My suspicion is that this is more of a work-around than a proper solution; if so, what should I be doing instead?
Found that it is a mistake on my part. I was incorrectly using CopyToAsync where I should be using CopyTo. Correcting this mistake, it works just fine.

Amazon S3 - Transfer utility won't work in paralel

I'm trying to upload multiple files to S3 using the TransferUtility class from Amazon SDK for .NET
My thinking was, that since the SDK doesn't allow to upload multiple files form different folders at once, I'd create multiple threads and upload there, but it looks like Amazon SDK has some kind of checking against this, since I don't notice any parallel execution of my uploading method.
Here is the pseudo-code I'm using:
int totalUploaded = 0;
foreach (var dItem in Detection.Items.AsParallel())
{
UploadFile(dItem);
totalUploaded++;
Action a = () => { lblStatus.Text = $"Uploaded {totalUploaded} from {Detection.ItemsCount}"; };
BeginInvoke(a);
}
I'm using .AsParallel to spawn multiple threads. My CPU (i7-5930K) has 6 cores and supports multi-threading, so AsParallel must spawn more threads as needed.
And here is the upload method
private void UploadFile(Detection.Item item)
{
Debug.WriteLine("Enter " + item.FileInfo);
Interlocked.Increment(ref _threadsCount);
...
using (var client = AmazonS3Client)
{
....
// if we are here we need to upload
TransferUtilityUploadRequest request = new TransferUtilityUploadRequest
{
Key = s3Key,
BucketName = settings.BucketName,
CannedACL = S3CannedACL.PublicRead,
FilePath = item.FileInfo.FullName,
ContentType = "image/png",
};
TransferUtility utility = new TransferUtility(client);
utility.Upload(request);
}
}
Сan't see what can be wrong here? Any idea highly appreciated.
Thx
The problem in your code is that you are only constructing the ParallelEnumerable, but treat it like a simple IEnumerable:
foreach (var dItem in Detection.Items.AsParallel())
This part of code simply iterates over collection. In case you want the parallel executing, you have to use the extension method ForAll():
Detection.Items.AsParallel().ForAll(dItem =>
{
//Do parallel stuff here
});
Also you can simply use the Parallel class:
Parallel.ForEach(Detection.Items, dItem =>
{
//Do parallel stuff here
});

NSAutoReleasePool and async functions

I have a function in my program that creates new widgets to represent data, however whenever a widget is created i get alot of "AutoRelease with no NSAutoReleasePool in place" error messages. Since an NSAutoReleasePool should be automatically created on the main thread, I have an inkling that these error messages appear because an async function might create my threads...
This is the function called to create widgets to represent the latest information. This function is called pretty often:
private void CreateAndDisplayTvShowWidget (TvShow show)
{
var Widget = new TvShowWidgetController (show);
Widget.OnRemoveWidget += ConfirmRemoveTvShow;
Widget.View.SetFrameOrigin (new PointF (0, -150));
Widget.View.SetFrameSize (new SizeF (ContentView.Frame.Width, 150));
ContentView.AddSubview (Widget.View);
show.ShowWidget = Widget;
}
This function is usually called when this async function returns:
private static void WebRequestCallback (IAsyncResult result)
{
HttpWebRequest request = (HttpWebRequest)result.AsyncState;
HttpWebResponse response = (HttpWebResponse)request.EndGetResponse (result);
StreamReader responseStream = new StreamReader (response.GetResponseStream ());
string responseString = responseStream.ReadToEnd ();
responseStream.Close ();
ProcessResponse (responseString, request);
}
ProcessResponse (responseString, request) looks like this:
private static void ProcessResponse (string responseString, HttpWebRequest request)
{
string requestUrl = request.Address.ToString ();
if (requestUrl.Contains (ShowSearchTag)) {
List<TvShow> searchResults = TvDbParser.ParseTvShowSearchResults (responseString);
TvShowSearchTimeoutClock.Enabled = false;
OnTvShowSearchComplete (searchResults);
} else if (requestUrl.Contains (MirrorListTag)) {
MirrorList = TvDbParser.ParseMirrorList (responseString);
SendRequestsOnHold ();
} else if (requestUrl.Contains (TvShowBaseTag)) {
TvShowBase showBase = TvDbParser.ParseTvShowBase (responseString);
OnTvShowBaseRecieved (showBase);
} else if (requestUrl.Contains (ImagePathReqTag)) {
string showID = GetShowIDFromImagePathRequest (requestUrl);
TvShowImagePath imagePath = TvDbParser.ParseTvShowImagePath (showID, responseString);
OnTvShowImagePathRecieved (imagePath);
}
}
CreateAndDisplayTvShowWidget (TvShow show) is called when the event OnTvShowBaseRecieved (TvShow) is called, which is when I get tons error messages regarding NSAutoReleasePool...
The last two functions are part of what is supposed to be a cross-platform assembly, so I can't have any MonoMac-specific code in there...
I never call any auto-release or release code for my widgets, so I assume that the MonoMac bindings does this automatically as part of its garbage collection?
You can create autorelease pools at point within the call stack, you can even have multiple nested autorelease pools with the same call stack. So you should be able to create your autorelease pools in the async entry functions.
You only need an NSAutoreleasePool if you use the auto-release features of objects. A solution is to create a NSAutoreleasePool around the code that manipulates auto-released objects (in the async callback).
Edit:
Have you tried to encapsulate the creation code with a NSAutoreleasePool ? As this is the only place where you call MonoMac code, this should solve the issue.
private void CreateAndDisplayTvShowWidget (TvShow show)
{
using(NSAutoreleasePool pool = new NSAutoreleasePool())
{
var Widget = new TvShowWidgetController (show);
Widget.OnRemoveWidget += ConfirmRemoveTvShow;
Widget.View.SetFrameOrigin (new PointF (0, -150));
Widget.View.SetFrameSize (new SizeF (ContentView.Frame.Width, 150));
ContentView.AddSubview (Widget.View);
show.ShowWidget = Widget;
}
}
Note that even if you don't use auto-released objects directly, there are some case where the Cococa API use them udner the hood.
I had a similar problem and it was the response.GetResponseStream that was the problem. I surrounded this code with...
using (NSAutoreleasePool pool = new NSAutoreleasePool()) {
}
... and that solved my problem.

Categories