We Keep Coding

Delayed rendering of the image - Skia ASP

I'm trying to draw a set of images with SkiaSharp, convert them to byte array, and send to HttpContext stream. For some reason, every image is being displayed in the browser only when the next image has been rendered and sent to the HTTP stream.
After rendering the 1st image, browser shows empty white area, no image
After rendering the 2nd image, browser shows 1st image
After rendering the 3rd image, browser shows 2nd image, and so on
Here is the Web API controller. Uncommenting second drawShapes call will make it work, but I would like to understand why image rendering is delayed. Maybe I just don't call some important method, like Flush or something similar?
Also, this is not an ASP issue, because I tried rendering images with System.Drawing.Common and images were rendered in real time without delay.
Does anybody know what is missing in this method?
[Route("/source")]
public async Task Get()
{
// Initialize Skia
var res = new byte[1000];
var boundary = Guid.NewGuid().ToString();
var generator = new Random();
var map = new SKBitmap(250, 250);
var canvas = new SKCanvas(map);
var inPaint = new SKPaint
{
Color = SKColors.Black,
Style = SKPaintStyle.Fill,
FilterQuality = SKFilterQuality.High
};
var exPaint = new SKPaint
{
Color = SKColors.White,
Style = SKPaintStyle.Fill,
FilterQuality = SKFilterQuality.Low
};
// Create HTTP stream
Response.ContentType = "multipart/x-mixed-replace;boundary=" + boundary;
var outputStream = Response.Body;
var cancellationToken = Request.HttpContext.RequestAborted;
// Create method posting images to HTTP stream
async Task drawShapes()
{
using (var image = SKImage.FromBitmap(map))
{
var pos = generator.Next(50, 150);
canvas.DrawRect(0, 0, 250, 250, exPaint);
canvas.DrawCircle(pos, pos, 20, inPaint);
res = image.Encode(SKEncodedImageFormat.Webp, 100).ToArray();
}
var header = $"--{ boundary }\r\nContent-Type: image/webp\r\nContent-Length: { res.Length }\r\n\r\n";
var headerData = Encoding.ASCII.GetBytes(header);
await outputStream.WriteAsync(headerData);
await outputStream.WriteAsync(res);
await outputStream.WriteAsync(Encoding.ASCII.GetBytes("\r\n"));
}
// Start HTTP stream
await Task.Run(async () =>
{
//while (true)
{
await drawShapes();
//await drawShapes(); // After uncommenting this shows the image (the previous one)
}
});
}
Reproducible sample

This was answered in SkiaSharp discussions.
https://github.com/mono/SkiaSharp/discussions/1975
Possible solutions are to make sure to Encode image only after all drawings are done or to call Encode on bitmap instead of image.
The simplest one is to replace this line.
//res = image.Encode(SKEncodedImageFormat.Webp, 100).ToArray(); // wrong
res = map.Encode(SKEncodedImageFormat.Webp, 100).ToArray(); // correct

How to downscale the pixel values in UWP App (C#)?

I built a UWP App based on the Squeeze Net example provided in the repository (C#) that uses a Deep Learning model (ONNX) for image classification. I have built the deep learning model in PyTorch where the pixel values of the image have been scaled down from the range [0, 255] to [0, 1] and then normalized with channel wise (RGB) standard deviation and mean. So, this model expects the pixel values other than [0, 255] range.
But in the UWP App, I'm unable to perform this downscaling of the pixel values before binding the inputs to the model. I have searched the SoftwareBitmap class but couldn't find a way to perform this downscaling operation. Any help would be very very appreciated.
I need this operation somewhere in between these lines of code.
await LoadModel();
// Trigger file picker to select an image file
var picker = new FileOpenPicker();
picker.ViewMode = PickerViewMode.Thumbnail;
picker.SuggestedStartLocation = PickerLocationId.PicturesLibrary;
picker.FileTypeFilter.Add(".jpg");
picker.FileTypeFilter.Add(".jpeg");
picker.FileTypeFilter.Add(".png");
StorageFile file = await picker.PickSingleFileAsync();
outputTextBlock.Text = $"The selected Image: {file.Name}";
SoftwareBitmap softwareBitmap;
using (IRandomAccessStream stream = await file.OpenAsync(FileAccessMode.Read))
{
// Create the decoder from the stream
BitmapDecoder decoder = await BitmapDecoder.CreateAsync(stream);
PixelDataProvider random = await decoder.GetPixelDataAsync();
// byte[] pD = random.DetachPixelData();
//await FileIO.WriteBytesAsync("path/file.ext", pD);
// System.IO.File.WriteAllBytes("path/file.ext", pD);
// byteData.Text = $"{pD}";
// Get the SoftwareBitmap representation of the file in BGRA8 format
softwareBitmap = await decoder.GetSoftwareBitmapAsync();
softwareBitmap = SoftwareBitmap.Convert(softwareBitmap, BitmapPixelFormat.Bgra8, BitmapAlphaMode.Ignore);
}
var streamD = await file.OpenReadAsync();
var imageSource = new BitmapImage();
await imageSource.SetSourceAsync(streamD);
selectedImage.Source = imageSource;
// Display the image
//SoftwareBitmapSource imageSource = new SoftwareBitmapSource();
//await imageSource.SetBitmapAsync(softwareBitmap);
//selectedImage.Source = imageSource;
// Encapsulate the image within a VideoFrame to be bound and evaluated
VideoFrame inputWoodImage = VideoFrame.CreateWithSoftwareBitmap(softwareBitmap);
await EvaluateVideoFrameAsync(inputWoodImage);

this is a repost of this thread here that contains some answers and is more actual:
https://github.com/Microsoft/Windows-Machine-Learning/issues/22

xamarin.forms save file/image

coding xamarin form project and also using pcl storage. have problem with saving image or file on device all examples i found show how to save stream of byte array into file but non of them show how to turn convert image into usable format for saving.
var webImage = new Image();
webImage.Source = new UriImageSource
{
CachingEnabled = false,
Uri = new Uri("http://server.com/image.jpg"),
};
byte[] image = ????(what i need)????(webImage.Source);
// get hold of the file system
IFolder folder = rootFolder ?? FileSystem.Current.LocalStorage;
// create a file, overwriting any existing file
IFile file = await folder.CreateFileAsync(fileName, CreationCollisionOption.ReplaceExisting);
// populate the file with image data
using (System.IO.Stream stream = await file.OpenAsync(FileAccess.ReadAndWrite))
{
stream.Write(image, 0, image.Length);
}
In other words looking for code to save image/file on device from url.
Tried ffimageloading since it contain converter to byte[], but always get error:"Object reference not set to an instance of an object." for GetImageAsJpgAsync and GetImageAsPngAsync method. problem might be that image isn't fully loaded. but finish event and finish command never get called even trough image is fully loaded on screen and bound to cachedImage.
var cachedImage = new CachedImage()
{
Source = "https://something.jpg",
FinishCommand = new Command(async () => await TEST()),
};
cachedImage.Finish += CachedImage_Finish;
var image = await cachedImage.GetImageAsJpgAsync();

With FFImageLoading (I noticed you use it):
await ImageService.Instance.LoadUrl("https://something.jpg").AsJpegStream();
If you want to get original file location, use this:
await ImageService.Instance.LoadUrl("https://something.jpg").Success((imageInfo) => { //TODO imageInfo.FilePath } ).DownloadOnly();

How to save media element's current frame?

I am developing a media player like vlc. One of it's features is saving snapshot during playback. I am able to successfully save the snapshot of any element using rendertargetbitmap class, using the following c# code.
RenderTargetBitmap renderTargetBitmap = new RenderTargetBitmap();
await renderTargetBitmap.RenderAsync(RenderedGrid, width, height);
RenderedImage.Source = renderTargetBitmap;
But it does not allow to capture media element. Please help me with a workaround. I guess, I am not sure, may it be possible by extracting a thumbnail from a specific position using position of StorageItemThumbnail Class. If even so, it will be a thumbnail not an image of current frame. Someone help please. It is essential feature of my app!

in a digital signage uwp app that I developed I had also a similar requirement. Searching the web on how I can do I found that in UWP you can't directly take a screenshot of a media element due to a protection policy (for example you could create a small app that captures all the frames of a video file without DRM management).
I came up with a workaround: I have the video file locally so I can extract a single frame at a certain point and save it to disk with the same dimensions of the video rendered on screen (in this way I had a full image of the frame to use later). To doing this I wrote this function:
private async Task<string> Capture(StorageFile file, TimeSpan timeOfFrame, Size imageDimension)
{
if (file == null)
{
return null;
}
//Get FrameWidth & FrameHeight
List<string> encodingPropertiesToRetrieve = new List<string>();
encodingPropertiesToRetrieve.Add("System.Video.FrameHeight");
encodingPropertiesToRetrieve.Add("System.Video.FrameWidth");
IDictionary<string, object> encodingProperties = await file.Properties.RetrievePropertiesAsync(encodingPropertiesToRetrieve);
uint frameHeight = (uint)encodingProperties["System.Video.FrameHeight"];
uint frameWidth = (uint)encodingProperties["System.Video.FrameWidth"];
//Get image stream
var clip = await MediaClip.CreateFromFileAsync(file);
var composition = new MediaComposition();
composition.Clips.Add(clip);
var imageStream = await composition.GetThumbnailAsync(timeOfFrame, (int)frameWidth, (int)frameHeight, VideoFramePrecision.NearestFrame);
//Create BMP
var writableBitmap = new WriteableBitmap((int)frameWidth, (int)frameHeight);
writableBitmap.SetSource(imageStream);
//Get stream from BMP
string mediaCaptureFileName = "IMG" + Guid.NewGuid().ToString().Substring(0, 4) + ".jpg";
var saveAsTarget = await CreateMediaFile(mediaCaptureFileName);
Stream stream = writableBitmap.PixelBuffer.AsStream();
byte[] pixels = new byte[(uint)stream.Length];
await stream.ReadAsync(pixels, 0, pixels.Length);
using (var writeStream = await saveAsTarget.OpenAsync(FileAccessMode.ReadWrite))
{
var encoder = await BitmapEncoder.CreateAsync(BitmapEncoder.JpegEncoderId, writeStream);
encoder.SetPixelData(
BitmapPixelFormat.Bgra8,
BitmapAlphaMode.Premultiplied,
(uint)writableBitmap.PixelWidth,
(uint)writableBitmap.PixelHeight,
96,
96,
pixels);
await encoder.FlushAsync();
using (var outputStream = writeStream.GetOutputStreamAt(0))
{
await outputStream.FlushAsync();
}
}
return saveAsTarget.Name;
}
This function save the frame to disk as image and return his filename.
In my application I had more "widget" (some of those with images, some with video) so I used this function to get the video frames and put a new image on top of the media element with the same dimensions and a higher Canvas Z-Index in order to use RenderTargetBitmap for take the final screenshot.
It's not a clean solution but it was a workaround for RenderTargetBitmap not considering MediaElement.
Edit:
I forgot a reference to an internal function in my application so I canghed the previous code and the line var file = await _mediaEngine.CreateMediaFile($"{Guid.NewGuid().ToString()}.jpg"); is now var file = await CreateMediaFile($"{Guid.NewGuid().ToString()}.jpg");
The async function CreateMediaFile(string fileName) simply create a new file on the disk:
public async Task<StorageFile> CreateMediaFile(string filename)
{
StorageFolder _mediaFolder = KnownFolders.PicturesLibrary;
return await _mediaFolder.CreateFileAsync(filename);
}

How to dynamically download and display images in Windows8 metro app

I get image URL at runtime from the xml file reader. This image URL is pass to the below method to download it dynamically.
public void Display_Image(string MyURL)
{
BitmapImage bi = new BitmapImage();
bi.UriSource = new Uri(this.BaseUri, MyURL);
Img_Poster.Source = bi;
}
But this doesn't work. I don't get any image source. The above code works fine with static URL provided at compile time. What do I need to do more?

The method I suggested below is obsolete. However, creating a new Bitmap image created dynamically with a Uri determined at runtime IS supported and working on the RTM build of Windows 8. Display_Image(url) should work as you expect.
You can get the image stream using the CreateFromUri helper: http://msdn.microsoft.com/en-us/library/windows/apps/windows.storage.streams.streamreference.createfromuri.aspx#Y0
var stream = RandomAccessStreamReference.CreateFromUri(new Uri(imageUrl))
You should then be able to set the source of your bitmap to the RandomAccessStream that the helper returns

I've had similar problems with previously-working Bitmap code not working on Windows RT, an early attempt convinces me that it refuses to download anything unless it is going to be displayed on the UI (here, I needed to insert a 1ms delay before assigning sources just to get it to trigger the image download):
var image = .... // reference to animage on the UI
var placeholder = ... // a placeholder BitmapImage
var source = ... // uri to download
image.Source = placeholder;
var src = new BitmapImage(new Uri(source));
src.ImageOpened += (s, e) =>
{
var bi = s as BitmapImage;
image.Source = bi;
};
image.Source = src;
// Delay required to trigger download
await Task.Delay(1);
image.Source = placeholder;
Here's another solution I've tried with success:
var image = .... // reference to animage on the UI
var source = ... // uri to download
var placeholder = ... // a placeholder BitmapImage
image.Source = placeholder;
var bytes = await new HttpClient().GetByteArrayAsync(source);
var img = new BitmapImage();
await img.SetSourceAsync(bytes.AsBuffer().AsStream().AsRandomAccessStream());
image.Source = img;