We Keep Coding

Save video frame in JPEG XR (UWP C#)

I'm new in C# coding and I would like to save a video frame in Jpeg-XR (16-bits image).
I wrote some lines of code that should be close to the solution but I'm only able to correctly save a 8-bits image.
//Use Windows.Media.Editing to get ImageStream
var clip = await MediaClip.CreateFromFileAsync(file);
var composition = new MediaComposition();
composition.Clips.Add(clip);
var imageStream = await composition.GetThumbnailAsync(Position, (int)frameWidth, (int)frameHeight, VideoFramePrecision.NearestFrame);
//generate bitmap
var writableBitmap = new WriteableBitmap((int)frameWidth, (int)frameHeight);
writableBitmap.SetSource(imageStream);
//generate some random name for file in PicturesLibrary
var name = Position.Minutes.ToString() + "_" + Position.Seconds.ToString() + "_"+ Position.Milliseconds.ToString();
var saveAsTarget = await KnownFolders.PicturesLibrary.CreateFileAsync(name + file.DisplayName + ".jxr");
//get stream from bitmap
const int BytesPerPixel = 4;
Stream stream = writableBitmap.PixelBuffer.AsStream();
byte[] pixels = new byte[(int)frameWidth * (int)frameHeight * BytesPerPixel];
await stream.ReadAsync(pixels, 0, pixels.Length);
using (var writeStream = await saveAsTarget.OpenAsync(FileAccessMode.ReadWrite))
{
var encoder = await BitmapEncoder.CreateAsync(BitmapEncoder.JpegXREncoderId, writeStream);
encoder.SetPixelData(
BitmapPixelFormat.Bgra8,
BitmapAlphaMode.Premultiplied,
(uint)writableBitmap.PixelWidth,
(uint)writableBitmap.PixelHeight,
96.0,
96.0,
pixels);
await encoder.FlushAsync();
using (var outputStream = writeStream.GetOutputStreamAt(0))
{
await outputStream.FlushAsync();
}
}
I tried to set the BitmapPixelFormat to Rgba16 but the results is a wrongly encoded image. Any clue on how to solve my problems?
Alternative solutions to my current code are welcome. Thank you :)

You should not need to do that much pixel copying by hand, this example uses the System.Windows.Media.Imaging namespace. I have no idea if that is available in uwp or not:
public static void SaveJxr(this BitmapSource bitmap, Stream stream, byte qualityLevel = 80)
{
var encoder = new WmpBitmapEncoder();
encoder.QualityLevel = qualityLevel;
encoder.Frames.Add(BitmapFrame.Create(bitmap));
encoder.Save(stream);
}
This uses the Windows Media Photo encoder, and but as far as I can tell this is just an older name for Jpeg XR. I'm also fairly sure the produced images was actually 16bits when I tested this. But you also need a tool that can decode 16 bit images, and these do not seem very common.
If you control both the saving and reading side you might consider using JpegLS instead, at least that seem to give better results for mono16 images when I tested it.

How to create a IDirect3DSurface from an IBuffer or byte array in UWP

I want to create a video from a few RenderTargetBitmaps in UWP. I am doing that by using MediaClips.
From RenderTargetBitmap i can get an IBuffer or byte array of pixels.
To create a MediaClip I need either an image file or an IDirect3DSurface.
Creating an image just to create a clip is very expensive, so I thought of using IDirect3DSurface.
How can I do this?
I have tried this:
RenderTargetBitmap renderTargetBitmap = new RenderTargetBitmap();
await renderTargetBitmap.RenderAsync(RenderedGrid, 100, 100);
IBuffer pixels = await renderTargetBitmap.GetPixelsAsync();
var values = Enum.GetValues(typeof(DirectXPixelFormat));
CanvasBitmap bitmap=null;
foreach (DirectXPixelFormat format in values)
{
try
{
videoClip = new MediaComposition();
bitmap = CanvasBitmap.CreateFromBytes(myWidget.Device, pixels, renderTargetBitmap.PixelWidth, renderTargetBitmap.PixelHeight, format);
StorageFile video2 = await storageFolder.CreateFileAsync("video2" + ".mp4", CreationCollisionOption.ReplaceExisting);
MediaClip d = MediaClip.CreateFromSurface(bitmap, DateTime.Now - previousFrame+new TimeSpan(100));
videoClip.Clips.Add(d);
await videoClip.RenderToFileAsync(video2);
break;
}
catch(Exception e)
{
}
}
I try all the formats in DirectXPixelFormat but none works.
I have a CanvasControl named myWidget that is empty.
I create a CanvasBitmap from Ibuffer (CanvasBitmap implements IDirect3DSurface)
Create a Mediaclip from CanvasBitmap
Add it to MediaComposition.
Then I try to render to video file.When i try to save to a file it throws an error
System.Runtime.InteropServices.COMException Stream is not in a state
to handle the request.
EDIT:
I figured out where the problem is, but not why and not how to fix it.
await videoClip.SaveAsync(video2);
videoClip= await MediaComposition.LoadAsync(video2);
var x=await videoClip.RenderToFileAsync(video2);
Now with these three lines i can save the video, but using only the third line it throws the error above. I cannot make sense of it. Why does saving and loading fix the problem??

Most probable reason is that, CanvasBitmap has an underlying IDirce3DSurface object as well as image data like byte[] or something else , though I am not sure about this.
If that's true, then creating a CanvasBitmap from byte[] or IBuffer won't effect the underlying IDirect3DSurface, the image part will be constructed only. You can see that by saving that image on the disk, it gives no error.
But I think there's a workaround if you want to skip saving data on the disk:
You can contruct the underlying IDirect3DSurface if you do Offscreen Drawing to a CanvasRenderTarget.
So, you can use the CanvasBitmap to construct a CanvasRenderTarget and then use that CanvasRenderTarget to contruct a MediaClip:
CanvasRenderTarget rendertarget;
using (CanvasBitmap canvas = CanvasBitmap.CreateFromBytes(CanvasDevice.GetSharedDevice(), pixels, renderTargetBitmap.PixelWidth, renderTargetBitmap.PixelHeight, format))
{
rendertarget = new CanvasRenderTarget(CanvasDevice.GetSharedDevice(), canvas.SizeInPixels.Width, canvas.SizeInPixels.Height, 96);
using (CanvasDrawingSession ds = rendertarget.CreateDrawingSession())
{
ds.Clear(Colors.Black);
ds.DrawImage(canvas);
}
}
MediaClip d = MediaClip.CreateFromSurface(renderTarget, TimeSpan.FromMilliseconds(80));
mc.Clips.Add(m);

The MediaComposition.RenderToFileAsync Method saves the composition to a video file that can be played back with standard media players. From the error info, it seems the stream content is not correct media data and can not be render into a video file directly.
So, to create a video from a few RenderTargetBitmaps in UWP, the way to use an image file should be your choice. using MediaClip.CreateFromImageFileAsync method by saving the RenderTargetBitmap into a file then using it to create a video.
private async void CreateVideoByConvertRenderBitmapToFile()
{
var folder = await ApplicationData.Current.LocalFolder.CreateFolderAsync("Test",
CreationCollisionOption.ReplaceExisting);
var composition = new MediaComposition();
for (int i = 0; i < 5; i++)
{
RenderTargetBitmap render = new RenderTargetBitmap();
await render.RenderAsync(RenderGrid);
MyImage.Source = render;
var pixel = await render.GetPixelsAsync();
var file = await folder.CreateFileAsync("test.png", CreationCollisionOption.GenerateUniqueName);
using (IRandomAccessStream stream = await file.OpenAsync(FileAccessMode.ReadWrite))
{
var logicalDpi = DisplayInformation.GetForCurrentView().LogicalDpi;
var encoder = await BitmapEncoder.CreateAsync(BitmapEncoder.PngEncoderId, stream);
encoder.SetPixelData(
BitmapPixelFormat.Bgra8,
BitmapAlphaMode.Ignore,
(uint)render.PixelWidth,
(uint)render.PixelHeight,
logicalDpi,
logicalDpi,
pixel.ToArray());
await encoder.FlushAsync();
stream.Dispose();
MediaClip clip = await MediaClip.CreateFromImageFileAsync(file, TimeSpan.FromSeconds(3));
composition.Clips.Add(clip);
MyText.Text = "First frame >>>" + i;
}
}
var video = await ApplicationData.Current.LocalFolder.CreateFileAsync("test.mp4",
CreationCollisionOption.ReplaceExisting);
var action = await composition.RenderToFileAsync(video, MediaTrimmingPreference.Precise);
await folder.DeleteAsync();
}

Send InkCanvas signature using WCF call

I have a Windows 10 UWP app that will run on Windows 10 Mobile. A requirement I have is to capture a signature from the user. So far, I am simply using an InkCanvas in XAML and have it wired up to my code behind.
I then have a button that when clicked, will take the signature on the InkCanvas and send it to the server via a WCF call. The server and WCF service is already existing. It takes in the signature image as a base64 serialized string.
I know how to get the base64 once I have either an image or a byte array. However, in my many hours of reading, I am finding that articles/examples were either written for WPF or Windows 8.1 and do not work on Windows 10 UWP. Also, of the examples that I have found that will work, it seems my only option is to save the signature to file as a GIF.
I see that I can call GetStrokes() like this
var strokeCollection = cvsSignature.InkPresenter.StrokeContainer.GetStrokes();
Which will return me a read only list of InkStroke. I guess I can iterate that list and build a byte array? How would I do that? It seems this is not efficient?
Otherwise, I thought I could just change the stream from a file stream to a memory stream but I guess either this is not possible or I am missing something. I am trying this
using (var inkMemStream = new MemoryStream())
{
await cvsSignature.InkPresenter.StrokeContainer.SaveAsync(inkMemStream);
}
But with this type of approach I get an exception that I cannot convert from System.IO.MemoryStream to Windows.Storage.Streams.IOutputStream
Thanks!

I found a way, if you don't want to save your InkCanvas to a file or as a GIF, to manipulate it in memory. I actually have three options below. Note, for some of this, you will need to add a reference to Win2D.uwp, which can be found on NuGet by searching that exact name. It is provided by Microsoft.
Convert the InkCanvas to a byte array:
private byte[] ConvertInkCanvasToByteArray()
{
//First, we need to get all of the strokes in the canvas
var canvasStrokes = myCanvas.InkPresenter.StrokeContainer.GetStrokes();
//Just as a check, make sure to only do work if there are actually strokes (ie not empty)
if (canvasStrokes.Count > 0)
{
var width = (int)myCanvas.ActualWidth;
var height = (int)myCanvas.ActualHeight;
var device = CanvasDevice.GetSharedDevice();
//Create a new renderTarget with the same width and height as myCanvas at 96dpi
var renderTarget = new CanvasRenderTarget(device, width,
height, 96);
using (var ds = renderTarget.CreateDrawingSession())
{
//This will clear the renderTarget with a clean slate of a white background.
ds.Clear(Windows.UI.Colors.White);
//Here is where we actually take the strokes from the canvas and draw them on the render target.
ds.DrawInk(myCanvas.InkPresenter.StrokeContainer.GetStrokes());
}
//Finally, this will return the render target as a byte array.
return renderTarget.GetPixelBytes();
}
else
{
return null;
}
}
If all you need is a byte array of the pixels, you are done. However, if you now want to generate an image, you use the WriteableBitmap to do so. Here are sync and async methods that can be called to do this.
private WriteableBitmap GetSignatureBitmapFull()
{
var bytes = ConvertInkCanvasToByteArray();
if (bytes != null)
{
var width = (int)cvsSignature.ActualWidth;
var height = (int)cvsSignature.ActualHeight;
var bmp = new WriteableBitmap(width, height);
using (var stream = bmp.PixelBuffer.AsStream())
{
stream.Write(bytes, 0, bytes.Length);
return bmp;
}
}
else
return null;
}
private async Task<WriteableBitmap> GetSignatureBitmapFullAsync()
{
var bytes = ConvertInkCanvasToByteArray();
if (bytes != null)
{
var width = (int)cvsSignature.ActualWidth;
var height = (int)cvsSignature.ActualHeight;
var bmp = new WriteableBitmap(width, height);
using (var stream = bmp.PixelBuffer.AsStream())
{
await stream.WriteAsync(bytes, 0, bytes.Length);
return bmp;
}
}
else
return null;
}
Finally, here is an async method that I am using to be able to do a crop on the bitmap if you want to do that. The key thing with this method is notice how with this approach, you don't have to get the pixel bytes as an array first. You just get the strokes from the canvas and then it is saved directly into a memory stream
private async Task<WriteableBitmap> GetSignatureBitmapCropped()
{
try
{
var canvasStrokes = cvsSignature.InkPresenter.StrokeContainer.GetStrokes();
if (canvasStrokes.Count > 0)
{
var bounds = cvsSignature.InkPresenter.StrokeContainer.BoundingRect;
var xOffset = (uint)Math.Round(bounds.X);
var yOffset = (uint)Math.Round(bounds.Y);
var pixelWidth = (int)Math.Round(bounds.Width);
var pixelHeight = (int)Math.Round(bounds.Height);
using (var memStream = new InMemoryRandomAccessStream())
{
await cvsSignature.InkPresenter.StrokeContainer.SaveAsync(memStream);
var decoder = await BitmapDecoder.CreateAsync(memStream);
var transform = new BitmapTransform();
var newBounds = new BitmapBounds();
newBounds.X = 0;
newBounds.Y = 0;
newBounds.Width = (uint)pixelWidth;
newBounds.Height = (uint)pixelHeight;
transform.Bounds = newBounds;
var pdp = await decoder.GetPixelDataAsync(
BitmapPixelFormat.Bgra8,
BitmapAlphaMode.Straight,
transform,
ExifOrientationMode.IgnoreExifOrientation,
ColorManagementMode.DoNotColorManage);
var pixels = pdp.DetachPixelData();
var cropBmp = new WriteableBitmap(pixelWidth, pixelHeight);
using (var stream = cropBmp.PixelBuffer.AsStream())
{
await stream.WriteAsync(pixels, 0, pixels.Length);
}
return cropBmp;
}
}
else
{
return null;
}
}
catch (Exception ex)
{
return null;
}
}
Hope this helps provide some alternatives when using InkCanvas in Windows 10 Universal.

it seems my only option is to save the signature to file as a GIF.
GIF is not the only choice. The official sample just show GIF, but you can also save the InkStokes collection to JPG and PNG. Code as follows:
var savePicker = new Windows.Storage.Pickers.FileSavePicker();
savePicker.SuggestedStartLocation = Windows.Storage.Pickers.PickerLocationId.PicturesLibrary;
savePicker.FileTypeChoices.Add("Gif,JPG,PNG", new System.Collections.Generic.List<string> { ".jpg" ,".gif",".png"});
Windows.Storage.StorageFile file = await savePicker.PickSaveFileAsync();
But with this type of approach I get an exception that I cannot convert from System.IO.MemoryStream to Windows.Storage.Streams.IOutputStream
As you see, it cannot convert from System.IO.MemoryStream to Windows.Storage.Streams.IOutputStream directly. You need to read the image file you just saved as ImageSource. As you known how to get the base64 once you have either an image or a byte array. So what we just need is to get an image or a byte array from the image file.
Read the image file as byte array and then a memory stream as follows
if (inkCanvas.InkPresenter.StrokeContainer.GetStrokes().Count > 0)
{
var savePicker = new Windows.Storage.Pickers.FileSavePicker();
savePicker.SuggestedStartLocation = Windows.Storage.Pickers.PickerLocationId.PicturesLibrary;
savePicker.FileTypeChoices.Add("Gif,JPG,PNG", new System.Collections.Generic.List<string> { ".jpg", ".gif", ".png" });
Windows.Storage.StorageFile file = await savePicker.PickSaveFileAsync();
if (null != file)
{
using (IRandomAccessStream stream = await file.OpenAsync(FileAccessMode.ReadWrite))
{
await inkCanvas.InkPresenter.StrokeContainer.SaveAsync(stream);
}
using (IRandomAccessStream streamforread = await file.OpenAsync(FileAccessMode.Read))
{
WriteableBitmap inkimagesource = new WriteableBitmap(50, 50);
inkimagesource.SetSource(streamforread);
byte[] imageBuffer = inkimagesource.PixelBuffer.ToArray();
MemoryStream ms = new MemoryStream(imageBuffer);
}
}
}
Read the file as image source just need to change the file reading code as follows:
using (IRandomAccessStream streamforread = await file.OpenAsync(FileAccessMode.Read)
{
var bitmap = new BitmapImage();
bitmap.SetSource(streamforread);
}
I guess I can iterate that list and build a byte array? How would I do that? It seems this is not efficient?
It seems like currently there is no API can build the InkStroke collections as byte array directly. SaveAsync() method has already help you save the InkStroke collection to a stream, you can use code above for using it.

UWP - scaled image is damaged

I have a problem to place an image with down-scale. Image is rasterized and I don't know find way to place it correctly.
Is it possible to place picture without losing quality ?
Code:
public async Task<BitmapImage> BitmapTransform(string filePath, uint width)
{
StorageFile file = await StorageFile.GetFileFromPathAsync(filePath);
if (file == null)
return null;
// create a stream from the file and decode the image
var fileStream = await file.OpenAsync(Windows.Storage.FileAccessMode.Read);
BitmapDecoder decoder = await BitmapDecoder.CreateAsync(fileStream);
// create a new stream and encoder for the new image
InMemoryRandomAccessStream ras = new InMemoryRandomAccessStream();
BitmapEncoder enc = await BitmapEncoder.CreateForTranscodingAsync(ras, decoder);
double ration =
enc.BitmapTransform.ScaledWidth = width;
enc.BitmapTransform.ScaledHeight = (uint)(((double)decoder.PixelHeight / (double)decoder.PixelWidth) * (double)width);
// write out to the stream
try
{
await enc.FlushAsync();
}
catch (Exception ex)
{
string s = ex.ToString();
}
// render the stream to the screen
BitmapImage bImg = new BitmapImage();
bImg.SetSource(ras);
return bImg;
}

You can choose the way of resizing - Nearest Neighbor, Linear, Cubic, Fant - by BitmapTransform.InterporationMode property.
Have you tried it?
https://msdn.microsoft.com/en-us/library/windows/apps/windows.graphics.imaging.bitmaptransform.interpolationmode
double ration =
enc.BitmapTransform.ScaledWidth = width;
enc.BitmapTransform.ScaledHeight = (uint)(((double)decoder.PixelHeight / (double)decoder.PixelWidth) * (double)width);
enc.BitmapTransform.InterpolationMode = BitmapInterpolationMode.Cubic; // <----- Set the Interporation mode here
But the quality of result is vary. It's depend on the original image.

Rotating images in WinRT

I have a Windows 8 app in which I want to rotate an image file.
In shot, I want to open an image file, rotate it and save the content back to the file.
Is that possible in WinRT? If so, how? Thanks.
Update:
Base on Vasile's answer, I could do some work on this. However I'm not sure what to do next:
public static async Task RotateImage(StorageFile file)
{
if (file == null)
return;
var data = await FileIO.ReadBufferAsync(file);
// create a stream from the file
var ms = new InMemoryRandomAccessStream();
var dw = new DataWriter(ms);
dw.WriteBuffer(data);
await dw.StoreAsync();
ms.Seek(0);
// find out how big the image is, don't need this if you already know
var bm = new BitmapImage();
await bm.SetSourceAsync(ms);
// create a writable bitmap of the right size
var wb = new WriteableBitmap(bm.PixelWidth, bm.PixelHeight);
ms.Seek(0);
// load the writable bitpamp from the stream
await wb.SetSourceAsync(ms);
wb.Rotate(90);
//How should I save the image to the file now?
}

Ofcourse it is possible. You can do it yourself with a pixel manipulation and create a new WriteableBitmapObject or, you could reuse the already implemented functionality from the WriteableBitmapEx (WriteableBitmap Extensions). You can get it via NuGet.
Here you can find a description of the implemented functionality which it offers, and few short samples.

Use this to save WriteableBitmap to StorageFile
private async Task<StorageFile> WriteableBitmapToStorageFile(WriteableBitmap writeableBitmap)
{
var picker = new FileSavePicker();
picker.FileTypeChoices.Add("JPEG Image", new string[] { ".jpg" });
StorageFile file = await picker.PickSaveFileAsync();
if (file != null && writeableBitmap != null)
{
using (IRandomAccessStream stream = await file.OpenAsync(FileAccessMode.ReadWrite))
{
BitmapEncoder encoder = await BitmapEncoder.CreateAsync(
BitmapEncoder.JpegEncoderId, stream);
Stream pixelStream = writeableBitmap.PixelBuffer.AsStream();
byte[] pixels = new byte[pixelStream.Length];
await pixelStream.ReadAsync(pixels, 0, pixels.Length);
encoder.SetPixelData(BitmapPixelFormat.Bgra8, BitmapAlphaMode.Ignore,
(uint)writeableBitmap.PixelWidth, (uint)writeableBitmap.PixelHeight, 96.0, 96.0, pixels);
await encoder.FlushAsync();
}
return file;
}
else
{
return null;
}
}