I am working on an UWP app for the HoloLens to read single frames from the devices camera. I want to use the camera mode with the lowest resolution available.
I took a look at the following links and examples and tried to create a minimal working app:
https://learn.microsoft.com/en-us/windows/uwp/audio-video-camera/set-media-encoding-properties
https://learn.microsoft.com/en-us/windows/uwp/audio-video-camera/use-opencv-with-mediaframereader
https://github.com/Microsoft/Windows-universal-samples/tree/master/Samples/CameraResolution
https://github.com/microsoft/Windows-universal-samples/tree/master/Samples/CameraOpenCV
This is the code snippet from MainPage.xaml.cs:
public async Task<int> Start()
{
// Find the sources
var allGroups = await MediaFrameSourceGroup.FindAllAsync();
var sourceGroups = allGroups.Select(g => new
{
Group = g,
SourceInfo = g.SourceInfos.FirstOrDefault(i => i.SourceKind == MediaFrameSourceKind.Color)
}).Where(g => g.SourceInfo != null).ToList();
if (sourceGroups.Count == 0)
{
// No camera sources found
return 0;
}
var selectedSource = sourceGroups.FirstOrDefault();
// Initialize MediaCapture
_mediaCapture = new MediaCapture();
var settings = new MediaCaptureInitializationSettings()
{
SourceGroup = selectedSource.Group,
SharingMode = MediaCaptureSharingMode.ExclusiveControl,
StreamingCaptureMode = StreamingCaptureMode.Video,
MemoryPreference = MediaCaptureMemoryPreference.Cpu
};
await _mediaCapture.InitializeAsync(settings);
// Query all properties of the device
IEnumerable<StreamResolution> allVideoProperties = _mediaCapture.VideoDeviceController.GetAvailableMediaStreamProperties(MediaStreamType.VideoRecord).Select(x => new StreamResolution(x));
// Order them by resolution then frame rate
allVideoProperties = allVideoProperties.OrderBy(x => x.Height * x.Width).ThenBy(x => x.FrameRate);
await _mediaCapture.VideoDeviceController.SetMediaStreamPropertiesAsync(MediaStreamType.VideoRecord, allVideoProperties.ElementAt(0).EncodingProperties);
// Create the frame reader
MediaFrameSource frameSource = _mediaCapture.FrameSources[selectedSource.SourceInfo.Id];
_reader = await _mediaCapture.CreateFrameReaderAsync(frameSource, MediaEncodingSubtypes.Bgra8);
_reader.FrameArrived += ColorFrameReader_FrameArrivedAsync;
await _reader.StartAsync();
return 1;
}
private async void ColorFrameReader_FrameArrivedAsync(MediaFrameReader sender, MediaFrameArrivedEventArgs args)
{
var frame = sender.TryAcquireLatestFrame();
if (frame != null)
{
var inputBitmap = frame.VideoMediaFrame?.SoftwareBitmap;
}
}
On my local machine (MacBookPro with Bootcamp partition) this code works using the webcam. It detects three supported video modes. I can change the resolution of the bitmap image in FrameArrivedAsync by changing the index from 0 to 1 or 2 at:
_mediaCapture.VideoDeviceController.SetMediaStreamPropertiesAsync(MediaStreamType.VideoRecord, allVideoProperties.ElementAt(0).EncodingProperties);
On HoloLens this code does not work. It detects the different modes like explained here (https://learn.microsoft.com/en-us/windows/mixed-reality/locatable-camera). But setting the MediaStreamProperties does not change anything regarding the received bitmap image. The bitmap is always 1280x720.
Just in case, we want to share how we setup the capture profile, you can refer to the following code with annotate to modify your project for testing. If in doubt, please feel free to add comments.
private async void SetupAndStartMediaCapture()
{
string deviceId = string.Empty;
_mediaCapture = new MediaCapture();
DeviceInformationCollection devices = await DeviceInformation.FindAllAsync(DeviceClass.VideoCapture);
foreach (var device in devices)
{
if(MediaCapture.IsVideoProfileSupported(device.Id))
{
deviceId = device.Id;
break; // The video device for which supported video profile support is queried.
}
}
MediaCaptureInitializationSettings mediaCapSettings = new MediaCaptureInitializationSettings
{
VideoDeviceId = deviceId
};
IReadOnlyList<MediaCaptureVideoProfile> profiles = MediaCapture.FindAllVideoProfiles(deviceId);
var profileMatch = (
from profile in profiles
from desc in profile.SupportedRecordMediaDescription
where desc.Width == 896 && desc.Height == 504 && desc.FrameRate == 24 // HL1
select new { profile, desc }
).FirstOrDefault();// Select the Profile with the required resolution from all available profiles.
if (profileMatch != null)
{
mediaCapSettings.VideoProfile = profileMatch.profile;
mediaCapSettings.RecordMediaDescription = profileMatch.desc;
}
await _mediaCapture.InitializeAsync(mediaCapSettings); //Initializes the MediaCapture object.
}
Related
I would like to know how to record 2 separate audio channel simultaneously.
I have 2 USB adapters with mic & speaker respectively.
The samples code which I can find only support single channel recording at a time.
Please help. Thanks.
For single channel my code as follow;
MediaCapture audioCapture = new MediaCapture();
MediaCaptureInitializationSettings captureInitSettings = new MediaCaptureInitializationSettings();
captureInitSettings.StreamingCaptureMode = StreamingCaptureMode.Audio;
captureInitSettings.MediaCategory = MediaCategory.Other;
captureInitSettings.AudioProcessing = AudioProcessing.Default;
await audioCapture.InitializeAsync(captureInitSettings);
private async void recordChannelA()
{
StorageFolder externalDevices = KnownFolders.RemovableDevices;
IReadOnlyList<StorageFolder> externalDrives = await externalDevices.GetFoldersAsync();
StorageFolder usbStorage = externalDrives[0];
if (usbStorage != null)
{
StorageFolder recordFolder = await usbStorage.CreateFolderAsync(recFolderName, CreationCollisionOption.OpenIfExists);
await usbStorage.GetFolderAsync(recFolderName);
StorageFile recordFile = await recordFolder.CreateFileAsync("Recording - " + DateTime.Now.ToString("yyyy-MM-dd_HH-mm-ss") + ".mp3", Windows.Storage.CreationCollisionOption.GenerateUniqueName);
MediaEncodingProfile profile = null;
profile = MediaEncodingProfile.CreateM4a(Windows.Media.MediaProperties.AudioEncodingQuality.Auto);
await audioCapture.StartRecordToStorageFileAsync(profile, recordFile);
Message.Text = "Recording ... ";
recordingtimerRun = new TimeSpan(0, 0, 0);
recordingTimer.Start();
}
else Message.Text = "Recording error !";
}
Update;
I created a 'listview' for the enumerated devices and to select the respective capture device. However, there is an Syntax Error which i cannot convert the enumaration.deviceinformation to imediasource.
captureInitSettings.AudioSource = captureDeviceList[audioCaptureList.SelectedIndex];
Update: I managed to get it to work
The solution is
captureInitSettingsA.AudioDeviceId = captureDeviceList[audioCaptureList.SelectedIndex].Id;
captureInitSettingsB.AudioDeviceId = captureDeviceList[audioCaptureList.SelectedIndex].Id;
However, how do i save these selections in app settings .. so that when I reboot I don't have to re-select again.
Update:
I manage to save the app setting for audiocapture & audiorender devices but I am not sure how to retrieve them & also to check if there is any previous settings saved.
Windows.Storage.ApplicationDataContainer localSettings = Windows.Storage.ApplicationData.Current.LocalSettings;
Windows.Storage.StorageFolder localFolder = Windows.Storage.ApplicationData.Current.LocalFolder;
localSettings.Values["audioACaptureSettings"] = captureAInitSettings.AudioDeviceId;
localSettings.Values["audioARenderSettings"] = mediaPlayerA.AudioDevice.Id;
localSettings.Values["audioBCaptureSettings"] = captureBInitSettings.AudioDeviceId;
localSettings.Values["audioBRenderSettings"] = mediaPlayerB.AudioDevice.Id;
private void loadAudioConfig()
{
Windows.Storage.ApplicationDataContainer localSettings = Windows.Storage.ApplicationData.Current.LocalSettings;
Windows.Storage.StorageFolder localFolder = Windows.Storage.ApplicationData.Current.LocalFolder;
if (localSettings.Values["audioACaptureSettings"] != null)
{
captureAInitSettings.AudioDeviceId = localSettings.Values["audioACaptureSettings"].ToString();
}
if (localSettings.Values["audioARenderSettings"] != null)
{
Object audioARenderValue = localSettings.Values["audioARenderSettings"];
mediaPlayerA.AudioDevice = audioARenderValue;
}
if (localSettings.Values["PAaudioCaptureSettings"] != null)
{
captureBInitSettings.AudioDeviceId = localSettings.Values["audioBCaptureSettings"].ToString();
}
if (localSettings.Values["PAaudioRenderSettings"] != null)
{
Object audioBRenderValue = localSettings.Values["audioBRenderSettings"];
mediaPlayerB.AudioDevice = audioBRenderValue;
}
You can refer to this document which introduced how to store and retrieve settings and other app data. You can save the data to Settings and Files.
When you use Settings, it only supports multiple data types as mentioned in the document.
If use files, you can store binary data or to enable your own, customized serialized types,.
In your provided code, it is correct to check if there is any previous settings saved:
if (localSettings.Values["audioACaptureSettings"] != null)
{
captureAInitSettings.AudioDeviceId = localSettings.Values["audioACaptureSettings"].ToString();
}
But it is incorrect about how to retrieve the setting as AudioDevice because it can not implicitly convert string to DeviceInformation. Please try in this way:
if (localSettings.Values["audioARenderSettings"] != null)
{
var aduioSource = localSettings.Values["audioARenderSettings"] as string;
mediaPlayerA.AudioDevice = await DeviceInformation.CreateFromIdAsync(aduioSource);
}
I am trying to set up a preview stream and recording loop with buttons to save the last 10 mins, 30 secs etc. This was working just fine until I started adding the code to handle rotation.
This is the line that throws.
await _mediaCapture.SetEncodingPropertiesAsync(MediaStreamType.VideoPreview,
videoEncodingProperties, mediaPropertySet);
here is the whole method
public async Task<MediaCapture> PrepareRecordingAsync() {
try {
_mediaCapture = new MediaCapture();
var allVideoDevices = await DeviceInformation.FindAllAsync(DeviceClass.VideoCapture);
var desiredDevice = allVideoDevices.FirstOrDefault(x => x.EnclosureLocation != null && x.EnclosureLocation.Panel == Panel.Back);
_cameraDevice = desiredDevice ?? allVideoDevices.FirstOrDefault();
_rotationHelper = new CameraRotationHelper(_cameraDevice.EnclosureLocation);
_mediaCapture.Failed += MediaCapture_Failed;
var settings = new MediaCaptureInitializationSettings { VideoDeviceId = _cameraDevice.Id };
await _mediaCapture.InitializeAsync(settings);
var encodingProfile = MediaEncodingProfile.CreateMp4(VideoEncodingQuality.Auto);
var rotationAngle = CameraRotationHelper.ConvertSimpleOrientationToClockwiseDegrees(_rotationHelper.GetCameraCaptureOrientation());
Guid RotationKey = new Guid("C380465D-2271-428C-9B83-ECEA3B4A85C1");
encodingProfile.Video.Properties.Add(RotationKey, PropertyValue.CreateInt32(rotationAngle));
var videoEncodingProperties = _mediaCapture.VideoDeviceController.GetMediaStreamProperties(MediaStreamType.VideoPreview);
MediaPropertySet mediaPropertySet = new MediaPropertySet();
await _mediaCapture.SetEncodingPropertiesAsync(MediaStreamType.VideoPreview, videoEncodingProperties, mediaPropertySet);
_ras = new InMemoryRandomAccessStream();
_recording = await _mediaCapture.PrepareLowLagRecordToStreamAsync(encodingProfile, _ras);
DisplayInformation.AutoRotationPreferences = DisplayOrientations.Portrait;
ConcurrentRecordAndPhotoSupported = _mediaCapture.MediaCaptureSettings.ConcurrentRecordAndPhotoSupported;
} catch (UnauthorizedAccessException) {
// This will be thrown if the user denied access to the camera in privacy settings
System.Diagnostics.Debug.WriteLine("The app was denied access to the camera");
} catch (Exception ex) {
System.Diagnostics.Debug.WriteLine("MediaCapture initialization failed. {0}", ex.Message);
}
return _mediaCapture;
}
None of the solutions found via google search are any help.
This is basically a modification of the MSDN How-to's.
EDIT: If I change the offending line to the following then it works fine.
_mediaCapture.SetPreviewRotation(VideoRotation.Clockwise90Degrees);
I can reproduce your issue on my side, it will throw the error exception at code line await _mediaCapture.SetEncodingPropertiesAsync(...);
The stream number provided was invalid.
PreviewState
According to the SetEncodingPropertiesAsync method
Note that this rotation is performed by the consumer of the stream, such as the CaptureElement or a video player app, while the actual pixels in the stream still retain their original orientation.
This method performed by the consumer of the stream. It seems like that you need to invoke StartPreviewAsync() firstly before you setting the preview rotation so that you have preview stream. More details please reference the "Add orientation data to the camera preview stream" section of Handle device orientation with MediaCapture.
After starting the preview, call the helper method SetPreviewRotationAsync to set the preview rotation.
So updating your code snippet as follows it will work.
_mediaCapture = new MediaCapture();
var allVideoDevices = await DeviceInformation.FindAllAsync(DeviceClass.VideoCapture);
var desiredDevice = allVideoDevices.FirstOrDefault(x => x.EnclosureLocation != null && x.EnclosureLocation.Panel == Windows.Devices.Enumeration.Panel.Back);
_cameraDevice = desiredDevice ?? allVideoDevices.FirstOrDefault();
_rotationHelper = new CameraRotationHelper(_cameraDevice.EnclosureLocation);
_mediaCapture.Failed += MediaCapture_Failed;
var settings = new MediaCaptureInitializationSettings { VideoDeviceId = _cameraDevice.Id };
await _mediaCapture.InitializeAsync(settings);
//Add the preview code snippet
PreviewControl.Source = _mediaCapture;
await _mediaCapture.StartPreviewAsync();
var encodingProfile = MediaEncodingProfile.CreateMp4(VideoEncodingQuality.Auto);
var rotationAngle = CameraRotationHelper.ConvertSimpleOrientationToClockwiseDegrees(_rotationHelper.GetCameraCaptureOrientation());
Guid RotationKey = new Guid("C380465D-2271-428C-9B83-ECEA3B4A85C1");
encodingProfile.Video.Properties.Add(RotationKey, PropertyValue.CreateInt32(rotationAngle));
var videoEncodingProperties = _mediaCapture.VideoDeviceController.GetMediaStreamProperties(MediaStreamType.VideoPreview);
MediaPropertySet mediaPropertySet = new MediaPropertySet();
await _mediaCapture.SetEncodingPropertiesAsync(MediaStreamType.VideoPreview, videoEncodingProperties, mediaPropertySet);
More details please reference the official sample.
I'm trying to create a callrecorder for winphone at the UWP. I'm trying to do this with Audio graphs. I need to make a node for an input device (microphone), a node for an output device (speaker) and submit them to the file(wave/mp3).
I'm receiving an exception.
AudioGraph graph;
AudioDeviceInputNode deviceInputNode;
AudioDeviceOutputNode deviceOutputNode;
AudioFileOutputNode fileOutputNode;
private async Task InitAudiographAsync()
{
AudioGraphSettings settings = new AudioGraphSettings(AudioRenderCategory.Speech);
CreateAudioGraphResult result = await AudioGraph.CreateAsync(settings);
if (result.Status == AudioGraphCreationStatus.Success)
{
graph = result.Graph;
CreateAudioDeviceOutputNodeResult deviceOutputNodeResult = await graph.CreateDeviceOutputNodeAsync();
if (deviceOutputNodeResult.Status == AudioDeviceNodeCreationStatus.Success)
{
deviceOutputNode = deviceOutputNodeResult.DeviceOutputNode;
var microphone = await DeviceInformation.CreateFromIdAsync(
MediaDevice.GetDefaultAudioCaptureId(AudioDeviceRole.Default));
var inProfile = MediaEncodingProfile.CreateWav(AudioEncodingQuality.High);
var deviceInputNodeResult = await graph.CreateDeviceInputNodeAsync(MediaCategory.Speech, inProfile.Audio, microphone);
if (deviceInputNodeResult.Status == AudioDeviceNodeCreationStatus.Success)
{
deviceInputNode = deviceInputNodeResult.DeviceInputNode;
FileSavePicker saveFilePicker = new FileSavePicker();
saveFilePicker.FileTypeChoices.Add("Windows Media Audio", new List<string>() { ".wma" });
saveFilePicker.FileTypeChoices.Add("MPEG Audio Layer-3", new List<string>() { ".mp3" });
saveFilePicker.SuggestedFileName = "New Audio Track";
StorageFile file = await saveFilePicker.PickSaveFileAsync();
// File can be null if cancel is hit in the file picker
if (file == null)
{
return;
}
MediaEncodingProfile mediaEncodingProfile;
switch (file.FileType.ToString().ToLowerInvariant())
{
case ".mp3":
mediaEncodingProfile = MediaEncodingProfile.CreateMp3(AudioEncodingQuality.High);
break;
case ".wav":
mediaEncodingProfile = MediaEncodingProfile.CreateWav(AudioEncodingQuality.High);
break;
default:
throw new ArgumentException();
}
CreateAudioFileOutputNodeResult fileOutputNodeResult = await graph.CreateFileOutputNodeAsync(file, mediaEncodingProfile);
if (fileOutputNodeResult.Status == AudioFileNodeCreationStatus.Success)
{
fileOutputNode = fileOutputNodeResult.FileOutputNode;
deviceInputNode.AddOutgoingConnection(deviceOutputNode);
deviceInputNode.AddOutgoingConnection(fileOutputNode);
graph.Start();
}
}
}
}
}
Did you check manifest\capabilities for microphone access and file access. Are you getting an access denied exception?
I am working on wpf application that uses GATT for bluetooth communication. I am using GattCharacteristic.ValueChanged to handle any value from device. It works for sometime and the event handler doesnt gets called after 1 minute. This problem observed in windows 8.1, but working in windows 10.
I am made GattCharacteristic member variable, so the GC doesnt collect the variable. Still the problem exists.
Can anyone tell me what could be the potential problem here?
private GattCharacteristic m_DataSenderCharacteristics;
foreach (var service in serviceInfo)
{
var uuid = MjGattDeviceService.UuidFromServiceInformation(service);
//Battery Service
if (uuid.ToString() == BatteryUUID)
{
var bService = await GattDeviceService.FromIdAsync(service.Id);
m_Device.batteryService = bService;
}
if (uuid.ToString() == CustomServiceUUID)
{
m_Device.gattService = await GattDeviceService.FromIdAsync(service.Id);
Guid controlGUID = new Guid(ControlUUID);
var controlGattCharacteristics = m_Device.gattService.GetCharacteristics(controlGUID);
Byte[] bsdata = new Byte[] { 0x0b, 0x00, 0x02 };
IBuffer buffUTF8 = CryptographicBuffer.CreateFromByteArray(bsdata);
GattCommunicationStatus status = await controlGattCharacteristics[0].WriteClientCharacteristicConfigurationDescriptorAsync(GattClientCharacteristicConfigurationDescriptorValue.Notify);
if (status == GattCommunicationStatus.Unreachable)
{
m_Device.isConnected = false;
nextPage = "Unreachable";
}
else if (status == GattCommunicationStatus.Success)
{
var ctrlStatus = await controlGattCharacteristics[0].WriteValueAsync(buffUTF8);
Guid guuid1 = new Guid(DataSenderUUID);
m_DataSenderCharacteristics = m_Device.gattService.GetCharacteristics(guuid1)[0];
GattCommunicationStatus notifyStatus = await m_DataSenderCharacteristics.
WriteClientCharacteristicConfigurationDescriptorAsync(GattClientCharacteristicConfigurationDescriptorValue.Notify);
//This event handler will be called for some 1 minute. After it falls dead
m_DataSenderCharacteristics[0].ValueChanged += GestureValueChanged;
if (notifyStatus == GattCommunicationStatus.Success)
{
nextPage = "Connected";
}
}
}
}
I'm using ZXing.net to create a UserControl for scanning barcode into a Windows Phone 8.1 RT app using the camera.
The barcode are decoded well but i'm having freezes on the UI when the method CapturePhotoToStreamAsync is called, even it is awaited.
It takes about 600 ms to execute.
I'm testing the app into the emulator.
The code below is executed in an async method :
// Preview of the camera
await _mediaCapture.InitializeAsync(settings);
VideoCapture.Source = _mediaCapture;
VideoCapture.FlowDirection = Windows.UI.Xaml.FlowDirection.LeftToRight;
await _mediaCapture.StartPreviewAsync();
VideoEncodingProperties res = _mediaCapture.VideoDeviceController.GetMediaStreamProperties(MediaStreamType.VideoPreview) as VideoEncodingProperties;
ImageEncodingProperties iep = ImageEncodingProperties.CreateBmp();
iep.Height = res.Height;
iep.Width = res.Width;
var barcodeReader = new BarcodeReader
{
TryHarder = true,
AutoRotate = true
};
WriteableBitmap wB = new WriteableBitmap((int)res.Width, (int)res.Height);
while (_result == null)
{
using (var stream = new InMemoryRandomAccessStream())
{
await _mediaCapture.CapturePhotoToStreamAsync(iep, stream);
stream.Seek(0);
await wB.SetSourceAsync(stream);
_result = barcodeReader.Decode(wB);
}
}
await _mediaCapture.StopPreviewAsync();
//callback to handle result
ScanCallback(_result.Text);
What can I do to prevent the UI from freezing ?
Luckily you don't have to capture photo to decode QRCode/Barcode on Windows Phone 8.1 Runtime. It is a quite new solution actually, but it works: https://github.com/mmaitre314/VideoEffect#realtime-video-analysis-and-qr-code-detection
After istalling the nuget package, you can easily decode barcodes realtime, without the need of calling CapturePhotoToStreamAsync. The only drawback is that you can only target ARM. You can find the sample code on the site. Or you can contact me, and I can send you the part of my project where I use this.
i always get better results when i use the camera to take a picture first (lets you focus on the correct place where the barcode is) and then send the picture for barcode recognition.
the stuttering is caused because you try to keep checking the live feed for barcodes which can be hard on the CPU (especially for ARM devices)
var dialog = new CameraCaptureUI();
StorageFile file = await dialog.CaptureFileAsync(CameraCaptureUIMode.Photo);
var stream = await file.OpenReadAsync();
// initialize with 1,1 to get the current size of the image
var writeableBmp = new WriteableBitmap(1, 1);
writeableBmp.SetSource(stream);
// and create it again because otherwise the WB isn't fully initialized and decoding
// results in a IndexOutOfRange
writeableBmp = new WriteableBitmap(writeableBmp.PixelWidth, writeableBmp.PixelHeight);
stream.Seek(0);
writeableBmp.SetSource(stream);
var result = ScanBitmap(writeableBmp);
string barcode = "";
if (result != null)
{
barcode = result.Text;
}
and here the ScanBitmap method:
private Result ScanBitmap(WriteableBitmap writeableBmp)
{
var barcodeReader = new BarcodeReader
{
Options = new ZXing.Common.DecodingOptions()
{
TryHarder = true
},
AutoRotate = true
};
var result = barcodeReader.Decode(writeableBmp);
if (result != null)
{
CaptureImage.Source = writeableBmp;
}
return result;
}