We Keep Coding

Text-To-Speech playback has very low volume after trying Speech-To-Text in Xamarin Forms App

Disclaimer: I am a newbie to c# and Xamarin.Forms - sorry for missing anything obvious.
I am trying to create an app that takes user input in the form of a voice command (using Speech-To-Text) and outputs an audio announcement from the application (using Text-To-Speech).
The issue is that when you start recording audio for the Speech-To-Text service, the device's audio is set to recording mode (not sure what the technical term for this is called) and playback audio is set to a very low volume (as described in this SO question and here) and here.
I'm ideally looking for a way to revert this so that once the appropriate voice command is recognised (i.e. 'Secret command') via Speech-To-Text, the user can hear the secret phrase back at full/normal volume through Text-To-Speech in a Xamarin Forms application.
I tried to produce a working example by adapting the sample code for Azure Cognitive Speech Service. I cloned the code and adapted the Xaml and CS for the MainPage slightly, as shown below, to stop the speech recognition service once a certain voice command is triggered and then activate a phrase to be spoken via the Text-To-Speech service. My sample demonstrates the issue. If the user starts by selecting the Transcribe button and enters the appropriate voice command, they should hear back the secret phrase, but the playback volume is so low when testing on a physical IOS device I can barely hear it.
XAML
<ContentPage xmlns="http://xamarin.com/schemas/2014/forms"
xmlns:x="http://schemas.microsoft.com/winfx/2009/xaml"
xmlns:d="http://xamarin.com/schemas/2014/forms/design"
xmlns:mc="http://schemas.openxmlformats.org/markup-compatibility/2006"
mc:Ignorable="d"
x:Class="CognitiveSpeechService.MyPage"
Title="Speech Services Transcription"
Padding="10,35,10,10">
<StackLayout>
<Frame BorderColor="DarkGray"
CornerRadius="10"
HeightRequest="300"
WidthRequest="280"
HorizontalOptions="Center"
VerticalOptions="Start"
BackgroundColor="LightGray">
<ScrollView x:Name="scroll">
<Label x:Name="transcribedText"
Margin="10,10,10,10" />
</ScrollView>
</Frame>
<ActivityIndicator x:Name="transcribingIndicator"
HorizontalOptions="Center"
VerticalOptions="Start"
WidthRequest="300"
IsRunning="False" />
<Button x:Name="transcribeButton"
WidthRequest="300"
HeightRequest="50"
Text="Transcribe"
TextColor="White"
CornerRadius="10"
BackgroundColor="Green"
BorderColor="DarkGray"
BorderWidth="1"
FontAttributes="Bold"
HorizontalOptions="Center"
VerticalOptions="Start"
Clicked="TranscribeClicked"/>
<Button x:Name="SpeakBtn"
WidthRequest="300"
HeightRequest="50"
Text="Speak"
TextColor="White"
CornerRadius="10"
BackgroundColor="Red"
BorderColor="DarkGray"
BorderWidth="1"
FontAttributes="Bold"
HorizontalOptions="Center"
VerticalOptions="Start"
Clicked="SpeakBtn_Clicked"/>
</StackLayout>
</ContentPage>
Code-behind
namespace CognitiveSpeechService
{
public partial class MyPage : ContentPage
{
AudioRecorderService recorder = new AudioRecorderService();
SpeechRecognizer recognizer;
IMicrophoneService micService;
bool isTranscribing = false;
public MyPage()
{
InitializeComponent();
micService = DependencyService.Resolve<IMicrophoneService>();
}
async void TranscribeClicked(object sender, EventArgs e)
{
bool isMicEnabled = await micService.GetPermissionAsync();
// EARLY OUT: make sure mic is accessible
if (!isMicEnabled)
{
UpdateTranscription("Please grant access to the microphone!");
return;
}
// initialize speech recognizer
if (recognizer == null)
{
var config = SpeechConfig.FromSubscription(Constants.CognitiveServicesApiKey, Constants.CognitiveServicesRegion);
recognizer = new SpeechRecognizer(config);
recognizer.Recognized += (obj, args) =>
{
UpdateTranscription(args.Result.Text);
};
}
// if already transcribing, stop speech recognizer
if (isTranscribing)
{
StopSpeechRecognition();
}
// if not transcribing, start speech recognizer
else
{
Device.BeginInvokeOnMainThread(() =>
{
InsertDateTimeRecord();
});
try
{
await recognizer.StartContinuousRecognitionAsync();
}
catch (Exception ex)
{
UpdateTranscription(ex.Message);
}
isTranscribing = true;
}
UpdateDisplayState();
}
// https://stackoverflow.com/questions/56514413/volume-has-dropped-significantly-in-text-to-speech-since-adding-speech-to-text
private async void StopSpeechRecognition()
{
if (recognizer != null)
{
try
{
await recognizer.StopContinuousRecognitionAsync();
Console.WriteLine($"IsRecording: {recorder.IsRecording}");
}
catch (Exception ex)
{
UpdateTranscription(ex.Message);
}
isTranscribing = false;
UpdateDisplayState();
}
}
void UpdateTranscription(string newText)
{
Device.BeginInvokeOnMainThread(() =>
{
if (!string.IsNullOrWhiteSpace(newText))
{
if (newText.ToLower().Contains("Secret command"))
{
Console.WriteLine("heart rate voice command detected");
// stop speech recognition
StopSpeechRecognition();
// do callout
string success = "this works!";
var settings = new SpeechOptions()
{
Volume = 1.0f,
};
TextToSpeech.SpeakAsync(success, settings);
// start speech recongition
} else
{
transcribedText.Text += $"{newText}\n";
}
}
});
}
void InsertDateTimeRecord()
{
var msg = $"=================\n{DateTime.Now.ToString()}\n=================";
UpdateTranscription(msg);
}
void UpdateDisplayState()
{
Device.BeginInvokeOnMainThread(() =>
{
if (isTranscribing)
{
transcribeButton.Text = "Stop";
transcribeButton.BackgroundColor = Color.Red;
transcribingIndicator.IsRunning = true;
}
else
{
transcribeButton.Text = "Transcribe";
transcribeButton.BackgroundColor = Color.Green;
transcribingIndicator.IsRunning = false;
}
});
}
async void SpeakBtn_Clicked(object sender, EventArgs e)
{
await TextToSpeech.SpeakAsync("Sample audio line. Blah blah blah. ");
}
}
}
Thanks for your help!

Found a working solution. Posting it below for whoever else it can help and future me.
I noticed this issue was only happening on IOS and not Android, it has to do with the category that AVAudioSession is set to when STT is enabled. As I best understand it, once STT is enabled, audio-ducking turns on for any non-STT-related audio.
You can resolve this issue by programmatically setting the right category using the AVAudioSession Xamarin.IOS API.
To get this working properly in a Xamarin.Forms project, you will need to use the Dependency Service to execute the Xamarin.IOS code in your shared project code.
I have set out the relevant bits of the code that worked for me below.
A full working example can be found in the solution branch of the Github repo mentioned in the comments above.
Mainpage (where STT and TTS services are happening)
public partial class MainPage : ContentPage
{
IAudioSessionService audioService;
public MainPage()
{
InitializeComponent();
micService = DependencyService.Resolve<IMicrophoneService>();
if (Device.RuntimePlatform == Device.iOS)
{
audioService = DependencyService.Resolve<IAudioSessionService>();
}
}
public void SpeechToText()
{
// wherever STT is required, call this first to set the right audio category
audioService?.ActivateAudioRecordingSession();
}
public void TextToSpeech()
{
// wherever TTS is required, let the OS know that you're playing audio so TTS interrupts instead of ducking.
audioService?.ActivateAudioPlaybackSession();
await TextToSpeech.SpeakAsync(TextForTextToSpeechAfterSpeechToText, settings);
// set audio session back to recording mode ready for STT
audioService?.ActivateAudioRecordingSession();
}
IAudioSessionService
// this interface should be in your shared project
namespace CognitiveSpeechService.Services
{
public interface IAudioSessionService
{
void ActivateAudioPlaybackSession();
void ActivateAudioRecordingSession();
}
}
project.Android/AndroidAudioSessionService
using System;
using CognitiveSpeechService.Services;
using Xamarin.Forms;
[assembly: Dependency(typeof(CognitiveSpeechService.Droid.Services.AndroidAudioSessionService))]
namespace CognitiveSpeechService.Droid.Services
{
public class AndroidAudioSessionService : IAudioSessionService
{
public void ActivateAudioPlaybackSession()
{
// do nothing as not required on Android
}
public void ActivateAudioRecordingSession()
{
// do nothing as not required on Android
}
}
}
Project.iOS/IOSAudioSessionService
using System;
using AVFoundation;
using CognitiveSpeechService.Services;
using Foundation;
using Xamarin.Forms;
[assembly: Dependency(typeof(CognitiveSpeechService.iOS.Services.IOSAudioSessionService))]
namespace CognitiveSpeechService.iOS.Services
{
public class IOSAudioSessionService : IAudioSessionService
{
public void ActivateAudioPlaybackSession()
{
var session = AVAudioSession.SharedInstance();
session.SetCategory(AVAudioSessionCategory.Playback, AVAudioSessionCategoryOptions.DuckOthers);
session.SetMode(AVAudioSession.ModeSpokenAudio, out NSError error);
session.SetActive(true);
}
public void ActivateAudioRecordingSession()
{
try
{
new System.Threading.Thread(new System.Threading.ThreadStart(() =>
{
var session = AVAudioSession.SharedInstance();
session.SetCategory(AVAudioSessionCategory.Record);
session.SetActive(true);
})).Start();
}
catch (Exception ex)
{
Console.WriteLine(ex.Message);
}
}
}
}

How to play a selected song from ListView in UWP App

I am new to UWP development and trying to play a song that is displayed in my ListView. The app I am writing is for a Windows IoT Core running on a raspberry pi3.
So basically what happens now is that when I select the song from my ListView and hit the play button nothing happens? I think I need to let the MediaPlayer know what song has been selected and to play that song?
This is my code I am using:
XAML:
<MediaElement x:Name="mediaPlaylist" AreTransportControlsEnabled="True" Margin="389,-10,10,10">
<MediaElement.TransportControls>
<MediaTransportControls IsZoomEnabled="False" IsZoomButtonVisible='False' IsSeekBarVisible="True" IsSeekEnabled="True" IsFullWindowButtonVisible="False" IsStopButtonVisible="True"/>
</MediaElement.TransportControls>
</MediaElement>
<ListView
x:Name="SelectSong"
Margin="389,28,10,113"
FontStyle="Oblique"
FontWeight="Bold"
Background="DarkGreen"
ItemClick="SongClicked"
IsItemClickEnabled="True"
/>
C#:
public MainPage()
{
this.InitializeComponent();
_ = InitFolderAsync();
}
private async Task InitFolderAsync()
{
var files = await Windows.Storage.KnownFolders.MusicLibrary.GetFilesAsync();
foreach (StorageFile item in files)
{
SelectSong.Items.Add(item.Name.ToString());
}
}
private async void SongClicked(object sender, ItemClickEventArgs e)
{
var file = e.ClickedItem as StorageFile;
if (file != null)
{
var stream = await file.OpenReadAsync();
mediaPlaylist.SetSource(stream, file.ContentType);
mediaPlaylist.Play();
}
}
I would like to have so that one can select a song and play it or to loop through all the songs.
Thanks

How to play a selected song from ListView in UWP App
Please check this line SelectSong.Items.Add(item.Name.ToString()); you just add the file's name to listview items, but not the file, so you can't get the file with var file = e.ClickedItem as StorageFile.
We need to edit the SongClicked and get music file with cliched item like the following.
private async void SongClicked(object sender, ItemClickEventArgs e)
{
var file = await KnownFolders.MusicLibrary.GetFileAsync(e.ClickedItem.ToString());
if (file != null)
{
var stream = await file.OpenReadAsync();
mediaPlaylist.SetSource(stream, file.ContentType);
mediaPlaylist.Play();
}
}

Exception throw out when trying to open the camera using Xam.Plugin.Media

When I'm trying to open the camera using await CrossMedia.Current.TakePhotAsync() Im getting below exception. It doesn't open any camera in the application.
Field overflow on store InterpretedSystem.Runtime.CompilerServices.IAsyncStateMachineVT.<>7__wrap2
My code as below,
public UserProfilePage()
{
InitializeComponent();
CameraButton.Clicked += CameraButton_Clicked;
BindingContext = new UserProfileViewModel(this.Navigation);
}
private async void CameraButton_Clicked(object sender, EventArgs e)
{
if (CrossMedia.Current.IsCameraAvailable && CrossMedia.Current.IsTakePhotoSupported)
{
// Supply media options for saving our photo after it's taken.
var mediaOptions = new Plugin.Media.Abstractions.StoreCameraMediaOptions
{
Directory = "Receipts",
Name = $"{DateTime.UtcNow}.jpg"
};
// Take a photo of the business receipt.
try
{
var file = await CrossMedia.Current.TakePhotoAsync(mediaOptions);
}
catch (Exception exception)
{
Console.WriteLine(exception);
throw;
}
}
}
Xaml page like below,
<Grid>
<Grid.RowDefinitions>
<RowDefinition Height="*" />
<RowDefinition Height="Auto" />
</Grid.RowDefinitions>
<Image x:Name="PhotoImage" />
<Button x:Name="CameraButton" Text="Take Photo" Grid.Row="1" />
</Grid>
How to get my camera open and take a photo in xamarin. Below are the confugarations,
Xam.Plugin.Media = v3.1.2
Xamarin.Forms = v2.5.0.121934
VS2017

This error occurs when you're trying to run Audio, Camera kind of features in the Xamarin Live Player.
Plugging actual physical device for debugging or release will solve the problem.

I think the reason could be in the target file name of the photo:
Name = $"{DateTime.UtcNow}.jpg"
The date contains characters that are invalid in a file path and that causes the exception.
You can use an alternative format that is suitable for file names:
Name = $"{DateTime.UtcNow.ToString("yyyy-dd-M--HH-mm-ss")}.jpg"

Toggling audio in an UWP app

I'm building an app with optional audio support, and it will feature some kind of background sounds (like a looping soundtrack that keeps on playing in every page).
What I did is to create an audio manager that allows me to manage my sounds, and this should mute/unmute the audio based on user's settings.
Unfortunately this doesn't happen, and the audio keeps on playing even if the user disables it.
This is a sample of my code:
public static async Task StartSoundManager()
{
// Get audio stream from app folder
// ...
BackgroundSound.SetSource(currentStream, currentFile.ContentType);
BackgroundSound.IsLooping = true;
ToggleSounds();
}
public static void ToggleSounds()
{
BackgroundSound.IsMuted = !Settings.IsAudioOn;
}
public bool IsAudioOn
{
// standard getter
set
{
// save value
SoundManager.ToggleSounds();
}
}
After some tests, IsMuted is set correctly (I've also tried setting volume to 0) but nothing happens when changing settings.
Do you guys have any idea on why such a simple task is not working as expected? It seems to me that you can't change volume after setting the source, and this feels really wrong.
EDIT: more complete class
public static class AudioManager
{
public const string BACKGROUND = "BACKGROUND.mp3";
private static readonly MediaElement BackgroundSound = new MediaElement();
public static async Task StartSoundManager()
{
// Get folder
var folder =
await (await Package.Current.InstalledLocation.GetFolderAsync("Assets")).GetFolderAsync("Audio");
var currentFile = await folder.GetFileAsync(BACKGROUND);
var currentStream = await currentFile.OpenAsync(FileAccessMode.Read);
BackgroundSound.SetSource(currentStream, currentFile.ContentType);
// Set mode and volume
BackgroundSound.IsLooping = true;
ToggleSounds();
}
public static void ToggleSounds()
{
BackgroundSound.IsMuted = !Settings.IsAudioOn; // IsAudioOn is false, still the sound plays
}
}

MediaElement is a XAML control, to make MediaElement.IsMuted property work, we need to add MediaElement into Visual Tree. For example, in your code, we can change BackgroundSound to a public field like:
public static readonly MediaElement BackgroundSound = new MediaElement();
And then in a page (e.g. MainPage) add it to the page:
protected override async void OnNavigatedTo(NavigationEventArgs e)
{
await AudioManager.StartSoundManager();
rootGrid.Children.Add(AudioManager.BackgroundSound);
}
After this, your ToggleSounds method should be able to work.
But since you want to keep on playing in every page, add MediaElement into the page may be not a good practice, here I'd suggest you use MediaPlayer class instead of MediaElement like:
public static class AudioManager
{
public const string BACKGROUND = "BACKGROUND.mp3";
private static readonly MediaPlayer BackgroundSound = new MediaPlayer();
public static void StartSoundManager()
{
BackgroundSound.Source = MediaSource.CreateFromUri(new Uri($"ms-appx:///Assets/Audio/{BACKGROUND}"));
BackgroundSound.IsLoopingEnabled = true;
ToggleSounds();
BackgroundSound.Play();
}
public static void ToggleSounds()
{
BackgroundSound.IsMuted = !Settings.IsAudioOn; // IsAudioOn is false, still the sound plays
}
}
For more info, please see Play audio and video with MediaPlayer.

I have a similar situation. I have a ToggleMenuFlyoutItem for turning sounds on and off app-wide that writes to the local settings:
Here is the XAML:
<Button IsTabStop="False" Style="{StaticResource qButtonStyleFinal}" TabIndex="2" x:Name="btnMore" BorderBrush="{x:Null}" Foreground="{x:Null}"
Content="1" Grid.Column="37" Margin="5" HorizontalAlignment="Center" Grid.Row="1" Grid.RowSpan="4" Grid.ColumnSpan="2" MaxWidth="60" MaxHeight="60">
<Button.Background>
<ImageBrush ImageSource="ms-appx:///Assets/more_project.png" Stretch="Uniform" />
</Button.Background>
<Button.Flyout>
<MenuFlyout>
<ToggleMenuFlyoutItem x:Name="mfiToggleSounds" Text="sound effects" IsChecked="{x:Bind Mode=TwoWay, Path=Model.Sounds}"></ToggleMenuFlyoutItem>
<MenuFlyoutItem x:Name="mfiExportExcel" Text="export to Excel" Tapped="mfiExportExcel_Tapped" />
<MenuFlyoutItem x:Name="mfiExportCSV" Text="export to CSV" Tapped="mfiExportCSV_Tapped" />
<MenuFlyoutItem x:Name="mfiEmailDeveloper" Text="email developer" Tapped="mfiEmailDeveloper_Tapped" />
</MenuFlyout>
</Button.Flyout>
</Button>
Changes are handled in the property set event:
private bool _sounds = true;
public bool Sounds
{
get => _sounds;
set
{
_sounds = value;
NotifyPropertyChanged();
var localSettings = ApplicationData.Current.LocalSettings;
localSettings.Values["sounds"] = _sounds;
}
}
Setting is loaded on the main page Page_Loaded event as follows:
// retrieve settings
var localSettings = ApplicationData.Current.LocalSettings;
if (localSettings.Values.ContainsKey("sounds"))
{
clsGlobal.statModel.Sounds = (bool)localSettings.Values["sounds"];
}
And when I play a sound, a simple if condition is included:
if (clsGlobal.statModel.Sounds && clsGlobal.fileInputs.ContainsKey("problem.wav"))
{
var snd1 = clsGlobal.fileInputs["problem.wav"];
if (snd1 != null)
{
snd1.Reset();
snd1.Start();
}
}
I am using the AudioGraph API for my sounds. Works better than the MediaPlayer route since that was giving me a weird situation where the sounds would not play sometimes on the first request. AudioGraph works better for me and is really fast to play sounds.

Universal Windows Platform - UI not updating when returning from different page

I'm learning Universal Windows Platform and I'm currently analysing music player using SoundCloud API from this site. Link to the github project is at the very bottom of the page. To get this project to work variable SoundCloudClientId from App.xaml.cs should be filled in (I used client id from previous example on the same page, not sure if I can paste that).
When application starts the NowPlaying page is loaded and changing tracks causes UI to update accordingly. The problem is when I navigate to any other page and return back to NowPlaying. I can still change music using buttons, but UI doesn't change (song title, album title etc.).
Important parts of the code:
NowPlaying.xaml
<ImageBrush x:Name="albumrtImage" ImageSource="Assets\Albumart.png" Stretch="UniformToFill" />
<TextBlock x:Name="txtSongTitle" Grid.Row="0" HorizontalAlignment="Center" Text="Song Title " FontSize="25" Foreground="White" Style="{StaticResource HeaderTextBlockStyle}" TextTrimming="WordEllipsis" />
<TextBlock x:Name="txtAlbumTitle" Grid.Row="0" Text="Label " HorizontalAlignment="Center" FontWeight="Light" FontSize="20" Foreground="#9799a5" Style="{StaticResource BodyTextBlockStyle}" TextTrimming="WordEllipsis"/>
NowPlaying.xaml.cs
async void BackgroundMediaPlayer_MessageReceivedFromBackground(object sender, MediaPlayerDataReceivedEventArgs e)
{
TrackChangedMessage trackChangedMessage;
if (MessageService.TryParseMessage(e.Data, out trackChangedMessage))
{
// When foreground app is active change track based on background message
await this.Dispatcher.RunAsync(CoreDispatcherPriority.Normal, () =>
{
var songIndex = GetSongIndexById(trackChangedMessage.TrackId);
if (songIndex >= 0)
{
var song = App.likes[songIndex];
LoadTrack(song); //Update UI
}
});
return;
}
BackgroundAudioTaskStartedMessage backgroundAudioTaskStartedMessage;
if (MessageService.TryParseMessage(e.Data, out backgroundAudioTaskStartedMessage))
{
backgroundAudioTaskStarted.Set();
return;
}
}
private async void LoadTrack(SoundCloudTrack currentTrack)
{
try
{
//Change album art
string albumartImage = Convert.ToString(currentTrack.artwork_url);
if (string.IsNullOrWhiteSpace(albumartImage))
{
albumartImage = #"ms-appx:///Assets/Albumart.png";
}
else
{
albumartImage = albumartImage.Replace("-large", "-t500x500");
}
//Next 3 lines when pages were switched don't cause UI to update
albumrtImage.ImageSource = new BitmapImage(new Uri(albumartImage));
txtSongTitle.Text = currentTrack.title;
txtAlbumTitle.Text = Convert.ToString(currentTrack.user.username);
}
catch (Exception ex)
{
MessageDialog showMessgae = new MessageDialog("Something went wrong. Please try again. Error Details : " + ex.Message);
await showMessgae.ShowAsync();
}
}
After navigating from NowPlaying->Me->NowPlaying and clicking next the track changes, but UI doesn't update as seen on the screen below:
UI problem
I'm trying to reproduce the problem on a simple example, but without any luck. What could cause this issue? Any help is appreciated.

I've found the solution. The problem was with cache. The NowPlaying page required putting following property:
NavigationCacheMode="Required"

I just want to add that you can also check navigation mode in OnNavigatedTo method. It can be helpful when for instance you would like to refresh the data when returning to the selected page.
protected override void OnNavigatedTo(NavigationEventArgs e)
{
base.OnNavigatedTo(e);
if(e.NavigationMode == NavigationMode.Back)
{
//refresh the data here...
}
}