I am working on an application in C# using XAudio2 from SharpDX. I know its outdated but instead of CSCore it offers the better Reverb. The aim itself is simple: load a wav file (48khz,24bit,1channel), get it through the XAudio2 Reverb with several Reverb presets and do a playback.
I managed to implement everything for a normal playback and also built in the Reverb. The audio file will be played back with the defaul reverb settings but as soon as I change the preset or change a specific value (RoomFilterFreq) the file isnt audible anymore and i don´t know why this happens. Does anyone has a clue about this? I worked through the few examples which are still online but could not find a reason for this behaviour.
I´m coding under Visual Studio 2019 with Net 4.7.2. The code looks as followed:
xaudio2 = new XAudio2();
xaudio2.StartEngine();
var masteringVoice = new MasteringVoice(xaudio2, 1, 48000);
var stream = new SoundStream(File.OpenRead(soundFilepath));
SharpDX.Multimedia.WaveFormat waveFormat = stream.Format;
var buffer = new AudioBuffer
{
Stream = stream.ToDataStream(),
AudioBytes = (int)stream.Length,
Flags = BufferFlags.EndOfStream
};
stream.Close();
sourceVoice = new SourceVoice(xaudio2, waveFormat, true);
// Set Loop
if (f1.loopcheckBox.Checked == true)
{
buffer.LoopBegin = buffer.PlayBegin;
buffer.LoopLength = buffer.PlayLength;
buffer.LoopCount = AudioBuffer.LoopInfinite;
}
// Set Reverb
if (f1.reverbenableButton.BackColor == Color.LightGreen)
{
var reverb = new Reverb(xaudio2);
var reverbparameters = (ReverbParameters)ReverbI3DL2Parameters.Presets.Room;
var effectDescriptor = new EffectDescriptor(reverb);
if (waveFormat.Channels == 1)
{
effectDescriptor.OutputChannelCount = 1;
}
else if (waveFormat.Channels == 2)
{
effectDescriptor.OutputChannelCount = 2;
}
else
{
MessageBox.Show("Channelrate not supported!");
return sourceVoice = null;
}
sourceVoice.SetEffectChain(effectDescriptor);
sourceVoice.SetEffectParameters(0, reverbparameters);
sourceVoice.EnableEffect(0);
}
sourceVoice.SubmitSourceBuffer(buffer, stream.DecodedPacketsInfo);
sourceVoice.Start();
I have successfully used Grpc in Unity and sent request to Dialog flow and received response. You can check the details here
However the whole returned result is the following only
{ "queryResult": { "languageCode": "ja" } }
The expected response id, query text, etc are not returned.
When testing in console.dialogflow.com I get the following result
{
"responseId": "cdf8003e-6599-4a28-9314-f4462c36e21b",
"queryResult": {
"queryText": "おはようございます",
"speechRecognitionConfidence": 0.92638445,
"languageCode": "ja"
}
}
However when I tried in console.dialogflow.com and didn't say anything I got
{ "queryResult": { "languageCode": "ja" } }
So perhaps the InputAudio encoding is wrong somehow.
Here's how I do it
var serializedByteArray = convertToBytes(samples);
request.InputAudio = Google.Protobuf.ByteString.CopyFrom(serializedByteArray);
And convert to bytes is like the following
public static byte[] convertToBytes(float[] audio)
{
List<byte> bytes = new List<byte>();
foreach (float audioI in audio) {
bytes.AddRange(BitConverter.GetBytes(audioI));
}
return bytes.ToArray();
}
The audio source is define as follows where sampleRate is 16000
audioSource.clip = Microphone.Start(null, true, 30, sampleRate);
I made sure to set sample rate hz properly.
queryInput.AudioConfig.SampleRateHertz = sampleRate;
Edit:
I have logged the recorded bytes from unity to a file (have all the bytes streamed appended together) and have written a console application to test the binary generated but using DetectIntent rather than streaming detect intent.
GoogleCredential credential = GoogleCredential.FromJson(privateKey);
var url = "dialogflow.googleapis.com";
Grpc.Core.Channel channel = new Grpc.Core.Channel(url, credential.ToChannelCredentials());
var client = SessionsClient.Create(channel);
CallOptions options = new CallOptions();
DetectIntentRequest detectIntentRequest = new DetectIntentRequest();
detectIntentRequest.Session = "projects/projectid/agent/sessions/" + "detectIntent";
QueryInput queryInput = new QueryInput();
queryInput.AudioConfig = new InputAudioConfig();
queryInput.AudioConfig.LanguageCode = "ja";
queryInput.AudioConfig.SampleRateHertz = sampleRate;//must be between 8khz and 48khz
queryInput.AudioConfig.AudioEncoding = AudioEncoding.Linear16;
detectIntentRequest.QueryInput = queryInput;
detectIntentRequest.InputAudio = Google.Protobuf.ByteString.CopyFrom(File.ReadAllBytes("D:\\temp\\audio.bytes"));
var response = client.DetectIntent(detectIntentRequest);
Console.WriteLine(response.ToString());
Console.WriteLine(response.ResponseId);
Console.Read();
I still get this (and empty response.ResponseId)
{ "queryResult": { "languageCode": "ja" } }
Thanks for advance.
Finally found the answer. The way I converted the datasource float to linear16 byte array was obviously wrong. Here's the code that worked
Credits to that post on unity forum.
https://forum.unity.com/threads/writing-audiolistener-getoutputdata-to-wav-problem.119295/#post-899142
public static byte[] convertToBytes(float[] dataSource)
{
var intData = new Int16[dataSource.Length];
//converting in 2 steps : float[] to Int16[], //then Int16[] to Byte[]
var bytesData = new Byte[dataSource.Length * 2];
//bytesData array is twice the size of
//dataSource array because a float converted in Int16 is 2 bytes.
var rescaleFactor = 32767; //to convert float to Int16
for (var i = 0; i < dataSource.Length; i++)
{
intData[i] = (short)(dataSource[i] * rescaleFactor);
var byteArr = new byte[2];
byteArr = BitConverter.GetBytes(intData[i]);
byteArr.CopyTo(bytesData, i * 2);
}
return bytesData;
}
Busy with a showcase / test of publishing webcam video through Apache Kafka to a Web (NancyFX C#) client in HTML 5. I know the resulting video stream will always have a few seconds lag. But I want to make the solution as fast as possible using newest web (HTML 5), streaming (Kafka), and video(Webm?) technology.
The Kafka part, both Producer and Consumer, are working: Sending Video Frame by Frame in JPeg format(C# System.Drawing.Bitmap) using the http://www.aforgenet.com/framework/. And I can use this library to create a Video File on disk but no streams.
When I use this Webm(VP9) Video File serving it through Nancy as Partial HTTP content (Thank you: http://richardssoftware.net/Home/Post/61) it works oke in the browser. This works fine because you know the total (file)length up front.
But now streaming the Jpeg Frame images on the fly to a Video player in HTML 5 seems not so easy.
My questions:
Does the HTML 5 Video players support streaming video: So video without an defined endpoint?
Which Video encoding is best for Live Streaming? I'm currently testing WebM(VP9) but have no experience with Live Video streaming.
Partial HTTP Content, using NancyFX, seems to support the video time slider offset. But is it also usable for Live video streaming? Maybe using the Kafka offset 1 on 1 or Frame id/key as slider value?
My code so far:
Kafka producer in C# using AForge.net lib to capture video cam:
class Program
{
const string brokerList = "127.0.0.1:2181";
const string topicName = "video_test";
static IDictionary<string, object> config = new Dictionary<string, object>
{
{"bootstrap.servers", "notebook039:9092" },
{"retries", 0 },
{"client.id", 200677},
{"batch.num.messages", 1},
{"socket.blocking.max.ms", 1},
{"socket.nagle.disable", true},
{"queue.buffering.max.ms", 0},
{"default.topic.config", new Dictionary<string, object>
{
{"acks", 1}
}
}
};
static Lazy<Producer<long, Bitmap>> publisher =
new Lazy<Producer<long, Bitmap>>(() => new Producer<long, Bitmap>(
config,
new LongSerializer(),
new BitmapSerializer()), true);
static long frameId = 0;
static private async void video_NewFrame(
object sender,
NewFrameEventArgs eventArgs)
{
try
{
// get new frame
Bitmap bitmap = eventArgs.Frame;
// process the frame
publisher.Value.ProduceAsync(topicName, ++frameId, bitmap, null);
if ((frameId % 24 == 0))
{
Console.WriteLine($"frameId: {frameId}");
}
}
catch ( Exception ex )
{
Console.WriteLine($"Error: " + ex.Message);
}
}
static void Main(string[] args)
{
var cancelled = false;
Console.CancelKeyPress += (_, e) =>
{
e.Cancel = true; // prevent the process from terminating.
cancelled = true;
};
// enumerate video devices
var videoDevices = new FilterInfoCollection(
FilterCategory.VideoInputDevice);
// create video source
VideoCaptureDevice videoSource = new VideoCaptureDevice(
videoDevices[0].MonikerString);
// set NewFrame event handler
videoSource.NewFrame += new NewFrameEventHandler(video_NewFrame);
// start the video source
videoSource.Start();
while (!cancelled)
{
Console.Write("> ");
Thread.Sleep(TimeSpan.FromSeconds(1));
}
// signal to stop
videoSource.SignalToStop();
}
}
And the C# Nancy Module subscribing to Kafka video Topic:
public class KafkaCam : NancyModule
{
const string brokerList = "notebook039:2181";
const string topicName = "video_test";
static IDictionary<string, object> config = new Dictionary<string, object>
{
{"group.id", Guid.NewGuid().ToString()},
{ "auto.offset.reset", "latest" },
{ "bootstrap.servers", "notebook039:9092" },
{"retries", 0 },
{"client.id", 210677},
{"batch.num.messages", 1},
{"socket.blocking.max.ms", 1},
{"socket.nagle.disable", true},
{"queue.buffering.max.ms", 0},
{"default.topic.config", new Dictionary<string, object>
{
{"acks", 1}
}
}
};
static Lazy<Consumer<long, Bitmap>> subscriber =
new Lazy<Consumer<long, Bitmap>>(
() => new Consumer<long, Bitmap>(
config,
new LongDeserializer(),
new BitmapDeserializer()),
true);
const string videoFilePath = #"D:\Kafka\clients\KafkaPublishSubscribeTests\data\video.webm";
Accord.Video.FFMPEG.VideoFileWriter videoFile = new Accord.Video.FFMPEG.VideoFileWriter();
public void FramesWriter()
{
try
{
Message<long, Bitmap> message = null;
subscriber.Value.Subscribe(topicName);
if (!videoFile.IsOpen)
{
videoFile.Open(videoFilePath, 640, 480, 24, Accord.Video.FFMPEG.VideoCodec.VP9);
}
do
{
subscriber.Value.Poll(TimeSpan.FromSeconds(2));
if (subscriber.Value.Consume(out message, TimeSpan.FromSeconds(2))
&& message != null )
{
videoFile.WriteVideoFrame(message.Value, (uint)message.Key);
videoFile.Flush();
}
}
while (true);
}
catch ( Exception ex )
{
throw ex;
}
finally
{
videoFile.Close();
videoFile = null;
}
}
public KafkaCam()
{
Task.Factory.StartNew(FramesWriter);
Get["/Camera.webm"] = parameters =>
{
return Response.FromPartialFile(Request, videoFilePath, "video/webm");
};
}
}
So I'm trying to use TweetSharp in VB.NET 2012 with C# to post a tweet with a image.
I found the code example of how to do it:
service.SendTweetWithMedia(new SendTweetWithMediaOptions
{
Status = "message",
Images = dictionary
}
);
However I'm not sure how to create the "dictionary" with the picture stream.
I tried this:
Dictionary<string, Stream> imageDict = new Dictionary<string, Stream>();
then referenced that later:
Images = imageDict
But it gives the error:
Error Screenshot
Anyone have any ideas of how this is supposed to work?
Another block of code I found and tried is:
using (var stream = new FileStream("Image.jpg", FileMode.Open))
{
var result = tservice.SendTweetWithMedia(new SendTweetWithMediaOptions
{
Status = "Message",
Images = new Dictionary<string, Stream> { { "john", stream } }
});
lblResult.Text = result.Text.ToString();
}
But it gives the same error about "FileStream".
You need to add a reference to the System.IO namespace, this is why you receive this error in the image you posted.
Here is an example:
var thumb = "http://somesite.net/imageurl";
var service = new TwitterService(key, secret);
service.AuthenticateWith(token, tokenSecret);
var req = WebRequest.Create(thumb);
using (var stream = req.GetResponse().GetResponseStream())
{
response = service.SendTweetWithMedia(new SendTweetWithMediaOptions
{
Status = tweet.Trim(),
Images = new Dictionary<string, Stream> { { fullname, stream } }
});
}
A GIF may fail during Tweet creation even if it is within the file size limit. Adhere to the following constraints to improve success rates.
Resolution should be <= 1280x1080 (width x height)
Number of frames <= 350
Number of pixels (width * height * num_frames) <= 300 million
Filesize <= 15Mb
I have MediaStreamSource implementation which can open files with multiple audio streams. In OpenMediaAsync method i deliver MediaStreamDescription for all video and audio streams but MediaElement detects only 1 audio stream. Also I have tested next logic which works:
detected 2 streams
report only first or second MediaStreamDescription of audio stream to ReportOpenMediaCompleted
But of course I want to report first and second audio stream to ReportOpenMediaCompleted and in result have MediaElement with 2 audio streams. Also I have discovered StreamId field inside MediaStreamSource class but it doesn't have set accessor, and when reporting streams with ReportOpenMediaCompleted all MediaStreamDescription have StreamId == 0.
OpenMediaAsync code:
protected override void OpenMediaAsync()
{
this.streamDesc = new Dictionary<int, MediaStreamDescription>();
List<MediaStreamDescription> availableStreams = new List<MediaStreamDescription>();
for (int i = 0; i < this.parser.StreamCount; i++)
{
Dictionary<MediaStreamAttributeKeys, string> streamAttributes = new Dictionary<MediaStreamAttributeKeys, string>();
MediaStreamDescription msd = null;
var type = this.parser.GetStreamType(i);
streamAttributes[MediaStreamAttributeKeys.CodecPrivateData] = this.parser.GetCodecPrivateData(i);
if (type == ParserComponent.StreamType.Video)
{
streamAttributes[MediaStreamAttributeKeys.VideoFourCC] = this.parser.GetCodecID(i);
streamAttributes[MediaStreamAttributeKeys.Width] = this.parser.GetWidth(i).ToString();
streamAttributes[MediaStreamAttributeKeys.Height] = this.parser.GetHeight(i).ToString();
msd = new MediaStreamDescription(MediaStreamType.Video, streamAttributes);
}
else if (type == ParserComponent.StreamType.Audio)
{
msd = new MediaStreamDescription(MediaStreamType.Audio, streamAttributes);
}
if (msd != null)
{
if (i == this.parser.CurrentAudioStreamIndex || i == this.parser.CurrentVideoStreamIndex)
{
this.parser.SetStreamActive(i, true);
// quick fix for multilanguage videos to submit only 1 audio stream
// availableStreams.Add(msd);
}
this.streamDesc.Add(i, msd);
availableStreams.Add(msd);
}
}
Dictionary<MediaSourceAttributesKeys, string> sourceAttributes = new Dictionary<MediaSourceAttributesKeys, string>();
sourceAttributes[MediaSourceAttributesKeys.CanSeek] = this.parser.Seekable.ToString();
sourceAttributes[MediaSourceAttributesKeys.Duration] = this.parser.Duration.Ticks.ToString();
ReportOpenMediaCompleted(sourceAttributes, availableStreams);
}
May be it's a good idea to use Player framework. It supports multiple audio streaming.