I've been trying to implement a WebRTC audio/video communication on Unity. So far, with the help of this blog post (and Google Translate) and the very few examples on the internet, I've managed to make quite a lot work on Unity using the WebRTC Unity Plugin.
But now I'm stuck. The great Unity sample project by mhama sadly doesn't have an example of how to convert the data I get from the native code to something that can be used as audio data in Unity.
The info I get from the callback is this
(IntPtr data, int bitsPerSample, int sampleRate, int numberOfChannels, int numberOfFrames)
That data in the native code is declared as
const void* audio_data
I know that to create an Audio Clip that Unity can use play some sound, I need a float array with sample values from -1 to 1. How do I go from that IntPtr data and all that extra info to that float array, is something I have no idea how to do.
Here's the sample I'm using as a base
I'm not sure you can do this without unsafe code. You'll need to make sure your project has unsafe code allowed.
// allocate the float arrays for the output.
// numberOfChannels x numberOfFrames
float[][] output = new float[numberOfChannels][];
for (int ch = 0 ; ch < numberOfChannels; ++ch)
output[ch] = new float[numberOfFrames];
// scaling factor for converting signed PCM into float (-1.0 to 1.0)
const double scaleIntToFloat = 1.0/0x7fffffff;
unsafe
{
// obtain a pointer to the raw PCM audio data.
byte *ptr = (byte *)data.ToPointer();
for (int frame = 0 ; frame < numberOfFrames; ++frame)
{
for (int ch = 0 ; ch < numberOfChannels; ++ch)
{
switch (bitsPerSample)
{
case 32:
// shift 4 bytes into the integer value
int intValue = *ptr++ << 24 & *ptr++ << 16 &
*ptr++ << 8 & *ptr++;
// scale the int to float and store.
output[ch][frame] = scaleIntToFloat * intValue;
break;
case 16:
// shift 2 bytes into the integer value. Note:
// shifting into the upper 16 bits to simplify things,
// e.g. multiply by the same scaling factor.
int intValue = *ptr++ << 24 & *ptr++ << 16;
output[ch][frame] = scaleIntToFloat * intValue;
break;
case 24:
...
case 8:
// not 8-bit is typically unsigned. Google it if
// you need to.
}
}
}
}
You can find the other conversions by searching this site for conversion of PCM to float. Also, depending on your circumstances, you might need a different endianness. If so, shift into the intValue in a different byte order.
I need to combine two Bytes into one int value.
I receive from my camera a 16bit Image were two successive bytes have the intensity value of one pixel. My goal is to combine these two bytes into one "int" vale.
I manage to do this using the following code:
for (int i = 0; i < VectorLength * 2; i = i + 2)
{
NewImageVector[ImagePointer] = ((int)(buffer.Array[i + 1]) << 8) | ((int)(buffer.Array[i]));
ImagePointer++;
}
My image is 1280*960 so VectorLength==1228800 and the incomming buffer size is 2*1228800=2457600 elements...
Is there any way that I can speed this up?
Maybe there is another way so I don't need to use a for-loop.
Thank you
You could use the equivalent to the union of c. Im not sure if faster, but more elegant:
[StructLayout(LayoutKind.Explicit)]
struct byte_array
{
[FieldOffset(0)]
public byte byte1;
[FieldOffset(1)]
public byte byte2;
[FieldOffset(0)]
public short int0;
}
use it like this:
byte_array ba = new byte_array();
//insert the two bytes
ba.byte1 = (byte)(buffer.Array[i]);
ba.byte2 = (byte)(buffer.Array[i + 1]);
//get the integer
NewImageVector[ImagePointer] = ba.int1;
You can fill your two bytes and use the int. To find the faster way take the StopWatch-Class and compare the two ways like this:
Stopwatch stopWatch = new Stopwatch();
stopWatch.Start();
//The code
stopWatch.Stop();
MessageBox.Show(stopWatch.ElapsedTicks.ToString()); //Or milliseconds ,...
Assuming you can (re-)define NewImageVector as a short[], and every two consecutive bytes in Buffer should be transformed into a short (which basically what you're doing now, only you cast to an int afterwards), you can use Buffer.BlockCopy to do it for you.
As the documentation tells, you Buffer.BlockCopy copies bytes from one array to another, so in order to copy your bytes in buffer you need to do the following:
Buffer.BlockCopy(Buffer, 0, NewImageVector, 0, [NumberOfExpectedShorts] * 2)
This tells BlockCopy that you want to start copying bytes from Buffer, starting at index 0, to NewImageVector starting at index 0, and you want to copy [NumberOfExpectedShorts] * 2 bytes (since every short is two bytes long).
No loops, but it does depend on the ability of using a short[] array instead of an int[] array (and indeed, on using an array to begin with).
Note that this also requires the bytes in Buffer to be in little-endian order (i.e. Buffer[index] contains the low byte, buffer[index + 1] the high byte).
You can achieve a small performance increase by using unsafe pointers to iterate the arrays. The following code assumes that source is the input byte array (buffer.Array in your case). It also assumes that source has an even number of elements. In production code you would obviously have to check these things.
int[] output = new int[source.Length / 2];
fixed (byte* pSource = source)
fixed (int* pDestination = output)
{
byte* sourceIterator = pSource;
int* destIterator = pDestination;
for (int i = 0; i < output.Length; i++)
{
(*destIterator) = ((*sourceIterator) | (*(sourceIterator + 1) << 8));
destIterator++;
sourceIterator += 2;
}
}
return output;
I'm developing a screen-sharing application. In this project I need to transport images over the internet. Obviously, I can't send a new picture over the internet every few seconds, it would be extremely slow.
I want to send one image of the server's screen to the client, and afterwards, instead of sending a new picture sending only the pixels that have been changed since the last image (the one the client already has).
I have written this code:
private List<Color> CompareBitmaps(Image old, Image _new)
{
List<Color> returnList = new List<Color>();
for(int i = 0; i < old.Width; i++)
for (int j = 0; j < old.Height; j++)
{
if (((Bitmap)old).GetPixel(i, j) != ((Bitmap)_new).GetPixel(i, j))
{
returnList.Add(((Bitmap)_new).GetPixel(i, j));
}
}
return returnList;
}
However, it works way too slow.
I'm looking for a faster algorithm, one with a better complexity.
Note: I don't want a built library that does that. I need an algorithm.
This routine finds the differences between two Bitmaps and returns them in the 1st Bitmap by setting everything else to almost black and pretty much transparent. It can also restore the original 2nd file by adding the result back into the previous image..
I shrunk a screenshot of 800MB 1o 12k - but there was really just a very small change at the Clocks hands ;-) If your images differ in many pixels, the compression will not be as spectacular..but I believe it will b good enough for tranmitting and I doubt anything on a pixel by pixel basis will compare to the compression routines of png or jpg file formats.. (you don't transmit bmps, I hope!)
The routine uses LockBits and is pretty fast.
The bool parameter decides whether to create the difference bitmap or to restore the changed bitmap.
public static Bitmap Difference(Bitmap bmp0, Bitmap bmp1, bool restore)
{
int Bpp = 4; // assuming an effective pixelformat of 32bpp
var bmpData0 = bmp0.LockBits(
new Rectangle(0, 0, bmp0.Width, bmp0.Height),
ImageLockMode.ReadWrite, bmp0.PixelFormat);
var bmpData1 = bmp1.LockBits(
new Rectangle(0, 0, bmp1.Width, bmp1.Height),
ImageLockMode.ReadOnly, bmp1.PixelFormat);
int len = bmpData0.Height * bmpData0.Stride;
byte[] data0 = new byte[len];
byte[] data1 = new byte[len];
Marshal.Copy(bmpData0.Scan0, data0, 0, len);
Marshal.Copy(bmpData1.Scan0, data1, 0, len);
for (int i = 0; i < len; i += Bpp)
{
if (restore)
{
bool toberestored = (data1[i ] != 2 && data1[i+1] != 3 &&
data1[i+2] != 7 && data1[i+2] != 42);
if (toberestored)
{
data0[i ] = data1[i]; // Blue
data0[i+1] = data1[i+1]; // Green
data0[i+2] = data1[i+2]; // Red
data0[i+3] = data1[i+3]; // Alpha
}
}
else
{
bool changed = ((data0[i ] != data1[i ]) ||
(data0[i+1] != data1[i+1]) || (data0[i+2] != data1[i+2]) );
data0[i ] = changed ? data1[i ] : (byte)2; // special markers
data0[i+1] = changed ? data1[i+1] : (byte)3; // special markers
data0[i+2] = changed ? data1[i+2] : (byte)7; // special markers
data0[i+3] = changed ? (byte)255 : (byte)42; // special markers
}
}
Marshal.Copy(data0, 0, bmpData0.Scan0, len);
bmp0.UnlockBits(bmpData0);
bmp1.UnlockBits(bmpData1);
return bmp0;
}
Notes:
- I have chosen a special color to mark those pixels that need to be restored at the recipient. Here I chose alpha=42 and R=7; G=3; B=2;.. Not 100% safe but almost; not a lot of pixels will be missed; and maybe you don't have tranparency anyway..?
I append two smaller images, both PNGs, around 400kB:
This is the difference image (3kB):
The restored image is the same as the 2nd image.
You need to return all the changed pixels, so the complexity will have to be m*n.
(Bitmap)_new).GetPixel(i, j) is called twice, use a temp value to store it might be a little better.
the pixel should have several values right? can you try to create a function called comprareTwoPixel(color A, color B)? and compare all the values one by one, if one of them is false, you don`t need to compare the rest, just return false. (Not sure if this will make it faster or not though.)
Like:
bool comprareTwoPixel(color A, color B)
{
if(A.a!=B.b)
return false;
if(A.b!=B.b)
return false;
if(A.c!=B.c)
return false;
return true;
}
I'm not sure of what you're trying to do with this code, because you aren't saving the indexes that have changed...
You may want to use an other constructor for your List, because List() haves a default capacity of 0
That mean you will re-allocate the internal buffer of the List maybe many times if many pixels have changed
Maybe recording the average number of pixels that has changed and setting the initial capacity of your List to that number can accelerate your code. At least, you can say that, E.g. 10% of the pixel change every frame :
List<Color> returnList = new List<Color>( (int)( 0.10 * numberOfPixel ) );
This may or may not work perfectly, but here goes. You can parallelize this if you see so fit by adding AsParallel.
I also copied and pasted a few lines in here, so feel free to let me know or edit if I have any typos or mismatched variables. But this is the gist of it. This should be pretty quick, within reason.
Essentially, since this might be a bit hard to understand as-is, the idea is to "lock" the bits, then use that pointer to copy them to a byte[]. That effectively copies out all the RGB(A) values, which you can then access really easily. This is a ton faster than GetPixel when you're reading, as you are, more than a pixel or two, because it takes a little while to grab the pixels out, but then it's just simple reads against memory.
Once I get them into the byte[]s, it's easy enough to just compare those for each pixel coordinate. I chose to use LINQ so that it's really easy to parallelize if need be, but you may or may not choose to actually implement that. I'm not sure whether you'll need to.
I've made a few assumptions here that I believe are fair, since it sounds like your implementation has all images coming from a single source. Namely, I assume that the images are the same size and format. So if that's not actually the case, you'll want to address that with some extra code in here, but that's still pretty easy.
private byte[] UnlockBits(Bitmap bmp, out int stride)
{
BitmapData bmpData = bmp.LockBits(new Rectangle(Point.Empty, bmp.Size), System.Drawing.Imaging.ImageLockMode.ReadOnly, bmp.PixelFormat);
IntPtr ptr = bmpData.Scan0;
stride = bmpData.Stride;
int bytes = Math.Abs(bmpData.Stride) * bmp.Height;
byte[] ret = new byte[bytes];
System.Runtime.InteropServices.Marshal.Copy(ptr, ret, 0, bytes);
bmp.UnlockBits(bmpData);
return ret;
}
private bool AreArraysEqual(byte[] a, byte[] b, int offset, int length)
{
for (int v = 0; v < length; v++)
{
int c = v + offset;
if (a[c] != b[c])
{
return false;
}
}
return true;
}
private IEnumerable<KeyValuePair<Point, Tuple<Color, Color>>> GetDifferences(Bitmap a, Bitmap b)
{
if (a.PixelFormat != b.PixelFormat)
throw new ArgumentException("Unmatched formats!");
if (a.Size != b.Size)
throw new ArgumentException("Unmatched length!");
int stride;
byte[] rgbValuesA = UnlockBits(a, out stride);
byte[] rgbValuesB = UnlockBits(b, out stride);
if (rgbValuesA.Length != rgbValuesB.Length)
throw new ArgumentException("Unmatched array lengths (unexpected error)!");
int bytesPerPixel = Image.GetPixelFormatSize(a.PixelFormat) / 8;
return Enumerable.Range(0, a.Height).SelectMany(y =>
Enumerable.Range(0, a.Width)
.Where(x => !AreArraysEqual(rgbValuesA,
rgbValuesB,
(y * stride) + (x * bytesPerPixel),
bytesPerPixel))
.Select(x =>
{
Point pt = new Point(x, y);
int pixelIndex = (y * stride) + (x * bytesPerPixel);
Color colorA = ReadPixel(rgbValuesA, pixelIndex, bytesPerPixel);
Color colorB = ReadPixel(rgbValuesB, pixelIndex, bytesPerPixel);
return new KeyValuePair<Point, Tuple<Color, Color>>(pt, colorA, colorB);
}
}
private Color ReadPixel(byte[] bytes, int offset, int bytesPerPixel)
{
int argb = BitConverter.ToInt32(pixelBytes, offset);
if (bytesPerPixel == 3) // no alpha
argb |= (255 << 24);
return Color.FromArgb(argb);
}
public IEnumerable<KeyValuePair<Point, Color>> GetNewColors(Bitmap _new, Bitmap old)
{
return GetDifferences(_new, old).Select(c => new KeyValuePair<Point, Color>(c.Key, c.Value.Item1));
}
In a true implementation, you might want to think about endianness and pixel format a bit more thoroughly than I have, but this should work more or less as a proof of concept, and I believe should handle a majority of practical cases.
And as #TaW said in a comment, you could also try blanking out (setting alpha to zero, probably) any that haven't changed. You can also benefit from pixel unlocking for that, too. Again, there are likely tutorials that tell you how to do it. But much of this could remain the same.
I'm trying to debug some bit shifting operations and I need to visualize the bits as they exist before and after a Bit-Shifting operation.
I read from this answer that I may need to handle backfill from the shifting, but I'm not sure what that means.
I think that by asking this question (how do I print the bits in a int) I can figure out what the backfill is, and perhaps some other questions I have.
Here is my sample code so far.
static string GetBits(int num)
{
StringBuilder sb = new StringBuilder();
uint bits = (uint)num;
while (bits!=0)
{
bits >>= 1;
isBitSet = // somehow do an | operation on the first bit.
// I'm unsure if it's possible to handle different data types here
// or if unsafe code and a PTR is needed
if (isBitSet)
sb.Append("1");
else
sb.Append("0");
}
}
Convert.ToString(56,2).PadLeft(8,'0') returns "00111000"
This is for a byte, works for int also, just increase the numbers
To test if the last bit is set you could use:
isBitSet = ((bits & 1) == 1);
But you should do so before shifting right (not after), otherwise you's missing the first bit:
isBitSet = ((bits & 1) == 1);
bits = bits >> 1;
But a better option would be to use the static methods of the BitConverter class to get the actual bytes used to represent the number in memory into a byte array. The advantage (or disadvantage depending on your needs) of this method is that this reflects the endianness of the machine running the code.
byte[] bytes = BitConverter.GetBytes(num);
int bitPos = 0;
while(bitPos < 8 * bytes.Length)
{
int byteIndex = bitPos / 8;
int offset = bitPos % 8;
bool isSet = (bytes[byteIndex] & (1 << offset)) != 0;
// isSet = [True] if the bit at bitPos is set, false otherwise
bitPos++;
}
I have this code for converting a byte[] to float[].
public float[] ConvertByteToFloat(byte[] array)
{
float[] floatArr = new float[array.Length / sizeof(float)];
int index = 0;
for (int i = 0; i < floatArr.Length; i++)
{
floatArr[i] = BitConverter.ToSingle(array, index);
index += sizeof(float);
}
return floatArr;
}
Problem is, I usually get a NaN result! Why should this be? I checked if there is data in the byte[] and the data seems to be fine. If it helps, an example of the values are:
new byte[] {
231,
255,
235,
255,
}
But this returns NaN (Not a Number) after conversion to float. What could be the problem? Are there other better ways of converting byte[] to float[]? I am sure that the values read into the buffer are correct since I compared it with my other program (which performs amplification for a .wav file).
If endianness is the problem, you should check the value of BitConverter.IsLittleEndian to determine if the bytes have to be reversed:
public static float[] ConvertByteToFloat(byte[] array) {
float[] floatArr = new float[array.Length / 4];
for (int i = 0; i < floatArr.Length; i++) {
if (BitConverter.IsLittleEndian) {
Array.Reverse(array, i * 4, 4);
}
floatArr[i] = BitConverter.ToSingle(array, i * 4);
}
return floatArr;
}
Isn't the problem that the 255 in the exponent represents NaN (see Wikipedia in the exponent section), so you should get a NaN. Try changing the last 255 to something else...
If it's the endianness that is wrong (you are reading big endian numbers) try these (be aware that they are "unsafe" so you have to check the unsafe flag in your project properties)
public static unsafe int ToInt32(byte[] value, int startIndex)
{
fixed (byte* numRef = &value[startIndex])
{
var num = (uint)((numRef[0] << 0x18) | (numRef[1] << 0x10) | (numRef[2] << 0x8) | numRef[3]);
return (int)num;
}
}
public static unsafe float ToSingle(byte[] value, int startIndex)
{
int val = ToInt32(value, startIndex);
return *(float*)&val;
}
I assume that your byte[] doesn't contain the binary representation of floats, but either a sequence of Int8s or Int16s. The examples you posted don't look like audio samples based on float (neither NaN nor -2.41E+24 are in the range -1 to 1. While some new audio formats might support floats outside that range, traditionally audio data consists of signed 8 or 16 bit integer samples.
Another thing you need to be aware of is that often the different channels are interleaved. For example it could contain the first sample for the left, then for the right channel, and then the second sample for both... So you need to separate the channels while parsing.
It's also possible, but uncommon that the samples are unsigned. In which case you need to remove the offset from the conversion functions.
So you first need to parse current position in the byte array into an Int8/16. And then convert that integer to a float in the range -1 to 1.
If the format is little endian you can use BitConverter. Another possibility that works with both endiannesses is getting two bytes manually and combining them with a bit-shift. I don't remember if little or big endian is common. So you need to try that yourself.
This can be done with functions like the following(I didn't test them):
float Int8ToFloat(Int8 i)
{
return ((i-Int8.MinValue)*(1f/0xFF))-0.5f;
}
float Int16ToFloat(Int16 i)
{
return ((i-Int16.MinValue)*(1f/0xFFFF))-0.5f;
}
Depends on how you want to convert the bytes to float. Start out by trying to pin-point what is actually stored in the byte array. Perhaps there is a file format specification? Other transformations in your code?
In the case each byte should be converted to a float between 0 and 255:
public float[] ConvertByteToFloat(byte[] array)
{
return array.Select(b => (float)b).ToArray();
}
If the bytes array contains binary representation of floats, there are several representation and if the representation stored in your file does not match the c# language standard floating point representation (IEEE 754) weird things like this will happen.