We Keep Coding

Optimization of conversion from opencv mat/Array to to OnnxRuntime Tensor?

I am using the ONNXRuntime to inference a UNet model and as a part of preprocessing I have to convert an EMGU OpenCV matrix to OnnxRuntime.Tensor.
I achieved it using two nested for loops which is unfortunately quite slow:
var data = new DenseTensor<float>(new[] { 1, 3, WIDTH, HEIGHT});
for (int y = 0; y < HEIGHT; y++)
{
for (int x = 0; x < WIDTH; x++)
{
data[0, 0, x, y] = image.GetValue(2, y, x)/255.0;
data[0, 1, x, y] = image.GetValue(1, y, x)/255.0;
data[0, 2, x, y] = image.GetValue(0, y, x)/255.0;
}
}
Then I found out that there exists a method which converts Array to DenseTensor. I wanted to use this method as follows:
var imgToPredictFloat = new Mat(image.Height, image.Width, DepthType.Cv32F, 3);
image.ConvertTo(imgToPredictFloat, DepthType.Cv32F, 1/255.0);
CvInvoke.CvtColor(imgToPredictFloat, imgToPredictFloat, ColorConversion.Bgra2Rgb);
var data = image.GetData().ToTensor<float>;
var reshaped = data.Reshape(new int[] { 1, 3, WIDTH, HEIGHT});
This would greatly improve the performance however the layout of the output tensor is not correct (the same as from the for loop) and the model obviously won't work. Any suggestions how to reshape the array to the correct layout?
In the code is also performed converting int 0-255 to float 0-1 and BGR layout to RGB layout.

This is how I have used cv::Mat with ONNX Runtime ( C++ ) :
const wchar_t* model_path = L"C:/data/DNN/ONNX/ResNet/resnet152v2/resnet152-v2-7.onnx";
printf("Using Onnxruntime C++ API\n");
Ort::Session session(env, model_path, session_options);
//*************************************************************************
// print model input layer (node names, types, shape etc.)
Ort::AllocatorWithDefaultOptions allocator;
size_t num_output_nodes = session.GetOutputCount();
std::vector<char*> outputNames;
for (size_t i = 0; i < num_output_nodes; ++i)
{
char* name = session.GetOutputName(i, allocator);
std::cout << "output: " << name << std::endl;
outputNames.push_back(name);
}
// print number of model input nodes
size_t num_input_nodes = session.GetInputCount();
std::vector<const char*> input_node_names(num_input_nodes);
std::vector<int64_t> input_node_dims; // simplify... this model has only 1 input node {1, 3, 224, 224}.
// Otherwise need vector<vector<>>
printf("Number of inputs = %zu\n", num_input_nodes);
// iterate over all input nodes
for (int i = 0; i < num_input_nodes; i++) {
// print input node names
char* input_name = session.GetInputName(i, allocator);
printf("Input %d : name=%s\n", i, input_name);
input_node_names[i] = input_name;
// print input node types
Ort::TypeInfo type_info = session.GetInputTypeInfo(i);
auto tensor_info = type_info.GetTensorTypeAndShapeInfo();
ONNXTensorElementDataType type = tensor_info.GetElementType();
printf("Input %d : type=%d\n", i, type);
// print input shapes/dims
input_node_dims = tensor_info.GetShape();
printf("Input %d : num_dims=%zu\n", i, input_node_dims.size());
for (int j = 0; j < input_node_dims.size(); j++)
printf("Input %d : dim %d=%jd\n", i, j, input_node_dims[j]);
}
cv::Size dnnInputSize;
cv::Scalar mean;
cv::Scalar std;
bool rgb = true;
//cv::Mat inputImage = cv::imread("C:/TestImages/kitten_01.jpg");
cv::Mat inputImage = cv::imread("C:/TestImages/slug_01.jpg");
rgb = true;
dnnInputSize = cv::Size(224, 224);
mean[0] = 0.485;
mean[1] = 0.456;
mean[2] = 0.406;
std[0] = 0.229;
std[1] = 0.224;
std[2] = 0.225;
cv::Mat blob;
// ONNX: (N x 3 x H x W)
cv::dnn::blobFromImage(inputImage, blob, 1.0 / 255.0, dnnInputSize, mean, rgb, false);
size_t input_tensor_size = blob.total();
std::vector<float> input_tensor_values(input_tensor_size);
for (size_t i = 0; i < input_tensor_size; ++i)
{
input_tensor_values[i] = blob.at<float>(i);
}
std::vector<const char*> output_node_names = { outputNames.front() };
// create input tensor object from data values
auto memory_info = Ort::MemoryInfo::CreateCpu(OrtArenaAllocator, OrtMemTypeDefault);
Ort::Value input_tensor = Ort::Value::CreateTensor<float>(memory_info, input_tensor_values.data(), input_tensor_size, input_node_dims.data(), 4);
assert(input_tensor.IsTensor());
// score model & input tensor, get back output tensor
auto output_tensors = session.Run(Ort::RunOptions{ nullptr }, input_node_names.data(), &input_tensor, 1, output_node_names.data(), 1);
assert(output_tensors.size() == 1 && output_tensors.front().IsTensor());
// Get pointer to output tensor float values
float* floatarr = output_tensors.front().GetTensorMutableData<float>();
assert(abs(floatarr[0] - 0.000045) < 1e-6);
cv::Mat1f result = cv::Mat1f(1000, 1, floatarr);
cv::Point classIdPoint;
double confidence = 0;
minMaxLoc(result, 0, &confidence, 0, &classIdPoint);
int classId = classIdPoint.y;
std::cout << "confidence: " << confidence << std::endl;
std::cout << "class: " << classId << std::endl;
The actual conversion part that you need is imho (adjust size and mean/std according to your network):
cv::Mat inputImage = cv::imread("C:/TestImages/slug_01.jpg");
rgb = true;
dnnInputSize = cv::Size(224, 224);
mean[0] = 0.485;
mean[1] = 0.456;
mean[2] = 0.406;
std[0] = 0.229;
std[1] = 0.224;
std[2] = 0.225;
cv::Mat blob;
// ONNX: (N x 3 x H x W)
cv::dnn::blobFromImage(inputImage, blob, 1.0 / 255.0, dnnInputSize, mean, rgb, false);

Convert Guid to 2 longs and 2 longs to Guid in c#

Guid is a 128bits structure, long is a Int64 so 64 bits structure, therefore Guid can be used to represent two long and two long can be stored in a Guid.
I have been searching several times for a reliable way to perform the transformation of a Guid to 2 longs and the way around, mainly to get a simple way to provide a tracking id to external services.
The objective is to get a reversable way to pass in a single parameter 2 longs, and decode it back later (of course it is not intended to be used "decoded" on the other side). It is like a session id for the external service.

Warning: these solutions do not take endianness into consideration, and the result may therefore differ from one platform to another
Taking advantage of new features of C# 7, I came out with the following tools class, which transforms long, ulong, int, uint to Guid and reverse:
public static class GuidTools
{
public static Guid GuidFromLongs(long a, long b)
{
byte[] guidData = new byte[16];
Array.Copy(BitConverter.GetBytes(a), guidData, 8);
Array.Copy(BitConverter.GetBytes(b), 0, guidData, 8, 8);
return new Guid(guidData);
}
public static (long, long) ToLongs(this Guid guid)
{
var bytes = guid.ToByteArray();
var long1 = BitConverter.ToInt64(bytes, 0);
var long2 = BitConverter.ToInt64(bytes, 8);
return (long1, long2);
}
public static Guid GuidFromULongs(ulong a, ulong b)
{
byte[] guidData = new byte[16];
Array.Copy(BitConverter.GetBytes(a), guidData, 8);
Array.Copy(BitConverter.GetBytes(b), 0, guidData, 8, 8);
return new Guid(guidData);
}
public static (ulong, ulong) ToULongs(this Guid guid)
{
var bytes = guid.ToByteArray();
var ulong1 = BitConverter.ToUInt64(bytes, 0);
var ulong2 = BitConverter.ToUInt64(bytes, 8);
return (ulong1, ulong2);
}
public static Guid GuidFromInts(int a, int b, int c, int d)
{
byte[] guidData = new byte[16];
Array.Copy(BitConverter.GetBytes(a), guidData, 4);
Array.Copy(BitConverter.GetBytes(b), 0, guidData, 4, 4);
Array.Copy(BitConverter.GetBytes(c), 0, guidData, 8, 4);
Array.Copy(BitConverter.GetBytes(d), 0, guidData, 12, 4);
return new Guid(guidData);
}
public static (int, int , int, int) ToInts(this Guid guid)
{
var bytes = guid.ToByteArray();
var a = BitConverter.ToInt32(bytes, 0);
var b = BitConverter.ToInt32(bytes, 4);
var c = BitConverter.ToInt32(bytes, 8);
var d = BitConverter.ToInt32(bytes, 12);
return (a, b, c, d);
}
public static Guid GuidFromUInts(uint a, uint b, uint c, uint d)
{
byte[] guidData = new byte[16];
Array.Copy(BitConverter.GetBytes(a), guidData, 4);
Array.Copy(BitConverter.GetBytes(b), 0, guidData, 4, 4);
Array.Copy(BitConverter.GetBytes(c), 0, guidData, 8, 4);
Array.Copy(BitConverter.GetBytes(d), 0, guidData, 12, 4);
return new Guid(guidData);
}
public static (uint, uint, uint, uint) ToUInts(this Guid guid)
{
var bytes = guid.ToByteArray();
var a = BitConverter.ToUInt32(bytes, 0);
var b = BitConverter.ToUInt32(bytes, 4);
var c = BitConverter.ToUInt32(bytes, 8);
var d = BitConverter.ToUInt32(bytes, 12);
return (a, b, c, d);
}
}
Also found another solution inspired from there: Converting System.Decimal to System.Guid
[StructLayout(LayoutKind.Explicit)]
struct GuidConverter
{
[FieldOffset(0)]
public decimal Decimal;
[FieldOffset(0)]
public Guid Guid;
[FieldOffset(0)]
public long Long1;
[FieldOffset(8)]
public long Long2;
}
private static GuidConverter _converter;
public static (long, long) FastGuidToLongs(this Guid guid)
{
_converter.Guid = guid;
return (_converter.Long1, _converter.Long2);
}
public static Guid FastLongsToGuid(long a, long b)
{
_converter.Long1 = a;
_converter.Long2 = b;
return _converter.Guid;
}

As an unsafe but very efficient version (no byte[] allocations, via BitConverter):
static void Main()
{
var g = Guid.NewGuid();
Console.WriteLine(g);
GuidToInt64(g, out var x, out var y);
Console.WriteLine(x);
Console.WriteLine(y);
var g2 = GuidFromInt64(x, y);
Console.WriteLine(g2);
}
public static unsafe void GuidToInt64(Guid value, out long x, out long y)
{
long* ptr = (long*)&value;
x = *ptr++;
y = *ptr;
}
public static unsafe Guid GuidFromInt64(long x, long y)
{
long* ptr = stackalloc long[2];
ptr[0] = x;
ptr[1] = y;
return *(Guid*)ptr;
}
You could actually do the same thing with a union struct, if you don't like using the unsafe keyword, but: it is more code, and a union struct is still fundamentally unverifiable, so this doesn't gain you much at the IL level (it just means you don't need the "allow unsafe code" flag):
static void Main()
{
var g = Guid.NewGuid();
Console.WriteLine(g);
var val = new GuidInt64(g);
var x = val.X;
var y = val.Y;
Console.WriteLine(x);
Console.WriteLine(y);
var val2 = new GuidInt64(x, y);
var g2 = val2.Guid;
Console.WriteLine(g2);
}
[StructLayout(LayoutKind.Explicit)]
struct GuidInt64
{
[FieldOffset(0)]
private Guid _guid;
[FieldOffset(0)]
private long _x;
[FieldOffset(8)]
private long _y;
public Guid Guid => _guid;
public long X => _x;
public long Y => _y;
public GuidInt64(Guid guid)
{
_x = _y = 0; // to make the compiler happy
_guid = guid;
}
public GuidInt64(long x, long y)
{
_guid = Guid.Empty;// to make the compiler happy
_x = x;
_y = y;
}
}

Following pair of methods could do what you need:
public static void GuidToInt16(Guid guidToConvert, out long guidAsLong1, out long guidAsLong2)
{
byte[] guidByteArray = guidToConvert.ToByteArray();
var segment1 = new ArraySegment<byte>(guidByteArray, 0, 8);
var segment2 = new ArraySegment<byte>(guidByteArray, 8, 8);
guidAsLong1 = BitConverter.ToInt64(segment1.ToArray(), 0);
guidAsLong2 = BitConverter.ToInt64(segment2.ToArray(), 0);
}
public static Guid Int16ToGuid(long guidAsLong1, long guidAsLong2)
{
var segment1 = BitConverter.GetBytes(guidAsLong1);
var segment2 = BitConverter.GetBytes(guidAsLong2);
return new Guid(segment1.Concat(segment2).ToArray());
}
And possible usage:
Guid guidToConvert = new Guid("cbd5bb87-a249-49ac-8b06-87c124205b99");
long guidAsLong1, guidAsLong2;
GuidToInt16(guidToConvert, out guidAsLong1, out guidAsLong2);
Console.WriteLine(guidAsLong1 + " " + guidAsLong2);
Guid guidConvertedBack = Int16ToGuid(guidAsLong1, guidAsLong2);
Console.WriteLine(guidConvertedBack);
Console.ReadKey();

My solution should help understand whole process with binary operations:
class Program
{
public static Guid LongsToGuid(long l1, long l2)
{
var a = (int)l1;
var b = (short)(l1 >> 32);
var c = (short)(l1 >> 48);
var d = (byte)l2;
var e = (byte)(l2 >> 8);
var f = (byte)(l2 >> 16);
var g = (byte)(l2 >> 24);
var h = (byte)(l2 >> 32);
var i = (byte)(l2 >> 40);
var j = (byte)(l2 >> 48);
var k = (byte)(l2 >> 56);
return new Guid(a, b, c, d, e, f, g, h, i, j, k);
}
public static long BytesToLong(byte[] bytes, int start, int end)
{
long toReturn = 0;
for (var i = start; i < end; i++)
{
toReturn |= ((long)bytes[i]) << (8 * i);
}
return toReturn;
}
static void Main(string[] args)
{
var l1 = long.MinValue;
var l2 = long.MaxValue;
var guid = LongsToGuid(l1, l2);
var guidBytes = guid.ToByteArray();
var readL1 = BytesToLong(guidBytes, 0, 8);
var readL2 = BytesToLong(guidBytes, 8, 16);
Console.WriteLine(l1 == readL1);
Console.WriteLine(l2 == readL2);
Console.ReadKey();
}
}

How can I get the data of 8bit wav file

I'm making a demo about sound in WindowsForm, I created 3 classes for taking data of wave file. Code is below:
RiffBlock
public class RiffBlock
{
private byte[] riffID;
private uint riffSize;
private byte[] riffFormat;
public byte[] RiffID
{
get { return riffID; }
}
public uint RiffSize
{
get { return (riffSize); }
}
public byte[] RiffFormat
{
get { return riffFormat; }
}
public RiffBlock()
{
riffID = new byte[4];
riffFormat = new byte[4];
}
public void ReadRiff(FileStream inFS)
{
inFS.Read(riffID, 0, 4);
BinaryReader binRead = new BinaryReader(inFS);
riffSize = binRead.ReadUInt32();
inFS.Read(riffFormat, 0, 4);
}
}
FormatBlock
public class FormatBlock
{
private byte[] fmtID;
private uint fmtSize;
private ushort fmtTag;
private ushort fmtChannels;
private uint fmtSamplesPerSec;
private uint fmtAverageBytesPerSec;
private ushort fmtBlockAlign;
private ushort fmtBitsPerSample;
public byte[] FmtID
{
get { return fmtID; }
}
public uint FmtSize
{
get { return fmtSize; }
}
public ushort FmtTag
{
get { return fmtTag; }
}
public ushort Channels
{
get { return fmtChannels; }
}
public uint SamplesPerSec
{
get { return fmtSamplesPerSec; }
}
public uint AverageBytesPerSec
{
get { return fmtAverageBytesPerSec; }
}
public ushort BlockAlign
{
get { return fmtBlockAlign; }
}
public ushort BitsPerSample
{
get { return fmtBitsPerSample; }
}
public FormatBlock()
{
fmtID = new byte[4];
}
public void ReadFmt(FileStream inFS)
{
inFS.Read(fmtID, 0, 4);
BinaryReader binRead = new BinaryReader(inFS);
fmtSize = binRead.ReadUInt32();
fmtTag = binRead.ReadUInt16();
fmtChannels = binRead.ReadUInt16();
fmtSamplesPerSec = binRead.ReadUInt32();
fmtAverageBytesPerSec = binRead.ReadUInt32();
fmtBlockAlign = binRead.ReadUInt16();
fmtBitsPerSample = binRead.ReadUInt16();
// This accounts for the variable format header size
// 12 bytes of Riff Header, 4 bytes for FormatId, 4 bytes for FormatSize & the Actual size of the Format Header
inFS.Seek(fmtSize + 20, System.IO.SeekOrigin.Begin);
}
}
DataBlock
public class DataBlock
{
private byte[] dataID;
private uint dataSize;
private Int16[] data;
private int dataNumSamples;
public byte[] DataID
{
get { return dataID; }
}
public uint DataSize
{
get { return dataSize; }
}
public Int16 this[int pos]
{
get { return data[pos]; }
}
public int NumSamples
{
get { return dataNumSamples; }
}
public DataBlock()
{
dataID = new byte[4];
}
public void ReadData(FileStream inFS)
{
inFS.Read(dataID, 0, 4);
BinaryReader binRead = new BinaryReader(inFS);
dataSize = binRead.ReadUInt32();
data = new Int16[dataSize];
inFS.Seek(40, SeekOrigin.Begin);
dataNumSamples = (int)(dataSize / 2);
for (int i = 0; i < dataNumSamples; i++)
{
data[i] = binRead.ReadInt16();
}
}
}
It works ok with only 16bit wave file, but when I choose a 8 bit wav file or another, the result of this command dataSize = binRead.ReadUInt32();is only 4 although the file size is big.
How I can get the data of 8bit, 24bit... wav file?
Some solutions is appreciated, thank you very much.

Your reading methodology is flawed. The length is correct but for an 8 bits per sample file you should be reading bytes not words; as it stands the data will be incorrect and the value returned by the NumSamples property will be wrong.
In my case, the sub chunk size is 1160, the number of channels is 1 (mono) and the bits per sample is 8 (byte). You will need to decode the bits per sample and adjust your reading accordingly. For the WAV file I used, your program allocated a data array the correct length but 16 bit, divided the data length by 2 and called this the number of samples (wrong, it should be 1160) and then proceeded to read 580 word values from the stream.
Edit: My ancient code will not cut it in the modern age (I seem to recall having to modify it some years ago to cope with at least one additional chunk type but the details escape me).
This is what you get; anything more and your question should read "Could someone write me a program to load WAV files", as it is, we are way beyond the original question and it is time for you to knuckle down and make it work how you need it to :-)
References:
http://www.neurophys.wisc.edu/auditory/riff-format.txt
http://www-mmsp.ece.mcgill.ca/Documents/AudioFormats/WAVE/WAVE.html
http://www.shikadi.net/moddingwikiResource_Interchange_File_Format_(RIFF)
http://soundfile.sapp.org/doc/WaveFormat/
All are very useful.
///<summary>
///Reads up to 16 bit WAV files
///</summary>
///<remarks> Things have really changed in the last 15 years</remarks>
public class RiffLoader
{
public enum RiffStatus { Unknown = 0, OK, FileError, FormatError, UnsupportedFormat };
LinkedList<Chunk> chunks = new LinkedList<Chunk>();
RiffStatus status = RiffStatus.Unknown;
List<String> errorMessages = new List<string>();
String source;
public String SourceName { get { return source; } }
public RiffStatus Status { get { return status; } }
public String[] Messages { get { return errorMessages.ToArray(); } }
enum chunkType { Unknown = 0, NoMore, Riff, Fmt, Fact, Data, Error = -1 };
static Int32 scan32bits(byte[] source, int offset = 0)
{
return source[offset] | source[offset + 1] << 8 | source[offset + 2] << 16 | source[offset + 3] << 24;
}
static Int32 scan16bits(byte[] source, int offset = 0)
{
return source[offset] | source[offset + 1] << 8;
}
static Int32 scan8bits(byte[] source, int offset = 0)
{
return source[offset];
}
abstract class Chunk
{
public chunkType Ident = chunkType.Unknown;
public int ByteCount = 0;
}
class RiffChunk : Chunk
{
public RiffChunk(int count)
{
this.Ident = chunkType.Riff;
this.ByteCount = count;
}
}
class FmtChunk : Chunk
{
int formatCode = 0;
int channels = 0;
int samplesPerSec = 0;
int avgBytesPerSec = 0;
int blockAlign = 0;
int bitsPerSample = 0;
int significantBits = 0;
public int Format { get { return formatCode; } }
public int Channels { get { return channels; } }
public int BlockAlign { get { return blockAlign; } }
public int BytesPerSample { get { return bitsPerSample / 8 + ((bitsPerSample % 8) > 0 ? 1 : 0); } }
public int BitsPerSample
{
get
{
if (significantBits > 0)
return significantBits;
return bitsPerSample;
}
}
public FmtChunk(byte[] buffer) : base()
{
int size = buffer.Length;
// if the length is 18 then buffer 16,17 should be 00 00 (I don't bother checking)
if (size != 16 && size != 18 && size != 40)
return;
formatCode = scan16bits(buffer, 0);
channels = scan16bits(buffer, 2);
samplesPerSec = scan32bits(buffer, 4);
avgBytesPerSec = scan32bits(buffer, 8);
blockAlign = scan16bits(buffer, 12);
bitsPerSample = scan16bits(buffer, 14);
if (formatCode == 0xfffe) // EXTENSIBLE
{
if (size != 40)
return;
significantBits = scan16bits(buffer, 18);
// skiping speaker map
formatCode = scan16bits(buffer, 24); // first two bytes of the GUID
// the rest of the GUID is fixed, decode it and check it if you wish
}
this.Ident = chunkType.Fmt;
this.ByteCount = size;
}
}
class DataChunk : Chunk
{
byte[] samples = null;
///<summary>
///Create a data chunk
///<para>
///The supplied buffer must be correctly sized with zero offset and must be purely for this class
///</para>
///<summary>
///<param name="buffer">source array</param>
public DataChunk(byte[] buffer)
{
this.Ident = chunkType.Data;
this.ByteCount = buffer.Length;
samples = buffer;
}
public enum SampleStatus { OK, Duff }
public class Samples
{
public SampleStatus Status = SampleStatus.Duff;
public List<int[]> Channels = new List<int[]>();
#if false // debugger helper method
/*
** Change #if false to #if true to include this
** Break at end of GetSamples on "return retval"
** open immediate window and type retval.DumpLast(16)
** look in output window for dump of last 16 entries
*/
public int DumpLast(int count)
{
for (int i = Channels[0].Length - count; i < Channels[0].Length; i++)
Console.WriteLine(String.Format("{0:X4} {1:X4},{2:X4}", i, Channels[0][i], Channels[1][i]));
return 0;
}
#endif
}
/*
** Return the decoded samples
*/
public Samples GetSamples(FmtChunk format)
{
Samples retval = new Samples();
int samplesPerChannel = this.ByteCount / (format.BytesPerSample * format.Channels);
int mask = 0, sign=0;
int [][] samples = new int [format.Channels][];
for (int c = 0; c < format.Channels; c++)
samples[c] = new int[samplesPerChannel];
if (format.BitsPerSample >= 8 && format.BitsPerSample <= 16) // 24+ is left as an excercise
{
sign = (int)Math.Floor(Math.Pow(2, format.BitsPerSample - 1));
mask = (int)Math.Floor(Math.Pow(2, format.BitsPerSample)) - 1;
int offset = 0, index = 0;
int s = 0;
while (index < samplesPerChannel)
{
for (int c = 0; c < format.Channels; c++)
{
switch (format.BytesPerSample)
{
case 1:
s = scan8bits(this.samples, offset) & mask;
break;
case 2:
s = scan16bits(this.samples, offset) & mask;
break;
}
// sign extend the data to Int32
samples[c][index] = s | ((s & sign) != 0 ? ~mask : 0);
offset += format.BytesPerSample;
}
++index;
}
retval.Channels = new List<int[]>(samples);
retval.Status = SampleStatus.OK;
}
return retval;
}
}
class FactChunk : Chunk
{
int samplesPerChannel;
public int SamplesPerChannel { get { return samplesPerChannel; } }
public FactChunk(byte[] buffer)
{
this.Ident = chunkType.Fact;
this.ByteCount = buffer.Length;
if (buffer.Length >= 4)
samplesPerChannel = scan32bits(buffer);
}
}
class DummyChunk : Chunk
{
public DummyChunk(int size, chunkType type = chunkType.Unknown)
{
this.Ident = type;
this.ByteCount = size;
}
}
public RiffLoader(String fileName)
{
source = fileName;
try
{
using (FileStream fs = new FileStream(fileName, FileMode.Open, FileAccess.Read))
using (BinaryReader reader = new BinaryReader(fs))
{
Chunk c = getChunk(fs, reader);
if (c.Ident != chunkType.Riff)
{
status = RiffStatus.FileError;
errorMessages.Add(String.Format("Error loading \"{0}\": No valid header"));
}
chunks.AddLast(c);
c = getChunk(fs, reader);
if (c.Ident != chunkType.Fmt)
{
status = RiffStatus.FileError;
errorMessages.Add(String.Format("Error loading \"{0}\": No format chunk"));
}
chunks.AddLast(c);
/*
** From now on we just keep scanning to the end of the file
*/
while (fs.Position < fs.Length)
{
c = getChunk(fs, reader);
switch (c.Ident)
{
case chunkType.Fact:
case chunkType.Data:
chunks.AddLast(c);
break;
case chunkType.Unknown:
break; // skip it - don't care what it is
}
}
FmtChunk format = null;
foreach (var chunk in chunks)
{
switch(chunk.Ident)
{
case chunkType.Fmt:
format = chunk as FmtChunk;
break;
case chunkType.Data:
if (format != null)
{
DataChunk dc = chunk as DataChunk;
var x = dc.GetSamples(format);
}
break;
}
}
}
}
catch (Exception e)
{
status = RiffStatus.FileError;
errorMessages.Add(String.Format("Error loading \"{0}\": {1}", e.Message));
}
}
/*
** Get a chunk of data from the file - knows nothing of the internal format of the chunk.
*/
Chunk getChunk(FileStream stream, BinaryReader reader)
{
byte[] buffer;
int size;
buffer = reader.ReadBytes(8);
if (buffer.Length == 8)
{
String prefix = new String(Encoding.ASCII.GetChars(buffer, 0, 4));
size = scan32bits(buffer, 4);
if (size + stream.Position <= stream.Length) // skip if there isn't enough data
{
if (String.Compare(prefix, "RIFF") == 0)
{
/*
** only "WAVE" type is acceptable
**
** Don't read size bytes or the entire file will end up in the RIFF chunk
*/
if (size >= 4)
{
buffer = reader.ReadBytes(4);
String ident = new String(Encoding.ASCII.GetChars(buffer, 0, 4));
if (String.CompareOrdinal(ident, "WAVE") == 0)
return new RiffChunk(size - 4);
}
}
else if (String.Compare(prefix, "fmt ") == 0)
{
if (size >= 16)
{
buffer = reader.ReadBytes(size);
if (buffer.Length == size)
return new FmtChunk(buffer);
}
}
else if (String.Compare(prefix, "fact") == 0)
{
if (size >= 4)
{
buffer = reader.ReadBytes(4);
if (buffer.Length == size)
return new FactChunk(buffer);
}
}
else if (String.Compare(prefix, "data") == 0)
{
// assume that there has to be data
if (size > 0)
{
buffer = reader.ReadBytes(size);
if ((size & 1) != 0) // odd length?
{
if (stream.Position < stream.Length)
reader.ReadByte();
else
size = -1; // force an error - there should be a pad byte
}
if (buffer.Length == size)
return new DataChunk(buffer);
}
}
else
{
/*
** there are a number of weird and wonderful block types - assume there has to be data
*/
if (size > 0)
{
buffer = reader.ReadBytes(size);
if ((size & 1) != 0) // odd length?
{
if (stream.Position < stream.Length)
reader.ReadByte();
else
size = -1; // force an error - there should be a pad byte
}
if (buffer.Length == size)
{
DummyChunk skip = new DummyChunk(size);
return skip;
}
}
}
}
}
return new DummyChunk(0, chunkType.Error);
}
}
You will need to add properties as required and code to navigate the returned linked list. Particular properties you may need are the sample rates in the format block, assuming you are going to process the data in some way.
Adding 24 to 32 bits is simple, if you need to go beyond 32 bits you will have to switch to int64's.
I have tested it with some good samples from http://www-mmsp.ece.mcgill.ca/Documents/AudioFormats/WAVE/Samples.html, as well as your 8 bit file, and it decodes OK and seems to have the correct data.
You should be more than capable of making it work now, good luck.
Edit (again, like the proverbial dog...):
Of course, I should have sign extended the value, so DataChunk.GetSamples() now does that. Looking at the Codeproject files (it isn't the greatest code by the way, but the guy does say that he is only just learning C#, so fair play for tackling a graphical user control) it is obvious that the data is signed. It's a shame that he didn't standardise his source to be an array of Int32's, then it wouldn't matter how many bits the WAV was encoded in.

You could use the Naudio library:
https://naudio.codeplex.com/ (take a look at their website, there are quite a lot of tutorials).
Hope this helps :).

After setting tree node height, treeview scroll incorrectly

public Form2()
{
InitializeComponent();
}
private void Form2_Load(object sender, EventArgs e)
{
Dictionary<int, TreeNode> map = new Dictionary<int, TreeNode>();
for (int i = 0; i < 100; i++)
{
TreeNode node = new TreeNode(i.ToString());
if (i % 10 == 0)
_tree.Nodes.Add(node);
map.Add(i, node);
if (i % 10 != 0)
{
map[Convert.ToInt32(Math.Floor(i / 10.0)) * 10].Nodes.Add(node);
node.SetNodeHeight(2);
}
}
}
}
public static class TreeViewExtendMethod
{
[StructLayout(LayoutKind.Sequential)]
struct TVITEMEX
{
public Mask mask;
public IntPtr item;
public uint state;
public uint stateMask;
public IntPtr pszText;
public int cchTextMax;
public int iImage;
public int iSelectedImage;
public int iChildren;
public IntPtr lParam;
public int iIntegral;
// ... plus some windows 6+ crap
}
[Flags]
enum Mask : uint
{
Text = 1,
Image = 2,
Param = 4,
State = 8,
Handle = 16,
SelectedImage = 32,
Children = 64,
Integral = 128,
}
public static int GetNodeHeight(this TreeNode tn)
{
TVITEMEX tvix = new TVITEMEX();
tvix.mask = Mask.Handle | Mask.Integral;
tvix.item = tn.Handle;
tvix.iIntegral = 0;
IntPtr hParam=Marshal.AllocHGlobal(Marshal.SizeOf(tvix));
Marshal.StructureToPtr(tvix, hParam, false);
win32.SendMessage(tn.TreeView.Handle, win32.TVM_GETITEM, IntPtr.Zero, hParam);
return tvix.iIntegral;
}
public static void SetNodeHeight(this TreeNode tn, int height)
{
TVITEMEX tvix = new TVITEMEX();
tvix.mask = Mask.Handle | Mask.Integral;
tvix.item = tn.Handle;
tvix.iIntegral = height;
IntPtr hParam=Marshal.AllocHGlobal(Marshal.SizeOf(tvix));
Marshal.StructureToPtr(tvix, hParam, false);
win32.SendMessage(tn.TreeView.Handle, win32.TVM_SETITEM, IntPtr.Zero, hParam);
}
}
The above is the sample code, after using the PINV to change the node height manually, when click in the bottom of the scroll bar, the treeview will scroll which should not scroll at all as it just a click.
At first, I think it may be caused by scroll bar, it doesn't work as expected after I set the scroll Msg para (8) to end scroll. At last, while catching the scroll msg (0x115), exits the WndProc, and then it works. I have no idea what is going on as I can't find any msg can cancel scroll except setting the wParam to 8 which is certainly not work.

I finally find a solution, though not elegant, but it should works.
Here is the solution,
protected override void WndProc(ref Message m)
{
if (m.Msg == 0x115)
{
if (((int)m.WParam & 5) == (5) ||
((int)m.WParam & 4) == (4)
)
{
int para = (int)m.WParam;
int temp = (para - (para % 2 == 0 ? 4 : 5)) >> 16;
// store the current position, if the position doesn't change, no need to handle
if (temp == pos)
return;
else
pos = temp;
}
if (((int)m.WParam & 8) == 8)
return;
//Debug.WriteLine(m.WParam);
//if (DisableScroll)
// return;
// //m.WParam = (IntPtr)8;
//SCROLLINFO si = new SCROLLINFO();
//si.cbSize = Marshal.SizeOf(si);
//si.fMask = (int)(ScrollInfoMask.SIF_POS | ScrollInfoMask.SIF_PAGE | ScrollInfoMask.SIF_RANGE | ScrollInfoMask.SIF_TRACKPOS);
//GetScrollInfo(this.Handle, 1, ref si);
////Debug.WriteLine(string.Format("Max pos:{0}, Min pos: {1}", si.nMin, si.nMax));
//Debug.WriteLine(string.Format("wParam:{0} hParam:{1}", m.WParam, m.LParam));
//Debug.WriteLine(string.Format("It page size: {0}, position: {1}, range: {2}, track pos: {3}", si.nPage, si.nPos, si.nPage, si.nTrackPos));
//if (si.nPage + Math.Max(si.nPos, si.nTrackPos) >= si.nMax)
//{
// return;
//}
}
base.WndProc(ref m);
}

How can I convert IP range to Cidr in C#?

There has lots of example of convert CIDR to ip range. But I want to know how can I use start/end ip address to generate a/some cidr in C#?
for example:
I have start ip address(192.168.0.1) and end ip address(192.168.0.254). So use these two address to generate cidr list {192.168.0.0/31, 192.168.0.2/32}. Is there any C# code example?

CIDR class with static methods to split an IP range into a minimal set of disjoint CIDR ranges, which cover exactly the original IP range.
The split methods (the "real" one working on BigIntegers doing the actual work, and the wrapper for IP addresses and CIDR creation) are at the bottom.
Use with foreach (IPRangeToCidr.CIDR c in IPRangeToCidr.CIDR.split(first, last)) ...
Requires System.Numerics.dll in the references.
using System;
using System.Numerics;
using System.Net;
using System.Net.Sockets;
using System.Collections.Generic;
namespace IPRangeToCidr {
public struct CIDR {
private IPAddress address;
private uint network_length, bits;
public CIDR(IPAddress address, uint network_length) {
this.address = address;
this.network_length = network_length;
this.bits = AddressFamilyBits(address.AddressFamily);
if (network_length > bits) {
throw new ArgumentException("Invalid network length " + network_length + " for " + address.AddressFamily);
}
}
public IPAddress NetworkAddress {
get { return address; }
}
public IPAddress LastAddress {
get { return IPAddressAdd(address, (new BigInteger(1) << (int) HostLength) - 1); }
}
public uint NetworkLength {
get { return network_length; }
}
public uint AddressBits {
get { return bits; }
}
public uint HostLength {
get { return bits - network_length; }
}
override public String ToString() {
return address.ToString() + "/" + NetworkLength.ToString();
}
public String ToShortString() {
if (network_length == bits) return address.ToString();
return address.ToString() + "/" + NetworkLength.ToString();
}
/* static helpers */
public static IPAddress IPAddressAdd(IPAddress address, BigInteger i) {
return IPFromUnsigned(IPToUnsigned(address) + i, address.AddressFamily);
}
public static uint AddressFamilyBits(AddressFamily family) {
switch (family) {
case AddressFamily.InterNetwork:
return 32;
case AddressFamily.InterNetworkV6:
return 128;
default:
throw new ArgumentException("Invalid address family " + family);
}
}
private static BigInteger IPToUnsigned(IPAddress addr) {
/* Need to reverse addr bytes for BigInteger; prefix with 0 byte to force unsigned BigInteger
* read BigInteger bytes as: bytes[n] bytes[n-1] ... bytes[0], address is bytes[0] bytes[1] .. bytes[n] */
byte[] b = addr.GetAddressBytes();
byte[] unsigned = new byte[b.Length + 1];
for (int i = 0; i < b.Length; ++i) {
unsigned[i] = b[(b.Length - 1) - i];
}
unsigned[b.Length] = 0;
return new BigInteger(unsigned);
}
private static byte[] GetUnsignedBytes(BigInteger unsigned, uint bytes) {
/* reverse bytes again. check that now higher bytes are actually used */
if (unsigned.Sign < 0) throw new ArgumentException("argument must be >= 0");
byte[] data = unsigned.ToByteArray();
byte[] result = new byte[bytes];
for (int i = 0; i < bytes && i < data.Length; ++i) {
result[bytes - 1 - i] = data[i];
}
for (uint i = bytes; i < data.Length; ++i) {
if (data[i] != 0) throw new ArgumentException("argument doesn't fit in requested number of bytes");
}
return result;
}
private static IPAddress IPFromUnsigned(BigInteger unsigned, System.Net.Sockets.AddressFamily family) {
/* IPAddress(byte[]) constructor picks family from array size */
switch (family) {
case System.Net.Sockets.AddressFamily.InterNetwork:
return new IPAddress(GetUnsignedBytes(unsigned, 4));
case System.Net.Sockets.AddressFamily.InterNetworkV6:
return new IPAddress(GetUnsignedBytes(unsigned, 16));
default:
throw new ArgumentException("AddressFamily " + family.ToString() + " not supported");
}
}
/* splits set [first..last] of unsigned integers into disjoint slices { x,..., x + 2^k - 1 | x mod 2^k == 0 }
* covering exaclty the given set.
* yields the slices ordered by x as tuples (x, k)
* This code relies on the fact that BigInteger can't overflow; temporary results may need more bits than last is using.
*/
public static IEnumerable<Tuple<BigInteger, uint>> split(BigInteger first, BigInteger last) {
if (first > last) yield break;
if (first < 0) throw new ArgumentException();
last += 1;
/* mask == 1 << len */
BigInteger mask = 1;
uint len = 0;
while (first + mask <= last) {
if ((first & mask) != 0) {
yield return new Tuple<BigInteger, uint>(first, len);
first += mask;
}
mask <<= 1;
++len;
}
while (first < last) {
mask >>= 1;
--len;
if ((last & mask) != 0) {
yield return new Tuple<BigInteger, uint>(first, len);
first += mask;
}
}
}
public static IEnumerable<CIDR> split(IPAddress first, IPAddress last) {
if (first.AddressFamily != last.AddressFamily) {
throw new ArgumentException("AddressFamilies don't match");
}
AddressFamily family = first.AddressFamily;
uint bits = AddressFamilyBits(family); /* split on numbers returns host length, CIDR takes network length */
foreach (Tuple<BigInteger, uint> slice in split(IPToUnsigned(first), IPToUnsigned(last))) {
yield return new CIDR(IPFromUnsigned(slice.Item1, family), bits - slice.Item2);
}
}
}
}

It is difficult to determine the what exactly is being asked here (the CIDR list you give doesn't seem to correspond with the given input addresses), however the following code will allow you to find the smallest single CIDR that contains the specified start and end addresses.
You need to first convert the start and end IP addresses into 32 bit integers (e.g. 192.168.0.1 becomes 0xc0a80001), then apply the following algorithm:
var startAddr = 0xc0a80001; // 192.168.0.1
var endAddr = 0xc0a800fe; // 192.168.0.254
// Determine all bits that are different between the two IPs
var diffs = startAddr ^ endAddr;
// Now count the number of consecutive zero bits starting at the most significant
var bits = 32;
var mask = 0;
while (diffs != 0)
{
// We keep shifting diffs right until it's zero (i.e. we've shifted all the non-zero bits off)
diffs >>= 1;
// Every time we shift, that's one fewer consecutive zero bits in the prefix
bits--;
// Accumulate a mask which will have zeros in the consecutive zeros of the prefix and ones elsewhere
mask = (mask << 1) | 1;
}
// Construct the root of the range by inverting the mask and ANDing it with the start address
var root = startAddr & ~mask;
// Finally, output the range
Console.WriteLine("{0}.{1}.{2}.{3}/{4}", root >> 24, (root >> 16) & 0xff, (root >> 8) & 0xff, root & 0xff, bits);
Running it on the two addresses in your question gives:
192.168.0.0/24

Necromancing.
No, there wasn't, and I don't understand why people keep upvoting wrong answers.
Here's the code for IP-range to CIDR & vice-versa:
// https://dev.maxmind.com/geoip/
// https://stackoverflow.com/questions/461742/how-to-convert-an-ipv4-address-into-a-integer-in-c
public static string IPrange2CIDR(string ip1, string ip2)
{
uint startAddr = IP2num(ip1);
uint endAddr = IP2num(ip2);
// uint startAddr = 0xc0a80001; // 192.168.0.1
// uint endAddr = 0xc0a800fe; // 192.168.0.254
// uint startAddr = System.BitConverter.ToUInt32(System.Net.IPAddress.Parse(ip1).GetAddressBytes(), 0);
// uint endAddr = System.BitConverter.ToUInt32(System.Net.IPAddress.Parse(ip2).GetAddressBytes(), 0);
if (startAddr > endAddr)
{
uint temp = startAddr;
startAddr = endAddr;
endAddr = temp;
}
// uint diff = endAddr - startAddr -1;
// int bits = 32 - (int)System.Math.Ceiling(System.Math.Log10(diff) / System.Math.Log10(2));
// return ip1 + "/" + bits;
uint diffs = startAddr ^ endAddr;
// Now count the number of consecutive zero bits starting at the most significant
int bits = 32;
// int mask = 0;
// We keep shifting diffs right until it's zero (i.e. we've shifted all the non-zero bits off)
while (diffs != 0)
{
diffs >>= 1;
bits--; // Every time we shift, that's one fewer consecutive zero bits in the prefix
// Accumulate a mask which will have zeros in the consecutive zeros of the prefix and ones elsewhere
// mask = (mask << 1) | 1;
}
string res = ip1 + "/" + bits;
System.Console.WriteLine(res);
return res;
}
// https://www.digitalocean.com/community/tutorials/understanding-ip-addresses-subnets-and-cidr-notation-for-networking
public static void CIDR2IP(string IP)
{
string[] parts = IP.Split('.', '/');
uint ipnum = (System.Convert.ToUInt32(parts[0]) << 24) |
(System.Convert.ToUInt32(parts[1]) << 16) |
(System.Convert.ToUInt32(parts[2]) << 8) |
System.Convert.ToUInt32(parts[3]);
int maskbits = System.Convert.ToInt32(parts[4]);
uint mask = 0xffffffff;
mask <<= (32 - maskbits);
uint ipstart = ipnum & mask;
uint ipend = ipnum | (mask ^ 0xffffffff);
string fromRange = string.Format("{0}.{1}.{2}.{3}", ipstart >> 24, (ipstart >> 16) & 0xff, (ipstart >> 8) & 0xff, ipstart & 0xff);
string toRange = string.Format("{0}.{1}.{2}.{3}", ipend >> 24, (ipend >> 16) & 0xff, (ipend >> 8) & 0xff, ipend & 0xff);
System.Console.WriteLine(fromRange + " - " + toRange);
}
public static uint IP2num(string ip)
{
string[] nums = ip.Split('.');
uint first = System.UInt32.Parse(nums[0]);
uint second = System.UInt32.Parse(nums[1]);
uint third = System.UInt32.Parse(nums[2]);
uint fourth = System.UInt32.Parse(nums[3]);
return (first << 24) | (second << 16) | (third << 8) | (fourth);
}
public static void Test()
{
string IP = "5.39.40.96/27";
// IP = "88.84.128.0/19";
CIDR2IP(IP);
// IPrange2CIDR("88.84.128.0", "88.84.159.255");
IPrange2CIDR("5.39.40.96", "5.39.40.127");
System.Console.WriteLine(System.Environment.NewLine);
System.Console.WriteLine(" --- Press any key to continue --- ");
System.Console.ReadKey();
}

I use this for IpV4, let me know if there is a problem in it.
You can find the extraction source code from the following link:
https://blog.ip2location.com/knowledge-base/how-to-convert-ip-address-range-into-cidr/
using System;
using System.Collections.Generic;
using System.Net;
namespace ConsoleApp
{
public class IPNetwork
{
private readonly long _firstIpAddress;
private readonly long _lastIpAddress;
public static IPNetwork[] FromIpRange(IPAddress firstIpAddress, IPAddress lastIpAddress)
=> FromIpRange(IpAddressToLong(firstIpAddress), IpAddressToLong(lastIpAddress));
public static IPNetwork[] FromIpRange(long firstIpAddress, long lastIpAddress)
{
var result = new List<IPNetwork>();
while (lastIpAddress >= firstIpAddress)
{
byte maxSize = 32;
while (maxSize > 0)
{
long mask = IMask(maxSize - 1);
long maskBase = firstIpAddress & mask;
if (maskBase != firstIpAddress)
break;
maxSize--;
}
double x = Math.Log(lastIpAddress - firstIpAddress + 1) / Math.Log(2);
byte maxDiff = (byte)(32 - Math.Floor(x));
if (maxSize < maxDiff)
{
maxSize = maxDiff;
}
var ipAddress = IpAddressFromLong(firstIpAddress);
result.Add(new IPNetwork(ipAddress, maxSize));
firstIpAddress += (long)Math.Pow(2, 32 - maxSize);
}
return result.ToArray();
}
private static long IMask(int s)
{
return (long)(Math.Pow(2, 32) - Math.Pow(2, 32 - s));
}
public static long IpAddressToLong(IPAddress ipAddress)
{
var bytes = ipAddress.GetAddressBytes();
return ((long)bytes[0] << 24) | ((long)bytes[1] << 16) | ((long)bytes[2] << 8) | bytes[3];
}
public static IPAddress IpAddressFromLong(long ipAddress)
=> new IPAddress((uint)IPAddress.NetworkToHostOrder((int)ipAddress));
public IPNetwork(IPAddress prefix, int prefixLength = 32)
{
if (prefix.AddressFamily != System.Net.Sockets.AddressFamily.InterNetwork)
throw new NotSupportedException("IPv6 is not supported");
Prefix = prefix;
PrefixLength = prefixLength;
var mask = (uint)~(0xFFFFFFFFL >> prefixLength);
_firstIpAddress = IpAddressToLong(Prefix) & mask;
_lastIpAddress = _firstIpAddress | ~mask;
}
public static IPNetwork Parse(string value)
{
try
{
var parts = value.Split('/');
return new IPNetwork(IPAddress.Parse(parts[0]), int.Parse(parts[1]));
}
catch
{
throw new FormatException($"Could not parse IPNetwork from {value}");
}
}
public override string ToString() => $"{Prefix}/{PrefixLength}";
public IPAddress Prefix { get; }
public int PrefixLength { get; }
public IPAddress LastAddress => IpAddressFromLong(_lastIpAddress);
public IPAddress FirstAddress => IpAddressFromLong(_firstIpAddress);
public long Total => _lastIpAddress - _firstIpAddress + 1;
}
}
Usage 1:
var startAddress = IPAddress.Parse("192.168.0.0");
var endAddress = IPAddress.Parse("192.168.0.255");
foreach (var item in IPNetwork.FromIpRange(startAddress, endAddress))
Console.WriteLine(item);
Result
192.168.0.0/24
Usage 2:
var startAddress = IPAddress.Parse("192.168.0.1");
var endAddress = IPAddress.Parse("192.168.0.254");
foreach (var item in IPNetwork.FromIpRange(startAddress, endAddress))
Console.WriteLine(item);
Result:
192.168.0.1/32
192.168.0.2/31
192.168.0.4/30
192.168.0.8/29
192.168.0.16/28
192.168.0.32/27
192.168.0.64/26
192.168.0.128/26
192.168.0.192/27
192.168.0.224/28
192.168.0.240/29
192.168.0.248/30
192.168.0.252/31
192.168.0.254/32

I would recommend the use of IPNetwork Library https://github.com/lduchosal/ipnetwork.
As of version 2, it supports IPv4 and IPv6 as well.
Supernet
IPNetwork network = IPNetwork.Parse("192.168.0.1");
IPNetwork network2 = IPNetwork.Parse("192.168.0.254");
IPNetwork ipnetwork = IPNetwork.Supernet(network, network2);
Console.WriteLine("Network : {0}", ipnetwork.Network);
Console.WriteLine("Netmask : {0}", ipnetwork.Netmask);
Console.WriteLine("Broadcast : {0}", ipnetwork.Broadcast);
Console.WriteLine("FirstUsable : {0}", ipnetwork.FirstUsable);
Console.WriteLine("LastUsable : {0}", ipnetwork.LastUsable);
Console.WriteLine("Usable : {0}", ipnetwork.Usable);
Console.WriteLine("Cidr : {0}", ipnetwork.Cidr);
Output
Network : 192.168.0.0
Netmask : 255.255.255.0
Broadcast : 192.168.0.255
FirstUsable : 192.168.0.1
LastUsable : 192.168.0.254
Usable : 254
Cidr : 24
Have fun !

I found this C code and converted it into C# and it's working now.