We Keep Coding

Scrolling through a waveform

I have a waveform visualiser I am trying to make for some audio editing, and need to be able to scroll through the wave form. The code I'm currently using comes from this question and works after I made some modification to allow the specifying of a start audio time and end audio time:
public Texture2D PaintWaveformSpectrum(AudioClip audio, int textWidth, int textHeight, int audioStart, int audioEnd, Color col) {
Texture2D tex = new Texture2D(textWidth, textHeight, TextureFormat.RGBA32, false);
float[] samples = new float[audioLength];
float[] waveform = new float[textWidth];
audio.GetData(samples, 0);
int packSize = ((audioEnd - audioStart) / textWidth) + 1;
if (audioStart != 0) {
audioStart += packSize % audioStart;
}
int s = 0;
for (int i = audioStart; i < audioEnd; i += packSize) {
waveform[s] = Mathf.Abs(samples[i]);
s++;
}
for (int x = 0; x < textWidth; x++) {
for (int y = 0; y < textHeight; y++) {
tex.SetPixel(x, y, Color.gray);
}
}
for (int x = 0; x < waveform.Length; x++) {
for (int y = 0; y <= waveform[x] * ((float)textHeight * .75f); y++) {
tex.SetPixel(x, (textHeight / 2) + y, col);
tex.SetPixel(x, (textHeight / 2) - y, col);
}
}
tex.Apply();
return tex;
}
The issue here however, is that when I'm scrolling through the audio, the waveform changes. It does indeed scroll, but the issue is that it is now showing different values in the waveform. This is because there are significantly more samples than pixels, so there is a need to down sample. At the moment, every nth sample is chosen, but the issue is with a different start point, different samples will be chosen. Images below for comparison (additionally, here's a video. This is what I want the scroll to look like):
As you can see they are slightly different. The overall structure is there but the waveform is ultimately different.
I thought this would be an easy fix - shift the start audio value to the nearest packSize (ie, audioStart += packSize % audioStart when audioStart != 0) but this didn't work. The same issue still occurred.
If anyone has any suggestions on how I can keep the waveform consistent while scrolling it would be much appreciated.

Despite years of programming experience, I still can't seem to correctly round a number. It was as simple as that.
The line
if (audioStart != 0) {
audioStart += packSize % audioStart;
}
should be
audioStart = (int) Mathf.Round(audioStart / packSize) * packSize;
Adding 1 extra byte to waveform is also necessary as half the time the rounding will cause there to be one extra sample included. As such, waveform should be defined as:
float[] waveform = new float[textWidth+1];
This solves the issue and the samples are chosen consistently. I'm not quite sure how programs like audacity manage to get nice looking waveforms that aren't super noisy (comparison below for the same song: mine on top, audacity below) but that's for another question.

Enhance performance to paint image, is SIMD perhapse a solution?

I have no experience with SIMD, but have a method that is too slow. I know get 40fps, and I need more.
Does anyone know how I could make this paint method faster? Perhaps the SIMD instructions are a solution?
The sourceData is now a byte[] (videoBytes) but could use a pointer too.
public bool PaintFrame(IntPtr layerBuffer, ushort vStart, byte vScale)
{
for (ushort y = 0; y < height; y++)
{
ushort eff_y = (ushort)(vScale * (y - vStart) / 128);
var newY = tileHeight > 0 ? eff_y % tileHeight : 0;
uint y_add = (uint)(newY * tileWidth * bitsPerPixel >> 3);
for (int x = 0; x < width; x++)
{
var newX = tileWidth > 0 ? x % tileWidth : 0;
ushort x_add = (ushort)(newX * bitsPerPixel >> 3);
uint tile_offset = y_add + x_add;
byte color = videoBytes[tile_offset];
var colorIndex = BitsPerPxlCalculation(color, newX);
// Apply Palette Offset
if (paletteOffset > 0)
colorIndex += paletteOffset;
var place = x + eff_y * width;
Marshal.WriteByte(layerBuffer + place, colorIndex);
}
}
return true;
}
private void UpdateBitPerPixelMethod()
{
// Convert tile byte to indexed color
switch (bitsPerPixel)
{
case 1:
BitsPerPxlCalculation = (color, newX) => color;
break;
case 2:
BitsPerPxlCalculation = (color, newX) => (byte)(color >> 6 - ((newX & 3) << 1) & 3);
break;
case 4:
BitsPerPxlCalculation = (color, newX) => (byte)(color >> 4 - ((newX & 1) << 2) & 0xf);
break;
case 8:
BitsPerPxlCalculation = (color, newX) => color;
break;
}
}
More info
Depending on the settings, the bpp can be changed. The indexed colors and the palette colors are separatly stored. Here I have to recreate the image pixels indexes, so later on I use the palette and color indexes in WPF(Windows) or SDL(Linux, Mac) to display the image.
vStart is the ability to crop the image on top.
The UpdateBitPerPixelMethod() will not change during a frame rendering, only before. During the for, no settings data can be changed.
So I was hoping that some parts can be written with SIMD, because the procedure is the same for all pixels.

Hy,
your code is not the clearest to me. Are you trying to create a new matrix / image ? If yes create a new 2D allocation and calculate the entire image into it. Set it to 0 after you do not need the calculations anymore.
Replace the Marshal.WriteByte(layerBuffer + place, colorIndex);with a 2D image ( maybe this is the image ?).
Regarding the rest it is a problem because you have non uniform offsets in indexing and jumps. That will make developing a SIMD solution difficult (you need masking and stuff). My bet would be to calculate everything for all the indices and save it into individual 2D matrices, that are allocated once at the begining.
For example:
ushort eff_y = (ushort)(vScale * (y - vStart) / 128);
Is calculated per every image row. Now you could calculate it once as an array since I do not believe that the format size of the images changes during the run.
I dont know if vStart and vScale are defined as a constant at program start. You should do this for every calculation that uses constant, and just read the matrices later to calculate.
SIMD can help but only if you do every iteration you calculate the same thing and if you avoid branching and switch cases.
Addition 1
You have multiple problems and design considerations from my stand point.
First of all you need to get away from the idea SIMD is going to help in your case. You would need to remove all conditional statements. SIMD-s are not build to deal with conditional statements.
Your idea should be to split up the logic into manageable pieces so you can see witch piece of the code takes most time.
One big problem is the write byte in the marshal, this is automatically saying to the compiler that you handle only and exclusively 1 byte. I'm guessing that this creates on big bottle neck.
By code analysis I see in each loop you are doing checks. This must be restructured.
Assumption is the image get rarely cropped this would be a separation from the image calculations.
List<ushort> eff_y = new List<ushort>();
List<uint> y_add = new List<uint>();
for (ushort y = 0; y < height; y++)
{
eff_y.add((ushort)(vScale * (y - vStart) / 128));
var newY = tileHeight > 0 ? eff_y % tileHeight : 0;
y_add = (uint)(newY * tileWidth * bitsPerPixel >> 3);
}
So this can be precalculated and changed only when the cropping changes.
Now it gets realy tricky.
paletteOffset - the if statement makes only sense in paletteOffset can be negative, then zero it out and remove the if statement
bitsPerPixel - this looks like a fixed value for the rendering duration
so remove the UpdateBitPerPixelMethod and send in a parameter.
for (ushort y = 0; y < height; y++)
{
for (int x = 0; x < width; x++)
{
var newX = tileWidth > 0 ? x % tileWidth : 0; // conditional stetement
ushort x_add = (ushort)(newX * bitsPerPixel >> 3);
uint tile_offset = y_add + x_add;
byte color = videoBytes[tile_offset];
var colorIndex = BitsPerPxlCalculation(color, newX);
// Apply Palette Offset
if (paletteOffset > 0) // conditional stetement
colorIndex += paletteOffset;
var place = x + eff_y * width;
Marshal.WriteByte(layerBuffer + place, colorIndex);
}
}
This are only few things that need to be done before you try anything with the SIMD. But by that time the changes will give the compiler hints about what you want to do. This could improve the machine code execution. You need also to test the performance of your code to pinpoint the bottle neck it is very hard to assume or guess correctly by code.
Good luck

Unity: GetPixels() always results in the colour black across whole image

I have an image (attached) which I'm using as a test. I'm trying to get and store all the colours of each pixel in an array.
I use the below code to do this;
Texture2D tex = mapImage.mainTexture as Texture2D;
int w = tex.width;
int h = tex.height;
Vector4[,] vals = new Vector4[w, h];
Color[] cols = tex.GetPixels();
for (int y = 0; y < h; y++)
{
for (int x = 0; x < w; x++)
{
if(cols[y+x] != Color.black)
{
Debug.Break();
}
vals[x, y] = cols[(y + x)];
}
}
Where mapImage is a public Material variable which I drag in into the scene on the prefab. As you can see, I've added a debug test there to pause the editor if a non-black colour is reached. This NEVER gets hit ever.
Interestingly, I've got another script which runs and tells me the colour values (GetPixel()) at the click position using the same image. It works fine (different methods, but both ultimately use the same material)
I'm at a loss as to why GetPixels() is always coming out black?
I've also been considering just loading the image data into a byte array, then parsing the values into a Vector4, but hoping this will work eventually.

You aren't indexing into the Color array properly. With the indices you are using, y+x, you keep checking the same values on the lowest rows of the texture, never getting past a certain point.
Instead, when calculating the index, you need to multiply the row that you are on by the row length and add that to the column you are on:
Texture2D tex = mapImage.mainTexture as Texture2D;
int w = tex.width;
int h = tex.height;
Vector4[,] vals = new Vector4[w, h];
Color[] cols = tex.GetPixels();
for (int y = 0; y < h; y++)
{
for (int x = 0; x < w; x++)
{
int index = y * w + x;
vals[x, y] = cols[index];
}
}
From the documentation on GetPixels:
The returned array is a flattened 2D array, where pixels are laid out left to right, bottom to top (i.e. row after row). Array size is width by height of the mip level used. The default mip level is zero (the base texture) in which case the size is just the size of the texture. In general case, mip level size is mipWidth=max(1,width>>miplevel) and similarly for height.

Find last drawn pixel of C# Metafile

I have a Metafile object. For reasons outside of my control, it has been provided much larger (thousands of times larger) than what would be required to fit the image drawn inside it.
For example, it could be 40 000 x 40 000, yet only contains "real" (non-transparent) pixels in an area 2000 x 1600.
Originally, this metafile was simply drawn to a control, and the control bounds limited the area to a reasonable size.
Now I am trying to split it into different chunks of dynamic size, depending on user input. What I want to do it count how many of those chunks will be there (in x and in y, even the splitting is into a two-dimensional grid of chunks).
I am aware that, technically, I could go the O(N²) way, and just check the pixels one by one to find the "real" bounds of the drawn image.
But this will be painfully slow.
I am looking for a way of getting the position (x,y) of the very last drawn pixel in the entire metafile, without iterating through every single one of them.
Since The DrawImage method is not painfully slow, at least not N² slow, I assume that the metafile object has some optimisations on the inside that would allow something like this. Just like the List object has a .Count Property that is much faster than actually counting the objects, is there some way of getting the practical bounds of a metafile?
The drawn content, in this scenario, will always be rectangular. I can safely assume that the last pixel will be the same, whether I loop in x then y, or in y then x.
How can I find the coordinates of this "last" pixel?

Finding the bounding rectangle of the non-transparent pixels for such a large image is indeed an interesting challenge.
The most direct approach would be tackling the WMF content but that is also by far the hardest to get right.
Let's instead render the image to a bitmap and look at the bitmap.
First the basic approach, then a few optimizations.
To get the bounds one need to find the left, top, right and bottom borders.
Here is a simple function to do that:
Rectangle getBounds(Bitmap bmp)
{
int l, r, t, b; l = t = r = b = 0;
for (int x = 0; x < bmp.Width - 1; x++)
for (int y = 0; y < bmp.Height - 1; y++)
if (bmp.GetPixel(x,y).A > 0) { l = x; goto l1; }
l1:
for (int x = bmp.Width - 1; x > l ; x--)
for (int y = 0; y < bmp.Height - 1; y++)
if (bmp.GetPixel(x,y).A > 0) { r = x; goto l2; }
l2:
for (int y = 0; y < bmp.Height - 1; y++)
for (int x = l; x < r; x++)
if (bmp.GetPixel(x,y).A > 0) { t = y; goto l3; }
l3:
for (int y = bmp.Height - 1; y > t; y--)
for (int x = l; x < r; x++)
if (bmp.GetPixel(x,y).A > 0) { b = y; goto l4; }
l4:
return Rectangle.FromLTRB(l,t,r,b);
}
Note that is optimizes the last, vertical loops a little to look only at the portion not already tested by the horizontal loops.
It uses GetPixel, which is painfully slow; but even Lockbits only gains 'only' about 10x or so. So we need to reduce the sheer numbers; we need to do that anyway, because 40k x 40k pixels is too large for a Bitmap.
Since WMF is usually filled with vector data we probably can scale it down a lot. Here is an example:
string fn = "D:\\_test18b.emf";
Image img = Image.FromFile(fn);
int w = img.Width;
int h = img.Height;
float scale = 100;
Rectangle rScaled = Rectangle.Empty;
using (Bitmap bmp = new Bitmap((int)(w / scale), (int)(h / scale)))
using (Graphics g = Graphics.FromImage(bmp))
{
g.ScaleTransform(1f/scale, 1f/scale);
g.Clear(Color.Transparent);
g.DrawImage(img, 0, 0);
rScaled = getBounds(bmp);
Rectangle rUnscaled = Rectangle.Round(
new RectangleF(rScaled.Left * scale, rScaled.Top * scale,
rScaled.Width * scale, rScaled.Height * scale ));
}
Note that to properly draw the wmf file one may need to adapt the resolutions. Here is an example i used for testing:
using (Graphics g2 = pictureBox.CreateGraphics())
{
float scaleX = g2.DpiX / img.HorizontalResolution / scale;
float scaleY = g2.DpiY / img.VerticalResolution / scale;
g2.ScaleTransform(scaleX, scaleY);
g2.DrawImage(img, 0, 0); // draw the original emf image.. (*)
g2.ResetTransform();
// g2.DrawImage(bmp, 0, 0); // .. it will look the same as (*)
g2.DrawRectangle(Pens.Black, rScaled);
}
I left this out but for fully controlling the rendering, it ought have been included in the snippet above as well..
This may or may not be good enough, depending on the accuracy needed.
To measure the bounds perfectly one can do this trick: Use the bounds from the scaled down test and measure unscaled but only a tiny stripe around the four bound numbers. When creating the render bitmap we move the origin accordingly.
Example for the right bound:
Rectangle rScaled2 = Rectangle.Empty;
int delta = 80;
int right = (int)(rScaled.Right * scale);
using (Bitmap bmp = new Bitmap((int)(delta * 2 ), (int)(h )))
using (Graphics g = Graphics.FromImage(bmp))
{
g.Clear(Color.Transparent);
g.DrawImage(img, - right - delta, 0);
rScaled2 = getBounds(bmp);
}
I could have optimized by not going over the full height but only the portion (plus delte) we already found..
Further optimization can be achieved if one can use knowledge about the data. If we know that the image data are connected we could use larger steps in the loops until a pixel is found and then trace back one step..

Drawing zig-zag lines is much slower than drawing straight lines

While using a self-written graphing control I noticed that the painting of the graph was much slower while displaying noisy data than when it displayed clean data.
I dug further into and narrowed the problem down to its bare minimum difference: Drawing the same amount of lines with varying Y values versus drawing lines with the same Y value.
So for example I put together the following tests. I generate lists of points, one with random Y values, one with the same Y, and one with a Zig-Zag Y pattern.
private List<PointF> GenerateRandom(int n, int width, int height)
{
//Generate random pattern
Random rnd = new Random();
float stepwidth = Convert.ToSingle(width / n);
float mid = Convert.ToSingle(height / 2);
float lastx = 0;
float lasty = mid;
List<PointF> res = new List<PointF>();
res.Add(new PointF(lastx, lasty));
for (int i = 1; i <= n; i++)
{
var x = stepwidth * i;
var y = Convert.ToSingle(height * rnd.NextDouble());
res.Add(new PointF(x, y));
}
return res;
}
private List<PointF> GenerateUnity(int n, int width, int height)
{
//Generate points along a simple line
float stepwidth = Convert.ToSingle(width / n);
float mid = Convert.ToSingle(height / 2);
float lastx = 0;
float lasty = mid;
List<PointF> res = new List<PointF>();
res.Add(new PointF(lastx, lasty));
for (int i = 1; i <= n; i++)
{
var x = stepwidth * i;
var y = mid;
res.Add(new PointF(x, y));
}
return res;
}
private List<PointF> GenerateZigZag(int n, int width, int height)
{
//Generate an Up/Down List
float stepwidth = Convert.ToSingle(width / n);
float mid = Convert.ToSingle(height / 2);
float lastx = 0;
float lasty = mid;
List<PointF> res = new List<PointF>();
res.Add(new PointF(lastx, lasty));
var state = false;
for (int i = 1; i <= n; i++)
{
var x = stepwidth * i;
var y = mid - (state ? 50 : -50);
res.Add(new PointF(x, y));
state = !state;
}
return res;
}
I now draw each list of points a few times and compare how long it takes:
private void DoTheTest()
{
Bitmap bmp = new Bitmap(970, 512);
var random = GenerateRandom(2500, bmp.Width, bmp.Height).ToArray();
var unity = GenerateUnity(2500, bmp.Width, bmp.Height).ToArray();
var ZigZag = GenerateZigZag(2500, bmp.Width, bmp.Height).ToArray();
using (Graphics g = Graphics.FromImage(bmp))
{
var tUnity = BenchmarkDraw(g, 200, unity);
var tRandom = BenchmarkDraw(g, 200, random);
var tZigZag = BenchmarkDraw(g, 200, ZigZag);
MessageBox.Show(tUnity.ToString() + "\r\n" + tRandom.ToString() + "\r\n" + tZigZag.ToString());
}
}
private double BenchmarkDraw(Graphics g, int n, PointF[] Points)
{
var Times = new List<double>();
for (int i = 1; i <= n; i++)
{
g.Clear(Color.White);
System.DateTime d3 = DateTime.Now;
DrawLines(g, Points);
System.DateTime d4 = DateTime.Now;
Times.Add((d4 - d3).TotalMilliseconds);
}
return Times.Average();
}
private void DrawLines(Graphics g, PointF[] Points)
{
g.DrawLines(Pens.Black, Points);
}
I come up with the following durations per draw:
Straight Line: 0.095 ms
Zig-Zag Pattern: 3.24 ms
Random Pattern: 5.47 ms
So it seems to get progressively worse, the more change there is in the lines to be drawn, and that is also a real world effect I encountered in the control painting I mentioned in the beginning.
My questions are thus the following:
Why does it make a such a brutal difference, which lines are to be drawn?
How can I improve the drawing speed for the noisy data?

Three reasons come to mind:
Line Length : Depending on the actual numbers sloped lines may be longer by just a few pixels or a lot or even by some substantial factor. Looking at your code I suspect the latter..
Algorithm : Drawing sloped lines does take some algorithm to find the next pixels. Even fast drawing routines need to do some computations as opposed to vertical or horizontal lines, which run straight through the pixel arrays.
Anti-Aliasing : Unless you turn off anti-aliasing completely (with all the ugly consequences) the number of pixels to paint will also be around 2-3 times more as all those anti-aliasing pixels above and below the center lines must also be calculated and drawn. Not to forget calculating their colors!
The remedy for the latter part is obviously to turn off anti-aliasing, but the other problems are simply the way things are. So best don't worry and be happy about the speedy straight lines :-)

If you really have a lot of lines or your lines could be very long (a few time the size of the screen), or if you have a lot of almost 0 pixel line, you have to wrote code to reduce useless drawing of lines.
Well, here are some ideas:
If you write many lines at the same x, then you could replace those by a single line between min and max y at that x.
If your line goes way beyond the screen boundary, you should clip them.
If a line is completly outside of the visible area, you should skip it.
If a line have a 0 length, you should not write it.
If a line has a single pixel length, you should write only that pixel.
Obviously, the benefit depends a lot on how many lines you draw... And also the alternative might not give the exact same result...
In practice, it you draw a chart on a screen, then if you display only useful information, it should be pretty fast on modern hardware.
Well if you use style or colors, it might not be as trivial to optimize the displaying of the data.
Alternatively, they are some charting component that are optimized for display large data... The good one are generally expensive but it might still worth it. Often trials are available so you can get a good idea on how much you might increase the performance and then decide what to do.