Running (one pass) calculation of covariance - c#

I got a set of 3d vectors (x,y,z), and I want to calculate the covariance matrix without storing the vectors.
I will do it in C#, but eventually I will implement it in C on a microcontroller, so I need the algorithm in itself, and not a library.
Pseudocode would be great also.


The formula is simple if you have Matrix and Vector classes at hand:
Vector mean;
Matrix covariance;
for (int i = 0; i < points.size(); ++i) {
Vector diff = points[i] - mean;
mean += diff / (i + 1);
covariance += diff * diff.transpose() * i / (i + 1);
}
covariance *= 1 / points.size()
I personally always prefer this style rather than the two-pass calculation. The code is short and the results are flawless.
Matrix and Vector can have fixed dimension and can be easily coded for this purpose. You can even rewrite the code into discrete floating-point calculations and avoid computing the symmetric part of the covariance matrix.
Note that there is a vector outer product on the second last row of code. Not all vector libraries interpret it correctly.


I think I have found the solution. It is based on this article about how to calculate covariance manually and this one about calculating running variance. And then I adapted the algorithm in the latter to calculate covariance instead of variance, given my understanding of it from the first article.
public class CovarianceMatrix
{
private int _n;
private Vector _oldMean, _newMean,
_oldVarianceSum, _newVarianceSum,
_oldCovarianceSum, _newCovarianceSum;
public void Push(Vector x)
{
_n++;
if (_n == 1)
{
_oldMean = _newMean = x;
_oldVarianceSum = new Vector(0, 0, 0);
_oldCovarianceSum = new Vector(0, 0, 0);
}
else
{
//_newM = _oldM + (x - _oldM) / _n;
_newMean = new Vector(
_oldMean.X + (x.X - _oldMean.X) / _n,
_oldMean.Y + (x.Y - _oldMean.Y) / _n,
_oldMean.Z + (x.Z - _oldMean.Z) / _n);
//_newS = _oldS + (x - _oldM) * (x - _newM);
_newVarianceSum = new Vector(
_oldVarianceSum.X + (x.X - _oldMean.X) * (x.X - _newMean.X),
_oldVarianceSum.Y + (x.Y - _oldMean.Y) * (x.Y - _newMean.Y),
_oldVarianceSum.Z + (x.Z - _oldMean.Z) * (x.Z - _newMean.Z));
/* .X is X vs Y
* .Y is Y vs Z
* .Z is Z vs X
*/
_newCovarianceSum = new Vector(
_oldCovarianceSum.X + (x.X - _oldMean.X) * (x.Y - _newMean.Y),
_oldCovarianceSum.Y + (x.Y - _oldMean.Y) * (x.Z - _newMean.Z),
_oldCovarianceSum.Z + (x.Z - _oldMean.Z) * (x.X - _newMean.X));
// set up for next iteration
_oldMean = _newMean;
_oldVarianceSum = _newVarianceSum;
}
}
public int NumDataValues()
{
return _n;
}
public Vector Mean()
{
return (_n > 0) ? _newMean : new Vector(0, 0, 0);
}
public Vector Variance()
{
return _n <= 1 ? new Vector(0, 0, 0) : _newVarianceSum.DivideBy(_n - 1);
}
}


The code from emu is elegant, but requires an additional step to be correct:
Vector mean;
Matrix covariance;
for (int i = 0; i < points.size(); ++i) {
Vector diff = points[i] - mean;
mean += diff / (i + 1);
covariance += diff * diff.transpose() * i / (i + 1);
}
covariance = covariance/(points.size()-1);
Note the final step of normalizing the covariance.


Here is a simple example in R to demonstrate the principle:
a <- matrix(rnorm(22), ncol = 2)
a1 <- a[1:10, ]
a2 <- a[2:11, ]
cov(a1)
cov(a2)
m <- 10
# initial step
m1.1 <- mean(a1[, 1])
m1.2 <- mean(a1[, 2])
c1.11 <- cov(a1)[1, 1]
c1.22 <- cov(a1)[2, 2]
c1.12 <- cov(a1)[1, 2]
#step 1->2
m2.1 <- m1.1 + (a[11, 1] - a[1, 1])/m
m2.2 <- m1.2 + (a[11, 2] - a[1, 2])/m
c2.11 <- c1.11 + (a[11, 1]^2 - a[1, 1]^2)/(m - 1) + (m1.1^2 - m2.1^2) * m/(m - 1)
c2.22 <- c1.22 + (a[11, 2]^2 - a[1, 2]^2)/(m - 1) + (m1.2^2 - m2.2^2) * m/(m - 1)
c2.12 <- c1.12 + (a[11, 1] * a[11, 2] - a[1, 1]*a[1, 2])/(m - 1) +
(m1.1 * m1.2 - m2.1 * m2.2) * m/(m - 1)
cov(a2) - matrix(c(c2.11, c2.12, c2.12, c2.22), ncol=2)

Related

Having trouble in finding roots of a cubic equation with C#

Recently my friends and I decided to make a multi-functional calculator to automatically find
the roots of quadratic equations.
It wasn't that challenging after all, so we decided to go onto the next level, to make a calculator for a cubic equation.
(ax^3 + bx^2 + cx + d)
However, we stumbled across some trivial problems, and no matter how hard we tried, the results are still the same.
we are beginners in terms of coding, so we're not sure if we are actually making some stupid mistakes here
but at least we want to learn something from others.
Basically, we have tried a lot of combinations of different cubic equations, or even re-coding the whole thing. The problem is that the results we yield are always wrong, but only for the real parts of the second root and third root.
For a better understanding, we have tried 9x^3 + 8x^2 + 7x + 6, as an example.
The correct answer, according to a cubic equation calculator website, is
(The website)
First root = -0.87285
Second root = -0.00802 + 0.87391 i
Third root = -0.00802 - 0.87391 i
However, our result to this equation is :
First root = -0.87285
Second root = -0.2963 + 0.87391 i
Third root = -0.2963 + -0.87391 i
It is apparently noticeable that only parts of the second and third roots are wrong.
We have tried finding similar threads to help, but those are a little bit too difficult for us to understand or those problems are not the same as ours.
We look forward to finding the solution and the causes to this problem.
We have separated the formulas for finding the roots of a cubic equation into 5 parts.
(rootp1-rootp5)
The formulas are coded according to the formulas that can be found in the Chinese version of Wikipedia's page of Cubic equation.
(The formulas)
We have also rounded the real parts of the roots to 5 digits.
(Some code lines might be redundant, but as I have mentioned, we are new to coding)
Code (C#) :
using System;
using System.Numerics;
namespace ComplexNumbers
{
public class ComplexNo
{
public static void Main()
{
Console.WriteLine("Cubic Equation Calculator :");
Console.Write("Insert first coefficient : ");
double a = Convert.ToDouble(Console.ReadLine());
Console.Write("Insert second coefficient : ");
double b = Convert.ToDouble(Console.ReadLine());
Console.Write("Insert third coefficient : ");
double c = Convert.ToDouble(Console.ReadLine());
Console.Write("Insert constant : ");
double d = Convert.ToDouble(Console.ReadLine());
Console.WriteLine(" ");
Console.WriteLine("Solve for " + a + "x" + "^3 " + b + "x^2 " + c + "x " + d + " :");
double rootp1 = -(b / (3 * a));
double rootp2 = (b * c / (6 * Math.Pow(a, 2))) - (Math.Pow(b, 3) / (27 * Math.Pow(a, 3))) - (d / (2 * a));
double rootp3 = (c / (3 * a)) - (Math.Pow(b, 2) / (9 * Math.Pow(a, 2)));
Complex root1 = rootp1 + Math.Cbrt(rootp2 + Math.Sqrt(Math.Pow(rootp2, 2) + Math.Pow(rootp3, 3))) +
Math.Cbrt(rootp2 - Math.Sqrt(Math.Pow(rootp2, 2) + Math.Pow(rootp3, 3)));
Complex rootp4 = new Complex(-1 / 2, Math.Sqrt(3) / 2);
Complex rootp5 = new Complex(-1 / 2, -(Math.Sqrt(3) / 2));
Complex root2 = rootp1 + (rootp4 * Math.Cbrt(rootp2 + Math.Sqrt(Math.Pow(rootp2, 2) + Math.Pow(rootp3, 3)))) +
(rootp5 * Math.Cbrt(rootp2 - Math.Sqrt(Math.Pow(rootp2, 2) + Math.Pow(rootp3, 3))));
Complex root3 = rootp1 + (rootp5 * Math.Cbrt(rootp2 + Math.Sqrt(Math.Pow(rootp2, 2) + Math.Pow(rootp3, 3)))) +
(rootp4 * Math.Cbrt(rootp2 - Math.Sqrt(Math.Pow(rootp2, 2) + Math.Pow(rootp3, 3))));
Console.WriteLine(" ");
Console.WriteLine("Results :");
Console.WriteLine("First Root :");
string root1rp = Convert.ToString(Math.Round(root1.Real, 5));
string root1ip = Convert.ToString(Math.Round(root1.Imaginary, 5));
Console.WriteLine(root1rp + " + " + root1ip + "i");
Console.WriteLine("Second Root :");
string root2rp = Convert.ToString(Math.Round(root2.Real, 5));
string root2ip = Convert.ToString(Math.Round(root2.Imaginary, 5));
Console.WriteLine(root2rp + " + " + root2ip + "i");
Console.WriteLine("Third Root :");
string root3rp = Convert.ToString(Math.Round(root3.Real, 5));
string root3ip = Convert.ToString(Math.Round(root3.Imaginary, 5));
Console.WriteLine(root3rp + " + " + root3ip + "i");
Console.ReadLine();
}
}
}
(Sorry for making this thread so long and my bad grammar)

The problem is with this line Complex rootp4 = new Complex(-1 / 2, Math.Sqrt(3) / 2);. -1/2 uses integer division and evaluates to 0.
This code will work.
Console.WriteLine(" ");
Console.WriteLine("Solve for " + a + "x" + "^3 " + b + "x^2 " + c + "x " + d + " :");
double minusBover3a = -(b / (3 * a));
double rootp2 = (b * c / (6 * Math.Pow(a, 2))) - (Math.Pow(b, 3) / (27 * Math.Pow(a, 3))) - (d / (2 * a));
double rootp3 = (c / (3 * a)) - (Math.Pow(b, 2) / (9 * Math.Pow(a, 2)));
double bigCubeRootPlus = Math.Cbrt(rootp2 + Math.Sqrt(Math.Pow(rootp2, 2) + Math.Pow(rootp3, 3)));
double bigCubeRootMinus = Math.Cbrt(rootp2 - Math.Sqrt(Math.Pow(rootp2, 2) + Math.Pow(rootp3, 3)));
// ** THIS IS THE PROBLEM. "-1/2" uses integer division, so this complex has 0 for real part
Complex complexPlus = new Complex(-1.0 / 2, Math.Sqrt(3) / 2);
Complex complexMinus = new Complex(-1.0 / 2, -(Math.Sqrt(3) / 2));
Complex root1 = minusBover3a + bigCubeRootPlus + bigCubeRootMinus;
Complex root2 = minusBover3a + complexPlus * bigCubeRootPlus + complexMinus * bigCubeRootMinus;
Complex root3 = minusBover3a + complexMinus * bigCubeRootPlus + complexPlus * bigCubeRootMinus;

I admit I haven't tried your code, but you may have a casting issue in the math in your original calculations. Try this:
double rootp2 = ((double)b * (double)c / (6D * Math.Pow(a, 2D))) - (Math.Pow(b, 3D) / (27D * Math.Pow(a, 3D))) - ((double)d / (2D * (double)a));
If that makes a difference, you'd have to propagate a similar change (cast all variables as (double) and inline constants as double with D) through the other calculations.

The following is an alternative to your code:
// from https://www.daniweb.com/programming/software-development/
// code/454493/solving-the-cubic-equation-using-the-complex-struct
// algorithm described in
// https://en.wikipedia.org/wiki/Cubic_equation#General_cubic_formula
const int NRoots = 3;
double SquareRootof3 = Math.Sqrt(3);
// the 3 cubic roots of 1
var CubicUnity = new List<Complex>(NRoots)
{ new Complex(1, 0),
new Complex(-0.5, -SquareRootof3 / 2.0),
new Complex(-0.5, SquareRootof3 / 2.0) };
// intermediate calculations
double DELTA = 18 * a * b * c * d
- 4 * b * b * b * d
+ b * b * c * c
- 4 * a * c * c * c
- 27 * a * a * d * d;
double DELTA0 = b * b - 3 * a * c;
double DELTA1 = 2 * b * b * b
- 9 * a * b * c
+ 27 * a * a * d;
Complex DELTA2 = -27 * a * a * DELTA;
Complex C = Complex.Pow((DELTA1 + Complex.Pow(DELTA2, 0.5)) / 2, 1 / 3.0);
for (int i = 0; i < NRoots; i++)
{
Complex M = CubicUnity[i] * C;
Complex Root = -1.0 / (3 * a) * (b + M + DELTA0 / M);
Console.WriteLine();
Console.WriteLine($"Root {i+1}:");
Console.WriteLine($"Real {Root.Real:0.#####}");
Console.WriteLine($"Imaginary {Root.Imaginary:0.#####}i");
}

Continuous Perceptron Weights Change Zero

I am working on 2 class simple perceptron problem. My project work getting user mouse click from GUI panel and make classification. Class 1 expected output: 1 and Class 2 expected output -1. My problem is discrete perceptron working fine but continuous perceptron after one point stop decrease error. I don't know what I am doing wrong. I look so much code and source.
My formulas;
E=1/2 Σ(d-o)^2
f(net)=(2/(1+ⅇ^(-net)))-1
ΔW=n(d-o)(1-o^2)y
like this.
d: Expected output,
net: weight*input sum,
y: input matrix ([x1 x2 -1]) and
o: Actual output.
Code for continuous perceptron below;
while (totalError > Emax)
{
totalError = 0;
for(i=0; i<point.Count; i++)
{
double x1 = point[i].X1;
double x2 = point[i].X2;
double net = (x1 * w0) + (x2 * w1) + (x0 * w2);
double o = (2 / (1 + Math.Exp(-net))) - 1;
double error = Math.Pow(point[i].Class - o, 2);
w0 += (x1 * c * (point[i].Class - o) * (1 - Math.Pow(o, 2))) / 2;
w1 += (x2 * c * (point[i].Class - o) * (1 - Math.Pow(o, 2))) / 2;
w2 += (x0 * c * (point[i].Class - o) * (1 - Math.Pow(o, 2))) / 2;
totalError += error;
}
totalError = totalError / 2;
ErrorShow(cycle, totalError);
objGraphic.Clear(Color.White);
DrawSeperationLine();
cycle++;
}
Emax=0.001 selected. Project working like this. You can see it not correct line location. Class 1 is blue and class 2 red.
I think problem in for loop.
Console Output of Code:
Edit:
After discuss with #TaW (Thanks for showing road), I find out my problem in output (activation function). It always return 1 or -1. After that in weight change function [1-Math.Pow(o,2)] part return 0 and that make weight change equal 0. So my question how can I solve this problem. Type casting not work.

My question's solution is using normalization. For normalization I use standard deviation. Standart deviation code is below;
for(i=0;i<point.Count;i++){
x1 += point[i].X1;
x2 += point[i].X2;
}
meanx1 = x1 / point.Count;
meanx2 = x2 / point.Count;
for(i=0;i<point.Count;i++){
totalX1 += Math.Pow(point[i].X1 - meanx1, 2);
totalX2 += Math.Pow(point[i].X2 - meanx2, 2);
}
normX1 = totalX1 / (point.Count - 1);
normX2 = totalX2 / (point.Count - 1);
normX1 = normX1 / 100;
normX2 = normX2 / 100;
The last division is used to decrease the value.

How would i work out magnitude quickly for 3 values?

How can I use a Fast Magnitude calculation for 3 values (instead of using square root)? (+/- 3% is good enough)
public void RGBToComparison(Color32[] color)
{
DateTime start = DateTime.Now;
foreach (Color32 i in color)
{
var r = PivotRgb(i.r / 255.0);
var g = PivotRgb(i.g / 255.0);
var b = PivotRgb(i.b / 255.0);
var X = r * 0.4124 + g * 0.3576 + b * 0.1805;
var Y = r * 0.2126 + g * 0.7152 + b * 0.0722;
var Z = r * 0.0193 + g * 0.1192 + b * 0.9505;
var LB = PivotXyz(X / 95.047);
var AB = PivotXyz(Y / 100);
var BB = PivotXyz(Z / 108.883);
var L = Math.Max(0, 116 * AB - 16);
var A = 500 * (LB - AB);
var B = 200 * (AB - BB);
totalDifference += Math.Sqrt((L-LT)*(L-LT) + (A-AT)*(A-AT) + (B-BT)*(B-BT));
}
totalDifference = totalDifference / color.Length;
text.text = "Amount of Pixels: " + color.Length + " Time(MilliSeconds):" + DateTime.Now.Subtract(start).TotalMilliseconds + " Score (0 to 100)" + (totalDifference).ToString();
RandomOrNot();
}
private static double PivotRgb(double n)
{
return (n > 0.04045 ? Math.Pow((n + 0.055) / 1.055, 2.4) : n / 12.92) * 100.0;
}
private static double PivotXyz(double n)
{
return n > 0.008856 ? CubicRoot(n) : (903.3 * n + 16) / 116;
}
private static double CubicRoot(double n)
{
return Math.Pow(n, 1.0 / 3.0);
}
This is the important part: totalDifference += Math.Sqrt((L-LT)*(L-LT) + (A-AT)*(A-AT) + (B-BT)*(B-BT));
I know there are FastMagnitude calculations online, but all the ones online are for two values, not three. For example, could i use the difference between the values to get a precise answer? (By implementing the difference value into the equation, and if the difference percentage-wise is big, falling back onto square root?)
Adding up the values and iterating the square root every 4 pixels is a last resort that I could do. But firstly, I want to find out if it is possible to have a good FastMagnitude calculation for 3 values.
I know I can multi-thread and parllelize it, but I want to optimize my code before I do that.

If you just want to compare the values, why not leave the square root out and work with the length squared?
Or use the taylor series of the square root of 1+x and cut off early :)

Try-catch affecting performance in XNA

I'm struggling a bit with a try-catch behaviour. I've read that it can affect performance, but not as much as this block of code does to my machine.
foreach (Crew CurrentBall in CrewList)
{
if (CurrentBall != WhiteBall)
{
double distance = 0;
try
{
distance = Math.Sqrt((CurrentBall.Rectangle.X - SnakeArray[CurrentBall.SnakePosition - 1].Rectangle.X)
* (CurrentBall.Rectangle.X - SnakeArray[CurrentBall.SnakePosition - 1].Rectangle.X)
+ (CurrentBall.Rectangle.Y - SnakeArray[CurrentBall.SnakePosition - 1].Rectangle.Y)
* (CurrentBall.Rectangle.Y - SnakeArray[CurrentBall.SnakePosition - 1].Rectangle.Y));
}
catch
{
distance = Math.Sqrt((CurrentBall.Rectangle.X - WhiteBall.Rectangle.X)
* (CurrentBall.Rectangle.X - WhiteBall.Rectangle.X)
+ (CurrentBall.Rectangle.Y - WhiteBall.Rectangle.Y)
* (CurrentBall.Rectangle.Y - WhiteBall.Rectangle.Y));
}
if (distance < 75 && CurrentBall != WhiteBall)
{
if (CurrentBall.IsFollowingWhite == false)
{
NumberOfBallsFollowingWhite++;
CrewCatch.Play();
Score += 5;
ReduceMovementSpeed();
ApplyText(CurrentBall);
SnakeArray[NumberOfBallsFollowingWhite] = CurrentBall;
CurrentBall.SnakePosition = NumberOfBallsFollowingWhite;
}
CurrentBall.Follow(SnakeArray[CurrentBall.SnakePosition - 1]);
CurrentBall.IsFollowingWhite = true;
}
}
}
It worked quite well until I used try-catch. I also tried without Math.Sqrt but it was no improvement.

This whole thing:
try
{
distance = Math.Sqrt((CurrentBall.Rectangle.X - SnakeArray[CurrentBall.SnakePosition - 1].Rectangle.X)
* (CurrentBall.Rectangle.X - SnakeArray[CurrentBall.SnakePosition - 1].Rectangle.X)
+ (CurrentBall.Rectangle.Y - SnakeArray[CurrentBall.SnakePosition - 1].Rectangle.Y)
* (CurrentBall.Rectangle.Y - SnakeArray[CurrentBall.SnakePosition - 1].Rectangle.Y));
}
catch
{
distance = Math.Sqrt((CurrentBall.Rectangle.X - WhiteBall.Rectangle.X)
* (CurrentBall.Rectangle.X - WhiteBall.Rectangle.X)
+ (CurrentBall.Rectangle.Y - WhiteBall.Rectangle.Y)
* (CurrentBall.Rectangle.Y - WhiteBall.Rectangle.Y));
}
Could be reduced to something like:
if (CurrentBall.SnakePosition - 1 >= 0 && CurrentBall.SnakePosition - 1 < SnakeArray.Length)
{
int xValue = CurrentBall.Rectangle.X - SnakeArray[CurrentBall.SnakePosition - 1].Rectangle.X;
int xSquared = xValue * xValue;
int yValue = CurrentBall.Rectangle.Y - SnakeArray[CurrentBall.SnakePosition - 1].Rectangle.Y;
int ySquared = yValue * yValue;
distance = Math.Sqrt(xSquared + ySquared);
}
else
{
//And similar for the above here...
distance = Math.Sqrt((CurrentBall.Rectangle.X - WhiteBall.Rectangle.X)
* (CurrentBall.Rectangle.X - WhiteBall.Rectangle.X)
+ (CurrentBall.Rectangle.Y - WhiteBall.Rectangle.Y)
* (CurrentBall.Rectangle.Y - WhiteBall.Rectangle.Y));
}
Never use a try/catch to perform logic operations. Also never Catch if you don't do something with the exception.
Also try to make your logic easy to read. See what I did in the first part of the "if"? See how much easier it is to understand (and debug). And no performance loss, the compiler will optimize it out!

Dicom Window width & level formula not giving greyvalues

I'm trying to implement the Window Width and level formula from de Dicom specification in my application. Only it's not returning any grayscales at the moment. The dicom specifies the formula as following:
These Attributes are applied according to the following pseudo-code, where x is the input value, y
is an output value with a range from ymin to ymax, c is Window Center (0028,1050) and w is
Window Width (0028,1051):
if (x <= c - 0.5 - (w-1)/2), then y = ymin
else if (x > c - 0.5 + (w-1)/2), then y = ymax,
else y = ((x - (c - 0.5)) / (w-1) + 0.5) * (ymax - ymin)+ ymin
So i've translated this into the following c# syntax:
if (pixelData[i] <= wLevel - 0.5 - (wWidth - 1) / 2)
oColor = 0;
else if (pixelData[i] > wLevel - 0.5 + (wWidth - 1) / 2)
oColor = 255;
else
oColor = (int)((pixelData[i] - (wLevel - 0.5)) / (wWidth - 1) + 0.5) * (255 - 0) + 0;
Howevery, the last part of the formula
oColor = (int)((pixelData[i] - (wLevel - 0.5)) / (wWidth - 1) + 0.5) * (255 - 0) + 0;
Only seems to return 0
Anyone sees how this is possible?

The meaning of VOI LUT is to map a given pixel range to displayable values (usually 0..0xFF), using clamping for out of range pixel values.
This means that for a given window/level we can compute the displayable range:
level-window/2 , level + window/2 .
For pixel values that are in that range, linear transformation is used:
((pixel - lower_window_limit) / window) * displayable_range
where lower_window_limit is level - window/2
This -window/2 is missing in your formula.

Categories