We Keep Coding

Continuous Perceptron Weights Change Zero

I am working on 2 class simple perceptron problem. My project work getting user mouse click from GUI panel and make classification. Class 1 expected output: 1 and Class 2 expected output -1. My problem is discrete perceptron working fine but continuous perceptron after one point stop decrease error. I don't know what I am doing wrong. I look so much code and source.
My formulas;
E=1/2 Σ(d-o)^2
f(net)=(2/(1+ⅇ^(-net)))-1
ΔW=n(d-o)(1-o^2)y
like this.
d: Expected output,
net: weight*input sum,
y: input matrix ([x1 x2 -1]) and
o: Actual output.
Code for continuous perceptron below;
while (totalError > Emax)
{
totalError = 0;
for(i=0; i<point.Count; i++)
{
double x1 = point[i].X1;
double x2 = point[i].X2;
double net = (x1 * w0) + (x2 * w1) + (x0 * w2);
double o = (2 / (1 + Math.Exp(-net))) - 1;
double error = Math.Pow(point[i].Class - o, 2);
w0 += (x1 * c * (point[i].Class - o) * (1 - Math.Pow(o, 2))) / 2;
w1 += (x2 * c * (point[i].Class - o) * (1 - Math.Pow(o, 2))) / 2;
w2 += (x0 * c * (point[i].Class - o) * (1 - Math.Pow(o, 2))) / 2;
totalError += error;
}
totalError = totalError / 2;
ErrorShow(cycle, totalError);
objGraphic.Clear(Color.White);
DrawSeperationLine();
cycle++;
}
Emax=0.001 selected. Project working like this. You can see it not correct line location. Class 1 is blue and class 2 red.
I think problem in for loop.
Console Output of Code:
Edit:
After discuss with #TaW (Thanks for showing road), I find out my problem in output (activation function). It always return 1 or -1. After that in weight change function [1-Math.Pow(o,2)] part return 0 and that make weight change equal 0. So my question how can I solve this problem. Type casting not work.

My question's solution is using normalization. For normalization I use standard deviation. Standart deviation code is below;
for(i=0;i<point.Count;i++){
x1 += point[i].X1;
x2 += point[i].X2;
}
meanx1 = x1 / point.Count;
meanx2 = x2 / point.Count;
for(i=0;i<point.Count;i++){
totalX1 += Math.Pow(point[i].X1 - meanx1, 2);
totalX2 += Math.Pow(point[i].X2 - meanx2, 2);
}
normX1 = totalX1 / (point.Count - 1);
normX2 = totalX2 / (point.Count - 1);
normX1 = normX1 / 100;
normX2 = normX2 / 100;
The last division is used to decrease the value.

How would i work out magnitude quickly for 3 values?

How can I use a Fast Magnitude calculation for 3 values (instead of using square root)? (+/- 3% is good enough)
public void RGBToComparison(Color32[] color)
{
DateTime start = DateTime.Now;
foreach (Color32 i in color)
{
var r = PivotRgb(i.r / 255.0);
var g = PivotRgb(i.g / 255.0);
var b = PivotRgb(i.b / 255.0);
var X = r * 0.4124 + g * 0.3576 + b * 0.1805;
var Y = r * 0.2126 + g * 0.7152 + b * 0.0722;
var Z = r * 0.0193 + g * 0.1192 + b * 0.9505;
var LB = PivotXyz(X / 95.047);
var AB = PivotXyz(Y / 100);
var BB = PivotXyz(Z / 108.883);
var L = Math.Max(0, 116 * AB - 16);
var A = 500 * (LB - AB);
var B = 200 * (AB - BB);
totalDifference += Math.Sqrt((L-LT)*(L-LT) + (A-AT)*(A-AT) + (B-BT)*(B-BT));
}
totalDifference = totalDifference / color.Length;
text.text = "Amount of Pixels: " + color.Length + " Time(MilliSeconds):" + DateTime.Now.Subtract(start).TotalMilliseconds + " Score (0 to 100)" + (totalDifference).ToString();
RandomOrNot();
}
private static double PivotRgb(double n)
{
return (n > 0.04045 ? Math.Pow((n + 0.055) / 1.055, 2.4) : n / 12.92) * 100.0;
}
private static double PivotXyz(double n)
{
return n > 0.008856 ? CubicRoot(n) : (903.3 * n + 16) / 116;
}
private static double CubicRoot(double n)
{
return Math.Pow(n, 1.0 / 3.0);
}
This is the important part: totalDifference += Math.Sqrt((L-LT)*(L-LT) + (A-AT)*(A-AT) + (B-BT)*(B-BT));
I know there are FastMagnitude calculations online, but all the ones online are for two values, not three. For example, could i use the difference between the values to get a precise answer? (By implementing the difference value into the equation, and if the difference percentage-wise is big, falling back onto square root?)
Adding up the values and iterating the square root every 4 pixels is a last resort that I could do. But firstly, I want to find out if it is possible to have a good FastMagnitude calculation for 3 values.
I know I can multi-thread and parllelize it, but I want to optimize my code before I do that.

If you just want to compare the values, why not leave the square root out and work with the length squared?
Or use the taylor series of the square root of 1+x and cut off early :)

Running (one pass) calculation of covariance

I got a set of 3d vectors (x,y,z), and I want to calculate the covariance matrix without storing the vectors.
I will do it in C#, but eventually I will implement it in C on a microcontroller, so I need the algorithm in itself, and not a library.
Pseudocode would be great also.

The formula is simple if you have Matrix and Vector classes at hand:
Vector mean;
Matrix covariance;
for (int i = 0; i < points.size(); ++i) {
Vector diff = points[i] - mean;
mean += diff / (i + 1);
covariance += diff * diff.transpose() * i / (i + 1);
}
covariance *= 1 / points.size()
I personally always prefer this style rather than the two-pass calculation. The code is short and the results are flawless.
Matrix and Vector can have fixed dimension and can be easily coded for this purpose. You can even rewrite the code into discrete floating-point calculations and avoid computing the symmetric part of the covariance matrix.
Note that there is a vector outer product on the second last row of code. Not all vector libraries interpret it correctly.

I think I have found the solution. It is based on this article about how to calculate covariance manually and this one about calculating running variance. And then I adapted the algorithm in the latter to calculate covariance instead of variance, given my understanding of it from the first article.
public class CovarianceMatrix
{
private int _n;
private Vector _oldMean, _newMean,
_oldVarianceSum, _newVarianceSum,
_oldCovarianceSum, _newCovarianceSum;
public void Push(Vector x)
{
_n++;
if (_n == 1)
{
_oldMean = _newMean = x;
_oldVarianceSum = new Vector(0, 0, 0);
_oldCovarianceSum = new Vector(0, 0, 0);
}
else
{
//_newM = _oldM + (x - _oldM) / _n;
_newMean = new Vector(
_oldMean.X + (x.X - _oldMean.X) / _n,
_oldMean.Y + (x.Y - _oldMean.Y) / _n,
_oldMean.Z + (x.Z - _oldMean.Z) / _n);
//_newS = _oldS + (x - _oldM) * (x - _newM);
_newVarianceSum = new Vector(
_oldVarianceSum.X + (x.X - _oldMean.X) * (x.X - _newMean.X),
_oldVarianceSum.Y + (x.Y - _oldMean.Y) * (x.Y - _newMean.Y),
_oldVarianceSum.Z + (x.Z - _oldMean.Z) * (x.Z - _newMean.Z));
/* .X is X vs Y
* .Y is Y vs Z
* .Z is Z vs X
*/
_newCovarianceSum = new Vector(
_oldCovarianceSum.X + (x.X - _oldMean.X) * (x.Y - _newMean.Y),
_oldCovarianceSum.Y + (x.Y - _oldMean.Y) * (x.Z - _newMean.Z),
_oldCovarianceSum.Z + (x.Z - _oldMean.Z) * (x.X - _newMean.X));
// set up for next iteration
_oldMean = _newMean;
_oldVarianceSum = _newVarianceSum;
}
}
public int NumDataValues()
{
return _n;
}
public Vector Mean()
{
return (_n > 0) ? _newMean : new Vector(0, 0, 0);
}
public Vector Variance()
{
return _n <= 1 ? new Vector(0, 0, 0) : _newVarianceSum.DivideBy(_n - 1);
}
}

The code from emu is elegant, but requires an additional step to be correct:
Vector mean;
Matrix covariance;
for (int i = 0; i < points.size(); ++i) {
Vector diff = points[i] - mean;
mean += diff / (i + 1);
covariance += diff * diff.transpose() * i / (i + 1);
}
covariance = covariance/(points.size()-1);
Note the final step of normalizing the covariance.

Here is a simple example in R to demonstrate the principle:
a <- matrix(rnorm(22), ncol = 2)
a1 <- a[1:10, ]
a2 <- a[2:11, ]
cov(a1)
cov(a2)
m <- 10
# initial step
m1.1 <- mean(a1[, 1])
m1.2 <- mean(a1[, 2])
c1.11 <- cov(a1)[1, 1]
c1.22 <- cov(a1)[2, 2]
c1.12 <- cov(a1)[1, 2]
#step 1->2
m2.1 <- m1.1 + (a[11, 1] - a[1, 1])/m
m2.2 <- m1.2 + (a[11, 2] - a[1, 2])/m
c2.11 <- c1.11 + (a[11, 1]^2 - a[1, 1]^2)/(m - 1) + (m1.1^2 - m2.1^2) * m/(m - 1)
c2.22 <- c1.22 + (a[11, 2]^2 - a[1, 2]^2)/(m - 1) + (m1.2^2 - m2.2^2) * m/(m - 1)
c2.12 <- c1.12 + (a[11, 1] * a[11, 2] - a[1, 1]*a[1, 2])/(m - 1) +
(m1.1 * m1.2 - m2.1 * m2.2) * m/(m - 1)
cov(a2) - matrix(c(c2.11, c2.12, c2.12, c2.22), ncol=2)

Correct way to use indexer in Parallel.For

I have a method which generates a (Waveform) bitmap based on a specific (Waveform)-function (in the examples below I am simply using Math.Sin to simplify things). Up until now this method was "serial" (no threads) however some of the functions used are relatively time consuming so I tried using Parallel.For and Parallel.ForEach, but I guess the index variable I use must get "corrupted" (or at least have another value than what I expect) the graphics being generated will either contain "spikes" or strange lines between points that are not neighbours.
Here first is my serial version (which works):
Point[] points = new Point[rect.Width];
byte[] ptTypes = new byte[rect.Width];
for (int i = 0; i < rect.Width; i++)
{
double phase = MathUtil.WrapRad((MathUtil.PI2 / (rect.Width / 1d)) * i);
double value = waveform.ValueAtPhase(phase);
newPoint = new Point(rect.Left + i,
rect.Top + (int) (halfHeight + (value * -1d * halfHeight * scaleY)));
points[i] = newPoint;
if (i == 0)
ptTypes[i] = (byte)PathPointType.Start;
else
ptTypes[i] = (byte)PathPointType.Line;
}
using (GraphicsPath wavePath = new GraphicsPath(points, ptTypes))
{
gph.DrawPath(wavePen, wavePath);
}
As you can see the code simply uses 2 arrays (one for Points and one for PointTypes) so the order these values are inserted into the array should not matter, as long as the values are inserted into the correct elements of the arrays.
Next example is using Parallel.For (to simplify the examples I have omitted the creation of the arrays and the actual draw method):
Parallel.For(0, rect.Width,
i =>
{
double phase = MathUtil.WrapRad((MathUtil.PI2 / (rect.Width / 1d)) * i);
double value = Math.Sin(phase);//waveform.ValueAtPhase(phase);
newPoint = new Point(rect.Left + i,
rect.Top + (int)(halfHeight + (value * -1d * halfHeight * scaleY)));
points[i] = newPoint;
if (i == 0)
ptTypes[i] = (byte)PathPointType.Start;
else
ptTypes[i] = (byte)PathPointType.Line;
});
Lastly I tried using a Partitioner with a Parallel.ForEach loop, but that did not fix the problem either:
var rangePartitioner = Partitioner.Create(0, rect.Width);
Parallel.ForEach(rangePartitioner, (range, loopState) =>
{
for (int i = range.Item1; i < range.Item2; i++)
{
double phase = MathUtil.WrapRad((MathUtil.PI2 / (rect.Width / 1d)) * i);
double value = Math.Sin(phase);//waveform.ValueAtPhase(phase);
newPoint = new Point(rect.Left + i,
rect.Top + (int)(halfHeight + (value * -1d * halfHeight * scaleY)));
points[i] = newPoint;
if (i == 0)
ptTypes[i] = (byte)PathPointType.Start;
else
ptTypes[i] = (byte)PathPointType.Line;
}
});
Pelle

newPoint = new Point(rect.Left + i, rect.Top + (int)(halfHeight + (value * -1d * halfHeight * scaleY)));
newPoint is not scoped to your for() loop - it's likely threads are updating it before you get to the next line points[i] = newPoint;
Change it to var newPoint = ...
Otherwise, your Parallel.For looks fine.
Also, does this have different behavior?
Math.Sin(phase);//waveform.ValueAtPhase(phase);
Providing ValueAtPhase does not modify anything, you should be able to use it within the loop.

Restart program from a certain line with an if statement?

could anyone help me restart my program from line 46 if the user enters 1 (just after the comment where it states that the next code is going to ask the user for 2 inputs) and if the user enters -1 end it. I cannot think how to do it. I'm new to C# any help you could give would be great!
class Program
{
static void Main(string[] args)
{
//Displays data in correct Format
List<float> inputList = new List<float>();
TextReader tr = new StreamReader("c:/users/tom/documents/visual studio 2010/Projects/DistanceCalculator3/DistanceCalculator3/TextFile1.txt");
String input = Convert.ToString(tr.ReadToEnd());
String[] items = input.Split(',');
Console.WriteLine("Point Latitude Longtitude Elevation");
for (int i = 0; i < items.Length; i++)
{
if (i % 3 == 0)
{
Console.Write((i / 3) + "\t\t");
}
Console.Write(items[i]);
Console.Write("\t\t");
if (((i - 2) % 3) == 0)
{
Console.WriteLine();
}
}
Console.WriteLine();
Console.WriteLine();
// Ask for two inputs from the user which is then converted into 6 floats and transfered in class Coordinates
Console.WriteLine("Please enter the two points that you wish to know the distance between:");
string point = Console.ReadLine();
string[] pointInput = point.Split(' ');
int pointNumber = Convert.ToInt16(pointInput[0]);
int pointNumber2 = Convert.ToInt16(pointInput[1]);
Coordinates distance = new Coordinates();
distance.latitude = (Convert.ToDouble(items[pointNumber * 3]));
distance.longtitude = (Convert.ToDouble(items[(pointNumber * 3) + 1]));
distance.elevation = (Convert.ToDouble(items[(pointNumber * 3) + 2]));
distance.latitude2 = (Convert.ToDouble(items[pointNumber2 * 3]));
distance.longtitude2 = (Convert.ToDouble(items[(pointNumber2 * 3) + 1]));
distance.elevation2 = (Convert.ToDouble(items[(pointNumber2 * 3) + 2]));
//Calculate the distance between two points
const double PIx = 3.141592653589793;
const double RADIO = 6371;
double dlat = ((distance.latitude2) * (PIx / 180)) - ((distance.latitude) * (PIx / 180));
double dlon = ((distance.longtitude2) * (PIx / 180)) - ((distance.longtitude) * (PIx / 180));
double a = (Math.Sin(dlat / 2) * Math.Sin(dlat / 2)) + Math.Cos((distance.latitude) * (PIx / 180)) * Math.Cos((distance.latitude2) * (PIx / 180)) * (Math.Sin(dlon / 2) * Math.Sin(dlon / 2));
double angle = 2 * Math.Atan2(Math.Sqrt(a), Math.Sqrt(1 - a));
double ultimateDistance = (angle * RADIO);
Console.WriteLine("The distance between your two points is...");
Console.WriteLine(ultimateDistance);
//Repeat the program if the user enters 1, end the program if the user enters -1
Console.WriteLine("If you wish to calculate another distance type 1 and return, if you wish to end the program, type -1.");
Console.ReadLine();
if (Convert.ToInt16(Console.ReadLine()) == 1);
{
//here is where I need it to repeat
}

bool exit = false;
do
{
Console.WriteLine("Please enter the two points that you wish to know the distance between:");
...
Console.WriteLine("If you wish to calculate another distance type 1 and return, if you wish to end the program, type -1.");
string input;
do
{
input = Console.ReadLine().Trim();
}
while (input != "1" && input != "-1");
if (input == -1) exit = true;
}
while (!exit);
But you would do much better to think about pushing logic into methods and functions such that you program is built up of much smaller building blocks.
You should be aiming towards something like this:
bool exit = false;
do
{
Point[] points = ReadCoordinates();
Whatever result = CalculateWhatever();
DisplayResults(results);
exit = ShouldExit();
}
while (!exit);
This makes the outer loop of your program self documenting and the methods self explanatory.

if (Convert.ToInt16(Console.ReadLine()) == 1);
{
Main(args)
}

That's quite an odd thing to do.
Have you looked into the while loop?
You could make a method of the above structure and loop it, while userinput equals 1
if it equals -1, call the following statement
Application.Exit()

Can use Goto to make the program loop on user input.
public static void Main(string[] args)
{
RestartApplication:
//// Displays data in correct Format
TextReader textReader = new StreamReader("c:/users/tom/documents/visual studio 2010/Projects/DistanceCalculator3/DistanceCalculator3/TextFile1.txt");
var input = Convert.ToString(textReader.ReadToEnd());
var items = input.Split(',');
Console.WriteLine("Point Latitude Longtitude Elevation");
for (var i = 0; i < items.Length; i++)
{
if (i % 3 == 0)
{
Console.Write((i / 3) + "\t\t");
}
Console.Write(items[i]);
Console.Write("\t\t");
if (((i - 2) % 3) == 0)
{
Console.WriteLine();
}
}
Console.WriteLine();
Console.WriteLine();
//// Ask for two inputs from the user which is then converted into 6 floats and transferred in class Coordinates
Console.WriteLine("Please enter the two points that you wish to know the distance between:");
var point = Console.ReadLine();
string[] pointInput;
if (point != null)
{
pointInput = point.Split(' ');
}
else
{
goto RestartApplication;
}
var pointNumber = Convert.ToInt16(pointInput[0]);
var pointNumber2 = Convert.ToInt16(pointInput[1]);
var distance = new Coordinates
{
Latitude = Convert.ToDouble(items[pointNumber * 3]),
Longtitude = Convert.ToDouble(items[(pointNumber * 3) + 1]),
Elevation = Convert.ToDouble(items[(pointNumber * 3) + 2]),
Latitude2 = Convert.ToDouble(items[pointNumber2 * 3]),
Longtitude2 = Convert.ToDouble(items[(pointNumber2 * 3) + 1]),
Elevation2 = Convert.ToDouble(items[(pointNumber2 * 3) + 2])
};
//// Calculate the distance between two points
const double PIx = 3.141592653589793;
const double Radio = 6371;
var dlat = (distance.Latitude2 * (PIx / 180)) - (distance.Latitude * (PIx / 180));
var dlon = (distance.Longtitude2 * (PIx / 180)) - (distance.Longtitude * (PIx / 180));
var a = (Math.Sin(dlat / 2) * Math.Sin(dlat / 2)) + Math.Cos(distance.Latitude * (PIx / 180)) * Math.Cos(distance.Latitude2 * (PIx / 180)) * (Math.Sin(dlon / 2) * Math.Sin(dlon / 2));
var angle = 2 * Math.Atan2(Math.Sqrt(a), Math.Sqrt(1 - a));
var ultimateDistance = angle * Radio;
Console.WriteLine("The distance between your two points is...");
Console.WriteLine(ultimateDistance);
//// Repeat the program if the user enters 1, end the program if the user enters -1
Console.WriteLine("If you wish to calculate another distance type 1 and return, if you wish to end the program, type -1.");
var userInput = Console.ReadLine();
if (Convert.ToInt16(userInput) == 1)
{
//// Here is where the application will repeat.
goto RestartApplication;
}
}
also did a bit of code formatting. Hope it helps.

You could do that with goto, be aware however that this is considered bad practice.
static void Main(string[] args)
{
...
MyLabel:
...
if (Convert.ToInt16(Console.ReadLine()) == 1);
{
//here is where I need it to repeat
goto MyLabel;
}