We Keep Coding

Linear Regression with NuML

I'm trying to do a really basic linear regression (Z = 2 * X + 1) prediction using NuML. Given the data is so linear I can't understand why the predicted value is so far off unless I am doing something wrong. I have the target class
public class Sample
{
public float V { get; set; }
public float X { get; set; }
public float Y { get; set; }
public float Z { get; set; }
public Func<float, float, float, float> OutputStrategy { get; set; }
public Sample(Func<float, float, float, float> outputStrategy)
{
OutputStrategy = outputStrategy;
}
public void Seed(int i)
{
V = (float) i;
X = (float) 2 * i;
Y = (float) 3 * i;
Z = OutputStrategy(V, X, Y);
}
}
and I have the NuML code to set up the source values and predict an answer for an arbitrary new data point:
NB: The output strategy is a simple 2 * A + 1. I've tried it with multivariate analysis and the prediction is further away
public static void Main(string[] args)
{
// Generate sample data
int sampleSize = 1000;
Sample[] samples = new Sample[sampleSize];
Func<float, float, float, float> outputStrategy = (A, B, C) => 2 * A + 1;
for (int i = 0; i < sampleSize; i++)
{
samples[i] = new Sample(outputStrategy);
samples[i].Seed(i);
}
// calculate model
var generator = new LinearRegressionGenerator();
var descriptor = Descriptor.New("Samples")
.With("V").As(typeof(float))
.With("X").As(typeof(float))
.With("Y").As(typeof(float))
.Learn("Z").As(typeof(float));
generator.Descriptor = descriptor;
var model = Learner.Learn(samples, 0.6, 50, generator);
// Use prediction
var targetSample = new Sample(outputStrategy);
targetSample.Seed(sampleSize + 1);
var predictedSample = model.Model.Predict(targetSample);
var predictedValue = predictedSample.Z;
var actualValue = outputStrategy(targetSample.V, targetSample.X, targetSample.Y);
Console.Write("Predicted Value = {0}, Actual Value = {1}, Difference = {2} {3:0.00}%", predictedValue, actualValue, actualValue - predictedValue, (decimal) (actualValue - predictedValue) / (decimal) predictedValue * 100M);
Console.ReadKey();
}
This gives a difference of about 0.5% which considering the line is completely straight was surprising. I have tried using different % of the dataset for training and number of iterations of the model but it makes no difference to the output.
If I use even a more slightly more complicated model I get much worse predictive capabilities. If I use logistic regression, the predicted output of Z is always 1?!

Square root line doesn’t work

I wrote a quick code for factorizing formulas, however the line that takes creates the square root of D doesn’t work. The line is line 10. Any help is appreciated.
using System;
public class MainClass {
//Source Numbers
public int A = 1;
public int B = 3;
public int C = 9;
//Calculation Numbers
public float Di;
public static double Sqrt(double Di); //This is the faulted line.
//Answers
public float X;
public float X1;
public float X2;
public static void Main() {
Console.Writeline("D=", Di);
//Calculation for the Square root of D
// (DSq)Math.Sqrt(Di);
Di = B^2-4*A*C;
//Calculation for the answers
if(Di>0) {
X1 = ((0-B)-DSq)/(A*2);
X2 = ((0-B)+DSq)/(A*2);
Console.Writeline("X=", X1, " or X=", X2);
}
else if(Di=0) {
X = 0-B;
Console.Writeline("X=", X);
}
else {
Console.Writeline("The formula cannot be solved.");
}
}
}

You are using a method definition with no body. In any case you dont need to invent the wheel, since Math has already a Math.Sqrt(), method. Try:
........
Di = B^2-4*A*C;
if (Di>0)
{
var sqrDi = Math.Sqrt(Di);
.....
}
...

You have several errors in your code, like the spelling of WriteLine, and comparing in if statements (use ==). This returns a list of valid solutions (X-values):
public IList<double> factorizeABC(double a, double b, double c)
{
var solutions = new List<double>();
var Di = b * b - 4 * a * c;
if (Di > 0)
{
var rtDi = Math.Sqrt(Di);
var X1 = (-b - rtDi) / (a * 2);
var X2 = (-b + rtDi) / (a * 2);
solutions.Add(X1);
solutions.Add(X2);
}
else if (Di == 0)
{
var X = -b / (a * 2);
solutions.Add(X);
}
return solutions;
}
usage:
var results = factorizeABC(1, 2, -8);
if (results.Count() == 0)
Console.WriteLine("The formula cannot be solved.");
if (results.Count() == 1)
Console.WriteLine("X=" + results[0].ToString());
if (results.Count() == 2)
Console.WriteLine("X=" + results[0].ToString() + " or X=" + results[1].ToString());

How can I generate a "random constant colour" for a given string?

how can I generate a "random constant colour" for a given string at runtime?
So a given string value will always have the same colour but different strings will have different colours.
Like how gmail assigns colours to the sender names.
Thanks
Responses to comments:
Thinking to generate the colour from a hashcode.
The colours won't be stored but generated from a hashcode.

I don't know any dedicated method for this, but here is a simple method generating Hexadecimal values with MD5 based on a given string:
using System.Security.Cryptography;
using System.Text;
static string GetColor(string raw)
{
using (MD5 md5Hash = MD5.Create())
{
byte[] data = md5Hash.ComputeHash(Encoding.UTF8.GetBytes(raw));
return BitConverter.ToString(data).Replace("-", string.Empty).Substring(0, 6);
}
}
Examples:
example#example.com
-> 23463B
info#google.com
-> 3C9015
stack#exchange.com
-> 7CA5E8
Edit:
I didn't tested it enough, so you may want to tweak it a little bit to get more different and unique values.
Edit2:
If you want transparency, check out this question/answer. By setting the Substring to Substring(0,8) , you should return a string with the alpha channel.

Similar to what the other answers are suggesting (hash the string in some form then use that hash to pick the color), but instead of using the hash to directly calculate the color use it as the index to an array of "Acceptable" colors.
class ColorPicker
{
public ColorPicker(int colorCount)
{
//The ".Skip(2)" makes it skip pure white and pure black.
// If you want those two, take out the +2 and the skip.
_colors = ColorGenerator.Generate(colorCount + 2).Skip(2).ToArray();
}
private readonly Color[] _colors;
public Color StringToColor(string message)
{
int someHash = CalculateHashOfStringSomehow(message);
return _colors[someHash % _colors.Length];
}
private int CalculateHashOfStringSomehow(string message)
{
//TODO: I would not use "message.GetHashCode()" as you are not
// guaranteed the same value between runs of the program.
// Make up your own algorithom or use a existing one that has a fixed
// output for a given input, like MD5.
}
}
This prevents issues like getting a white color when you plan on showing the text with a white background and other similar problems.
To populate your Color[] see this answer for the ColorGenerator class or just make your own pre-defined list of colors that look good on whatever background they will be used on.
Appendix:
In case the link goes down, here is a copy of the ColorGenerator class
public static class ColorGenerator
{
// RYB color space
private static class RYB
{
private static readonly double[] White = { 1, 1, 1 };
private static readonly double[] Red = { 1, 0, 0 };
private static readonly double[] Yellow = { 1, 1, 0 };
private static readonly double[] Blue = { 0.163, 0.373, 0.6 };
private static readonly double[] Violet = { 0.5, 0, 0.5 };
private static readonly double[] Green = { 0, 0.66, 0.2 };
private static readonly double[] Orange = { 1, 0.5, 0 };
private static readonly double[] Black = { 0.2, 0.094, 0.0 };
public static double[] ToRgb(double r, double y, double b)
{
var rgb = new double[3];
for (int i = 0; i < 3; i++)
{
rgb[i] = White[i] * (1.0 - r) * (1.0 - b) * (1.0 - y) +
Red[i] * r * (1.0 - b) * (1.0 - y) +
Blue[i] * (1.0 - r) * b * (1.0 - y) +
Violet[i] * r * b * (1.0 - y) +
Yellow[i] * (1.0 - r) * (1.0 - b) * y +
Orange[i] * r * (1.0 - b) * y +
Green[i] * (1.0 - r) * b * y +
Black[i] * r * b * y;
}
return rgb;
}
}
private class Points : IEnumerable<double[]>
{
private readonly int pointsCount;
private double[] picked;
private int pickedCount;
private readonly List<double[]> points = new List<double[]>();
public Points(int count)
{
pointsCount = count;
}
private void Generate()
{
points.Clear();
var numBase = (int)Math.Ceiling(Math.Pow(pointsCount, 1.0 / 3.0));
var ceil = (int)Math.Pow(numBase, 3.0);
for (int i = 0; i < ceil; i++)
{
points.Add(new[]
{
Math.Floor(i/(double)(numBase*numBase))/ (numBase - 1.0),
Math.Floor((i/(double)numBase) % numBase)/ (numBase - 1.0),
Math.Floor((double)(i % numBase))/ (numBase - 1.0),
});
}
}
private double Distance(double[] p1)
{
double distance = 0;
for (int i = 0; i < 3; i++)
{
distance += Math.Pow(p1[i] - picked[i], 2.0);
}
return distance;
}
private double[] Pick()
{
if (picked == null)
{
picked = points[0];
points.RemoveAt(0);
pickedCount = 1;
return picked;
}
var d1 = Distance(points[0]);
int i1 = 0, i2 = 0;
foreach (var point in points)
{
var d2 = Distance(point);
if (d1 < d2)
{
i1 = i2;
d1 = d2;
}
i2 += 1;
}
var pick = points[i1];
points.RemoveAt(i1);
for (int i = 0; i < 3; i++)
{
picked[i] = (pickedCount * picked[i] + pick[i]) / (pickedCount + 1.0);
}
pickedCount += 1;
return pick;
}
public IEnumerator<double[]> GetEnumerator()
{
Generate();
for (int i = 0; i < pointsCount; i++)
{
yield return Pick();
}
}
IEnumerator IEnumerable.GetEnumerator()
{
return GetEnumerator();
}
}
public static IEnumerable<Color> Generate(int numOfColors)
{
var points = new Points(numOfColors);
foreach (var point in points)
{
var rgb = RYB.ToRgb(point[0], point[1], point[2]);
yield return Color.FromArgb(
(int)Math.Floor(255 * rgb[0]),
(int)Math.Floor(255 * rgb[1]),
(int)Math.Floor(255 * rgb[2]));
}
}
}

3 integer variables, r,g and b.
Loop through each character in the string in steps of 3 and add the character code.
r += n + 0
g += n + 1
b += n + 2
after the loop take r,g, and b modulo 255 and create a color using Color.FromARGB.
No guarantees the color will be pretty though, and some strings may happen to have colors very close to each other.

I see some pretty good answeers but though it whould contribute with a little fun solutuion to generate colors from string, the Hash version looks like the best way to go but if this gives any one some inspiration to bould off, have at it
ConsoleKeyInfo ch = new ConsoleKeyInfo();
while (ch.KeyChar != 'e')
{
Console.WriteLine("type string to seed color");
string s = Console.ReadLine(); // gets text from input, in this case the command line
double d=0;
foreach(char cha in s.ToCharArray())
{
d=+ (int)cha; // get the value and adds it
}
d= (255/(Math.Pow(0.2,-0.002 *d))); // Generates a seed like value from i where 255 is the maximum. basicly 255/0.2^(-0.002*d)
int i = Convert.ToInt32(d); //then convets and get rid of the decimels
Color c = Color.FromArgb(i, i, i);// add a bit more calculation for varieng colers.
Console.WriteLine(c.Name);
Console.WriteLine("To Exit press e");
ch = Console.ReadKey()
}
Edit1: It definantly needs some tweeking, since the longer the string the ligther the color, but i think something can come from it with a little work :)

How to efficiently calculate a moving Standard Deviation

Below you can see my C# method to calculate Bollinger Bands for each point (moving average, up band, down band).
As you can see this method uses 2 for loops to calculate the moving standard deviation using the moving average. It used to contain an additional loop to calculate the moving average over the last n periods. This one I could remove by adding the new point value to total_average at the beginning of the loop and removing the i - n point value at the end of the loop.
My question now is basically: Can I remove the remaining inner loop in a similar way I managed with the moving average?
public static void AddBollingerBands(SortedList<DateTime, Dictionary<string, double>> data, int period, int factor)
{
double total_average = 0;
for (int i = 0; i < data.Count(); i++)
{
total_average += data.Values[i]["close"];
if (i >= period - 1)
{
double total_bollinger = 0;
double average = total_average / period;
for (int x = i; x > (i - period); x--)
{
total_bollinger += Math.Pow(data.Values[x]["close"] - average, 2);
}
double stdev = Math.Sqrt(total_bollinger / period);
data.Values[i]["bollinger_average"] = average;
data.Values[i]["bollinger_top"] = average + factor * stdev;
data.Values[i]["bollinger_bottom"] = average - factor * stdev;
total_average -= data.Values[i - period + 1]["close"];
}
}
}

The problem with approaches that calculate the sum of squares is that it and the square of sums can get quite large, and the calculation of their difference may introduce a very large error, so let's think of something better. For why this is needed, see the Wikipedia article on Algorithms for computing variance and John Cook on Theoretical explanation for numerical results)
First, instead of calculating the stddev let's focus on the variance. Once we have the variance, stddev is just the square root of the variance.
Suppose the data are in an array called x; rolling an n-sized window by one can be thought of as removing the value of x[0] and adding the value of x[n]. Let's denote the averages of x[0]..x[n-1] and x[1]..x[n] by µ and µ’ respectively. The difference between the variances of x[0]..x[n-1] and x[1]..x[n] is, after canceling out some terms and applying (a²-b²) = (a+b)(a-b):
Var[x[1],..,x[n]] - Var[x[0],..,x[n-1]]
= (\sum_1^n x[i]² - n µ’²)/(n-1) - (\sum_0^{n-1} x[i]² - n µ²)/(n-1)
= (x[n]² - x[0]² - n(µ’² - µ²))/(n-1)
= (x[n]-µ’ + x[0]-µ)(x[n]-x[0])/(n-1)
Therefore the variance is perturbed by something that doesn't require you to maintain the sum of squares, which is better for numerical accuracy.
You can calculate the mean and variance once in the beginning with a proper algorithm (Welford's method). After that, every time you have to replace a value in the window x[0] by another x[n] you update the average and variance like this:
new_Avg = Avg + (x[n]-x[0])/n
new_Var = Var + (x[n]-new_Avg + x[0]-Avg)(x[n] - x[0])/(n-1)
new_StdDev = sqrt(new_Var)

The answer is yes, you can. In the mid-80's I developed just such an algorithm (probably not original) in FORTRAN for a process monitoring and control application. Unfortunately, that was over 25 years ago and I do not remember the exact formulas, but the technique was an extension of the one for moving averages, with second order calculations instead of just linear ones.
After looking at your code some, I am think that I can suss out how I did it back then. Notice how your inner loop is making a Sum of Squares?:
for (int x = i; x > (i - period); x--)
{
total_bollinger += Math.Pow(data.Values[x]["close"] - average, 2);
}
in much the same way that your average must have originally had a Sum of Values? The only two differences are the order (its power 2 instead of 1) and that you are subtracting the average each value before you square it. Now that might look inseparable, but in fact they can be separated:
SUM(i=1; n){ (v[i] - k)^2 }
is
SUM(i=1..n){v[i]^2 -2*v[i]*k + k^2}
which becomes
SUM(i=1..n){v[i]^2 -2*v[i]*k} + k^2*n
which is
SUM(i=1..n){v[i]^2} + SUM(i=1..n){-2*v[i]*k} + k^2*n
which is also
SUM(i=1..n){v[i]^2} + SUM(i=1..n){-2*v[i]}*k + k^2*n
Now the first term is just a Sum of Squares, you handle that in the same way that you do the sum of Values for the average. The last term (k^2*n) is just the average squared times the period. Since you divide the result by the period anyway, you can just add the new average squared without the extra loop.
Finally, in the second term (SUM(-2*v[i]) * k), since SUM(v[i]) = total = k*n you can then change it into this:
-2 * k * k * n
or just -2*k^2*n, which is -2 times the average squared, once the period (n) is divided out again. So the final combined formula is:
SUM(i=1..n){v[i]^2} - n*k^2
or
SUM(i=1..n){values[i]^2} - period*(average^2)
(be sure to check the validity of this, since I am deriving it off the top of my head)
And incorporating into your code should look something like this:
public static void AddBollingerBands(ref SortedList<DateTime, Dictionary<string, double>> data, int period, int factor)
{
double total_average = 0;
double total_squares = 0;
for (int i = 0; i < data.Count(); i++)
{
total_average += data.Values[i]["close"];
total_squares += Math.Pow(data.Values[i]["close"], 2);
if (i >= period - 1)
{
double total_bollinger = 0;
double average = total_average / period;
double stdev = Math.Sqrt((total_squares - Math.Pow(total_average,2)/period) / period);
data.Values[i]["bollinger_average"] = average;
data.Values[i]["bollinger_top"] = average + factor * stdev;
data.Values[i]["bollinger_bottom"] = average - factor * stdev;
total_average -= data.Values[i - period + 1]["close"];
total_squares -= Math.Pow(data.Values[i - period + 1]["close"], 2);
}
}
}

I've used commons-math (and contributed to that library!) for something very similar to this. It's open-source, porting to C# should be easy as store-bought pie (have you tried making a pie from scratch!?). Check it out: http://commons.apache.org/math/api-3.1.1/index.html. They have a StandardDeviation class. Go to town!

Most important information has already been given above --- but maybe this is still of general interest.
A tiny Java library to calculate moving average and standard deviation is available here:
https://github.com/tools4j/meanvar
The implementation is based on a variant of Welford's method mentioned above. Methods to remove and replace values have been derived that can be used for moving value windows.
Disclaimer: I am the author of the said library.

I just did it with Data From Binance Future API
Hope this helps:
using Newtonsoft.Json;
using System;
using System.Collections.Generic;
using System.Collections.ObjectModel;
using System.ComponentModel;
using System.Data;
using System.Drawing;
using System.IO;
using System.Linq;
using System.Net;
using System.Text;
using System.Threading.Tasks;
using System.Windows.Forms;
using static System.Windows.Forms.VisualStyles.VisualStyleElement;
namespace Trading_Bot_1
{
public class BOLL
{
private BollingerBandData graphdata = new BollingerBandData();
private List<TickerData> data = new List<TickerData>();
public BOLL(string url)
{
string js = getJsonFromUrl(url);
//dynamic data = JObject.Parse(js);
object[][] arrays = JsonConvert.DeserializeObject<object[][]>(js);
data = new List<TickerData>();
for (int i = 1; i < 400; i++)
{
data.Add(new TickerData
{
Date = DateTime.Now,
Open = Convert.ToDouble(arrays[arrays.Length - i][1]),
High = Convert.ToDouble(arrays[arrays.Length - i][2]),
Low = Convert.ToDouble(arrays[arrays.Length - i][3]),
Close = Convert.ToDouble(arrays[arrays.Length - i][4]),
Volume = Math.Round(Convert.ToDouble(arrays[arrays.Length - i][4]), 0),
AdjClose = Convert.ToDouble(arrays[arrays.Length - i][6])
});
}
graphdata.LowerBand.Add(1);
graphdata.LowerBand.Add(2);
graphdata.LowerBand.Add(3);
graphdata.LowerBand.Add(1);
graphdata.UpperBand.Add(1);
graphdata.UpperBand.Add(2);
graphdata.UpperBand.Add(3);
graphdata.UpperBand.Add(4);
graphdata.MovingAverageWindow.Add(10);
graphdata.MovingAverageWindow.Add(20);
graphdata.MovingAverageWindow.Add(40);
graphdata.MovingAverageWindow.Add(50);
graphdata.Length.Add(10);
graphdata.Length.Add(30);
graphdata.Length.Add(50);
graphdata.Length.Add(100);
// DataContext = graphdata;
}
public static string getJsonFromUrl(string url1)
{
var uri = String.Format(url1);
WebClient client = new WebClient();
client.UseDefaultCredentials = true;
var data = client.DownloadString(uri);
return data;
}
List<double> UpperBands = new List<double>();
List<double> LowerBands = new List<double>();
public List<List<double>> GetBOLLDATA(int decPlaces)
{
int datalength = graphdata.SelectedMovingAverage + graphdata.SelectedLength;
string bands = "";
for (int i = graphdata.SelectedLength - 1; i >= 0; i--)
{
List<double> price = new List<double>();
for (int j = 0; j < graphdata.SelectedMovingAverage; j++)
{
price.Add(data[i + j].Close);
}
double sma = CalculateAverage(price.ToArray());
double sigma = CalculateSTDV(price.ToArray());
double lower = sma - (graphdata.SelectedLowerBand * sigma);
double upper = sma + (graphdata.SelectedUpperBand * sigma);
UpperBands.Add(Math.Round( upper,decPlaces));
LowerBands.Add(Math.Round(lower, decPlaces));
bands += (Math.Round(upper, decPlaces) + " / " + Math.Round(lower, decPlaces)) + Environment.NewLine;
// graphdata.ChartData.Add(new ChartData() { SMA = sma, LowerBandData = lower, UpperBandData = upper });
}
//MessageBox.Show(bands);
return new List<List<double>> { UpperBands, LowerBands };
}
public double[] GetBOLLDATA(int decPlaces, string a)
{
List<double> price = new List<double>();
for (int j = 0; j < graphdata.SelectedMovingAverage; j++)
{
price.Add(data[j].Close);
}
double sma = CalculateAverage(price.ToArray());
double sigma = CalculateSTDV(price.ToArray());
double lower = sma - (graphdata.SelectedLowerBand * sigma);
double upper = sma + (graphdata.SelectedUpperBand * sigma);
return new double[] { Math.Round(upper, decPlaces), Math.Round(lower, decPlaces) };
}
private double CalculateAverage(double[] data)
{
int count = data.Length;
double sum = 0;
for (int i = 0; i < count; i++)
{
sum += data[i];
}
return sum / count;
}
private double CalculateVariance(double[] data)
{
int count = data.Length;
double sum = 0;
double avg = CalculateAverage(data);
for (int i = 0; i < count; i++)
{
sum += (data[i] - avg) * (data[i] - avg);
}
return sum / (count - 1);
}
private double CalculateSTDV(double[] data)
{
double var = CalculateVariance(data);
return Math.Sqrt(var);
}
}
public class ChartData
{
public double UpperBandData
{ get; set; }
public double LowerBandData
{ get; set; }
public double SMA
{ get; set; }
}
public class BollingerBandData : INotifyPropertyChanged
{
private ObservableCollection<int> _lowerBand;
private ObservableCollection<int> _upperBand;
private ObservableCollection<int> _movingAvg;
private ObservableCollection<int> _length;
private ObservableCollection<ChartData> _chartData;
private int _selectedLowerBand;
private int _selectedUpperBand;
private int _selectedMovingAvg;
private int _selectedLength;
public BollingerBandData()
{
_lowerBand = new ObservableCollection<int>();
_upperBand = new ObservableCollection<int>();
_movingAvg = new ObservableCollection<int>();
_length = new ObservableCollection<int>();
_chartData = new ObservableCollection<ChartData>();
SelectedLowerBand = 2;
SelectedUpperBand = 2;
SelectedMovingAverage = 20;
SelectedLength = 5;
}
public ObservableCollection<ChartData> ChartData
{
get
{
return _chartData;
}
set
{
_chartData = value;
RaisePropertyChanged("ChartData");
}
}
public ObservableCollection<int> LowerBand
{
get
{
return _lowerBand;
}
set
{
_lowerBand = value;
RaisePropertyChanged("LowerBand");
}
}
public ObservableCollection<int> UpperBand
{
get
{
return _upperBand;
}
set
{
_upperBand = value;
RaisePropertyChanged("UpperBand");
}
}
public ObservableCollection<int> MovingAverageWindow
{
get
{
return _movingAvg;
}
set
{
_movingAvg = value;
RaisePropertyChanged("MovingAverageWindow");
}
}
public ObservableCollection<int> Length
{
get
{
return _length;
}
set
{
_length = value;
RaisePropertyChanged("Length");
}
}
public int SelectedLowerBand
{
get
{
return _selectedLowerBand;
}
set
{
_selectedLowerBand = value;
RaisePropertyChanged("SelectedLowerBand");
}
}
public int SelectedUpperBand
{
get
{
return _selectedUpperBand;
}
set
{
_selectedUpperBand = value;
RaisePropertyChanged("SelectedUpperBand");
}
}
public int SelectedMovingAverage
{
get
{
return _selectedMovingAvg;
}
set
{
_selectedMovingAvg = value;
RaisePropertyChanged("SelectedMovingAverage");
}
}
public int SelectedLength
{
get
{
return _selectedLength;
}
set
{
_selectedLength = value;
RaisePropertyChanged("SelectedLength");
}
}
public event PropertyChangedEventHandler PropertyChanged;
private void RaisePropertyChanged(string propertyName)
{
PropertyChangedEventHandler handler = this.PropertyChanged;
if (handler != null)
{
handler(this, new PropertyChangedEventArgs(propertyName));
}
}
}
public class TickerData
{
public DateTime Date
{ get; set; }
public double Open
{ get; set; }
public double High
{ get; set; }
public double Low
{ get; set; }
public double Close
{ get; set; }
public double Volume
{ get; set; }
public double AdjClose
{ get; set; }
}
}

How do I determine the standard deviation (stddev) of a set of values?

I need to know if a number compared to a set of numbers is outside of 1 stddev from the mean, etc..

While the sum of squares algorithm works fine most of the time, it can cause big trouble if you are dealing with very large numbers. You basically may end up with a negative variance...
Plus, don't never, ever, ever, compute a^2 as pow(a,2), a * a is almost certainly faster.
By far the best way of computing a standard deviation is Welford's method. My C is very rusty, but it could look something like:
public static double StandardDeviation(List<double> valueList)
{
double M = 0.0;
double S = 0.0;
int k = 1;
foreach (double value in valueList)
{
double tmpM = M;
M += (value - tmpM) / k;
S += (value - tmpM) * (value - M);
k++;
}
return Math.Sqrt(S / (k-2));
}
If you have the whole population (as opposed to a sample population), then use return Math.Sqrt(S / (k-1));.
EDIT: I've updated the code according to Jason's remarks...
EDIT: I've also updated the code according to Alex's remarks...

10 times faster solution than Jaime's, but be aware that,
as Jaime pointed out:
"While the sum of squares algorithm works fine most of the time, it
can cause big trouble if you are dealing with very large numbers. You
basically may end up with a negative variance"
If you think you are dealing with very large numbers or a very large quantity of numbers, you should calculate using both methods, if the results are equal, you know for sure that you can use "my" method for your case.
public static double StandardDeviation(double[] data)
{
double stdDev = 0;
double sumAll = 0;
double sumAllQ = 0;
//Sum of x and sum of x²
for (int i = 0; i < data.Length; i++)
{
double x = data[i];
sumAll += x;
sumAllQ += x * x;
}
//Mean (not used here)
//double mean = 0;
//mean = sumAll / (double)data.Length;
//Standard deviation
stdDev = System.Math.Sqrt(
(sumAllQ -
(sumAll * sumAll) / data.Length) *
(1.0d / (data.Length - 1))
);
return stdDev;
}

The accepted answer by Jaime is great, except you need to divide by k-2 in the last line (you need to divide by "number_of_elements-1").
Better yet, start k at 0:
public static double StandardDeviation(List<double> valueList)
{
double M = 0.0;
double S = 0.0;
int k = 0;
foreach (double value in valueList)
{
k++;
double tmpM = M;
M += (value - tmpM) / k;
S += (value - tmpM) * (value - M);
}
return Math.Sqrt(S / (k-1));
}

The Math.NET library provides this for you to of the box.
PM> Install-Package MathNet.Numerics
var populationStdDev = new List<double>(1d, 2d, 3d, 4d, 5d).PopulationStandardDeviation();
var sampleStdDev = new List<double>(2d, 3d, 4d).StandardDeviation();
See PopulationStandardDeviation for more information.

Code snippet:
public static double StandardDeviation(List<double> valueList)
{
if (valueList.Count < 2) return 0.0;
double sumOfSquares = 0.0;
double average = valueList.Average(); //.NET 3.0
foreach (double value in valueList)
{
sumOfSquares += Math.Pow((value - average), 2);
}
return Math.Sqrt(sumOfSquares / (valueList.Count - 1));
}

You can avoid making two passes over the data by accumulating the mean and mean-square
cnt = 0
mean = 0
meansqr = 0
loop over array
cnt++
mean += value
meansqr += value*value
mean /= cnt
meansqr /= cnt
and forming
sigma = sqrt(meansqr - mean^2)
A factor of cnt/(cnt-1) is often appropriate as well.
BTW-- The first pass over the data in Demi and McWafflestix answers are hidden in the calls to Average. That kind of thing is certainly trivial on a small list, but if the list exceed the size of the cache, or even the working set, this gets to be a bid deal.

I found that Rob's helpful answer didn't quite match what I was seeing using excel. To match excel, I passed the Average for valueList in to the StandardDeviation calculation.
Here is my two cents... and clearly you could calculate the moving average (ma) from valueList inside the function - but I happen to have already before needing the standardDeviation.
public double StandardDeviation(List<double> valueList, double ma)
{
double xMinusMovAvg = 0.0;
double Sigma = 0.0;
int k = valueList.Count;
foreach (double value in valueList){
xMinusMovAvg = value - ma;
Sigma = Sigma + (xMinusMovAvg * xMinusMovAvg);
}
return Math.Sqrt(Sigma / (k - 1));
}

With Extension methods.
using System;
using System.Collections.Generic;
namespace SampleApp
{
internal class Program
{
private static void Main()
{
List<double> data = new List<double> {1, 2, 3, 4, 5, 6};
double mean = data.Mean();
double variance = data.Variance();
double sd = data.StandardDeviation();
Console.WriteLine("Mean: {0}, Variance: {1}, SD: {2}", mean, variance, sd);
Console.WriteLine("Press any key to continue...");
Console.ReadKey();
}
}
public static class MyListExtensions
{
public static double Mean(this List<double> values)
{
return values.Count == 0 ? 0 : values.Mean(0, values.Count);
}
public static double Mean(this List<double> values, int start, int end)
{
double s = 0;
for (int i = start; i < end; i++)
{
s += values[i];
}
return s / (end - start);
}
public static double Variance(this List<double> values)
{
return values.Variance(values.Mean(), 0, values.Count);
}
public static double Variance(this List<double> values, double mean)
{
return values.Variance(mean, 0, values.Count);
}
public static double Variance(this List<double> values, double mean, int start, int end)
{
double variance = 0;
for (int i = start; i < end; i++)
{
variance += Math.Pow((values[i] - mean), 2);
}
int n = end - start;
if (start > 0) n -= 1;
return variance / (n);
}
public static double StandardDeviation(this List<double> values)
{
return values.Count == 0 ? 0 : values.StandardDeviation(0, values.Count);
}
public static double StandardDeviation(this List<double> values, int start, int end)
{
double mean = values.Mean(start, end);
double variance = values.Variance(mean, start, end);
return Math.Sqrt(variance);
}
}
}

/// <summary>
/// Calculates standard deviation, same as MATLAB std(X,0) function
/// <seealso cref="http://www.mathworks.co.uk/help/techdoc/ref/std.html"/>
/// </summary>
/// <param name="values">enumumerable data</param>
/// <returns>Standard deviation</returns>
public static double GetStandardDeviation(this IEnumerable<double> values)
{
//validation
if (values == null)
throw new ArgumentNullException();
int lenght = values.Count();
//saves from devision by 0
if (lenght == 0 || lenght == 1)
return 0;
double sum = 0.0, sum2 = 0.0;
for (int i = 0; i < lenght; i++)
{
double item = values.ElementAt(i);
sum += item;
sum2 += item * item;
}
return Math.Sqrt((sum2 - sum * sum / lenght) / (lenght - 1));
}

The trouble with all the other answers is that they assume you have your
data in a big array. If your data is coming in on the fly, this would be
a better approach. This class works regardless of how or if you store your data. It also gives you the choice of the Waldorf method or the sum-of-squares method. Both methods work using a single pass.
public final class StatMeasure {
private StatMeasure() {}
public interface Stats1D {
/** Add a value to the population */
void addValue(double value);
/** Get the mean of all the added values */
double getMean();
/** Get the standard deviation from a sample of the population. */
double getStDevSample();
/** Gets the standard deviation for the entire population. */
double getStDevPopulation();
}
private static class WaldorfPopulation implements Stats1D {
private double mean = 0.0;
private double sSum = 0.0;
private int count = 0;
#Override
public void addValue(double value) {
double tmpMean = mean;
double delta = value - tmpMean;
mean += delta / ++count;
sSum += delta * (value - mean);
}
#Override
public double getMean() { return mean; }
#Override
public double getStDevSample() { return Math.sqrt(sSum / (count - 1)); }
#Override
public double getStDevPopulation() { return Math.sqrt(sSum / (count)); }
}
private static class StandardPopulation implements Stats1D {
private double sum = 0.0;
private double sumOfSquares = 0.0;
private int count = 0;
#Override
public void addValue(double value) {
sum += value;
sumOfSquares += value * value;
count++;
}
#Override
public double getMean() { return sum / count; }
#Override
public double getStDevSample() {
return (float) Math.sqrt((sumOfSquares - ((sum * sum) / count)) / (count - 1));
}
#Override
public double getStDevPopulation() {
return (float) Math.sqrt((sumOfSquares - ((sum * sum) / count)) / count);
}
}
/**
* Returns a way to measure a population of data using Waldorf's method.
* This method is better if your population or values are so large that
* the sum of x-squared may overflow. It's also probably faster if you
* need to recalculate the mean and standard deviation continuously,
* for example, if you are continually updating a graphic of the data as
* it flows in.
*
* #return A Stats1D object that uses Waldorf's method.
*/
public static Stats1D getWaldorfStats() { return new WaldorfPopulation(); }
/**
* Return a way to measure the population of data using the sum-of-squares
* method. This is probably faster than Waldorf's method, but runs the
* risk of data overflow.
*
* #return A Stats1D object that uses the sum-of-squares method
*/
public static Stats1D getSumOfSquaresStats() { return new StandardPopulation(); }
}

We may be able to use statistics module in Python. It has stedev() and pstdev() commands to calculate standard deviation of sample and population respectively.
details here: https://www.geeksforgeeks.org/python-statistics-stdev/
import statistics as st
print(st.ptdev(dataframe['column name']))

This is Population standard deviation
private double calculateStdDev(List<double> values)
{
double average = values.Average();
return Math.Sqrt((values.Select(val => (val - average) * (val - average)).Sum()) / values.Count);
}
For Sample standard deviation, just change [values.Count] to [values.Count -1] in above code.
Make sure you don't have only 1 data point in your set.