We Keep Coding

Average loss stuck on local minima (Self developed Perceptron in C#)

I am trying to build a custom machine learning library in C# , I have researched fairly about the topic. My first example (XOR estimator) was a success, I was able to lower the average loss to almost cero. Then I tried to build a model to classify handwritten digits (using MNIST text database).The problem is ,no matter how I configure the model, I always get stuck on a certain average loss over the data set.Second problem, because the MNIST dataset is very large, the model takes a lot of time to compute, maybe I can use some advice on how to carry on the slowest parts of the algorithm (I am using stochastic gradient descent).I am going to show the main method that performs most of the work.
I have tried using MSE and CrossEntropy loss functions, also tanh , sigmoid , reLu and softPlus activation functions. The model I am trying to build is a 4 layer one. First layer, 784 input neurons ; Second , 16 neurons,sigmoid ; Third , 16 neurons , sigmoid and Output layer, 10 neurons (one hot encoded digits) with sigmoid. I am aware that the code below may not be a minimal reproducible example, but it represents the algorithm I am trying to figure. I have also uploaded the solution to GitHub, maybe somebody can give me a hand figuring the problem. This is the link https://github.com/juan-carvajal/MachineLearningFramework
Running the Main method of the app will first, execute the XOR classifier, that runs good. Then the MNIST classifier.
The model is best represented here:
DataSet dataSet = new DataSet("mnist2.txt", ' ', 10, false);
//This creates a model with batching=128 , learningRate=0.5 and
//CrossEntropy loss function
var p = new Perceptron(128, 0.5, ErrorFunction.CrossEntropy())
.Layer(784, ActivationFunction.Sigmoid())
.Layer(16, ActivationFunction.Sigmoid())
.Layer(16, ActivationFunction.Sigmoid())
.Layer(10, ActivationFunction.Sigmoid());
//1000 is the number of epochs
p.Train2(dataSet, 1000);
Actual Algorithm (stochastic gradiente descent):
Console.WriteLine("Initial Loss:"+ CalculateMeanErrorOverDataSet(dataSet));
for (int i = 0; i < epochs; i++)
{
//Shuffle the data in every step
dataSet.Shuffle();
List<DataRow> batch = dataSet.NextBatch(this.Batching);
//Gets random batch from the dataSet
int count = 0;
foreach (DataRow example in batch)
{
count++;
double[] result = this.FeedForward(example.GetFeatures());
double[] labels = example.GetLabels();
if (result.Length != labels.Length)
{
throw new Exception("Inconsistent array size, Incorrect implementation.");
}
else
{
//What follows is the calculation of the gradient for this example, every example affects the current gradient, then all those changes are averaged an every parameter is updated.
double error = CalculateExampleLost(example);
for (int l = this.Layers.Count - 1; l > 0; l--)
{
if (l == this.Layers.Count - 1)
{
for (int j = 0; j < this.Layers[l].CostDerivatives.Length; j++)
{
this.Layers[l].CostDerivatives[j] = ErrorFunction.GetDerivativeValue(labels[j], this.Layers[l].Activations[j]);
}
}
else
{
for (int j = 0; j < this.Layers[l].CostDerivatives.Length; j++)
{
double acum = 0;
for (int j2 = 0; j2 < Layers[l + 1].Size; j2++)
{
acum += Layers[l + 1].WeightMatrix[j2, j] * this.Layers[l+1].ActivationFunction.GetDerivativeValue(Layers[l + 1].WeightedSum[j2]) * Layers[l + 1].CostDerivatives[j2];
}
this.Layers[l].CostDerivatives[j] = acum;
}
}
for (int j = 0; j < this.Layers[l].Activations.Length; j++)
{
this.Layers[l].BiasVectorChangeRecord[j] += this.Layers[l].ActivationFunction.GetDerivativeValue(Layers[l].WeightedSum[j]) * Layers[l].CostDerivatives[j];
for (int k = 0; k < Layers[l].WeightMatrix.GetLength(1); k++)
{
this.Layers[l].WeightMatrixChangeRecord[j, k] += Layers[l - 1].Activations[k]
* this.Layers[l].ActivationFunction.GetDerivativeValue(Layers[l].WeightedSum[j])
* Layers[l].CostDerivatives[j];
}
}
}
}
}
TakeGradientDescentStep(batch.Count);
if ((i + 1) % (epochs / 10) == 0)
{
Console.WriteLine("Epoch " + (i + 1) + ", Avg.Loss:" + CalculateMeanErrorOverDataSet(dataSet));
}
}
This is an example of what the local minima looks like in the current model.
In my research I found out that similar models may archieve accuracy up to 90%. My model barely got 10%.

C# string.split is splitting letter by letter when pulled from IBM Emulator

I work in an IBM emulator environment. I'm trying to grab a row of text and split by spaces to return each word in the row. However, when printed to the console it's showing only first letters. This code has worked in the past when utilized on raw text or api results. The same code when implemented on an IBM Emulator does not however.
For example if the row from the emulator is HEALTHSELECT OF TEXAS , the code below would do the first and second console print correctly. With the double for-loop printing only the first letters. The real listed console output is below:
looking for healthselect of texas in away from home care
healthselect
of
texas
looking for h in a
looking for h in w
looking for h in a
looking for h in y
looking for e in a
looking for e in w
looking for e in a
looking for e in y
looking for a in a
looking for a in w
looking for a in a
looking for a in y
C# code:
public bool inStringAll(string needle, string haystack)
{
needle = Regex.Replace(needle.ToLower(), #"\r\n", string.Empty);
haystack = Regex.Replace(haystack.ToLower(), #"\r\n", string.Empty);
Console.WriteLine($"looking for {needle} in {haystack}");
string[] needleArr = needle.Split(' ');
string[] haystackArr = haystack.Split(' ');
for(int j = 0; j < needleArr.Length; j++)
Console.WriteLine(needleArr[j]);
int percent = 0;
int increment;
bool decision = false;
if (needleArr.Length > 3)
increment = 100 / 3;
else increment = 100 / needleArr.Length;
for (int i = 0; i < 3; i++)
{
for (int j = 0; j < haystackArr.Length; j++)
{
Console.WriteLine("looking for " + needle[i] + " in " + haystack[j]);
if (needleArr[i] == haystackArr[j])
{
percent += increment;
}
}
}
Console.WriteLine($"Percent: {percent}%");
if (percent >= 50)
decision = true;
return decision;
}
Has anyone worked with an emulator of this kind and had any issues getting these values correctly? I've worked with old attachmate systems using EXTRA! Basic and there was no issues, but this is my first client with IBM systems using C#.net so it may be different.

Simple C# program using arrays throws an error

Well I'm trying to write a program in which if you add for example 3 integers in the array, let's say 3 2 1, it will add them again after it so it becomes 321 321.
Here is the code I need to fix. And sorry for the stupid question I am a beginner with arrays.
I get this error
Index was outside the bounds of the array
My code:
using System;
public class Program
{
public static void Main()
{
int arraylength = int.Parse(Console.ReadLine());
int[] array = new int[arraylength];
for (int i = 0; i < arraylength + 1 / 2; i++)
{
int typed = int.Parse(Console.ReadLine());
array[i] = typed;
if (i == arraylength / 2)
{
for (int a = arraylength + 1 / 2; a < arraylength + 1; a++)
{
array[a] = typed;
}
}
}
}
}

Array indices in C# start at 0 and end at length - 1. You need to remove the + 1 from each of your for loop conditions:
for (int i = 0; i < arraylenght / 2; i++)
and
for (int a = (arraylenght + 1) / 2; a < arraylenght; a++)
I also suggest that you change arraylenght to arraylength. Since you probably autocompleted this every time you used it, the misspelling occurs consistently throughout your code and the compiler is satisfied. However, misspellings make it difficult for humans to read your code.
p.s. Your code doesn't do what you think it does. I suggest you step away from the computer for a moment and write in words what you are trying to accomplish. Describe each step of your solution in as much detail as you can. Then look at how your words match with the code you wrote. You will probably find that you do not need nested loops.

Drawing an ASCII tree using nested for-loops

I am practising my programming skills with a rather common problem, to draw an ASCII tree composed of a letter (let's say x's) to draw a tree.
Ie:
Prompt: Enter the number of lines for your tree: 5
Output:
x
xxx
xxxxx
xxxxxxx
x
The solution I have is currently working as I want it to. I am only missing one feature (which is part of my question). The issue is, I do not understand the for-loop conditionals I have used. Yes, I did get help from another individual and I still do not understand their logic, but it works. Please, please, please explain to me how the for loop conditionals are actually working for me to draw what I want. x.x
Furthermore, I am unable to draw the last "stump" of the tree, the final x in the last row.
Here is my code.
static void Main(string[] args)
{
int lines = 0;
Console.WriteLine("Enter number of lines: ");
string LinesAsString = Console.ReadLine();
lines = Convert.ToInt32(LinesAsString);
print(lines);
}
public static void print(int lines)
{
for (int i = 0; i < lines; i++)
{
for (int a = 0; a < lines-i-1; a++)
{
Console.Write(" ");
}
for (int y = 0; y < i*2 + 1; y++)
{
Console.Write("x");
}
Console.WriteLine();
}
}
Any help is appreciated.
Thanks.

The first for loop prints the left padding (all spaces). The expression lines - i - 1 comes form the fact that you need to center the tree and you know that line 0 has a single x, line 1 has 3 (xxx), line 2 has 5 (xxxxx) and so on. But the number of left spaces depends on the total number of lines the tree has, therefore you need to take the lines into account as well as the index of the current line (which is the value of the variable i). To figure out the relationship between these two, you can just try with a small number of levels, let's say 3:
__x 2 for the 1st line.
_xxx 1 for the 2nd line.
xxxxx 0 for the 3rd line.
The second for loop prints the xs. It knows that for the first line (i = 0) it needs to print a single x, so when i is 0, y needs to be 1, therefore the + 1. The * 2 comes from the progression i * 2 + 1 = {1, 3, 5, 7...} which is the number of xs in each line.
To print the last stub character you can use the same logic as in the first for loop, inserting the number of spaces for the first line followed by a single x.
Console.Write(new String(' ', lines - 1) + "x");
This should be added just after the for loop that contains the two other loops.

It's quite simple:
Each iteration of the first for loop draws a line. Supposing you have 4 lines (I am not considering the one with the lonely x):
line 1 (i=0): 3 spaces 1 x
line 2 (i=1): 2 spaces 3 x
line 3 (i=2): 1 spaces 5 x
line 4 (i=3): 0 spaces 7 x
From these, you can obtain a relation between the number of spaces, x's, the line index i and the number of lines.
spaces = (lines - i) - 1 // decreases by one every iteration starting at lines - 1
x = i * 2 + 1 // the odd numbers in increasing order
The first loop draw the spaces and the second one the x's

int count = 0;
for (int i = 0; i < 8; i++)
{
for (int x = 0; x < 8-i-1; x++)
{
Console.Write(" ");
}
for (int j = 0; j < i*2+1; j++)
{
Console.Write("X");
}
Console.WriteLine();
}
//solution to your stump problem
for (int i = 0; i < 4; i++)//the no. 4 means that you stump will be made from four X's
{
for (int x = 0; x < 8-count-1; x++)
{
Console.Write(" ");
}
Console.Write("X");
Console.WriteLine();
}
Console.ReadKey();

Index outside array inside loop

I get an error Index was outside the bounds of the array, im trying to do some simple math and holding it in my list.
List<int> integerList = new List<int>();
for (int a = 0; a < textBox1.Text.Length; a++)
{
for (int b = 8; b > 1 ; b--)
{
integerList.Add(int.Parse(textBox1.Text[a * b].ToString())); //this line
}
}
listBox1.DataSource = integerList;
What I am trying to achieve is this, a user must enter a 7 digit number into the textbox say for instance 4565457, I wanted to store this number in my integerList, then take each number starting from the beginning of the users input and multiply down from 8 untill 2 is reached.
For instance:
4 x 8
5 x 7
6 x 6
5 x 5
4 x 4
5 x 3
7 x 2
I wanted to then store the sum of these multiplications for later use.

It's probably the textBox1.Text that's out of bounds. Try adding a check before using the indexer:
if (a*b < textBox1.Length)
integerList.Add(int.Parse(textBox1.Text[a * b].ToString())); //this line

Try this (since you probably want to calc the value of your a number by the value of b):
List<int> integerList = new List<int>();
for (int a = 0; a < textBox1.Text.Length; a++)
{
for (int b = 8; b > 1 ; b--)
{
integerList.Add(int.Parse(textBox1.Text[a].ToString()) * b); //this line
}
}
listBox1.DataSource = integerList;

The index a*b is outside the bounds of the TextBox content. You have to put in place safeguards to make sure you don't index the string content of the textbox using an index that is out of bounds

This line textBox1.Text[a * b] is likely the source of the problem. There's no bounds checking and a * b likely evaluates to an integer greater than the final index of textBox1. I don't have an great solution for you because I don't really understand what you're trying to do... The expression inside the inner for loop just generally does not make much sense.
You could simply check that a * b is less than textBox1.Text.Length in order to prevent the exception but that probably won't make the code do what you actually want it to.

Sorry didn't got time to read the edited question. According to new description, if user enters 4565457 you want to multiply each character in that string from position 0 to last (6 in this case) with 8 - position. Following loop single for loop will do the trick.
List<int> integerList = new List<int>();
var l = textBox1.Text.Length;
for (int a = 0, b = 8; a < l && b > 1; a++, b--)
{
integerList.Add((Convert.ToInt16(textBox1.Text[a]) * b)); //this line
}
listBox1.DataSource = integerList;
At the end of loop, listBox1 will contain following values:
416
371
324
265
208
159
110
Try this:
List<int> integerList = new List<int>();
var l = textBox1.Text.Length;
for (int a = 0; a < l; a++)
{
for (int b = 8; b > 1 && (a*b) < l ; b--)
{
integerList.Add(int.Parse(textBox1.Text[a * b].ToString())); //this line
}
}
listBox1.DataSource = integerList;
a * b is getting past the length of text in textbox so adding && (a*b) > l will keep this in control.