been trying to implement a Q deep learning algorithm, having an issue though, its not working, after 100 000 game plays and using 1000 iterations to train each step (although i have tried lower numbers for both) it's still not learning. Network and game are in the linked image, http://imgur.com/a/hATfB here is what happens in each training step:
double maxQval;
double[] inputvec;
int MaxQ = GetRandDir(state, out maxQval, out inputvec);//input vec is board
double[] QtarVec = new double[] { 0, 0, 0, 0 };
double r = GetR((int)state[0], (int)state[1]); // GetR is reward
QtarVec[MaxQ] = Qtar(r, maxQval); // backprop vector of 0's except Qtar replaces a value
associator.Train(50, new double[][] { inputvec }, new double[][] { QtarVec });
Training data pair for backprop is (input i linked in image,QTarget = r + gamma * MaxQ) , MaxQ is max network output layer activation or a random one (epsilon greedy). r is reward obtained from each move, -10 for obstacle and 10 for goal. (althogh I have tried just 10 for goal and 0 for everything else. Here is training code.
public void Train(int nTrails)
{
double[] state = new double[] { 1, 1 }; // inital position
int its = 0;
for (int i = 0; i < nTrails; i++)
{
while (((state[0] < 4) && (state[1] < 4))&&((state[0] * 100 >0) && (state[1] * 100 >0)) && (state[0] != 3 && state[1] != 3))//while on board and not at goal postion
{
double temp = r.NextDouble();
int next = -1;
lines.Add(new Vector2((float)(state[0] * 100), (float)(state[1] * 100)));
if (temp < epsilon)
{
next = TrainRandIt(state); // move random direction, backprop
}
else
{
next = TrainMaxIt(state); // move in max activation direction, backprop
}
if (next == 0) .//updating postion
{
state[0]++;
}
else if (next == 1)
{
state[0]--;
}
else if (next == 2)
{
state[1]++;
}
else if (next == 3)
{
state[1]--;
}
}
}
state[0] = 1;
state[1] = 1; // resetting game
}
Any Help appreciated.
Judging from the linked image you provided, it is just like a maze game where you have inputs for the player's position and the output as the direction the player should move to (up, down, left or right).
Here is a machine learning engine which is able to solve exactly that and more - the Ryskamp Learning Machine (RLM). The RLM has a different approach compared to the typical machine learning engines that you may have tried so far so I suggest you go to the link I've provided to learn more about it and what makes it different.
It is written in C# and we have an example of the Maze game just like the one you are trying out which you can browse through our Github page or even try it yourself by cloning/downloading the source code together with the examples apps provided.
For documentation, you may refer to the Documentations files provided or even through the github wiki.
The RLM is also available via Nuget.
Related
I'm trying to find a faster implementation of a flood fill algorithm for a program I'm making using C# in Unity 2020.
This is my current method, which in my program takes about 400ms to run on a 1000 x 1000 map. Instead of a target colour to replace, I am using a height map (called noiseMap in this code snippet) and all values above a threshold should be considered inside the flooded area.
public void Flood()
{
landMasses.Clear();
globalSet.Clear();
HashSet<Vector2Int> samples = new HashSet<Vector2Int>();
for (int x = 0; x < mapGen.mapSize; x += mapGen.scanStride)
{
for (int y = 0; y < mapGen.mapSize; y += mapGen.scanStride)
{
samples.Add(new Vector2Int(x, y));
}
}
float[,] noiseMap = mapGen.noiseMap;
int mapSize = mapGen.mapSize;
float threshold = mapGen.threshold;
foreach (var sample in samples)
{
CalculateSets(sample, noiseMap, mapSize, threshold);
}
}
public bool Inside(Vector2Int point)
{
return Inside(point.x, point.y);
}
public bool Inside(int x, int y)
{
if (x < mapGen.mapSize && x >= 0 && y < mapGen.mapSize && y >= 0)
{
return mapGen.noiseMap[x, y] > mapGen.threshold;
}
return false;
}
public void CalculateSets(Vector2Int sample, float[,] noiseMap, int mapSize, float threshold)
{
if (globalSet.Contains(sample) || noiseMap[sample.x, sample.y] < threshold)
{
return;
}
HashSet<Vector2Int> set = new HashSet<Vector2Int>();
Queue<Vector2Int> queue = new Queue<Vector2Int>();
queue.Enqueue(sample);
while (queue.Count > 0)
{
Vector2Int n = queue.Dequeue();
if (set.Contains(n))
{
continue;
}
if(Inside(n))
{
set.Add(n);
globalSet.Add(n);
queue.Enqueue(new Vector2Int(n.x, n.y - 1));
queue.Enqueue(new Vector2Int(n.x, n.y + 1));
queue.Enqueue(new Vector2Int(n.x - 1, n.y));
queue.Enqueue(new Vector2Int(n.x + 1, n.y));
}
}
landMasses.Add(landMasses.Count.ToString(), set);
}
I've looked around at places like Wikipedia and other online forums for an implementation of the scan line flood fill, but every implementation I find has very little documentation to go along with it, or has no definitions of what their variable names represent. Regardless of this, I have tried to decipher these other implementations and have had 0 luck.
For example, on the Floodfill Wikipedia Page, there are a few different methods along with pseudocode to go along with it - but I cannot find definitions for what most of the variables mean in the later methods which are supposedly faster. Perhaps it's simple, but as someone overall new to computing algorithms I am struggling to figure it out.
So at the end of all this, I am essentially just looking for a faster way to implement something like a floodfill algorithm than what I currently have. It doesn't need to exactly fit into my program of course, even just a general C# implementation or more clarified pseudocode example with comments will be a great help.
Thank you for reading!!
Good day everyone, I'm writing my own pathfinding script, first had it on paper then started coding and let me tell you, it's much harder in practice than in theory. So, I've ran into a problem, which I of course can't solve.
The problem presents itself in the following images:1) In this shot, the waypoint is set to (6,6) and returns no errors.
2) Note the 2 points in the upper right corner, one shows no direction the other shows up. Error is the node that points upwards. In this shot, the waypoint is moved to (7,5) at which point it starts throwing errors starting from the last index. The more I move the waypoint closer to the bottom-right corner, the more points at X=13 down the Y axis throws exceptions.
Relevant code:
for (int x = 0; x < map.sizeX; x++)
{
for (int y = 0; y < map.sizeY; y++)
{
if (!(x == pVal.x && y == pVal.y) && map.grid[x, y].passable )
{
float dot = 1;
var heading = (grid[x, y].position - t.position).normalized;
heading.y = 0;
foreach (Vector3 direction in map.GetDirections())
{
var dot2 = Vector3.Dot(heading, direction.normalized);
if (dot > dot2 )
{
if (map.grid[x + (int)direction.x, y + (int)direction.y].passable)
{ // Error thrown when it reaches this if-statement \\
grid[x, y].direction = direction;
dot = dot2;
}
}
}
}
}
}
This Index out of Bounds error is only thrown when I add the check to see if the point towards the direction is passable or not. Another thing to note is that I use direction.y where the directions are actually stored in x and z. For some reason if I use the z instead of y, it stops working completely.
When in doubt, try walking through a test case to see what goes wrong.
Let's say we're in the second image, x = 13, y = 12
if (!(x == pVal.x && y == pVal.y) && map.grid[x, y].passable )
(13, 12) is not our target point, and is passable, so we pass this test and proceed to the next line...
float dot = 1;
var heading = (grid[x, y].position - t.position).normalized;
heading.y = 0;
heading ends up being something like (0.659, 0, 0.753) here, though if you have some y offsets it might be shorter since you normalize it before zeroing the y.
foreach (Vector3 direction in map.GetDirections())
I don't know what order your directions are stored in, so I'll just guess here:
{(0, 0, 1), (1, 0, 1), (1, 0, 0), (1, 0, -1)...}
Starting with (0, 0, 1) then...
var dot2 = Vector3.Dot(heading, direction.normalized);
if (dot > dot2 )
dot is still 1, and dot2 is 0.753 so we pass this test, check that the cell above is passable (even though that points wildly away from the direction we want to go! More on that soon), set dot = dot2 and try the next direction:
(1, 0, 1) normalizes to (0.707, 0, 0.707). dot is 0.753 and dot2 is 0.466 + 0.532 = 0.998 so we fail the dot > dot2 test and skip this one.
Here's the killer: (1, 0, 0)
dot is still 0.753 and dot2 is 0.659, so we pass the dot > dot2 test, and proceed to check the cell in that direction:
if (map.grid[x + (int)direction.x, y + (int)direction.y].passable)
{ // Error thrown when it reaches this if-statement \\
No kidding an error is thrown! We were already at x = 13 (ie. map.sizeX - 1) and we added 1, so we're off the edge of the board!
So, this error is easy to detect just by walking through the problem case.
Possible fixes (from most to least hacky):
Do bounds checking whenever you try to access an adjacent cell, and skip it if it would lead off the map.
Add a border of unused grid cells around your map, so checking one cell off the edge never poses a problem.
Consider switching to a more conventional, well-studied pathfinding algorithm, like Breadth-first search (all moves cost the same) or Djikstra's algorithm (distinct move costs) if you want to populate the entire grid with pathing information, or A* if you want the shortest point-to-point path.
Make a simple function that checks if you are within the bounds of the map.grid 2D array before doing if (map.grid[x + (int)direction.x, y + (int)direction.y].passable).
Check if map.grid[x + (int)direction.x is less than map.sizeX-1
then check if map.grid[ y + (int)direction.y] is less than map.sizeY-1.
If both conditions are met, go ahead with the if (map.grid[x + (int)direction.x, y + (int)direction.y].passable).
Here is a simple function to simplify that:
bool isWithinBound(Vector3 direction, int sizeX, int sizeY, int x, int y)
{
return ((x + (int)direction.x < sizeX - 1) && (y + (int)direction.y < sizeY - 1));
}
Now you can just do:
for (int x = 0; x < map.sizeX; x++)
{
for (int y = 0; y < map.sizeY; y++)
{
if (!(x == pVal.x && y == pVal.y) && map.grid[x, y].passable)
{
float dot = 1;
var heading = (grid[x, y].position - t.position).normalized;
heading.y = 0;
foreach (Vector3 direction in map.GetDirections())
{
var dot2 = Vector3.Dot(heading, direction.normalized);
if (dot > dot2)
{
//Check if we are within bounds
if (isWithinBound(direction, map.sizeX, map.sizeY, x, y))
{
if (map.grid[x + (int)direction.x, y + (int)direction.y].passable)
{ // Error thrown when it reaches this if-statement \\
grid[x, y].direction = direction;
dot = dot2;
}
}
}
}
}
}
}
Picture 1.
Picture 2.
I made method for checking if all pieces are in base or goal if yes it returns true,now i need another method.
If its like on Picture 1. i need to change number of throws to 3 ,if its ordered like on Picture 2 i can allow only 1 throw.
I got 4 goalPositions and 4 piecePositions,and need to check if pieces are ordered on them properly from 54-51 path positions(path is array of 55 fields 0-54) ,if yes return true if not return false.
I am new to C# never had chance to work with order checking till now.
I was trying to do it with 3 int lists goalPositions (populated with 51,52,53,54 path positions),piecePositions(populated with pieces positions with getPosition() method),and piecesOnGoal (reserved for counting pieces on goal positions). but no luck with that.
I'll add some code. Part of Player class with that lists and method for checking pieces in goal or base
class Player {
protected PieceSet[] pieces;
Color color;
int numberOfThrows;
Dice dice;
public List<int> goalPositions;
public List<int> piecePositions;
public List<int> piecesOnGoal;
public enum Color
{
Yellow, Green, Blue, Red
}
public Player(Color color)
{
int[] path = new int[55];
this.color = color;
dice = new Dice();
numberOfThrows = 3;
switch (color)
{
case Color.Yellow:
path = BoardHelper.getYellowPath();
break;
case Color.Green:
path = BoardHelper.getGreenPath();
break;
case Color.Blue:
path = BoardHelper.getBluePath();
break;
case Color.Red:
path = BoardHelper.getRedPath();
break;
}
pieces = new PieceSet[4];
pieces[0] = new PieceSet(path, 0);
pieces[1] = new PieceSet(path, 1);
pieces[2] = new PieceSet(path, 2);
pieces[3] = new PieceSet(path, 3);
piecePositions = new List<int>(4);
piecePositions.Add(pieces[0].getPosition());
piecePositions.Add(pieces[1].getPosition());
piecePositions.Add(pieces[2].getPosition());
piecePositions.Add(pieces[3].getPosition());
goalPositions = new List<int>(4);
goalPositions.Add(51);
goalPositions.Add(52);
goalPositions.Add(53);
goalPositions.Add(54);
piecesOnGoal =new List<int>();
}
public bool isAllPiecesInBaseOrGoal()
{
if ((pieces[0].getPosition() < 4 || pieces[0].getPosition() > 50) &&
(pieces[1].getPosition() < 4 || pieces[1].getPosition() > 50) &&
(pieces[2].getPosition() < 4 || pieces[2].getPosition() > 50) &&
(pieces[3].getPosition() < 4 || pieces[3].getPosition() > 50))
return true;
else
return false;
}
And this is how I was thinking to solve my problem: Check if goalPositions contains piecePositions. If yes, add that position into piecesOnGoal. Now I need somehow to check if these piecesOnGoal are ordered. If yes, return true. If not, false.
I am open for any suggestion.
public bool isAllPiecesAreOrderedInGoal()
{
for (int i = 0; i < 4; i++)
{
if (goalPositions.Contains(piecePositions[i]))
{
piecesOnGoal.Add(piecePositions[i]);
}
}
}
Any help is appreciated.
I would suggest a method that checks if any move is possible. A move is possible if there are pieces outside the home or goal or if there are pieces in the goal which have an empty spot before them:
bool IsMovePossible()
{
// check every piece
for(int iPiece = 0; iPiece < 4; ++iPiece)
{
// if it is outside of home or goal, there is a possible move
if(piecePositions[iPiece] > 3 && piecePositions[iPiece] < goalPositions.First())
return true;
// if it is in the goal, check the next position
if(piecePositions[iPiece] >= goalPositions.First()
&& piecePositions[iPiece] < goalPositions.Last()
&& !piecePositions.Any(p => p == piecePositions[iPiece] + 1))
return true;
}
// we have found no piece with a possible move
return false;
}
Then, decide how often the user can roll the dice based on this method's return value. Note that the last check (piecePositions.Any) could be done more efficiently if you maintain an inverted map (i.e. for every position, store which piece is on that position). But checking four pieces should be fine.
I am trying to recognize 0 to 9 digits using Aforge.net . I tried everything but I am still unable to get result please look at my program and why I am unable to recognize digits. Problem may be in number of hidden layers, learning rate or input data , I have tried it by changing number of hidden layers and learning rate. Please suggest ideas.
// opening file
OpenFileDialog open = new OpenFileDialog();
ActivationNetwork enactivation = new ActivationNetwork(new BipolarSigmoidFunction(1), 3886,10, 10);
double[][] input = new double[10][];
double[][] output = new double[10][];
//generating input data using Feature class -- which code is given below
Feature feature = new Feature();
//iterating for all 10 digits.
for (int i = 0; i < 10; i++)
{
open.ShowDialog();
Bitmap bitmap = new Bitmap(open.FileName);
double[] features = feature.features(bitmap);
input[i] = features;
features = feature.features(bitmap);
output[i] = feature.features(bitmap);
}
enactivation.Randomize();
BackPropagationLearning learn = new BackPropagationLearning(enactivation);
//learning
learn.LearningRate = 0.005f;
learn.Momentum = 0.005f;
double errora;
int iteration = 0;
while (true)
{
errora = learn.RunEpoch(input, output);
if (errora < 0.0006)
break;
else if (iteration > 23000)
break;
iteration++;
// Console.WriteLine("error {0} {1} ", errora, iteration);
}
double[] sample;
open.ShowDialog();
Bitmap temp = new Bitmap(open.FileName);
// providing input for computation using feature class
sample = feature.features(temp);
foreach (double daa in enactivation.Compute(sample))
{
Console.WriteLine(daa);
}
Class Feature for providing input for training nural network
class Feature
{
public double[] features(Bitmap bitmap)
{
//feature
double[] feature = new double[bitmap.Width * bitmap.Height];
int featurec = 0;
for (int vert = 0; vert < bitmap.Height; vert++)
{
for (int horizantal = 0; horizantal < bitmap.Width; horizantal++)
{
feature[featurec] = bitmap.GetPixel(horizantal, vert).ToArgb();
if (feature[featurec] < 1)
{
feature[featurec] = -0.5;
}
else
{
feature[featurec] = 0.5;
}
featurec++;
}
}
return feature;
}
}
I haven't used aforge, but re. using backprop neural nets for this problem:
You need something like a 10x10 input grid with each cell in the grid getting 1/100 of the image
You need at least one, possibly 2, hidden layers
The net will train faster with a bias input - meaning a source of a fixed value - for each cell (this lets the cells train faster: Role of Bias in Neural Networks)
I'd never start in bp mode but always run something a statistical annealing first. Bp is for descending inside a local minimum once one is found
Also:
Have you successfully used aforge for other problems?
What happens when you try to train the net?
It's for a game of checkers. See revision history for older versions of code.
private static Move GetBestMove(Color color, Board board, int depth)
{
var bestMoves = new List<Move>();
var validMoves = board.GetValidMoves(color);
int highestScore = int.MinValue;
Board boardAfterMove;
int tmpScore;
var rand = new Random();
Debug.WriteLine("{0}'s Moves:", color);
foreach (var move in validMoves)
{
boardAfterMove = board.Clone().ApplyMove(move);
if(move.IsJump && !move.IsCrowned && boardAfterMove.GetJumps(color).Any())
tmpScore = NegaMax(color, boardAfterMove, depth);
else
tmpScore = -NegaMax(Board.Opposite(color), boardAfterMove, depth);
Debug.WriteLine("{0}: {1}", move, tmpScore);
if (tmpScore > highestScore)
{
bestMoves.Clear();
bestMoves.Add(move);
highestScore = tmpScore;
}
else if (tmpScore == highestScore)
{
bestMoves.Add(move);
}
}
return bestMoves[rand.Next(bestMoves.Count)];
}
private static int NegaMax(Color color, Board board, int depth)
{
var validMoves = board.GetValidMoves(color);
int highestScore = int.MinValue;
Board boardAfterMove;
if (depth <= 0 || !validMoves.Any())
return BoardScore(color, board);
foreach (var move in validMoves)
{
boardAfterMove = board.Clone().ApplyMove(move);
if(move.IsJump && !move.IsCrowned && boardAfterMove.GetJumps(color).Any())
highestScore = Math.Max(highestScore, NegaMax(color, boardAfterMove, depth));
else
highestScore = Math.Max(highestScore, -NegaMax(Board.Opposite(color), boardAfterMove, depth - 1));
}
return highestScore;
}
private static int BoardScore(Color color, Board board)
{
if (!board.GetValidMoves(color).Any()) return -1000;
return board.OfType<Checker>().Sum(c => (c.Color == color ? 1 : -1) * (c.Class == Class.Man ? 2 : 3));
}
I'm trying it with depth 0, and the scores are correct for about half the game, and then all of a sudden it starts screwing up. One of the players will start proclaiming his score is higher than it really is. Why would it only work for half a game?!
Interesting approach, the first time I see MaxiMax. But I see a problem here:
var minMove = GetBestMove(... board.Clone().ApplyMove(move), ...);
float score = ... BoardScore(color, board.Clone().ApplyMove(minMove));
In this code, move and minMove are moves for different sides and yet you apply them equally at the same level here. The second line should be something like:
float score = ... BoardScore(... board.Clone().ApplyMove(move).ApplyMove(minMove));
You can of course store and re-use the board.Clone().ApplyMove(move) part.
But then you still loose information: At Depth 100 you filter out the best boardScore at depth 99 but you don't have/use anything from levels 98..0 except when there was no move (null), but as you noticed yourself that part goes wrong.
Tried looking at some pseudo
algorithms, but all the seem to return
a score. That confuses me, because I
don't really want to get a score back,
I want to get a Move back.
Still, that is the way to go. The main result from a tree-search is the value of the best branch. The move itself is only essential at the root level. Leave it until you start implementing alpha/beta, then you will be able to store the best branch in a single table.
I would advice switching to a regular NegaMax,
also see this SO question.
Found the bug: What could cause this to start miscalculating after awhile?
New code:
private static Move GetBestMove(Color color, Board board, int depth)
{
var bestMoves = new List<Move>();
IEnumerable<Move> validMoves = board.GetValidMoves(color);
int highestScore = int.MinValue;
Board boardAfterMove;
int tmpScore;
var rand = new Random();
Debug.WriteLine("{0}'s Moves:", color);
foreach (Move move in validMoves)
{
boardAfterMove = board.Clone().ApplyMove(move);
if (move.IsJump && !move.IsCrowned && boardAfterMove.GetJumps(color).Any())
tmpScore = NegaMax(color, boardAfterMove, depth);
else
tmpScore = -NegaMax(Board.Opposite(color), boardAfterMove, depth);
Debug.WriteLine("{0}: {1}", move, tmpScore);
if (tmpScore > highestScore)
{
bestMoves.Clear();
bestMoves.Add(move);
highestScore = tmpScore;
}
else if (tmpScore == highestScore)
{
bestMoves.Add(move);
}
}
return bestMoves[rand.Next(bestMoves.Count)];
}
private static int NegaMax(Color color, Board board, int depth)
{
IEnumerable<Move> validMoves = board.GetValidMoves(color);
int highestScore = int.MinValue;
Board boardAfterMove;
if (depth <= 0 || !validMoves.Any())
return BoardScore(color, board);
foreach (Move move in validMoves)
{
boardAfterMove = board.Clone().ApplyMove(move);
if (move.IsJump && !move.IsCrowned && boardAfterMove.GetJumps(color).Any())
highestScore = Math.Max(highestScore, NegaMax(color, boardAfterMove, depth));
else
highestScore = Math.Max(highestScore, -NegaMax(Board.Opposite(color), boardAfterMove, depth - 1));
}
return highestScore;
}
private static int BoardScore(Color color, Board board)
{
if (!board.GetValidMoves(color).Any()) return -1000;
return board.OfType<Checker>().Sum(c => (c.Color == color ? 1 : -1) * (c.Class == Class.Man ? 2 : 3));
}
I'm not 100% convinced this works perfectly. It seems to work for depth 0, and usually for depth 1... beyond that, I have no idea what the computer is thinking. Still doesn't appear to play super intelligently.
Edit: Running this and max speed... NegaMax agent vs Random. NegaMax always wins. Watching the scores for occurrences of "1000". He always wins within a few turns after that, so it does appear to be working, finally!