We Keep Coding

C# masking an array to exclude indexes as fast as in python

What I have:
//This data set contains columns (second index) having the same value in each row (first index)
double[][] dataSet = new double[][]
{
new double[] {1, 2, 3, 4},
new double[] {5, 6, 7, 4},
new double[] {8, 9, 10, 4},
};
What i want to get:
// This data set has no column where the value in each row is the same
double[][] reducedDataSet = new double[][]
{
new double[] {1, 2, 3},
new double[] {5, 6, 7},
new double[] {8, 9, 10},
};
In python this can be easily done by:
all_equal_value_indices = numpy.all(data_set == data_set[0, :], axis=0) // Finds the indices of all columns that have equal values in each row
reduced_data_set = data_set[:, ~all_equal_value_indices] // Takes all rows with only those columns where all_equal_value_indices is not 1
In C# I can get an array containing the indices that should be excluded relatively fast, but how can I use these indices as mask to get only those columns not contained in these indices?
What i tried:
var numberOfDeletedColumns = 0;
var reducedDataSet = dataSet;
foreach (var columnToDelete in columnsToDelete)
{
reducedDataSet = reducedDataSet.RemoveColumn(columnToDelete - numberOfDeletedColumns++);
}
RemoveColumn is an extension provided by Accord.Net and has the following code:
/// <summary>Returns a new matrix without one of its columns.</summary>
public static T[][] RemoveColumn<T>(this T[][] matrix, int index)
{
T[][] objArray = new T[matrix.Length][];
for (int index1 = 0; index1 < matrix.Length; ++index1)
{
objArray[index1] = new T[matrix[index1].Length - 1];
for (int index2 = 0; index2 < index; ++index2)
objArray[index1][index2] = matrix[index1][index2];
for (int index2 = index + 1; index2 < matrix[index1].Length; ++index2)
objArray[index1][index2 - 1] = matrix[index1][index2];
}
return objArray;
}
But this is much slower than the implementation in python. Could someone suggest a faster method to achieve a reduced data set?

Array.Copy helps it run about 2x faster on my computer.
static T[][] FastRemoveColumn<T>(T[][] matrix, int index)
{
T[][] objArray = new T[matrix.Length][];
for (int i = 0; i < matrix.Length; i++)
{
var line = matrix[i];
var reducedline = new T[line.Length - 1];
Array.Copy(line, 0, reducedline, 0, index);
Array.Copy(line, index + 1, reducedline, index, line.Length - index - 1);
objArray[i] = reducedline;
}
return objArray;
}
and I also tried multithread. It runs very slow:
static T[][] MultiThreadRemoveColumn<T>(T[][] matrix, int index)
{
T[][] objArray = new T[matrix.Length][];
Parallel.For(0, matrix.Length, i =>
{
var line = matrix[i];
var reducedline = new T[line.Length - 1];
Array.Copy(line, 0, reducedline, 0, index);
Array.Copy(line, index + 1, reducedline, index, line.Length - index - 1);
objArray[i] = reducedline;
});
return objArray;
}
Test:
// init
double[][] arr = new double[2000][];
for (int i = 0; i < arr.Length; i++)
arr[i] = new double[2000];
double v = 0;
for (int i = 0; i < arr.Length; i++)
{
for (int j = 0; j < arr[i].Length; j++)
{
arr[i][j] = v;
v++;
}
}
Stopwatch sw = Stopwatch.StartNew();
var reducedArr = RemoveColumn(arr, 200);
sw.Stop();
Console.WriteLine(sw.ElapsedMilliseconds);
sw.Restart();
var reducedArr2 = FastRemoveColumn(arr, 200);
sw.Stop();
Console.WriteLine(sw.ElapsedMilliseconds);
sw.Restart();
var reducedArr3 = MultiThreadRemoveColumn(arr, 200);
sw.Stop();
Console.WriteLine(sw.ElapsedMilliseconds);
// Check the result
for (int i = 0; i < reducedArr.Length; i++)
{
for (int j = 0; j < reducedArr[i].Length; j++)
{
if(reducedArr[i][j] != reducedArr2[i][j]) throw new Exception();
if(reducedArr[i][j] != reducedArr3[i][j]) throw new Exception();
}
}
Update
Solution to remove several columns:
public static T[][] DeleteColumns<T>(T[][] matrix, int[] columns)
{
if (matrix.Length == 0) return new T[0][];
bool[] delColumns = new bool[matrix[0].Length];
foreach (int col in columns) delColumns[col] = true;
List<int> remainCols = new List<int>();
for (int i = 0; i < delColumns.Length; i++)
{
if (!delColumns[i]) remainCols.Add(i);
}
var target = new T[matrix.Length][];
for (int rowIndex = 0; rowIndex < matrix.Length; rowIndex++)
{
T[] sourceRow = matrix[rowIndex];
T[] targetRow = new T[remainCols.Count];
for (int i = 0; i < remainCols.Count; i++)
{
targetRow[i] = sourceRow[remainCols[i]];
}
target[rowIndex] = targetRow;
}
return target;
}
Test on a 2000x2000 matrix. Comparing with Adam Brown's solution, testing removing all columns is absolutely unfair, but my solution is faster even if removing only one column.

Sometimes, components don't work nicely in aggregate. In this case, your remove column function is reallocating the entire matrix, so the operation is linear in the number of columns you want to remove (ouch). To fix this, remove all the columns in one pass.
class Program
{
static void Main(string[] args)
{
var matrix = new[]
{
new [] {1, 2, 3, 4, 5},
new [] {1, 2, 3, 4, 5},
new [] {1, 2, 3, 4, 5},
new [] {1, 2, 3, 4, 5},
};
var result = matrix.DeleteColums(new [] {0, 2, 4});
foreach (var row in result)
{
foreach (var column in row)
{
Console.Write(column);
Console.Write(" ");
}
Console.WriteLine();
}
Console.ReadKey();
}
}
static class MatrixHelper
{
public static T[][] DeleteColums<T>(this T[][] matrix, int[] columns)
{
var sorted = columns.Distinct().OrderBy(e => e).Concat(new [] { int.MaxValue }).ToArray();
var target = new T[matrix.Length][];
for (int row = 0; row < matrix.Length; row++)
{
var sourceRow = matrix[row];
var targetRow = new T[sourceRow.Length - columns.Length];
var sortedIndex = 0;
for (int i = 0; i < sourceRow.Length; i++)
{
if (i == sorted[sortedIndex])
{
sortedIndex++;
continue;
}
targetRow[i - sortedIndex] = sourceRow[i];
}
target[row] = targetRow;
}
return target;
}
}
If this isn't enough, then you'll need to think about whether you need to use arrays. For instance, you could have a data structure for your matrix that dynamically masks columns, instead of an array-of-arrays.
UPDATE:
Given that other solutions in this page have assumed that the matrix, despite being represented by a jagged array, has the same indices per row, I thought I would give another go at making a faster solution. Here are two solutions that beat all previous ones in this thread, under those assumptions, including a faster parallel one.
public static T[][] DeleteColumns<T>(this T[][] matrix, int[] columns)
{
if (matrix.Length == 0) return matrix;
//Previous code assumed matrix could be jagged - new code assumes all columns
//present and all rows same length
var rowLength = matrix[0].Length;
if (rowLength == 0) return matrix;
var sorted = columns.Distinct().ToArray();
var target = new T[matrix.Length][];
var remainingLength = rowLength - sorted.Length;
//Allocate the targets all in one go - to avoid doing allocation in parallel.
for (var row = 0; row < matrix.Length; row++)
{
target[row] = new T[remainingLength];
}
//Work out remaining columns (previous code assumed these could
//be different per row, this assumes all rows have the same
//contents.
var remaining = Enumerable.Range(0, rowLength).Except(sorted).ToArray();
for (int row = 0; row < matrix.Length; row++)
{
var sourceRow = matrix[row];
var targetRow = target[row];
for (int i = 0; i < targetRow.Length; i++)
{
targetRow[i] = sourceRow[remaining[i]];
}
}
return target;
}
And the faster parallel one (allocation for the parallel one is now about 90% of the total time):
public static T[][] DeleteColumnsParallel<T>(this T[][] matrix, int[] columns)
{
if (matrix.Length == 0) return matrix;
//Previous code assumed matrix could be jagged - new code assumes all columns
//present and all rows same length
var rowLength = matrix[0].Length;
if (rowLength == 0) return matrix;
var sorted = columns.Distinct().ToArray();
var target = new T[matrix.Length][];
var remainingLength = rowLength - sorted.Length;
//Allocate the targets all in one go - to avoid doing allocation in parallel.
for (var row = 0; row < matrix.Length; row++)
{
target[row] = new T[remainingLength];
}
//Work out remaining columns (previous code assumed these could
//be different per row, this assumes all rows have the same
//contents.
var remaining = Enumerable.Range(0, rowLength).Except(sorted).ToArray();
Parallel.For(0, matrix.Length, row =>
{
var sourceRow = matrix[row];
var targetRow = target[row];
for (int i = 0; i < targetRow.Length; i++)
{
targetRow[i] = sourceRow[remaining[i]];
}
});
return target;
}
Results for 10000x10000 matrix with half the columns randomly removed.
My previous attempt: 1300ms
Best previous attempt: 450ms
My new serial version: 390ms
My new parallel version: 310ms
Time just to allocate the result matrix (i.e. lower bound on best achievable time): 265ms
But, I think it's important to call out a far, far faster solution. At the moment, in the fastest parallel solution, 90% of the time is spent allocating memory. If, on the other hand, you were to make a Matrix class that had it's own indexer, you would be able to dynamically pretend that certain columns of the underlying data structure didn't exist. Depending on how you're using the matrices, versus how often you're masking rows or columns, this could be dramatically faster.

How to convert Multidimensional int Array to a String List?

MultiDimensional 2D int array to string List
I would like to convert my array2D:
int[,] array2D = new int[,] { { 1, 2 }, { 3, 4 }, { 5, 6 }, { 7, 8 } };
to a List:
List<String> numbers = new List<string>(array2d.ToString());

You can flatten a multi-dimensional array with Enumerable.Cast
List<String> number = array2D.Cast<int>().Select(i => i.ToString()).ToList();

You can convert 2d array into jagged array and then convert it to List.
int[,] arr = new int[,] { { 1, 2, 3 }, { 4, 5, 6 } };
int[][] jagged = new int[arr.GetLength(0)][];
for (int i = 0; i < arr.GetLength(0); i++)
{
jagged[i] = new int[arr.GetLength(1)];
for (int j = 0; j < arr.GetLength(1); j++)
{
jagged[i][j] = arr[i, j];
}
}
List<int[]> list = jagged.ToList();

LINQ and Array.Copy

I have the following:
var list = new List<double[]>();
list.Add(new double[] { 300, 12, 22 });
list.Add(new double[] { 310, 13, 23 });
list.Add(new double[] { 320, 14, 24 });
list.Add(new double[] { 330, 15, 25 });
I would like to get from this a multidimensional array containing the first 2 columns:
double[,] a = { {300,12}, {310,13}, {320,14}, {330,15}}
Can I do this using linq? And how?

Try the following
var a = list.Select(x => new [] { x[0], x[1] }).ToArray();
EDIT
Didn't realize at first the intent was to get a non-jagged 2d array out of the source. Unfortunately there isn't really a way to do that with the standard LINQ methods. They deal mostly in terms of IEnumerable<T> and T[]. However there is nothing stopping you from creating a new method which does this
public static T[,] ToMultidimensionArray<T>(this List<T[]> list, int columns)
{
var array = new T[list.Count, columns];
for (int i = 0; i < list.Count; i++)
{
var source = list[i];
for (int j = 0; j < columns; j++)
{
array[i, j] = source[j];
}
}
return array;
}
Now you can convert the original list with a simple query
var a = list.ToMultidimensionArray(2);

LINQ and multi-dimensional arrays do not mix well. Array.Copy is not applicable here as well.
Use a traditional for loop:
double[,] result = new double[list.Count, 2];
for (int i = 0; i < list.Count; i++)
{
result[i, 0] = list[i][0];
result[i, 1] = list[i][1];
}

public static T[,] GetColumns<T>(IList<IEnumerable<T>> source, int numColumns)
{
T[,] output = new T[source.Count, numColumns];
for (int i = 0; i < source.Count; i++)
{
int j = 0;
foreach (T item in source[j].Take(numColumns))
{
output[i, j] = item;
j++;
}
}
return output;
}
Note that in this case if any of the lists contain an array that doesn't have enough values to fill all of the columns you want then they'll be left with default values, it won't throw an exception. You'll need to check for it and throw one yourself if you want that to happen.

var array = list.Select(item => item.Take(2).ToArray()).ToArray();
foreach (var item in array)
{
Console.WriteLine("{0}, {1}", item[0], item[1]);
}

matrix to array c#

Which would be the most efficient way to convert a squared matrix like
1 2 3
4 5 6
7 8 9
into
[1 2 3 4 5 6 7 8 9]
in c#
I was doing
int[,] array2D = new int[,] { { 1, 2, 3 }, { 4, 5, 6 }, { 7, 8, 9 } };
int[] array1D = new int[9];
int ci=0;
for (int i = 0; i < 3; i++)
{
for (int j = 0; j < 3; j++)
{
array1D[ci++] = array2D[i, j]);
}
}

LINQ makes this trivial.
int[,] array2d = ...;
var array1d = array2d.Cast<int>().ToArray();
Otherwise, your way is adequate but could be generalized:
int[,] array2d = ...;
var rows = array2d.GetLength(0);
var cols = array2d.GetLength(1);
var array1d = new int[rows * cols];
var current = 0;
for (int i = 0; i < rows; i++)
{
for (int j = 0; j < cols; j++)
{
array1d[current++] = array2d[i, j];
}
}
Or even:
int[,] array2d = ...;
var array1d = new int[array2d.GetLength(0) * array2d.GetLength(1)];
var current = 0;
foreach (var value in array2d)
{
array1d[current++] = value;
}

As Jeff said, LINQ makes this trivial. OfType<>() should generally be a little faster than Cast<> though:
array1D = array2D.OfType<int>().ToArray();
The implementation of OfType<> however will still suffer from boxing/unboxing penalties, as #phoog mentioned.
Just for the fun of it, if you want a fast LINQ-based solution (avoiding the cost of boxing) you could use this small extension method:
static class LinqEx
{
public static IEnumerable<T> Flatten<T>(this T[,] matrix)
{
foreach (var item in matrix) yield return item;
}
}
Or this, based on Jeff's 2nd solution:
public static IEnumerable<T> Flatten<T>(this T[,] matrix)
{
var rows = matrix.GetLength(0);
var cols = matrix.GetLength(1);
for (var i = 0; i < rows;i++ )
{
for (var j = 0; j < cols; j++ )
yield return matrix[i, j];
}
}
usage:
int[,] array2D = new int[,] { { 1, 2, 3 }, { 4, 5, 6 }, { 7, 8, 9 } };
int[] array1D = array2D.Flatten().ToArray();
I didn't fully profile this but I expect this will get you much better performance than the built-in options based on LINQ/IEnumerable. Jeff's second solution will however always be the fasted, it seems.

Alternate solution using Buffer.BlockCopy:
int[,] array2D = new int[,] { { 1, 2, 3 }, { 4, 5, 6 }, { 7, 8, 9 } };
int[] array1D = new int[ array2D.Length ];
Buffer.BlockCopy(array2D, 0, array1D, 0, array1D.Length * sizeof(int));

You're always better off allocating the complete result array in one hit, then copying the data in.
You should find the total size like this;
var size = arrays.Sum(a=> a.Length);
var result = new int[size];
And then copy the arrays using Array.CopyTo, instead of looping yourself;
var cursor = 0;
foreach(var a in arrays) {
a.CopyTo(result, cursor);
cursor += a.Length;
}
Array.CopyTo will be faster than your own loop; at least, not slower. It will probably use C's memcpy function internally to do a low-level block copy. This is as efficient as you can be.

C#, rotating 2D arrays

I've looked at other post on rotating 2D arrays, but it's not quite what I want.
I want something like this
int[,] original= new int[4,2]
{
{1,2},
{5,6},
{9,10},
{13,14}
};
I want to turn it like this,
rotatedArray = { {1,5,9,13}, {2,6,10,14}};
I want to do some analysis by column, as opposed to by rows.
This works, but is there an easier way??
private static int[,] RotateArray(int[,] myArray)
{
int org_rows = myArray.GetLength(0);
int org_cols = myArray.GetLength(1);
int[,] myRotate = new int[org_cols, org_rows];
for (int i = 0; i < org_rows; i++)
{
for(int j = 0; j < org_cols; j++)
{
myRotate[j, i] = myArray[i, j];
}
}
return myRotate;
}
Is there an easy way to iterate through columns in c#?
B

If you change your array to be an array of arrays it gets easier. I found this if you change it to an int[][]:
int[][] original = new[]
{
new int[] {1, 2},
new int[] {5, 6},
new int[] {9, 10},
new int[] {13, 14}
};
and the rotate method:
private static int[][] Rotate(int[][] input)
{
int length = input[0].Length;
int[][] retVal = new int[length][];
for(int x = 0; x < length; x++)
{
retVal[x] = input.Select(p => p[x]).ToArray();
}
return retVal;
}