We Keep Coding

Merge first list with second list based on standard deviation of second list C#

Given 2 datasets (which are both a sequence of standard deviations away from a number, we are looking for the overlapping sections):
var list1 = new decimal[] { 357.06, 366.88, 376.70, 386.52, 406.15 };
var list2 = new decimal[] { 370.51, 375.62, 380.72, 385.82, 390.93 };
I would like to perform a merge with items from List2 being placed closest to items of List1, within a certain range, i.e. merge List2 element within 5.10 (standard deviation) of List1 element:
357.06
366.88 => 370.51
376.70 => 375.52, 380.72
386.52 => 390.93
406.15
The idea is to cluster values from List2 and count them, in this case element with value 376.70 would have the highest significance as it has 2 close neighbors of 375.52 and 380.72 (where as 366.88 and 386.52 have only 1 match, and the remaining none within range).
Which C# math/stats libraries could be used for this (or would there be a better way to combine statistically)?
If this is more of a computer science or stats question apologies in advance will close and reopen on relevant SO site.

Assuming that list2 is sorted (if not, put Array.Sort(list2);) you can try Binary Search:
Given:
var list1 = new decimal[] { 357.06m, 366.88m, 376.70m, 386.52m, 406.15m };
var list2 = new decimal[] { 370.51m, 375.62m, 380.72m, 385.82m, 390.93m };
decimal sd = 5.10m;
Code:
// Array.Sort(list2); // Uncomment, if list2 is not sorted
List<(decimal value, decimal[] list)> result = new List<(decimal value, decimal[] list)>();
foreach (decimal value in list1) {
int leftIndex = Array.BinarySearch<decimal>(list2, value - sd);
if (leftIndex < 0)
leftIndex = -leftIndex - 1;
else // edge case
for (; leftIndex >= 1 && list1[leftIndex - 1] == value - sd; --leftIndex) ;
int rightIndex = Array.BinarySearch<decimal>(list2, value + sd);
if (rightIndex < 0)
rightIndex = -rightIndex - 1;
else // edge case
for (; rightIndex < list1.Length - 1 && list1[rightIndex + 1] == value + sd; ++rightIndex) ;
result.Add((value, list2.Skip(leftIndex).Take(rightIndex - leftIndex).ToArray()));
}
Let's have a look:
string report = string.Join(Environment.NewLine, result
.Select(item => $"{item.value} => [{string.Join(", ", item.list)}]"));
Console.Write(report);
Outcome:
357.06 => []
366.88 => [370.51]
376.70 => [375.62, 380.72]
386.52 => [385.82, 390.93]
406.15 => []

Something like this should work
var list1 = new double[] { 357.06, 366.88, 376.70, 386.52, 406.15 };
var list2 = new double[] { 370.51, 375.62, 380.72, 385.82, 390.93 };
double dev = 5.1;
var result = new Dictionary<double, List<double>>();
foreach (var l in list2) {
var diffs = list1.Select(r => new { diff = Math.Abs(r - l), r })
.Where(d => d.diff <= dev)
.MinBy(r => r.diff)
.FirstOrDefault();
if (diffs == null) {
continue;
}
List<double> list;
if (! result.TryGetValue(diffs.r, out list)) {
list = new List<double>();
result.Add(diffs.r, list);
}
list.Add(l);
}
It uses MinBy from MoreLinq, but it is easy to modify to work without it.

In fact, you don't need extra libs or something else. You can use just LINQ for this.
internal class Program
{
private static void Main(string[] args)
{
var deviation = 5.1M;
var list1 = new decimal[] { 357.06M, 366.88M, 376.70M, 386.52M, 406.15M };
var list2 = new decimal[] { 370.51M, 375.62M, 380.72M, 385.82M, 390.93M };
var result = GetDistribution(list1.ToList(), list2.ToList(), deviation);
result.ForEach(x => Console.WriteLine($"{x.BaseValue} => {string.Join(", ", x.Destribution)} [{x.Weight}]"));
Console.ReadLine();
}
private static List<Distribution> GetDistribution(List<decimal> baseList, List<decimal> distrebutedList, decimal deviation)
{
return baseList.Select(x =>
new Distribution
{
BaseValue = x,
Destribution = distrebutedList.Where(y => x - deviation < y && y < x + deviation).ToList()
}).ToList();
}
}
internal class Distribution
{
public decimal BaseValue { get; set; }
public List<decimal> Destribution { get; set; }
public int Weight => Destribution.Count;
}
I hope it was useful for you.

Permutations with repetition

If you can find a better title, please edit.
I will start by saying I've looked at several q&a's on this topic, mainly this one and this article without having found a way to do this:
Given the word "HALLOWEEN" I would like to find all permutations and combinations for all lengths. The first thing I tried was iterating through the below code giving it length of 1 to begin with and continuing until reaching the length of the word (9).
public static IEnumerable<IEnumerable<T>>
GetPermutations<T>(IEnumerable<T> list, int length)
{
if (length == 1) return list.Select(t => new T[] {t});
return GetPermutations(list, length - 1)
.SelectMany(t => list.Where(e => !t.Contains(e)),
(t1, t2) => t1.Concat(new T[] {t2}));
}
This gave me unexpected results as the double 'E' and 'L's were omitted, leaving the final set short.
A simpler example could be 'MOM' {M,O,M} where the final set of outcomes would be:
M
O
MO
OM
MM
MOM
MMO
OMM
Notice that I want to see both 'M's as available, but I don't want to see "MMM" as a result. "MOM" would appear twice in the result due to leaving original order (1,2,3) and swapping positions 1 and 3 (3,2,1) would both result in 'M','O','M' but this character sequence only appears once is the result list (which can be done by a string comparison)
Again, with set {1,1,2,3} I would expect to see:
{1,1}
but NOT {2,2} or {3,3}

Here's another solution that should be clear and easily understandable:
public static IEnumerable<string> GetPermutations(string input)
{
if (string.IsNullOrEmpty(input))
{
return new List<string>();
}
var length = input.Length;
var indices = Enumerable.Range(0, length).ToList();
var permutationsOfIndices = GetNumericalPermutations(indices, length);
var permutationsOfInput = permutationsOfIndices.Select(x => new string(x.Select(y => input[y]).ToArray()))
.Distinct();
return permutationsOfInput;
}
private static List<List<int>> GetNumericalPermutations(List<int> values, int maxLength)
{
if (maxLength == 1)
{
return values.Select(x => new List<int>{x}).ToList();
}
else
{
var permutations = GetNumericalPermutations(values, maxLength - 1);
foreach (var index in values)
{
var newPermutations = permutations.Where(x => !x.Contains(index))
.Select(x => x.Concat(new List<int> { index }))
.Where(x => !permutations.Any(y => y.SequenceEqual(x)))
.Select(x => x.ToList())
.ToList();
permutations.AddRange(newPermutations);
}
return permutations;
}
}
For example, the output for "MOM" is:
M
O
OM
MM
MO
MMO
OMM
MOM

I suggest looking at the permutations of the letter positions 0,1,2,3,4,etc mapping those to letters, and then eliminating the duplicates.
Without changing the GetPermutations function, I added another function to get the permutations of the letter positions, map those result to character strings and then eliminate the duplicates.
public void PermutationsTestMethod()
{
GetPermutationsOfString("MOM").ForEach(v => Debug.Print(v));
}
public List<string> GetPermutationsOfString(string value)
{
var resultList = new List<string>();
for (var i = 1; i <= value.Length; i++)
{
var permutations = GetPermutations(Enumerable.Range(0, value.Length), i);
resultList.AddRange(
permutations
.Select(v => new string(v.Select(z => value[z]).ToArray()))
.Distinct()
);
}
return resultList;
}
public static IEnumerable<IEnumerable<T>> GetPermutations<T>(IEnumerable<T> list, int length)
{
if (length == 1) return list.Select(t => new T[] { t });
return GetPermutations(list, length - 1)
.SelectMany(t => list.Where(e => !t.Contains(e)),
(t1, t2) => t1.Concat(new T[] { t2 }));
}

This works fine:
Func<string, IEnumerable<string>> getAllSubsets = null;
getAllSubsets = x =>
(x == null || !x.Any())
? Enumerable.Empty<string>()
: (x.Length > 1
? getAllSubsets(x.Substring(1))
.SelectMany(y => new [] { y, x.Substring(0, 1) + y })
: new [] { "", x.Substring(0, 1) });
So given getAllSubsets("ABC") I get:
"", "A", "B", "AB", "C", "AC", "BC", "ABC"
And, for your "MOM" example I get:
"", "M", "O", "MO", "M", "MM", "OM", "MOM"
It would be trivial to filter out the empty string and duplicate values if need be, but as it stands it strictly produces all subsets.

I think it is generally better try to avoid generating and eliminating permutations. Text like "aaaaaaaaaaaaaaab" can generate really big amount of duplications.
public static IEnumerable<IEnumerable<T>>
GetPermutationsInner<T>(IEnumerable<IGrouping<T, T>> groupedList, int length)
{
if (length == 1) return groupedList.Select(t => new T[] { t.Key });
return GetPermutationsInner<T>(groupedList, length - 1)
.SelectMany(t => groupedList
.Where(e => t.Count(w => w.Equals(e.Key)) < e.Count())
.Select(s => s.Key),
(t1, t2) => t1.Concat(new T[] { t2 }));
}
public static IEnumerable<IEnumerable<T>>
GetPermutations<T>(IEnumerable<T> list)
{
var resultList = new List<IEnumerable<T>>();
for (int i = 1; i <= list.Count(); ++i)
{
resultList.AddRange(GetPermutationsInner<T>(list.GroupBy(g => g), i));
}
return resultList;
}

Powerset with upper and lower limits

I have a process where I need to find all of the best combinations that fit a certain criteria. Powersets are what I think I am looking for, but i'd like to build in some filters in to avoid millions of results.
For example I have the following Object:
public class CableReel
{
public string ReelId { get; set; }
public int Length { get; set; }
}
I'd like to get a powerset of the above object where the combined lengths meet a lower limit and also an upper limit as well.
The following list:
"a", 5200
"b", 2500
"c", 1000
I'd like to put in the following limits, min 3000, max 5000. I'd like to then only see the BC combination.
I have found the following code for returning generic powersets:
public IEnumerable<IEnumerable<T>> GetPowerSet<T>(List<T> list)
{
return from m in Enumerable.Range(0, 1 << list.Count)
select
from i in Enumerable.Range(0, list.Count)
where (m & (1 << i)) != 0
select list[i];
}
I was just wondering how this might be adapted to accomplish what I am looking to do?
Solution:
Using Alex Sikilinda's advice I came up with:
var ls = new List<CableReel>
{
new CableReel{Length = 5,ReelId = "A"},
new CableReel{Length = 3,ReelId = "B"},
new CableReel{Length = 2,ReelId = "C"},
new CableReel{Length = 7,ReelId = "e"},
new CableReel{Length = 3,ReelId = "f"},
new CableReel{Length = 2,ReelId = "g"}
};
var p = GetPowerSet(ls).Where(psl => psl.Sum(ps => ps.Length) > 5 && psl.Sum(ps => ps.Length) < 8);

Supposing your powerset code works fine, you can just filter the list before calling your method:
var filteredList = list.Where(e => e > 4000 || e < 5000);
var result = GetPowerSet(filteredList.ToList());
Edited:
because you want to filter not elements, but sets, you can filter the result.
This code demonstrates how you can filters sets, sum of which are bigger than 2:
public static IEnumerable<IEnumerable<T>> GetPowerSet<T>(List<T> list, Func<IEnumerable<T>, Boolean> filter)
{
var fullPowerSet = from m in Enumerable.Range(0, 1 << list.Count)
select
from i in Enumerable.Range(0, list.Count)
where (m & (1 << i)) != 0
select list[i];
return fullPowerSet.Where(e => filter(e));
}
static void Main(String[] args)
{
List<Int32> inputList = new List<Int32>()
{
1,2
};
var result = GetPowerSet(inputList, ps => ps.Sum() > 1);
}
Now you can change the delegate ps => ps.Sum() > 1 to get what you want.

LINQ swap columns into rows

Is there a fancy LINQ expression that could allow me to do the following in a much more simpler fashion. I have a List<List<double>>, assuming the List are columns in a 2d matrix, I want to swap the list of columns into a list of rows. I have the following obvious solution:
int columns = 5;
var values; // assume initialised as List<List<double>>()
var listOfRows = new List<List<double>>();
for (int i = 0; i < columns ; i++)
{
List<double> newRow = new List<double>();
foreach (List<double> value in values)
{
newRow.Add(value[i]);
}
listOfRows.Add(newRow);
}

You could LINQify the inner loop pretty easily:
vector.AddRange(values.Select(value => value[i]));
Whether or not that improves the readability is left entirely up to you!

Here's a Linq expression that would do what you want - looking at it I'd personally stick with the nested foreach loops though - much easier to read:
var columnList= new List<List<double>>();
columnList.Add(new List<double>() { 1, 2, 3 });
columnList.Add(new List<double>() { 4, 5, 6 });
columnList.Add(new List<double>() { 7, 8, 9 });
columnList.Add(new List<double>() { 10, 11, 12 });
int columnCount = columnList[0].Count;
var rowList = columnList.SelectMany(x => x)
.Select((x, i) => new { V = x, Index = i })
.GroupBy(x => (x.Index + 1) % columnCount)
.Select(g => g.Select( x=> x.V).ToList())
.ToList();
This example also would only work on a matrix with a fixed column count. Basically it's flattening the matrix into a list, then creating the list of rows by grouping by the index of the element in the list modulo the column count.
Edit:
A different approach, much closer to a nested loop and probably similar performance besides the overhead.
int columnCount = columnList[0].Count;
int rowCount = columnList.Count;
var rowList = Enumerable.Range(0, columnCount)
.Select( x => Enumerable.Range(0, rowCount)
.Select(y => columnList[y][x])
.ToList())
.ToList();

var inverted = Enumerable.Range(0, columnCount)
.Select(index => columnList.Select(list => list[index]));
In short, we enumerate the column index from a range and use it to collect the nth element of each list.
Please note that you'll need to check that every list has the same number of columns.

Here's one that works for rectangular (non-ragged) matrices. The C# code here works cut-and-paste into LinqPad, a free, interactive C# programming tool.
I define a postfix operator (that is, an extension method) "Transpose." Use the operator as follows:
var rand = new Random();
var xss = new [] {
new [] {rand.NextDouble(), rand.NextDouble()},
new [] {rand.NextDouble(), rand.NextDouble()},
new [] {rand.NextDouble(), rand.NextDouble()},
};
xss.Dump("Original");
xss.Transpose().Dump("Transpose");
resulting in something like this:
Original
0.843094345109116
0.981432441613373
0.649207864724662
0.00594645645746331
0.378864820291691
0.336915332515219
Transpose
0.843094345109116
0.649207864724662
0.378864820291691
0.981432441613373
0.00594645645746331
0.336915332515219
The gist of the implementation of this operator is the following
public static IEnumerable<IEnumerable<T>> Transpose<T>(this IEnumerable<IEnumerable<T>> xss)
{
var heads = xss.Heads();
var tails = xss.Tails();
var empt = new List<IEnumerable<T>>();
if (heads.IsEmpty())
return empt;
empt.Add(heads);
return empt.Concat(tails.Transpose());
}
Here is the full implementation, with some lines commented out that you can uncomment to monitor how the function works.
void Main()
{
var rand = new Random();
var xss = new [] {
new [] {rand.NextDouble(), rand.NextDouble()},
new [] {rand.NextDouble(), rand.NextDouble()},
new [] {rand.NextDouble(), rand.NextDouble()},
};
xss.Dump("Original");
xss.Transpose().Dump("Transpose");
}
public static class Extensions
{
public static IEnumerable<T> Heads<T>(this IEnumerable<IEnumerable<T>> xss)
{
Debug.Assert(xss != null);
if (xss.Any(xs => xs.IsEmpty()))
return new List<T>();
return xss.Select(xs => xs.First());
}
public static bool IsEmpty<T>(this IEnumerable<T> xs)
{
return xs.Count() == 0;
}
public static IEnumerable<IEnumerable<T>> Tails<T>(this IEnumerable<IEnumerable<T>> xss)
{
return xss.Select(xs => xs.Skip(1));
}
public static IEnumerable<IEnumerable<T>> Transpose<T>(this IEnumerable<IEnumerable<T>> xss)
{
// xss.Dump("xss in Transpose");
var heads = xss.Heads()
// .Dump("heads in Transpose")
;
var tails = xss.Tails()
// .Dump("tails in Transpose")
;
var empt = new List<IEnumerable<T>>();
if (heads.IsEmpty())
return empt;
empt.Add(heads);
return empt.Concat(tails.Transpose())
// .Dump("empt")
;
}
}

I am combining some of the answers above, which sometimes had columns and rows inverted form the original answer or from the convention I am used to : row refers to the first index and column to the inner ( second) index. e.g. values[row][column]
public static List<List<T>> Transpose<T>(this List<List<T>> values)
{
if (values.Count == 0 || values[0].Count == 0)
{
return new List<List<T>>();
}
int ColumnCount = values[0].Count;
var listByColumns = new List<List<T>>();
foreach (int columnIndex in Enumerable.Range(0, ColumnCount))
{
List<T> valuesByColumn = values.Select(value => value[columnIndex]).ToList();
listByColumns.Add(valuesByColumn);
}
return listByColumns;
}
Actually the word row and column is just our convention of thinking about the data in rows and columns , and sometimes adds more confusion than solving them.
We are actually just swapping the inner index for the outer index. (or flipping the indexes around). So one could also just define the following extension method. . Again I borrowed from above solutions, just put it into something I find readable and fairly compact.
Checks that the inner lists are of equal sized are required.
public static List<List<T>> InsideOutFlip<T>(this List<List<T>> values)
{
if (values.Count == 0 || values[0].Count == 0)
{
return new List<List<T>>();
}
int innerCount = values[0].Count;
var flippedList = new List<List<T>>();
foreach (int innerIndex in Enumerable.Range(0, innerCount))
{
List<T> valuesByOneInner = values.Select(value => value[innerIndex]).ToList();
flippedList.Add(valuesByOneInner);
}
return flippedList;
}

C#: Altering values for every item in an array

I'm wondering if there is built-in .NET functionality to change each value in an array based on the result of a provided delegate. For example, if I had an array {1,2,3} and a delegate that returns the square of each value, I would like to be able to run a method that takes the array and delegate, and returns {1,4,9}. Does anything like this exist already?

LINQ provides support for projections using the Select extension method:
var numbers = new[] {1, 2, 3};
var squares = numbers.Select(i => i*i).ToArray();
You can also use the slightly less fluent Array.ConvertAll method:
var squares = Array.ConvertAll(numbers, i => i*i);
Jagged arrays can be processed by nesting the projections:
var numbers = new[] {new[] {1, 2}, new[] {3, 4}};
var squares = numbers.Select(i => i.Select(j => j*j).ToArray()).ToArray();
Multidimensional arrays are a little more complex. I've written the following extension method which projects every element in a multidimensional array no matter what its rank.
static Array ConvertAll<TSource, TResult>(this Array source,
Converter<TSource, TResult> projection)
{
if (!typeof (TSource).IsAssignableFrom(source.GetType().GetElementType()))
{
throw new ArgumentException();
}
var dims = Enumerable.Range(0, source.Rank)
.Select(dim => new {lower = source.GetLowerBound(dim),
upper = source.GetUpperBound(dim)});
var result = Array.CreateInstance(typeof (TResult),
dims.Select(dim => 1 + dim.upper - dim.lower).ToArray(),
dims.Select(dim => dim.lower).ToArray());
var indices = dims
.Select(dim => Enumerable.Range(dim.lower, 1 + dim.upper - dim.lower))
.Aggregate(
(IEnumerable<IEnumerable<int>>) null,
(total, current) => total != null
? total.SelectMany(
item => current,
(existing, item) => existing.Concat(new[] {item}))
: current.Select(item => (IEnumerable<int>) new[] {item}))
.Select(index => index.ToArray());
foreach (var index in indices)
{
var value = (TSource) source.GetValue(index);
result.SetValue(projection(value), index);
}
return result;
}
The above method can be tested with an array of rank 3 as follows:
var source = new int[2,3,4];
for (var i = source.GetLowerBound(0); i <= source.GetUpperBound(0); i++)
for (var j = source.GetLowerBound(1); j <= source.GetUpperBound(1); j++)
for (var k = source.GetLowerBound(2); k <= source.GetUpperBound(2); k++)
source[i, j, k] = i*100 + j*10 + k;
var result = (int[,,]) source.ConvertAll<int, int>(i => i*i);
for (var i = source.GetLowerBound(0); i <= source.GetUpperBound(0); i++)
for (var j = source.GetLowerBound(1); j <= source.GetUpperBound(1); j++)
for (var k = source.GetLowerBound(2); k <= source.GetUpperBound(2); k++)
{
var value = source[i, j, k];
Debug.Assert(result[i, j, k] == value*value);
}

Not that I'm aware of (replacing each element rather than converting to a new array or sequence), but it's incredibly easy to write:
public static void ConvertInPlace<T>(this IList<T> source, Func<T, T> projection)
{
for (int i = 0; i < source.Count; i++)
{
source[i] = projection(source[i]);
}
}
Use:
int[] values = { 1, 2, 3 };
values.ConvertInPlace(x => x * x);
Of course if you don't really need to change the existing array, the other answers posted using Select would be more functional. Or the existing ConvertAll method from .NET 2:
int[] values = { 1, 2, 3 };
values = Array.ConvertAll(values, x => x * x);
This is all assuming a single-dimensional array. If you want to include rectangular arrays, it gets trickier, particularly if you want to avoid boxing.

Using System.Linq you could do something like:
var newArray = arr.Select(x => myMethod(x)).ToArray();

LINQ queries could easily solve this for you - make sure you're referencing System.Core.dll and have a
using System.Linq;
statement. For example, if you had your array in a variable named numberArray, the following code would give you exactly what you're looking for:
var squares = numberArray.Select(n => n * n).ToArray();
The final "ToArray" call is only needed if you actually need an array, and not an IEnumerable<int>.

you can use linq to accomplish this in shorthand but be careful remember that a foreach occurs underneath anyway.
int[] x = {1,2,3};
x = x.Select(( Y ) => { return Y * Y; }).ToArray();

Here is another solution for M x N arrays, where M and N are not known at compile time.
// credit: https://blogs.msdn.microsoft.com/ericlippert/2010/06/28/computing-a-cartesian-product-with-linq/
public static IEnumerable<IEnumerable<T>> CartesianProduct<T>(IEnumerable<IEnumerable<T>> sequences)
{
IEnumerable<IEnumerable<T>> result = new[] { Enumerable.Empty<T>() };
foreach (var sequence in sequences)
{
// got a warning about different compiler behavior
// accessing sequence in a closure
var s = sequence;
result = result.SelectMany(seq => s, (seq, item) => seq.Concat<T>(new[] { item }));
}
return result;
}
public static void ConvertInPlace(this Array array, Func<object, object> projection)
{
if (array == null)
{
return;
}
// build up the range for each dimension
var dimensions = Enumerable.Range(0, array.Rank).Select(r => Enumerable.Range(0, array.GetLength(r)));
// build up a list of all possible indices
var indexes = EnumerableHelper.CartesianProduct(dimensions).ToArray();
foreach (var index in indexes)
{
var currentIndex = index.ToArray();
array.SetValue(projection(array.GetValue(currentIndex)), currentIndex);
}
}