We Keep Coding

Merge first list with second list based on standard deviation of second list C#

Given 2 datasets (which are both a sequence of standard deviations away from a number, we are looking for the overlapping sections):
var list1 = new decimal[] { 357.06, 366.88, 376.70, 386.52, 406.15 };
var list2 = new decimal[] { 370.51, 375.62, 380.72, 385.82, 390.93 };
I would like to perform a merge with items from List2 being placed closest to items of List1, within a certain range, i.e. merge List2 element within 5.10 (standard deviation) of List1 element:
357.06
366.88 => 370.51
376.70 => 375.52, 380.72
386.52 => 390.93
406.15
The idea is to cluster values from List2 and count them, in this case element with value 376.70 would have the highest significance as it has 2 close neighbors of 375.52 and 380.72 (where as 366.88 and 386.52 have only 1 match, and the remaining none within range).
Which C# math/stats libraries could be used for this (or would there be a better way to combine statistically)?
If this is more of a computer science or stats question apologies in advance will close and reopen on relevant SO site.

Assuming that list2 is sorted (if not, put Array.Sort(list2);) you can try Binary Search:
Given:
var list1 = new decimal[] { 357.06m, 366.88m, 376.70m, 386.52m, 406.15m };
var list2 = new decimal[] { 370.51m, 375.62m, 380.72m, 385.82m, 390.93m };
decimal sd = 5.10m;
Code:
// Array.Sort(list2); // Uncomment, if list2 is not sorted
List<(decimal value, decimal[] list)> result = new List<(decimal value, decimal[] list)>();
foreach (decimal value in list1) {
int leftIndex = Array.BinarySearch<decimal>(list2, value - sd);
if (leftIndex < 0)
leftIndex = -leftIndex - 1;
else // edge case
for (; leftIndex >= 1 && list1[leftIndex - 1] == value - sd; --leftIndex) ;
int rightIndex = Array.BinarySearch<decimal>(list2, value + sd);
if (rightIndex < 0)
rightIndex = -rightIndex - 1;
else // edge case
for (; rightIndex < list1.Length - 1 && list1[rightIndex + 1] == value + sd; ++rightIndex) ;
result.Add((value, list2.Skip(leftIndex).Take(rightIndex - leftIndex).ToArray()));
}
Let's have a look:
string report = string.Join(Environment.NewLine, result
.Select(item => $"{item.value} => [{string.Join(", ", item.list)}]"));
Console.Write(report);
Outcome:
357.06 => []
366.88 => [370.51]
376.70 => [375.62, 380.72]
386.52 => [385.82, 390.93]
406.15 => []

Something like this should work
var list1 = new double[] { 357.06, 366.88, 376.70, 386.52, 406.15 };
var list2 = new double[] { 370.51, 375.62, 380.72, 385.82, 390.93 };
double dev = 5.1;
var result = new Dictionary<double, List<double>>();
foreach (var l in list2) {
var diffs = list1.Select(r => new { diff = Math.Abs(r - l), r })
.Where(d => d.diff <= dev)
.MinBy(r => r.diff)
.FirstOrDefault();
if (diffs == null) {
continue;
}
List<double> list;
if (! result.TryGetValue(diffs.r, out list)) {
list = new List<double>();
result.Add(diffs.r, list);
}
list.Add(l);
}
It uses MinBy from MoreLinq, but it is easy to modify to work without it.

In fact, you don't need extra libs or something else. You can use just LINQ for this.
internal class Program
{
private static void Main(string[] args)
{
var deviation = 5.1M;
var list1 = new decimal[] { 357.06M, 366.88M, 376.70M, 386.52M, 406.15M };
var list2 = new decimal[] { 370.51M, 375.62M, 380.72M, 385.82M, 390.93M };
var result = GetDistribution(list1.ToList(), list2.ToList(), deviation);
result.ForEach(x => Console.WriteLine($"{x.BaseValue} => {string.Join(", ", x.Destribution)} [{x.Weight}]"));
Console.ReadLine();
}
private static List<Distribution> GetDistribution(List<decimal> baseList, List<decimal> distrebutedList, decimal deviation)
{
return baseList.Select(x =>
new Distribution
{
BaseValue = x,
Destribution = distrebutedList.Where(y => x - deviation < y && y < x + deviation).ToList()
}).ToList();
}
}
internal class Distribution
{
public decimal BaseValue { get; set; }
public List<decimal> Destribution { get; set; }
public int Weight => Destribution.Count;
}
I hope it was useful for you.

generate all possibilities for entities in 2 collections

I'm simplifying my blind spot to the following
assume you have 4 entities a, b, c and d; 2 collections x and y; possibilities could be for
(x;y) => (abcd;)(abc;d)(abd;c)(acd;b)(bcd;a)(ac;bd)(ad;bc)(ab;dc)(bc;ad)etc..
in short, i need to generate all possibilities for n entities in m collections in an efficient way
(bit more about the domain, i don't care for ordering, so (ab;cd) is essentially the same as (ba;cd) for my case use, if that's going to make it easier for you)

Very closely related is an answer given here, which deals with generation of all subsets. All subsets of a given sequence can be generated by tho following snippet, which is taken from there.
static IEnumerable<T[]> GetSubsets<T>(T[] set) {
bool[] state = new bool[set.Length+1];
for (int x; !state[set.Length]; state[x] = true ) {
yield return Enumerable.Range(0, state.Length)
.Where(i => state[i])
.Select(i => set[i])
.ToArray();
for (x = 0; state[x]; state[x++] = false);
}
}
Based on enumeration of all subsets,the desired solution can be ganerated by determining the complement as follows.
public class Partition
{
public IEnumerable<string> First;
public IEnemurable<string> Second;
};
var input = new string[]{ "a", "b", "c", "d" };
var subsets = Getsubsets(input);
var Partitions = new List<Partition>();
foreach (var subset in subsets)
{
var iPart = new Partition();
iPart.First = subset;
iPart.Second = input.Where(iEl => false == subset.Contains(iEl));
Partitions.Add(iPart);
}

Here is an implementation that uses as state a BigInteger instead of a bool[] or a BitArray:
using System.Numerics;
public static IEnumerable<(T[], T[])> GetCollectionPairs<T>(T[] source)
{
BigInteger combinations = BigInteger.One << source.Length;
for (BigInteger i = 0; i < combinations; i++)
{
yield return
(
Enumerable.Range(0, source.Length)
.Where(j => (i & (BigInteger.One << j)) != 0)
.Select(j => source[j])
.ToArray(),
Enumerable.Range(0, source.Length)
.Where(j => (i & (BigInteger.One << j)) == 0)
.Select(j => source[j])
.ToArray()
);
}
}
Usage example:
var items = new string[] { "A", "B", "C", "D" };
var pairs = GetCollectionPairs(items);
foreach (var pair in pairs)
{
Console.WriteLine(
$"({String.Join("", pair.Item1)};{String.Join("", pair.Item2)})");
}
Output:
(;ABCD)
(A;BCD)
(B;ACD)
(AB;CD)
(C;ABD)
(AC;BD)
(BC;AD)
(ABC;D)
(D;ABC)
(AD;BC)
(BD;AC)
(ABD;C)
(CD;AB)
(ACD;B)
(BCD;A)
(ABCD;)
This generates (AB;CD) and (CD;AB) as different pairs. If this is not desirable, then simply loop until i < combinations / 2 instead of i < combinations.

algorithm searching through dictionary

Im working from the Q https://www.testdome.com/for-developers/solve-question/10282
Write a function that, given a list and a target sum, returns zero-based indices of any two distinct elements whose sum is equal to the target sum. If there are no such elements, the function should return null.
For example, FindTwoSum(new List<int>() { 1, 3, 5, 7, 9 }, 12) should return a Tuple<int, int> containing any of the following pairs of indices:
1 and 4 (3 + 9 = 12)
2 and 3 (5 + 7 = 12)
3 and 2 (7 + 5 = 12)
4 and 1 (9 + 3 = 12)
So far iv got:
class TwoSum
{
public static Tuple<int, int> FindTwoSum(IList<int> list, int sum)
{
//throw new NotImplementedException("Waiting to be implemented.");
IList<int> duplicateList = list;
foreach (int i in list)
{
foreach (int j in duplicateList)
{
if (i != j)
{
if (i + j == sum)
{
return Tuple.Create(i, j);
}
}
}
}
return null;
}
public static void Main(string[] args)
{
Tuple<int, int> indices = FindTwoSum(new List<int>() { 1, 3, 5, 7, 9 }, 12);
Console.WriteLine(indices.Item1 + " " + indices.Item2);
}
}
This returns the correct answer in my code but is failing 3 out of 4 cases in the quesitong because:
Example case: Wrong answer
No solution: Correct answer
One solution: Wrong answer
Performance test with a large number of elements: Wrong answer
Ive looked at the hints
Hint 1: Nested for loops can iterate over the list and calculate a sum in O(N^2) time.
Hint 2: A dictionary can be used to store pre-calculated values, this may allow a solution with O(N) complexity.
So im using nested loops but Im guessing in this instance in order to pass hint2 I need to use a dictionary...How can I refactor this into using a dictionary?
Thanks for any help!

You are not returning indexes, you are returning values. for loops are not foreach loops.
A nested for loops solution would be something like this:
for(int i=0; i<list.Count-1; i++)
{
for(int j=i+1;j<list.Count;j++)
{
if(list[i]+list[j] == sum)
{
return Tuple.Create(i, j);
}
}
}
return null;
I'll leave the dictionary solution for you to create.

Hi this one received 50%
public static Tuple<int, int> FindTwoSum(IList<int> list, int sum)
{
int n = list.Count-1;
while(n != 0)
{
for (int i = 0; i <= list.Count-1 ; i++)
{
if (list[n] + list[i] == sum)
{
return Tuple.Create(i, n);
}
}
n--;
}
return null;
}
// get list value:
var aat = (from l1 in list
from l2 in list
where l1 + l2 == 12
group new { l1, l2} by new { l1, l2 } into gp
select new {gp.Key}).ToDictionary( a => a.Key.l1, b => b.Key.l2 );
// get list index of the value:
var aav = (from l1 in list
from l2 in list
where l1 + l2 == 12
group new { l1, l2 } by new { l1, l2 } into gp
select new { gp.Key })
.ToDictionary( a => list.IndexOf(a.Key.l1), b => list.IndexOf(b.Key.l2)
);

finding all possible sum of two arrays element

I have two arrays and i am trying to get all possible sum of each element with other element of two array and index of each element
int[] width = new int[2] {10,20 };
int[] height = new int[2] {30,40 };
result should like this (value / indexes)
10 width0
10+20 width0+width1
10+30 width0+height0
10+40 width0+height1
10+20+30 width0+width1+height0
10+20+40 width0+width1+height1
10+20+30+40 width0+width1+height0+height1
And so for each element in two array
I tried using permutation but I get other output

It is more easy to get all combinations from one array than two arrays. And as we see, you need to store indices and array names along with the value of the elements in collections. So, in my opinion the best option is to combine these two arrays in one dictionary, where the key will be the value of the numbers and the value will be [ArrayName + Index of item] (f.e width0, height1 and so on....)
So, let's combine these arrays in one dictionary:
int[] width = new int[2] { 10, 20 };
int[] height = new int[2] { 30, 40 };
var widthDictionary = width.ToList().Select((number, index) => new { index, number })
.ToDictionary(key => key.number, value => string.Format("width{0}", value.index));
var heightDictionary = height.ToList().Select((number, index) => new { index, number })
.ToDictionary(key => key.number, value => string.Format("height{0}", value.index));
// And here is the final dictionary
var totalDictionary = widthDictionary.Union(heightDictionary);
Then add this method to your class: (source)
public static IEnumerable<IEnumerable<T>> GetPowerSet<T>(List<T> list)
{
return from m in Enumerable.Range(0, 1 << list.Count)
select
from i in Enumerable.Range(0, list.Count)
where (m & (1 << i)) != 0
select list[i];
}
Then send your dictionary as an argument to this method and project this collection as you want with the help of the Select() method:
var sumOfCombinations = GetPowerSet(totalDictionary.ToList())
.Where(x => x.Count() > 0)
.Select(x => new
{
Numbers = x.Select(pair => pair.Key).ToList(),
DisplayValues = x.Select(pair => pair.Value).ToList()
})
.ToList();
And at the end you can display expected result as this:
sumOfCombinations.ForEach(x =>
{
x.Numbers.ForEach(number => Console.Write("{0} ", number));
x.DisplayValues.ForEach(displayValue => Console.Write("{0} ", displayValue));
Console.WriteLine();
});
And, the result is:

This is a play off of #Farhad Jabiyev's answer.
Declares a class called IndexValuePair. and uses foreach on widthList and heightList. to populate the 'Index' property of item instance.
Note: Index is a string.
Class & Static Function
public class IndexValuePair
{
public string Index {get;set;}
public int Value {get;set;}
}
public static IEnumerable<IEnumerable<T>> GetPowerSet<T>(List<T> list)
{
return from m in Enumerable.Range(0, 1 << list.Count)
select
from i in Enumerable.Range(0, list.Count)
where (m & (1 << i)) != 0
select list[i];
}
Main (Console)
static void Main(string[] args)
{
int[] width = new int[2] { 10, 20 };
int[] height = new int[2] { 30, 40 };
var wholeList = width.Select(val => new IndexValuePair() { Index = "width", Value = val }).ToList();
var heightList = height.Select(val => new IndexValuePair() { Index = "height", Value = val }).ToList();
var iteration = 0;
wholeList.ForEach(ivp => { ivp.Index = ivp.Index + count; count = iteration + 1; });
iteration = 0;
heightList.ForEach(ipv => { ivp.Index = ivp.Index + count; count = iteration + 1; });
wholeList.AddRange(heightList);
var sumOfCombinations = GetPowerSet(wholeList).Where(x => x.Count() > 0)
.Select(x => new { Combination = x.ToList(), Sum = x.Sum(ivp => ivp.Value) }).ToList();
StringBuilder sb = new StringBuilder();
sumOfCombinations.ForEach(ivp =>
{
ivp.Combination.ForEach(pair => sb.Append(string.Format("{0} ", pair.Value)));
sb.Append(string.Format("= {0} = ", x.Sum));
ivp.Combination.ForEach(pair=> sb.Append(string.Format("{0} + ", pair.Index)));
sb.Length -= 3;
Console.WriteLine(sb);
sb.Clear();
});
var key = Console.ReadKey();
}

LINQ swap columns into rows

Is there a fancy LINQ expression that could allow me to do the following in a much more simpler fashion. I have a List<List<double>>, assuming the List are columns in a 2d matrix, I want to swap the list of columns into a list of rows. I have the following obvious solution:
int columns = 5;
var values; // assume initialised as List<List<double>>()
var listOfRows = new List<List<double>>();
for (int i = 0; i < columns ; i++)
{
List<double> newRow = new List<double>();
foreach (List<double> value in values)
{
newRow.Add(value[i]);
}
listOfRows.Add(newRow);
}

You could LINQify the inner loop pretty easily:
vector.AddRange(values.Select(value => value[i]));
Whether or not that improves the readability is left entirely up to you!

Here's a Linq expression that would do what you want - looking at it I'd personally stick with the nested foreach loops though - much easier to read:
var columnList= new List<List<double>>();
columnList.Add(new List<double>() { 1, 2, 3 });
columnList.Add(new List<double>() { 4, 5, 6 });
columnList.Add(new List<double>() { 7, 8, 9 });
columnList.Add(new List<double>() { 10, 11, 12 });
int columnCount = columnList[0].Count;
var rowList = columnList.SelectMany(x => x)
.Select((x, i) => new { V = x, Index = i })
.GroupBy(x => (x.Index + 1) % columnCount)
.Select(g => g.Select( x=> x.V).ToList())
.ToList();
This example also would only work on a matrix with a fixed column count. Basically it's flattening the matrix into a list, then creating the list of rows by grouping by the index of the element in the list modulo the column count.
Edit:
A different approach, much closer to a nested loop and probably similar performance besides the overhead.
int columnCount = columnList[0].Count;
int rowCount = columnList.Count;
var rowList = Enumerable.Range(0, columnCount)
.Select( x => Enumerable.Range(0, rowCount)
.Select(y => columnList[y][x])
.ToList())
.ToList();

var inverted = Enumerable.Range(0, columnCount)
.Select(index => columnList.Select(list => list[index]));
In short, we enumerate the column index from a range and use it to collect the nth element of each list.
Please note that you'll need to check that every list has the same number of columns.

Here's one that works for rectangular (non-ragged) matrices. The C# code here works cut-and-paste into LinqPad, a free, interactive C# programming tool.
I define a postfix operator (that is, an extension method) "Transpose." Use the operator as follows:
var rand = new Random();
var xss = new [] {
new [] {rand.NextDouble(), rand.NextDouble()},
new [] {rand.NextDouble(), rand.NextDouble()},
new [] {rand.NextDouble(), rand.NextDouble()},
};
xss.Dump("Original");
xss.Transpose().Dump("Transpose");
resulting in something like this:
Original
0.843094345109116
0.981432441613373
0.649207864724662
0.00594645645746331
0.378864820291691
0.336915332515219
Transpose
0.843094345109116
0.649207864724662
0.378864820291691
0.981432441613373
0.00594645645746331
0.336915332515219
The gist of the implementation of this operator is the following
public static IEnumerable<IEnumerable<T>> Transpose<T>(this IEnumerable<IEnumerable<T>> xss)
{
var heads = xss.Heads();
var tails = xss.Tails();
var empt = new List<IEnumerable<T>>();
if (heads.IsEmpty())
return empt;
empt.Add(heads);
return empt.Concat(tails.Transpose());
}
Here is the full implementation, with some lines commented out that you can uncomment to monitor how the function works.
void Main()
{
var rand = new Random();
var xss = new [] {
new [] {rand.NextDouble(), rand.NextDouble()},
new [] {rand.NextDouble(), rand.NextDouble()},
new [] {rand.NextDouble(), rand.NextDouble()},
};
xss.Dump("Original");
xss.Transpose().Dump("Transpose");
}
public static class Extensions
{
public static IEnumerable<T> Heads<T>(this IEnumerable<IEnumerable<T>> xss)
{
Debug.Assert(xss != null);
if (xss.Any(xs => xs.IsEmpty()))
return new List<T>();
return xss.Select(xs => xs.First());
}
public static bool IsEmpty<T>(this IEnumerable<T> xs)
{
return xs.Count() == 0;
}
public static IEnumerable<IEnumerable<T>> Tails<T>(this IEnumerable<IEnumerable<T>> xss)
{
return xss.Select(xs => xs.Skip(1));
}
public static IEnumerable<IEnumerable<T>> Transpose<T>(this IEnumerable<IEnumerable<T>> xss)
{
// xss.Dump("xss in Transpose");
var heads = xss.Heads()
// .Dump("heads in Transpose")
;
var tails = xss.Tails()
// .Dump("tails in Transpose")
;
var empt = new List<IEnumerable<T>>();
if (heads.IsEmpty())
return empt;
empt.Add(heads);
return empt.Concat(tails.Transpose())
// .Dump("empt")
;
}
}

I am combining some of the answers above, which sometimes had columns and rows inverted form the original answer or from the convention I am used to : row refers to the first index and column to the inner ( second) index. e.g. values[row][column]
public static List<List<T>> Transpose<T>(this List<List<T>> values)
{
if (values.Count == 0 || values[0].Count == 0)
{
return new List<List<T>>();
}
int ColumnCount = values[0].Count;
var listByColumns = new List<List<T>>();
foreach (int columnIndex in Enumerable.Range(0, ColumnCount))
{
List<T> valuesByColumn = values.Select(value => value[columnIndex]).ToList();
listByColumns.Add(valuesByColumn);
}
return listByColumns;
}
Actually the word row and column is just our convention of thinking about the data in rows and columns , and sometimes adds more confusion than solving them.
We are actually just swapping the inner index for the outer index. (or flipping the indexes around). So one could also just define the following extension method. . Again I borrowed from above solutions, just put it into something I find readable and fairly compact.
Checks that the inner lists are of equal sized are required.
public static List<List<T>> InsideOutFlip<T>(this List<List<T>> values)
{
if (values.Count == 0 || values[0].Count == 0)
{
return new List<List<T>>();
}
int innerCount = values[0].Count;
var flippedList = new List<List<T>>();
foreach (int innerIndex in Enumerable.Range(0, innerCount))
{
List<T> valuesByOneInner = values.Select(value => value[innerIndex]).ToList();
flippedList.Add(valuesByOneInner);
}
return flippedList;
}