We Keep Coding

IQueryable cannot translate numeric string comparer

I'm trying to sort the data by the type "V0003", "4323Fw" but an error occurs with comparer, what is going wrong and how can I sort in IQueryable?
My comparer:
public class NumericStringComparer : IComparer<string>
{
public int Compare(string s1, string s2)
{
const int S1GreaterThanS2 = 1;
const int S2GreaterThanS1 = -1;
const int S2EqualsS1 = 0;
var isS1Numeric = IsNumeric(s1);
var isS2Numeric = IsNumeric(s2);
if (isS1Numeric && isS2Numeric)
{
var firstNum = Convert.ToInt64(s1);
var secondNum = Convert.ToInt64(s2);
if (firstNum > secondNum)
{
return S1GreaterThanS2;
}
if (firstNum < secondNum)
{
return S2GreaterThanS1;
}
return S2EqualsS1;
}
if (isS1Numeric)
{
return S2GreaterThanS1;
}
if (isS2Numeric)
{
return S1GreaterThanS2;
}
return string.Compare(s1, s2, true, CultureInfo.InvariantCulture);
static bool IsNumeric(string value)
{
return long.TryParse(value, out _);
}
}
}
My sort:
query.OrderBy(bdc => bdc.Code, comparer), // query is IQueryable<MyClass>
Error:
"The LINQ expression""DbSet()\r\n .Where(bdc => bdc.DateEnd >= DateTime.UtcNow && bdc.Deleted == False)\r\n .OrderByDescending(bdc => bdc.Id)\r\n .Include(bdc => bdc.InverseParent)\r\n .ThenInclude(bdc => bdc.InverseParent)\r\n .ThenInclude(bdc => bdc.InverseParent)\r\n .Where(bdc => (int)bdc.StagesBudgetCycle == (int)(short)__StagesBudgetCycle_0 || bdc.InverseParent\r\n .AsQueryable()\r\n .Any(children => (int)children.StagesBudgetCycle == (int)(short)__StagesBudgetCycle_0 || children.InverseParent\r\n .AsQueryable()\r\n .Any(children => (int)children.StagesBudgetCycle == (int)(short)__StagesBudgetCycle_0 || children.InverseParent\r\n .AsQueryable()\r\n .Any(children => (int)children.StagesBudgetCycle == (int)(short)__StagesBudgetCycle_0))))\r\n .OrderBy(\r\n keySelector: bdc => bdc.Code, \r\n comparer: __p_1)""could not be translated. Either rewrite the query in a form that can be translated",
"or switch to client evaluation explicitly by inserting a call to""AsEnumerable",
"AsAsyncEnumerable",
"ToList",
"or""ToListAsync"". See https"://go.microsoft.com/fwlink/?linkid=2101038 for more information.

Looking for a better pattern for a visitor-IAsyncEnumerable?

Consider this snippet (much simplified than the original code):
async IAsyncEnumerable<(DateTime, double)> GetSamplesAsync()
{
// ...
var cbpool = new CallbackPool(
HandleBool: (dt, b) => { },
HandleDouble: (dt, d) =>
{
yield return (dt, d); //not allowed
});
while (await cursor.MoveNextAsync(token))
{
this.Parse(cursor.Current, cbpool);
}
}
private record CallbackPool(
Action<DateTime, bool> HandleBool,
Action<DateTime, double> HandleDouble
);
Then, the below Parse is just a behavior-equivalent of the original.
Random _rnd = new Random();
void Parse(object cursor, CallbackPool cbpool)
{
double d = this._rnd.NextDouble(); //d = [0..1)
if (d >= 0.5)
{
cbpool.HandleDouble(new DateTime(), d);
}
else if (d >= 0.25)
{
cbpool.HandleBool(new DateTime(), d >= 0.4);
}
}
However, I do like the GetSamplesAsync code, but the compiler does not: the yield cannot be used within a lambda.
So, I changed the function as follows, although it became much less readable (and also error-prone):
async IAsyncEnumerable<(DateTime, double)> GetSamplesAsync()
{
// ...
(DateTime, double) pair = default;
bool flag = false;
var cbpool = new CallbackPool(
HandleBool: (dt, b) => { },
HandleDouble: (dt, d) =>
{
pair = (dt, d);
flag = true;
});
while (await cursor.MoveNextAsync(token))
{
this.Parse(cursor.Current, cbpool);
if (flag)
{
yield return pair;
}
flag = false;
}
}
I wonder if there is a better way to solve this kind of pattern.

The external flag/pair is pretty dangerous and unnecessary (and it forces a closure); it seems like this bool could be returned from the Parse method, for example:
await foreach (var item in cursor)
{
if (Parse(item, cbpool, out var result))
yield return result;
}
(everything could also be returned via a value-tuple if you don't like the out)

How to write a function that generates ID by taking missing items in a sequence?

How can I write an algorithm that can take unused ID's out of a sequence starting from 1 to 99 in the format "C00"? For example NewId(['C01', 'C02', 'C03']) should emit 'C04', but NewId(['C02', 'C03', 'C04']) should emit C01, and NewId(['C01', 'C03', 'C04']) should result in C02.
I wrote an implementation but the result is wrong.
Example : CAT_ID : C01, C02, C05, C06, C11. When I run it, the expected result is C03. My algorithm is as follows:
Sort ID asc
Go through every item in the list
Compare first value with the next, if they are not the same, add 1 and exit loop.
This is my code:
public static string Get_AreaID_Auto()
{
string result = "";
if (db.TESTs.ToList().Count <= 0)
{
result = "01";
}
else
{
int maxId = 0;
foreach (var item in db.TESTs.OrderBy(e => e.CAT_ID).ToList())
{
if (int.Parse(item.CAT_ID.Substring(1)) + 1 != int.Parse(item.CAT_ID.Substring(1)))
{
maxId = int.Parse(item.CAT_ID.Substring(1) + 1);
break;
}
}
switch (maxId.ToString().Length)
{
case 1:
if (maxId == 9)
{
result = "10";
}
else
result = "0" + (maxId + 1);
break;
case 2:
result = "" + (maxId + 1);
break;
default:
break;
}
}
return "C" + result;
}
Can someone point out what is wrong?

This should work for you:
public static string Get_AreaID_Auto()
{
var existing = db.TESTs.Select(e => e.CAT_ID).OrderBy(x => x).ToList();
if (existing.Count == 0)
{
return "C01";
}
else
{
return
existing
.Concat(new [] { "" })
.Select((x, n) => new
{
actual = x,
expected = String.Format("C{0:00}", n + 1),
})
.Where(x => x.actual != x.expected)
.Select(x => x.expected)
.First();
}
}
This uses a generate and test approach. No parsing necessary.
I just realised with the .Concat(new [] { "" }) change that the if statement is now no longer required. You can do this instead:
public static string Get_AreaID_Auto()
{
return
db.TESTs
.Select(e => e.CAT_ID)
.OrderBy(x => x)
.ToArray()
.Concat(new [] { "" })
.Select((x, n) => new
{
actual = x,
expected = String.Format("C{0:00}", n + 1),
})
.Where(x => x.actual != x.expected)
.Select(x => x.expected)
.First();
}

Try this
public static string Get_AreaID_Auto()
{
string result = "";
if (db.TESTs.ToList().Count <= 0)
{
result = "01";
}
else
{
var item = db.TESTs.OrderByDescending(e => e.CAT_ID).First();
result = int.Parse(item.CAT_ID.Substring(1)) + 1;
}
return string.Format("C{0:D3}",result);
}
Updated Code...Now Try this
public static string Get_AreaID_Auto()
{
string result = "";
if (db.TESTs.ToList().Count <= 0)
{
result = "01";
}
else
{
var items = db.TESTs.OrderBy(e => e.CAT_ID).ToArray();
for(int i=0;i<items.count;i++)
{
if ((i==items.count-1) || (int.Parse(items[i].CAT_ID.Substring(1)) + 1 != int.Parse(items[i+1].CAT_ID.Substring(1))))
{
result = int.Parse(items[i].CAT_ID.Substring(1) + 1);
break;
}
}
}
return string.Format("C{0:D2}",result);
}

Here is a solution I think would work:
var items = db.TESTs.Select(x => int.Parse(x.CAT_ID.Substring(1))).OrderBy(v => v).ToArray();
if(!items.Any())
return "C01";
int current = 0;
for (int i = 0; i < items.Length; i++)
{
if (items[i] > current + 1)
return "C" + (current + 1) .ToString("00");
current = items[i];
}
return "C" + (items.Max() + 1).ToString("00");

Mathematical Equality of Func<int, int> Part Two: Using the Syntax Tree

So, for those who have not been involved in my earlier question on this topic, I have some C# types to represent Sequences, like in math, and I store a formula for generating the nth-term as
Func NTerm;
Now, I would like to compare one Sequence's nth-term formula to another for mathematical equality. I have determined that practically, this requires a limitation to five simple binary operators: +,-,/,*,%.
With tremendous help from Scott Chamberlain on the OP, I've managed to get here. I'm envisioning some kind of recursive solution, but I'm not quite sure where to proceed. Will keep chugging away. Currently, this handles single binary expressions like x + 5 or 7 * c, well, but doesn't deal with other cases. I think the idea is to start from the end of the List, since I think it parses with last operations at the top of the syntax tree and first ones at the bottom. I haven't yet done extensive testing for this method.
UPDATE
public sealed class LambdaParser : ExpressionVisitor
{
public List<BinaryExpression> Expressions { get; } = new List<BinaryExpression> ();
protected override Expression VisitBinary (BinaryExpression node)
{
Expressions.Add (node);
return base.VisitBinary (node);
}
public bool MathEquals (LambdaParser g)
{
try
{
return MathEquals (Expressions, g.Expressions);
} catch (Exception e)
{
throw new Exception ("MathEquals", e);
}
}
public static bool MathEquals (List<BinaryExpression> f, List<BinaryExpression> g)
{
//handle simple cases
if (ReferenceEquals (f, g))
return true;
if (f == null || g == null)
return false;
if (f.Count == 0 || g.Count == 0)
return false;
try
{
//handle single one element cases
if (f.Count == 1 && g.Count == 1)
{
return MathEquals (f[0], g[0]);
}
} catch (Exception e)
{
throw new Exception ("MathEquals", e);
}
throw new NotImplementedException ("Math Equals");
}
static bool MathEquals (BinaryExpression f, BinaryExpression g)
{
if (ReferenceEquals (f, g))
return true;
if (f == null || g == null)
return false;
if (f.NodeType != g.NodeType)
return false;
try
{
switch (f.NodeType)
{
case ExpressionType.Add:
case ExpressionType.Multiply:
return CompareCommutative (f, g);
case ExpressionType.Subtract:
case ExpressionType.Divide:
case ExpressionType.Modulo:
return CompareNonCommutative (f, g);
default:
throw new NotImplementedException ($"Math Equals {nameof(f)}: {f.NodeType}, {nameof(g)}: {g.NodeType}");
}
} catch (Exception e)
{
throw new Exception ($"MathEquals {nameof(f)}: {f.NodeType}, {nameof(g)}: {g.NodeType}", e);
}
}
static bool IsParam (Expression f)
{
return f.NodeType == ExpressionType.Parameter;
}
static bool IsConstant (Expression f)
{
return f.NodeType == ExpressionType.Constant;
}
static bool CompareCommutative (BinaryExpression f, BinaryExpression g)
{
bool left, right;
try
{
//parse left f to left g and right g
left = CompareParamOrConstant (f.Left, g.Left) || CompareParamOrConstant (f.Left, g.Right);
//parse right f to left g and right g
right = CompareParamOrConstant (f.Right, g.Left) || CompareParamOrConstant (f.Right, g.Right);
return left && right;
} catch (Exception e)
{
throw new Exception ($"CompareCommutative {nameof(f)}: {f.NodeType}, {nameof(g)}: {g.NodeType}", e);
}
}
static bool CompareNonCommutative (BinaryExpression f, BinaryExpression g)
{
bool left, right;
try
{
//compare f left to g left
left = CompareParamOrConstant (f.Left, g.Left);
//compare f right to f right
right = CompareParamOrConstant (f.Right, g.Right);
} catch (Exception e)
{
throw new Exception ($"CompareNonCommutative {nameof(f)}: {f.NodeType}, {nameof(g)}: {g.NodeType}", e);
}
return left && right;
}
static bool CompareParamOrConstant (Expression f, Expression g)
{
var ParamF = IsParam (f);
var ConstantF = IsConstant (f);
if (!(ParamF || ConstantF))
{
throw new ArgumentException ($"{nameof(f)} is neither a param or a constant", $"{nameof(f)}");
}
var ParamG = IsParam (g);
var ConstantG = IsConstant (g);
if (!(ParamG || ConstantG))
{
throw new ArgumentException ($"{nameof(g)} is neither a param or a constant", $"{nameof(g)}");
}
if (ParamF)
{
return ParamG;
}
if (ConstantF)
{
return ConstantG && (f as ConstantExpression).Value.Equals ((g as ConstantExpression).Value);
}
}
}
END_UPDATE
I updated the above code to reflect the changes made (mostly refactoring, but also a slightly different approach) after comments reminded me that I'd ignored the non-commutability of some operators.
How do I extend it for expressions with multiple operators of the five defined above? Instead of just 5 * x, something like 2 * x % 3 - x / 5.

Using LinqToSql without load base in memory

I have one problem. If i'm using LinqToSql, my program load my database in memory.
little example:
//pageNumber = 1; pageSize = 100;
var result =
(
from a in db.Stats.AsEnumerable()
where (DictionaryFilter(a, sourceDictionary) && DateFilter(a, beginTime, endTime) && ErrorFilter(a, WarnLevel))
select a
);
var size = result.Count(); // size = 1007
var resultList = result.Skip((pageNumber-1)*pageSize).Take(pageSize).ToList();
return resultList;
DictionaryFilter, DateFilter and ErrorFilter are functions that filter my datebase.
after this my program use ~250Mb of Ram.
if i dont use:
var size = result.Count();
My program use ~120MB Ram.
Before use this code, my program use ~35MB Ram.
How can I use count and take functions not loading all my datebase in memory?
static bool DateFilter(Stat table, DateTime begin, DateTime end)
{
if ((table.RecordTime >= begin.ToFileTime()) && (table.RecordTime <= end.ToFileTime()))
{
return true;
}
return false;
}
static bool ErrorFilter(Stat table, bool[] WarnLevel)
{
if (WarnLevel[table.WarnLevel]) return true;
else return false;
}
static bool DictionaryFilter(Stat table, Dictionary<GetSourcesNameResult, bool> sourceDictionary)
{
foreach (var word in sourceDictionary)
{
if (table.SourceName == word.Key.SourceName)
{
return word.Value;
}
}
//
return false;
}

Simple: don't use .AsEnumerable(). That means "switch to LINQ-to-Objects". Before that, db.Stats was IQueryable<T>, which is a composable API, and would do what you expect.
That, however, means that you can't use C# methods like DictionaryFilter and DateFilter, and must instead compose things in terms of the Expression API. If you can illustrate what they do I can probably advise further.
With your edit, the filtering can be tweaked, for example:
static IQueryable<Stat> ErrorFilter(IQueryable<Stat> source, bool[] WarnLevel) {
// extract the enabled indices (match to values)
int[] levels = WarnLevel.Select((val, index) => new { val, index })
.Where(pair => pair.val)
.Select(pair => pair.index).ToArray();
switch(levels.Length)
{
case 0:
return source.Where(x => false);
case 1:
int level = levels[0];
return source.Where(x => x.WarnLevel == level);
case 2:
int level0 = levels[0], level1 = levels[1];
return source.Where(
x => x.WarnLevel == level0 || x.WarnLevel == level1);
default:
return source.Where(x => levels.Contains(x.WarnLevel));
}
}
the date filter is simpler:
static IQueryable<Stat> DateFilter(IQueryable<Stat> source,
DateTime begin, DateTime end)
{
var from = begin.ToFileTime(), to = end.ToFileTime();
return source.Where(table => table.RecordTime >= from
&& table.RecordTime <= to);
}
and the dictionary is a bit like the levels:
static IQueryable<Stat> DictionaryFilter(IQueryable<Stat> source,
Dictionary<GetSourcesNameResult, bool> sourceDictionary)
{
var words = (from word in sourceDictionary
where word.Value
select word.Key.SourceName).ToArray();
switch (words.Length)
{
case 0:
return source.Where(x => false);
case 1:
string word = words[0];
return source.Where(x => x.SourceName == word);
case 2:
string word0 = words[0], word1 = words[1];
return source.Where(
x => x.SourceName == word0 || x.SourceName == word1);
default:
return source.Where(x => words.Contains(x.SourceName));
}
}
and:
IQueryable<Stat> result = db.Stats;
result = ErrorFilter(result, WarnLevel);
result = DateFiter(result, beginTime, endTime);
result = DictionaryFilter(result, sourceDictionary);
// etc - note we're *composing* a filter here
var size = result.Count(); // size = 1007
var resultList = result.Skip((pageNumber-1)*pageSize).Take(pageSize).ToList();
return resultList;
The point is we're now using IQueryable<T> and Expression exclusively.

The following SO question might explain things: Understanding .AsEnumerable in Linq To Sql
.AsEnumerable() loads the entire table.