We Keep Coding

Retrieve user defined 'Combination of Columns' from database with Entity Framework

I need to retrieve user defined columns from database with Entity Framework.
I need to create column Projection based on passed collection, a List of strings, where each element contains column names, with Entity Framework
I have a list of string which contains strings like this;
List<string> columnsToSelect = new List<string>();
columnsToSelect.Add("col1 + col2");
columnsToSelect.Add("col2");
columnsToSelect.Add("col3 + col4");
I have a table called 'RawData' with 6 columns like this:
Id EmpId col1 col2 col3 col4
Now, if i were to query simple
var rawDatacolums = Context.RawData.where(a => a.EmpId = #EmpId)
this will generate SQL statement like this,
Select Id,EmpId,col1,col2,col3,col4 from RawData
where EmpId = #EmpId
Here i want to pass columnsToSelect as argument and my result should be based on column selector i am passing in List
What i want to do is,
var rawDatacolums = Context.RawData.select(columnsToSelect).where(a =>
a.EmpId = #EmpId)
Which should generate SQL like;
Select col1 + col2 as Col1Col2, col2 as Col2, col3 + col4 as Col3Col4
from RawData where EmpId = #EmpId
I have tried to use "SelectProperties" from this article here:
https://byalexblog.net/entity-framework-dynamic-columns
https://github.com/lAnubisl/entity-framework-dynamic-queries/blob/master/entity-framework-dynamic-queries/SelectiveQuery.cs
var rawDatacolums = Context.RawData.SelectProperties(columnsToSelect)
if a pass exact columns like col1, col2 as list it works
but it doesn't work the way i want for example Sum of Two columns
My Requirement is i need to project addition of columns
like 'col1 + col2' & 'col3 + col4'
Updated Answer
Based on couple of suggestions, i played more with Dynamic LINQ
and i made it work, i was able to apply various math conditions on my projection
and was able to create Dynamic Class out of it
The original github reference is as below:
https://github.com/kahanu/System.Linq.Dynamic
but i found explanation here more useful please take a look here:
http://ak-dynamic-linq.azurewebsites.net/GettingStarted#conversions
Some other material i referred and had to use - which maybe helpful to someone is here - http://www.albahari.com/nutshell/predicatebuilder.aspx
Sample working code would look like this;
var sampleQuery = "New(Col1+Col2 as Stage1Count)";
IEnumerable queryResult= Context.RawData.AsQueryable().Select(sampleQuery );
System.Diagnostics.Debug.WriteLine("Debug Sample Query: " + queryResult.ToString());
foreach (var cust in queryResult)
{
System.Diagnostics.Debug.WriteLine("Debug Sample StageCount : " + cust.ToString());
}
Thanks all for your comments and suggestions! Cheers!

It is obviously possible to create classes at runtime, or even new anonymous types, but they are extremely limited in how you can use them in your code.
If you prefer to work within the modern generic Queryable framework, and avoid creating classes at runtime that have limited compile time access, you can roll your own expression parser and build Expression trees. The trick is to use an Array type as the return from the Select to make the members accessible. This does mean all the expressions must return the same type, but this implementation will convert all the expressions to one type if necessary.
Here is a sample implementation:
public static class IQueryableExt {
public static Expression<Func<TRec, TVal?[]>> SelectExpr<TRec, TVal>(this IEnumerable<string> strExprs) where TVal : struct {
var p = Expression.Parameter(typeof(TRec), "p");
var exprs = strExprs.Select(se => {
var e = se.ParseExpression(p);
return e.Type.IsNullableType() && e.Type.GetGenericArguments()[0] == typeof(TVal) ? e : Expression.Convert(e, typeof(TVal?));
}).ToArray();
return Expression.Lambda<Func<TRec, TVal?[]>>(Expression.NewArrayInit(typeof(TVal?), exprs), p);
}
static char[] operators = { '+', '-', '*', '/' };
static Regex tokenRE = new Regex($#"(?=[-+*/()])|(?<=[-+*/()])", RegexOptions.Compiled);
static HashSet<char> hsOperators = operators.ToHashSet();
static Dictionary<char, ExpressionType> opType = new Dictionary<char, ExpressionType>() {
{ '*', ExpressionType.Multiply },
{ '/', ExpressionType.Divide },
{ '+', ExpressionType.Add },
{ '-', ExpressionType.Subtract }
};
static int opPriority(char op) => hsOperators.Contains(op) ? Array.IndexOf(operators, op) >> 1 : (op == ')' ? -1 : -2);
public static Expression ParseExpression(this string expr, ParameterExpression dbParam) {
var opStack = new Stack<char>();
opStack.Push('(');
var operandStack = new Stack<Expression>();
foreach (var t in tokenRE.Split(expr).Where(t => !String.IsNullOrEmpty(t)).Append(")")) {
if (t.Length > 1) // process column name
operandStack.Push(Expression.PropertyOrField(dbParam, t));
else {
while (t[0] != '(' && opPriority(opStack.Peek()) >= opPriority(t[0])) {
var curOp = opStack.Pop();
var right = operandStack.Pop();
var left = operandStack.Pop();
if (right.Type != left.Type) {
if (right.Type.IsNullableType())
left = Expression.Convert(left, right.Type);
else if (left.Type.IsNullableType())
right = Expression.Convert(right, left.Type);
else
throw new Exception($"Incompatible types for operator{curOp}: {left.Type.Name}, {right.Type.Name}");
}
operandStack.Push(Expression.MakeBinary(opType[curOp], left, right));
}
if (t[0] != ')')
opStack.Push(t[0]);
else
opStack.Pop(); // pop (
}
}
return operandStack.Pop();
}
public static bool IsNullableType(this Type nullableType) =>
// instantiated generic type only
nullableType.IsGenericType &&
!nullableType.IsGenericTypeDefinition &&
Object.ReferenceEquals(nullableType.GetGenericTypeDefinition(), typeof(Nullable<>));
}
Unfortunately type inference can't easily get the answer type, so you have to manually pass in the record type and answer type. Note there's special code in the parser to handle conversion to (common in SQL) nullable types when mixing nullable and non-nullable.
Given the columnsToSelect you provided as an example:
List<string> columnsToSelect = new List<string>();
columnsToSelect.Add("col1 + col2");
columnsToSelect.Add("col2");
columnsToSelect.Add("col3 + col4");
You can query the database like so:
var queryResult= Context.RawData.Select(columnsToSelect.SelectExpr<TRawData, int>());
And queryResult would be of type IQueryable<int[]> or IQueryable<int?[]> depending on the SQL column types.

Mapping from join - The type 'CarDTO' appears in two structurally incompatible initializations within a single LINQ to Entities query

I am trying to map into CarDTO from right side of left join, but gets the following error:
The type 'CarDTO' appears in two structurally incompatible
initializations within a single LINQ to Entities query. A type can be
initialized in two places in the same query, but only if the same
properties are set in both places and those properties are set in the
same order.
I am trying to make self left join and my query looks like this:
var query = from left in db.Cars
join right in db.Cars on left.id_Base equals right.ID into rightGrouped
from rightGr in rightGrouped.DefaultIfEmpty()
select new CarDTO()
{
ID = left.ID,
Description = left.DisplayName,
Name = left.Name,
IsSystem = left != null ? left.Template.isSystem : (byte?)null,
IdBase = left.id_Base,
InnerCar = new CarDTO()
{
ID = rightGr.ID,
Description = rightGr.DisplayName,
Name = rightGr.Name,
IsSystem = rightGr != null ? rightGr.Template.isSystem : (byte?)null,
IdBase = rightGr.id_Base,
InnerCar = new CarDTO()
}
};
If I change the following row InnerCar = new CarDTO() to InnerCar = null,
then I get the following error:
Unable to create a null constant value of type 'CarDTO'. Only entity
types, enumeration types or primitive types are supported in this
context.`
Does anybody know what I am doing wrong?

Error message explains that EF expects all initializations of the same type (CarDto in this case) in the same expression to follow a rule: in each initialization the same properties should be set, and in the same order. Your code violates this rule, because you have 3 initializations. 2 of them set the same properties in same order, but third one (last InnerCar = new CarDTO()) does not.
The only possible value to use in InnerCar.InnerCar initialization to satisfy this rule is null (because of recursive definition). However, EF will not allow you to do that, because it cannot create constant values of arbitrary types (doesn't know how to convert such thing into SQL).
So doing this with recursive type is just not possible. Instead, we should use different type for InnerCar. This can be done for example with anonymous types:
select new
{
ID = left.ID,
Description = left.DisplayName,
Name = left.Name,
IsSystem = left != null ? left.Template.isSystem : (byte?)null,
IdBase = left.id_Base,
InnerCar = new
{
ID = rightGr.ID,
Description = rightGr.DisplayName,
Name = rightGr.Name,
IsSystem = rightGr != null ? rightGr.Template.isSystem : (byte?)null,
IdBase = rightGr.id_Base
}
};
This will work, because two anonymous types here are different. First one has 6 properties (including InnerCar) and second one has 5 properties (without InnerCar). Of course for type to be anonymous is not a requirement here.
You can also just flatten this structure:
select new
{
LeftID = left.ID,
RightID = rightGr.ID,
// etc
};
After you performed such mapping, and have done any other filtering\paging\whatever - convert it to your type after materializing:
// query here is complete
var result = query.AsEnumerable().Select(c => {
new CarDto {
ID = c.ID,
Description = c.Description,
// etc
}
});

How can I refine a linq query that binds to a DTO via another list?

I'm doing some SalesForce work via C# and I have a list of Cases. You should not need to be familiar with SalesForce to understand what the issue is. I'm trying to retrieve a list of CaseComment objects where the parentId of CaseComment matches the Id of Case. i.e:
Case 1
Id = 123456
Case Comment 1
Id = 5532094
ParentId = 234242
Case Comment 2
Id = 984984
ParentId = 123456
In this case, the parentId of case comment 2 matches Case 1 and thus should be returned. Here is the code I'm currently using:
static List<SalesForceCaseComment> GetCaseCommentsFromListOfCases(List<SalesForceCase> Cases)
{
using (salesforce_backupsEntities le = new salesforce_backupsEntities())
{
List<SalesForceCaseComment> casecomments = (from c in le.CaseComments select new SalesForceCaseComment
{
CommentBody = c.CommentBody,
CreatedById = c.CreatedById,
Id = c.Id,
IsDeleted = c.IsDeleted,
IsPublished = c.IsPublished,
LastModifiedById = c.LastModifiedById,
ParentId = c.ParentId
}).Where(n => Cases.Any(s=> s.Id == n.ParentId)).ToList();
return casecomments;
}
}
This unfortunately results in the following error:
Unable to create a constant value of type 'DTO.SalesForceCase'. Only primitive types or enumeration types are supported in this context.

It looks like you're effectively joining a local List<SalesForceCase> to a table on the provider side, which probably isn't going to work. If the provider supports the Contains operator, you can do this:
var ids = Cases.Select(c => c.Id).ToArray();
var comments = from c in le.CaseComments
where ids.Contains(c.ParentId)
select new /* projection */;

Basically it means that underline provider doesnt support enumeration types, so you need to either write your own visitor to enumerate it (harder) or just convert it to expression with primitive types with "or", which simply means looping through your array and adding conditions.
Second way could be easier if salesforce_backupsEntities inherited from ObjectContext. In this case you can use IN esql operator.
Also, if there is navigation property in Case object, you may want to attach object to context and do load on navigation property ( case.Comments.Load() )

I would guess something like this should work (join instead of Where):
static List<SalesForceCaseComment> GetCaseCommentsFromListOfCases(List<SalesForceCase> Cases)
{
using (salesforce_backupsEntities le = new salesforce_backupsEntities())
{
List<SalesForceCaseComment> casecomments = (from c in le.CaseComments
where Cases.Select(x => x.Id).Contains(c.ParentId)
select new SalesForceCaseComment
{
CommentBody = c.CommentBody,
CreatedById = c.CreatedById,
Id = c.Id,
IsDeleted = c.IsDeleted,
IsPublished = c.IsPublished,
LastModifiedById = c.LastModifiedById,
ParentId = c.ParentId
})).ToList();
return casecomments;
}
}

PredicateBuilder returns all users predicate.Or

I'm developing a library with C#, Entity Framework Code First 4.4.0.0 and .NET Framework 4.0.
I have this code:
string[] keywords = null;
if (query_string != null)
keywords = query_string.Split(' ');
// Check parameter. It will throw an exception.
ParameterCheck.CheckObjectId(user_id);
// Convert parameter to int.
int userId = Int32.Parse(user_id);
try
{
using (var context = new AdnLineContext())
{
context.Configuration.ProxyCreationEnabled = false;
//IQueryable<User> query = context.Users;
IQueryable<User> query = context.Users.AsExpandable();
var predicate = PredicateBuilder.False<User>();
foreach (string keyword in keywords)
{
string temp = keyword;
predicate = predicate.Or(u => u.Name.Contains(temp) ||
u.State.Contains(temp) ||
u.HashTags.Contains(temp));
}
query.Where(predicate);
query.Include("WhoHasBlockedMe").Include("Friends");
var users = query.ToList();
[ ... ]
}
}
if keywords = {"john", "happy"} I want to get this SQL:
select * from users where
users.name like '%john%' or
users.state like '%john%' or
users.hashtags like '%john%' or
users.name like '%happy%' or
users.state like '%happy%' or
users.hashtags like '%happy%';
How can I do that with PredicateBuilder?
By the way, I have downloaded LinqKit from http://www.albahari.com/nutshell/predicatebuilder.aspx
UPDATE
I have added a breakpoint at var users = query.ToList(); and query has this SQL:
{SELECT
[Extent1].[UserId] AS [UserId],
[...]
FROM [dbo].[Users] AS [Extent1]}
It doesn't have a Where clause.

Your problem is here:
query.Where(predicate);
query.Include("WhoHasBlockedMe").Include("Friends");
var users = query.ToList();
.Where and .Include don't modify the underlying query, they return a new query.
Use
query = query.Where(predicate);
query = query.Include("WhoHasBlockedMe").Include("Friends");
var users = query.ToList();
instead.

What's the result of true OR something Or SomethingElse?
Do you even need to know what something is?
It will always be true.
You need to start out with False, not True. False Or anything is equivalent to anything.
If you were AND-ing items, you'd start with true, given that true AND anything is equivalent to anything.

LINQ query with SELECT and two GROUP-BY condition

What's the equivalent LINQ instruction for a Datatable of the following SQL query:
SELECT code_direction, count(TP) AS CN
FROM table1
WHERE cod_time = 'A011'
GROUP BY TP,code_direction;
and how to get the result into a new datatable?
I tried to convert it but I there're some errors. Someone could take a look on this:
var query = from t in table1.AsEnumerable()
group t by new { t.TP, t.code_direction }
into grp
select new
{
grp.Key.code_direction,
CN = grp.Count(t.TP)
};
foreach (var x in query)
{
Console.Write(x.code_direction);
Console.Write(x.CN);
}

As far as your first question goes. The LINQ equivalent of the SQL query is:
var query = from t in table1.AsEnumerable()
where t.cod_time == "A011"
group t by new { t.TP, t.code_direction }
into grp
select new
{
grp.Key.code_direction,
CN = grp.Count()
};
Note that you don't have to pass any argument to grp.Count(). (For the obvious reason that in SQL COUNT(TP) is the same as COUNT(*), i.e. just count the number of rows. The story would be different if you'd use COUNT(DISTINCT TP) or similar.)
As far as the second question goes, if your query just returned an IEnumerable<T> where T is DataRow (i.e. a query like table1.AsEnumerable().Where(r => r.cod_time == "A011")) then you could just the DataTableExtensions.CopyToDataTable extension method. As your query returns an anonymous type however, you will have to follow these instructions found on MSDN.

I Have been using LINQ to work on a JSON object returned from a remote sharepoint web service. I have posted this because most of the answers I found online were slightly different from what I needed.
a json list of daily activities is returned from a remote sharepoint list & is then summarised using LINQ
The simplified version of a custom object definition is shown below( & which is defined in the models area of an MVC application)
public class MyCustomObjectList
{
public string eventdate { get; set; }
public string userid { get; set; }
public string action { get; set; }
}
The JSON object is serialised into a MyCustomObjectList array.
var customobject = serializer.Deserialize<MyCustomObjectList>(jsonobject);
I wanted to work out how many actions of each type happened on a given day. NB eventdate is stored as a string in format yyyy-mm-dd hh:MM:ss. This was to simplify conversions between c#, JSON & Jquery ( where required I create DateTime objects elsewhere in the code using the
eventdate.
Some will argue this is inefficient, but I prefer to split processes into a sequential set of really simple operations, for the sake of easier debugging & to help other people follow my code. Thats why there are 2 Linq queries .
querya strips out the time component from the eventdate This ensures our later grouping happens by day, & not by second. To be doubly sure that there is no caching, I create it in a new field called actionday. I also rename action to activity, because intellisense was getting confused!! The other columns are copied as is.
var querya =
from c in customobject.rows
select new { actionday = c.eventdate.Substring(0, 10), activity = c.action, c.userid,
c.eventdate };
/* queryb produces a grouped count of querya, grouped on actionday & activity, creating new columns actionkey,ActionCount,Dte,action & DetailList ( which is a summary for debugging purposes)
*/
var queryb=
from p in querya group p by new { p.actionday, p.activity} into idGroup
actionkey = idGroup.Key,
ActionCount = idGroup.Count(),
Dte = idGroup.Key.actionday,
action = idGroup.Key.activity,
DetailList = idGroup
};
Here’s a version that sumarises by 3 columns
var queryc = from p in querya
group p by new { p.actionday, p.userid, p.activity} into idGroup
select new
{
actionday = idGroup.Key,
ActionCount = idGroup.Count(),
userid = idGroup.Key.userid,
Dte = idGroup.Key.actionday,
action = idGroup.Key.activity,
DetailList = idGroup
};