var Result = addressContext.Address_Lookup
.Where(c => c.Address_Full.ToUpper().Contains(term.ToUpper())
|| c.Address_Full.ToUpper().Contains(TermModified.ToUpper()))
.Select(e => new {
id = e.Address_ID,
label = e.Address_Full,
value = e.Address_Full })
.ToList();
To ensure search will be non-case sensitive I am using ToUpper().
I am searching for something like Jimmy (with a capital J). jimmy (all lower case) doesnt catch? why?
Since you're using entity-framework, a linq-to-sql framework, you're actually trying to make the database perform a .ToUpper rather than performing one in-memory as you would if running through an IEnumerable. If the query translation in your framework doesn't support the function, it either won't be used or throw an Exception.
You can generally predict such behaviour by checking whether you're calling a function against an IQueryable object, which queues all calls as an expression tree for translation, or an IEnumerable, which uses foreach and yield returns to handle evaluation. Since the Linq functions are extension methods, polymorphism doesn't apply here.
If you're not worried about the performance hit of getting EVERY entry from that table in-memory, add a .AsEnumerable() call, and your functionwill evaluate on localized data.
var Result = addressContext.Address_Lookup
.AsEnumerable()
.Where(c => c.Address_Full.ToUpper().Contains(term.ToUpper())
|| c.Address_Full.ToUpper().Contains(TermModified.ToUpper()))
.Select(e => new
{
id = e.Address_ID,
label = e.Address_Full,
value = e.Address_Full
})
.ToList();
Related
I am trying to make a query to a database view based on earlier user-choices. The choices are stored in lists of objects.
What I want to achieve is for a record to be added to the reportViewList if the stated value exists in one of the lists, but if for example the clientList is empty the query should overlook this statement and add all clients in the selected date-range. The user-choices are stored in temporary lists of objects.
The first condition is a time-range, this works fine. I understand why my current solution does not work, but I can not seem to wrap my head around how to fix it. This example works when both a client and a product is chosen. When the lists are empty the reportViewList is obviously also empty.
I have played with the idea of adding all the records in the date-range and then removing the ones that does not fit, but this would be a bad solution and not efficient.
Any help or feedback is much appreciated.
List<ReportView> reportViews = new List<ReportView>();
using(var dbc = new localContext())
{
reportViewList = dbc.ReportViews.AsEnumerable()
.Where(x => x.OrderDateTime >= from && x.OrderDateTime <= to)
.Where(y => clientList.Any(x2 => x2.Id == y.ClientId)
.Where(z => productList.Any(x3 => x3.Id == z.ProductId)).ToList();
}
You should not call AsEnumerable() before you have added eeverything to your query. Calling AsEnumerable() here will cause your complete data to be loaded in memory and then be filtered in your application.
Without AsEnumerable() and before calling calling ToList() (Better call ToListAsync()), you are working with an IQueryable<ReportView>. You can easily compose it and just call ToList() on your final query.
Entity Framework will then examinate your IQueryable<ReportView> and generate an SQL expression out of it.
For your problem, you just need to check if the user has selected any filters and only add them to the query if they are present.
using var dbc = new localContext();
var reportViewQuery = dbc.ReportViews.AsQueryable(); // You could also write IQuryable<ReportView> reportViewQuery = dbc.ReportViews; but I prefer it this way as it is a little more save when you are refactoring.
// Assuming from and to are nullable and are null if the user has not selected them.
if (from.HasValue)
reportViewQuery = reportViewQuery.Where(r => r.OrderDateTime >= from);
if (to.HasValue)
reportViewQuery = reportViewQuery.Where(r => r.OrderDateTime <= to);
if(clientList is not null && clientList.Any())
{
var clientIds = clientList.Select(c => c.Id).ToHashSet();
reportViewQuery = reportViewQuery.Where(r => clientIds.Contains(y.ClientId));
}
if(productList is not null && productList.Any())
{
var productIds = productList.Select(p => p.Id).ToHashSet();
reportViewQuery = reportViewQuery.Where(r => productIds .Contains(r.ProductId));
}
var reportViews = await reportViewQuery.ToListAsync(); // You can also use ToList(), if you absolutely must, but I would not recommend it as it will block your current thread.
I am getting the error about LINQ to Entities does not recognize the method 'System.String Format but in the past I was able to do this when I have included .AsEnumerable() is there something different I need to do because of the GroupBy section?
select new PresentationLayer.Models.PanelMeeting
{
GroupId = pg.GroupId,
MeetingId = pmd.MeetingId,
GuidelineName = pmv.GuidelineName,
PanelDisclosuresAttendanceURL = string.Format("{0}?MeetingId={1}&GroupId=0",PanelDisclosureLink, pmd.MeetingId),
}).GroupBy(g => new
{
g.MeetingId,
g.GroupId
})
.AsEnumerable()
.SelectMany(grp => grp.AsEnumerable()).ToList(),
You have to be aware of the difference between an IEnumerable<...> and an IQueryable<...>.
IEnumerable
An object that implements IEnumerable<...> represents a sequence of similar items. You can ask for the first element of the sequence, and as long as you've got elements you can ask for the next element. IEnumerable objects are supposed to be executed within your own process. IEnumerable objects hold everything to enumerate the sequence.
At its lowest level, this is done using GetEnumerator() / MoveNext() / Current:
IEnumerable<Customer> customers = ...
IEnumerator<Customer> enumerator = customers.GetEnumerator();
while (enumerator.MoveNext())
{
// There is a next Customer
Customer customer = enumerator.Current;
ProcessCustomer(customer);
}
If you use foreach, then internally GetEnumerator / MoveNext / Current are called.
If you look closely to LINQ, you will see that there are two groups of LINQ methods. Those that return IEnumerable<TResult> and those that dont't return IEnumerable<...>
LINQ functions from the first group won't enumerate the query. They use deferred execution, or lazy execution. In the comments section of every LINQ method, you'll find this description.
The LINQ functions of the other group will execute the query. If you look at the reference source of extension class Enumerable, you'll see that they internally use foreach, or at lower level use GetEnumerator / MoveNext / Current
IQueryable
An object that implements IQueryable<...> seems like an IEnumerable. However, it represents the potential to fetch data for an Enumerable sequence. The data is usually provided by a different process.
For this, the IQueryable holds an Expression and a Provider. The Expression represents what must be fetched in some generic format. The Provider knows who will provide the data (usually a database management system) and how to communicate with this DBMS (usually SQL).
When you start enumerating the sequence, deep inside using GetEnumerator, the Expression is sent to the Provider, who will try to translate it into SQL. The data is fetched from the DBMS, and returned as an Enumerable object. The fetched data is accessed by repeatedly calling MoveNext / Current.
Because the database is not contacted until you start enumerating, you'll have to keep the connection to the database open until you've finished enumerating. You've probably made the following mistake once:
IQueryable<Customer> customers;
using (var dbContext = new OrderDbContext(...))
{
customers = dbContext.Customers.Where(customer => customer...);
}
var fetchedCustomers = customers.ToList();
Back to your question
In your query, you use string.Format(...). Your Provider doesn't know how to translate this method into SQL. Your Provider also doesn't know any of your local methods. In fact, there are even several standard LINQ methods that are not supported by LINQ to entities. See Supported and Unsupported LINQ methods.
How to solve the problem?
If you need to call unsupported methods, you can use AsEnumerable to fetch the data. All LINQ methods after AsEnumerable are executed by your own process. Hence you can call any of your own functions.
Database Management systems are extremely optimized in table handling. One of the slower parts of a database query is the transport of the selected data to your local process. Hence, let the DBMS do all selecting, try to transport as little data as possible.
So let your DBMS do your Where / (Group-)Join / Sum / FirstOrDefault / Any etc. String formatting can be done best by you.
In your String.Format you use PanelDisclosureLink and pmd.MeetingId. It will probably be faster if your process does the formatting. Alas you forgot to give us the beginning or your query.
I'm not sure where your PanelDisclosureLink comes from. Is it a local variable? If that is the case, then PanelDisclosuresAttendanceURL will be the same string for every item in your group. Is this intended?
var panelDisclosureLine = ...;
var result = dbContext... // probably some joining with Pgs, Pmds and Pmvs,
.Select(... => new
{
GroupId = pg.GroupId,
MeetingId = pmd.MeetingId,
GuidelineName = pmv.GuidelineName,
})
// make groups with same combinations of [MeetingId, GroupId]
.GroupBy(joinResult => new
{
MeetingId = joinResult.MeetingId,
GroupId = joinResult.GroupId,
},
// parameter resultSelector: use the Key, and all JoinResult items that have this key
// to make one new:
(key, joinResultItemsWithThisKey) => new
{
MeetingId = key.MeetingId,
GroupId = key.GroupId,
GuideLineNames = joinResultsItemsWithThisKey
.Select(joinResultItem => joinResultItem.GuideLineName)
.ToList(),
})
So by now the DBMS has transformed your join result into objects with
[MeetingId, GroupId] combinations and a list of all GuideLineNames that have belong to
this [MeetingId, GroupId] combination.
Now you can move it to your local process and use String.Format.
.AsEnumerable()
.SelectMany (fetchedItem => fetchedItem.GuideLineNames,
(fetchedItem, guideLineName) => PresentationLayer.Models.PanelMeeting
{
GroupId = fetchedItem.GroupId,
MeetingId = fetchedItem.MeetingId,
GuidelineName = guidelineName,
PanelDisclosuresAttendanceURL = string.Format("...",
PanelDisclosureLink,
fetchedItem.MeetingId);
Note: in my parameter choice plurals are collections; singulars are elements of these collections.
PanelDisclosuresAttendanceURL = string.Format("{0}?MeetingId={1}&GroupId=0",PanelDisclosureLink, pmd.MeetingId),
}).
.GroupBy
If you want to use string.Format you first have to get the data from the server.
You can just move the .GroupBy( ... ) and then the .AsEnumerable() call to the top, before select new PresentationLayer.Models.PanelMeeting { ... }. If you are not selecting too much data that way...
Im curious about how the compiler handles the following expression:
var collapsed = elements.GroupBy(elm => elm.OrderIdentifier).Select(group => new ModelsBase.Laser.Element()
{
CuttingDurationInSeconds = group.Sum(itm => itm.CuttingDurationInSeconds),
FladderDurationInSeconds = group.Sum(itm => itm.FladderDurationInSeconds),
DeliveryDate = group.Min(itm => itm.DeliveryDate),
EfterFladderOpstilTid = group.First().EfterFladderOpstilTid,
EfterRadanOpstilTid = group.First().EfterRadanOpstilTid,
});
As you can see, I'm using group sum twice, so does anyone know if the "group" list will be iterated twice to get both sums, or will it be optimized so there is actually only 1 complete iteration of the list.
LINQ ist most often not the best way to reach high performance, what you get is productivity in programming, you get a result without much lines of code.
The possibilities to optimize is limited. In case of Querys to SQL, there is one rule of thumb: One Query is better than two queries.
1) there is only one round trip to the SQL_Server
2) SQL Server is made to optimize those queries, and optimization is getting better if the server knows, what you want to do in the next step. Optimization is done per query, not over multiple queries.
In case of Linq to Objects, there is absolutely no gain in building huge queries.
As your example shows, it will probably cause multiple iterations. You keep your code simpler and easier to read - but you give up control and therefore performance.
The compiler certainly won't optimize any of that.
If this is using LINQ to Objects, and therefore delegates, the delegate will iterate over each group 5 times, for the 5 properties.
If this is using LINQ to SQL, Entity Framework or something similar, and therefore expression trees, then it's basically up to the query provider to optimize this appropriately.
You can optimise your request by adding two field in the grouping key
var collapsed = elements.GroupBy(elm => new{
OrderIdentifier=elm.OrderIdentifier,
EfterFladderOpstilTid=elm.EfterFladderOpstilTid,
EfterRadanOpstilTid=elm.EfterRadanOpstilTid
})
.Select(group => new ModelsBase.Laser.Element()
{
CuttingDurationInSeconds = group.Sum(itm => itm.CuttingDurationInSeconds),
FladderDurationInSeconds = group.Sum(itm => itm.FladderDurationInSeconds),
DeliveryDate = group.Min(itm => itm.DeliveryDate),
EfterFladderOpstilTid = group.Key.EfterFladderOpstilTid,
EfterRadanOpstilTid = group.Key.EfterRadanOpstilTid,
});
Or by using LET statement
var collapsed = from groupedElement in
(from element in elements
group element by element.OrderIdentifier into g
select g)
let First = groupedElement.First()
select new ModelsBase.Laser.Element()
{
CuttingDurationInSeconds = groupedElement.Sum(itm => itm.CuttingDurationInSeconds),
FladderDurationInSeconds = groupedElement.Sum(itm => itm.FladderDurationInSeconds),
DeliveryDate = groupedElement.Min(itm => itm.DeliveryDate),
EfterFladderOpstilTid = First.EfterFladderOpstilTid,
EfterRadanOpstilTid = First.EfterRadanOpstilTid
};
I have a database that users can run a variety of calculations on. The calculations run on 4 different columns each calculation does not necessarily use every column i.e. calculation1 might turn into sql like
SELECT SUM(Column1)
FROM TABLE
WHERE Column1 is not null
and calculation2 would be
SELECT SUM(Column2)
WHERE Column2 is null
I am trying to generate this via linq and I can get the correct data back by calculating everything every time such as
table.Where(x => x.Column1 != null)
.Where(x => x.Column2 == null)
.GroupBy(x => x.Date)
.Select(dateGroup => new
{
Calculation1 = dateGroup.Sum(x => x.Column1 != null),
Calculation2 = dateGroup.Sum(x => x.Column2 == null)
}
The problem is that my dataset is very large, and so I do not want to perform a calculation unless the user has requested it. I have looked into dynamically generating Linq queries. All I have found so far is PredicateBuilder and DynamicSQL, which appear to only be useful for dynamically generating the Where predicate, and hardcoding the sql query itself as a string with the Sum(Column1) or Sum(Column2) being inserted when necessary.
How would one go about dynamically adding the different parts of the Select query into an anonymous type like this? Or should I be looking at an entirely different way of handling this
You can return your query without executing it, which will allow you to dynamically choose what to return.
That said, you cannot dynamically modify an anonymous type at runtime. They are statically typed at compile time. However, you can use a different return object to allow for dynamic properties without needing an external library.
var query = table
.Where(x => x.Column1 != null)
.Where(x => x.Column2 == null)
.GroupBy(x => x.Date);
You can then dyamically resolve queries with any one of the following:
dynamic
dynamic returnObject = new ExpandoObject();
if (includeOne)
returnObject.Calculation1 = groupedQuery.Select (q => q.Sum(x => x.Column1));
if (includeTwo)
returnObject.Calculation2 = groupedQuery.Select (q => q.Sum (x => x.Column2));
Concrete Type
var returnObject = new StronglyTypedObject();
if (includeOne)
returnObject.Calculation1 = groupedQuery.Select (q => q.Sum(x => x.BrandId));
Dictionary<string, int>
I solved this and kept myself from having to lose type safety with Dynamic Linq by using a hacky workaround. I have a object containing bools that correspond to what calculations I want to do such as
public class CalculationChecks
{
bool doCalculation1 {get;set;}
bool doCalculation2 {get;set;}
}
and then do a check in my select for whether or not I should do the calculation or return a constant, like so
Select(x => new
{
Calculation1 = doCalculation1 ? DoCalculation1(x) : 0,
Calculation2 = doCalculation2 ? DoCalculation2(x) : 0
}
However, this appears to be an edge case with linq to sql or ef, that causes the generated sql to still do the calculations specified in DoCalculation1() and DoCalculation2 and then use a case statement to decide whether or not its going to return the data to me. It runs signficantly slower,40-60% in testing, and the execution plan shows that it uses a much more inefficient query.
The solution to this problem was to use an ExpressionVisitor to go through the expression and remove the calculations if the corresponding bool was false. The code showing how implement this ExpressionVisitor was provided by #StriplingWarrior on this question Have EF Linq Select statement Select a constant or a function
Using both of these solutions together is still not creating sql that runs at 100% the speed of plain sql. In testing it was within 10s of plain sql no matter the size of the test set, and the major portions of the execution plan were the same
Whenever the tags argument is not empty I get a NotSupportedException:
Local sequence cannot be used in LINQ to SQL implementation of query operators except the
Contains() operator.
[WebMethod]
public static object GetAnswersForSurvey(string surveyName, int? surveyYear, IEnumerable<string> tags, IEnumerable<string> benchmarks)
{
IQueryable<DAL.Answer> results = new DataClassesDataContext().Answers
.OrderBy(a => a.Question.Variable);
if (!String.IsNullOrEmpty(surveyName)) results = results.Where(a => a.Survey.Name == surveyName);
if (surveyYear.HasValue) results = results.Where(a => a.Survey.Year == surveyYear.Value);
if (tags.Any()) results = results.Where(answer => answer.Question.Tags.Select(t => t.Label).Intersect(tags).Any());
if (benchmarks.Any()) results = results.Where(answer => benchmarks.Contains(answer.Question.BenchmarkCode));
return results.Select(a => new {
a.Question.Wording,
a.Demographic,
Benchmark = a.Question.BenchmarkCode,
a.Question.Scale,
a.Mean,
a.MEPMean,
a.NSSEMean
});
}
I understand it may not be possible to do it the way I'm trying. If it is impossible, can anyone offer any alternatives?
A number of the general purpose Linq to Object methods are not support within Linq to SQL.
http://msdn.microsoft.com/en-us/library/bb399342.aspx
An alternative might be to complete several sub-queries against sql and then perform your intersection operations as Linq to Objects
I think the reason is that the implementation of System.Data.Linq.Table.Intersect returns an IEnumerable, which in turn does not implement a parameterless version of Any().