I've been struggling and trying to get my mind wrapped around these concepts. Any help would be greatly appreciated. Given a collection of Orders in one table what is the best way to iterate over it, and perform queries against other tables.
This loop does what I want, but I've been told and shown how this causes performance problems. On other posts I've read where it is best to query the dbase outside of loops.
foreach (var item in myOrders)
{
var lookUpValues=(from f in ORDERS
where f.ORDERS ==item.ORDER_ID
&& f.ORDERS_SUFFIX==item.LINE
select f.Something);
}
Rather than pulling down the entire Order table prior to iterating over I've tried something like this, but as you see it won't return the accurate rows.
var orderIds=myOrders.Select (x =>x.ORDER_ID).Distinct().ToList();
var lineIds=myOrders.Select (x =>x.LINE).Distinct().ToList();
var query=(from f in ORDERS
where orderIds.Contains(f.ORDERS)
&& lineIds.Contains(f.ORDERS_SUFFIX)
select f).ToList();
The below would do what I think is needed. It would return the correct Orders that are required to iterate over. However, it throws an error, which I believe is because you can't query memory objects and entities together.
var ordersAndLine=(from f in myOrders select new{sId=f.ORDER_ID,line=f.LINE}).ToList();
var query=(from f in ORDERS
from s in ordersAndLine
where s.sId==f.ORDERS
&& s.line==f.ORDERS_SUFFIX
select f).ToList();
NotSupportedException: Unable to create a constant value of type 'Anonymous type'. Only primitive types ('such as Int32, String, and Guid') are supported in this context.
I really hope my question makes sense. I apologize in advance if I didn't explain my situation correctly. What I've been doing is pulling down much of the table to memory to iterate over, but I was concerned that this would cause performance issues and wanted to see if there is a better way to filter the result query.
If you combine your first attempt and your last attempt you'd have it. The last attempt would be fine if you weren't creating an anonymous type.
var ordersAndLine=(from f in myOrders select f).ToList();
var query=(from f in ORDERS
from s in ordersAndLine
where s.ORDER_ID==f.ORDERS
&& s.LINE==f.ORDERS_SUFFIX
select f).ToList();
Have a look at these:
http://social.msdn.microsoft.com/forums/en-US/adodotnetentityframework/thread/095745fe-dcf0-4142-b684-b7e4a1ab59f0/
Linq to Entities - SQL "IN" clause
Try this:
var orders = myOrders.SelectMany(item => (from f in ORDERS
where f.ORDERS ==item.ORDER_ID
&& f.ORDERS_SUFFIX==item.LINE
select f.Something));
Related
var ids = IdsList.Select(pID => pID.ID).ToArray();
var response= await MyService.GetByMyQuery(
new QueryDefinition(
"SELECT * FROM p WHERE p.id NOT IN(#ids)"
)
.WithParameter("#ids", string.Join(",", ids)),
);
So this is not working. The operator returns all the items, instead of just the ones not in the list. In the cosmos DB SQL query editor i can easily do
SELECT * FROM p WHERE p.id NOT IN("id1","id2")
and it returns the expected results without any problems. SO i guess that the problem is in the code layer, the way I'm passing the ids to the WithParameter() menthod.
Any insights is greatly appreciated.
The problem
Your C# code is not sending multiple values as the #ids parameter, but a single string value, effectively like this:
SELECT * FROM p
WHERE p.id NOT IN("id1, id2")
Since this compound id does not exist then it returns all items like you observed.
Solution
It may be possible with IN keyword as well, but I do know for sure that this pattern will work:
SELECT * FROM p
WHERE NOT ARRAY_CONTAINS(#ids, p.id)
NB! Correct me if I'm mistaken, but most likely this condition will NOT be served by index. So, you may want to reconsider your design, unless your real case would have an additional good indexable predicate, or else it will be slow and costly.
I am still in the midst of converting our web app from vb to C# and this complex query, that I just finished converting and have tested and it works properly, needs refactoring.
This section could probably perform a little better if lambdas were used (I've read in numerous places that lambda expressions perform faster)
foreach (var datapersymbol in (from symId in symblist
select (
from q in ctx.HistoricalPriceData.Where(q => q.SymbolId == symId)
join x in ctx.Symbols on q.SymbolId equals x.SymbolId
orderby q.Date descending
select new {q.Date, q.LastPrice, x.Symbol1})).ToList())
As you can see I've added lambdas where I thought appropriate but Im certain they can be in other parts of this expressions as well. I'm still getting up to speed in C# so Im not an expert here.
Assuming you have a navigation property on HistoricalPriceData to Symbols called "Symbol":
foreach(var datapersymbol in ctx
.HistoricalPriceData
.Where(h=>symblist.Contains(h.SymbolId))
.OrderByDescending(h=>h.Date)
.Select(h=>new {h.Date,h.LastPrice,h.Symbol.Symbol1}))
As Rahul pointed out, there is no performance difference between lamda/method syntax and query syntax, however, this will perform slightly better because it drops the .ToList(), which will allow the loop to begin processing as soon as it gets the first record from the database rather than waiting for the entire result (and it doesn't need to create a List). I've also simplified it somewhat by using .Contains instead of a subquery as well, and switched to using navigation properties rather than a explicit join which makes it easier to read/more maintainable.
For the clarity of the code I will rather refactor it like below;
var datapersymbols = from symId in symblist
select (
from q in ctx.HistoricalPriceData.Where(q => q.SymbolId == symId)
join x in ctx.Symbols on q.SymbolId equals x.SymbolId
orderby q.Date descending
select new {q.Date, q.LastPrice, x.Symbol1});
foreach (var datapersymbol in datapersymbols)
{
}
As far as I can see, in this case, there is not much that you can gain from the C# side. You have to check the query plan for the SQL query and then add necessary indexes. The SymbolId field should be an indexed field.
I'm trying to join a row of parent data to a related collection which has been squeezed into a single piece of data.
I have an IQueryable of Orders:
IQueryable<Order> orderList = context.Set<Order>
.Where("OrderDate >= #0", startDate)
.Where("OrderDate <= #0", endDate)
and a related IList<KeyValuePair<int, string>> where each KVP contains the OrderID and a concatenated string of the product names from each Order Line. I want to join the Value from the correct KeyValuePair to the Order info based on the OrderID Key.
To illustrate, the desired output would look something like:
OrderNum OrderDate Customer State OrderTotal Products_Ordered
12345 12/12/2012 J.Bloggs WA $25.50 Bolts, Hammer, Suregrip Clamp
I am trying a linq join that looks like this:
IQueryable result = from o in orders
join line in orderLines on o.OrderID equals line.Key
select new
{
o.OrderNum
o.OrderDate,
o.Customer.CustomerFullName,
o.DeliverAddress.State,
o.TotalPrice,
line.Value
}
The method performing the join seems to work, but when I access the returned IQueryable, I get a NotSupportedException: Unable to create a constant value of type 'System.Collections.Generic.KeyValuePair`2'. Only primitive types or enumeration types are supported in this context.
What am I doing wrong?
Your IQueryable is actually a specialized entity framework implementation that, when iterated, attempts to construct an SQL query by examining an expression tree, execute this query, and return an enumerable over the results. This is fragile, and your projections and queries can't be arbitrarily complex, or the expression -> SQL converter has no idea what to do with it.
Fixing this by materializing your IQueryable first is fine, but you don't even really need to do that. Why have a list of what is essentially tuples, when what you want is a dictionary that maps the order ID to a bunch of data?
IDictionary<int, string> orderLines = new Dictionary<int, string>();
// Add a dummy item
orderLines[1234] = "Hello, this, is, a, test";
// Get your combined view
// Assuming you have an order with 1234 as the ID, this should work
var result = from o in orders
select new
{
o.OrderNum
o.OrderDate,
o.Customer.CustomerFullName,
o.DeliverAddress.State,
o.TotalPrice,
Products = orderLines[o.OrderID]
}
Entity Framework has issues with some types, because it tries to push them to the backing database. The quick and easy solution is to pull the data down before performing the join.
var orderList = context.Set<Order>
.Where("OrderDate >= #0", startDate)
.Where("OrderDate <= #0", endDate)
.ToList(); // ------> Relevant line <------
var result = from o in orderList
join line in orderLines on o.OrderID equals line.Key
select new
{
o.OrderNum
o.OrderDate,
o.Customer.CustomerFullName,
o.DeliverAddress.State,
o.TotalPrice,
line.Value
};
It's just a change in where the joining happens, RDBMS- or client-side.
Performance won't be an issue, assuming you don't have a ton of data and all rows will be matched. Of course, you will have to pull down data that doesn't get joined, so if only a few of orderList will be in result, that might be worth a second thought, design-wise. The only alternative would be in pushing the KeyValuePair items to the server first, which probably isn't what you're looking for.
Joining a query result (IQueryable) with a normal enumerable stored in memory (IEnumerable<...>) doesn't make much sense. Think about it, the query doesn't get processed until it's materialized, injecting your own data in queries is done in the query text itself -- that's not what you really want, now is it?
I think what you expect from this is best achieved by first materializing the IQueryable into an IEnumerable<>, then doing a plain LINQ join on two IEnumerable<>s, which is trivial.
It's not like you'd be using the IEnumerable<> to filter the result set on the server side, you're not really losing any performance here.
Edit: Note that if you are using the IEnumerable<> to filter the results on the server side, you can do that! EF (I assume that's what you're using) has very strong special cases for things like IQueriable<>.Any<>() with an IEnumerable<>.Contains<>() inside it -- it inserts the literal values in the query text. It's just the actual join that doesn't make much sense in this context.
I guess it should be really simple, but i cannot find how to do it.
I have a linq query, that selects one column, of type int, and i need it sorted.
var values = (from p in context.Products
where p.LockedSince == null
select Convert.ToInt32(p.SearchColumn3)).Distinct();
values = values.OrderBy(x => x);
SearchColumn3 is op type string, but i only contains integers. So i thought, converting to Int32 and ordering would definitely give me a nice 1,2,3 sorted list of values. But instead, the list stays ordered like it were strings.
199 20 201
Update:
I've done some tests with C# code and LinqPad.
LinqPad generates the following SQL:
SELECT [t2].[value]
FROM (
SELECT DISTINCT [t1].[value]
FROM (
SELECT CONVERT(Int,[t0].[SearchColumn3]) AS [value], [t0].[LockedSince], [t0].[SearchColumn3]
FROM [Product] AS [t0]
) AS [t1]
WHERE ([t1].[LockedSince] IS NULL)
) AS [t2]
ORDER BY [t2].[value]
And my SQL profiler says that my C# code generates this piece of SQL:
SELECT DISTINCT a.[SearchColumn3] AS COL1
FROM [Product] a
WHERE a.[LockedSince] IS NULL
ORDER BY a.[SearchColumn3]
So it look like C# Linq code just omits the Convert.ToInt32.
Can anyone say something useful about this?
[Disclaimer - I work at Telerik]
You can solve this problem with Telerik OpenAccess ORM too. Here is what i would suggest in this case.
var values = (from p in context.Products
where p.LockedSince == null
orderby "cast({0} as integer)".SQL<int>(p.SearchColumn3)
select "cast({0} as integer)".SQL<int>(p.SearchColumn3)).ToList().Distinct();
OpenAccess provides the SQL extension method, which gives you the ability to add some specific sql code to the generated sql statement.
We have started working on improving this behavior.
Thank you for pointing this out.
Regards
Ralph
Same answer as one my other questions, it turns out that the Linq provider i'm using, the one that comes with Telerik OpenAccess ORM does things different than the standard Linq to SQL provider! See the SQL i've posted in my opening post! I totally wasn't expecting something like this, but i seem that the Telerik OpenAccess thing still needs a lot of improvement. So be careful before you start using it. It looks nice, but it has some serious shortcomings.
I can't replicate this problem. But just make sure you're enumerating the collection when you inspect it. How are you checking the result?
values = values.OrderBy(x => x);
foreach (var v in values)
{
Console.WriteLine(v.ToString());
}
Remember, this won't change the order of the records in the database or anywhere else - only the order that you can retrieve them from the values enumeration.
Because your values variable is a result of a Linq expression, so that it doest not really have values until you calling a method such as ToList, ToArray, etc.
Get back to your example, the variable x in OrderBy method, will be treated as p.SearchColumn3 and therefore, it's a string.
To avoid that, you need to let p.SearchColumn3 become integer before OrderBy method.
You should add a let statement in to your code as below:
var values = (from p in context.Products
where p.LockedSince == null
let val = Convert.ToInt32(p.SearchColumn3)
select val).Distinct();
values = values.OrderBy(x => x);
In addition, you can combine order by statement with the first, it will be fine.
I have a local collection of record Id's (integers).
I need to retrieve records that have every one of their child records' ids in that local collection.
Here is my query:
public List<int> OwnerIds { get; private set; }
...
filteredPatches = from p in filteredPatches
where OwnerIds.All(o => p.PatchesOwners.Select(x => x.OwnerId).Contains(o))
select p;
I am getting this error:
Local sequence cannot be used in Linq to SQL implementation of query operators except the Contains() operator.
I understand that .All() isn't supported by Linq to SQL, but is there a way to do what I am trying to do?
Customers where
OrderIds in the child collection are a subset of the IDs in the in-memory collection.
from c in myDC.Customer
where c.Orders.All(o => myList.Contains(o.ID))
select c;
Customers where
OrderIds in the in-memory collection are a subset of the IDs in the child collection.
from c in myDC.Customers
where (from o in c.Orders
where myList.Contains(o.ID)
group o.ID by o.ID).Distinct().Count() == myList.Count()
select c;
Customers where
OrderIds in the in-memory collection are set-equal to the IDs in the child collection.
from c in myDC.Customers
let Ids = c.Orders.Select(o => o.ID).Distinct()
where Ids.Count() == myList.Count()
&& Ids.All(id => myList.Contains(id))
select c;
All of these generated sql for me.
PS - these presume the IDs are already distinct in myList. If they aren't yet, use:
myList = myList.Distinct().ToList();
PSS - good for lists up to ~2000 items. Higher than that will get translated to sql, and then sql server will barf at the number of parameters.
I don't know of a way to do it with Linq to SQL. The problem is that you need to get your list over to the server so that it can query against it. (your list is in memory on your machine, SQL Server needs to do the filtering on the server)
With straight SQL, you could use a regular SELECT statement with the "in()" operator to do that. (don't go over 1,000 items in the "in")
You could insert all of the ID's into a temp table in SQL, and then join to the table (you could use LINQ with this solution, but it requires 2 steps - the insert (assuming you have a "sets" table), and then the joined query (and then a cleanup query to remove your set).
You could LINQ query without the filter condition and then filter on your in-memory set (not recommended if the unfiltered result set could be large).
what the compiler says is...
OwnerIds.Contains(someVariable)
is supported and it will be translated as:
WHERE someVariable IN (OwnerId1, OwnerId2, OwnerIdN)
now, we don't have all the informations of you query but if you can reformulate what you're trying to do to use Contains, you'll be ok.
Could you do a join OwnerIds?
Error said "Local sequence (means OwnerIds) cannot be used in Linq to SQL implementation of query operators except the Contains() operator."
So you can do:
1) load ALL filteredPatches rows from SQL
var loadedData = filteredPatches.Select(i => i).ToList();
2) filter data as simple local sequence
var result = loadedData.Where(i => i.PatchesOwners.All(o => OwnerIds.Contains(o.ID)));