We Keep Coding

Faster way to get distinct values in LINQ?

I have a web part in SharePoint, and I am trying to populate a drop-down control with the unique/distinct values from a particular field in a list.
Unfortunately, due to the nature of the system, it is a text field, so there is no other definitive source to get the data values (i.e., if it were a choice field, I could get the field definition and just get the values from there), and I am using the chosen value of the drop-down in a subsequent CAML query, so the values must be accurate to what is present on the list items. Currently the list has arpprox. 4K items, but it is (and will continue) growing slowly.
And, it's part of a sandbox solution, so it is restricted by the user code service time limit - and it's timing out more often than not. In my dev environment I stepped through the code in debug, and it seems like the line of LINQ where I actually get the distinct values is the most time consuming, and I then commented out the call to this method entirely, and the timeouts stop, so I am fairly certain this is where the problem is.
Here's my code:
private void AddUniqueValues(SPList list, SPField filterField, DropDownList dropDownControl)
{
SPQuery query = new SPQuery();
query.ViewFields = string.Format("<FieldRef Name='{0}' />", filterField.InternalName);
query.ViewFieldsOnly = true;
SPListItemCollection results = list.GetItems(query); // retrieves ~4K items
List<string> uniqueValues = results.Cast<SPListItem>().Select(item => item[filterField.Id].ToString()).Distinct().ToList(); // this takes too long with 4K items
uniqueValues.Sort();
dropDownControl.Items.AddRange(uniqueValues.Select(itm => new ListItem(itm)).ToArray());
}
As far as I am aware, there's no way to get "distinct" values directly in a CAML query, so how can I do this more quickly? Is there a way to restructure the LINQ to run faster?
Is there an easy/fast way to do this from the client side? (REST would be preferred, but I'd do JSOM if necessary).
Thought I'd add some extra information here since I did some further testing and found some interesting results.
First, to address the questions of whether the Cast() and Select() are needed: yes, they are.
SPListItemCollection is IEnumerable but not IEnumerable<T>, so we need to cast just to be able to get to use LINQ at all.
Then after it's cast to IEnumerable<SPListItem>, SPListItem is a fairly complex object, and I am looking to find distinct values from just one property of that object. Using Distinct() directly on the IEnumerable<SPListItem> yields.. all of them. So I have to Select() just the single values I want to compare.
So yes, the Cast() and Select() are absolutely necessary.
As noted in the comments by M.kazem Akhgary, in my original line of code, calling ToString() every time (for 4K items) did add some time. But in testing some other variations:
// original
List<string> uniqueValues = results.Cast<SPListItem>().Select(item => item[filterField.Id].ToString()).Distinct().ToList();
// hash set alternative
HashSet<object> items = new HashSet<object>(results.Cast<SPListItem>().Select(itm => itm[filterField.Id]));
// don't call ToString(), just deal with base objects
List<object> obs = results.Cast<SPListItem>().Select(itm => itm[filterField.Id]).Distinct().ToList();
// alternate LINQ syntax from Pieter_Daems answer, seems to remove the Cast()
var things = (from SPListItem item in results select item[filterField.Id]).Distinct().ToList();
I found that all of those methods took multiple tens of seconds to complete. Strangely, the DataTable/DataView method from Pieter_Daems answer, to which I added a bit to extract the values I wanted:
DataTable dt = results2.GetDataTable();
DataView vw = new DataView(dt);
DataTable udt = vw.ToTable(true, filterField.InternalName);
List<string> rowValues = new List<string>();
foreach (DataRow row in udt.Rows)
{
rowValues.Add(row[filterField.InternalName].ToString());
}
rowValues.Sort();
took only 1-2 seconds!
In the end, I am going with Thriggle's answer, because it deals nicely with SharePoint's 5000 item list view threshold, which I will probably be dealing with some day, and it is only marginally slower (2-3 seconds) than the DataTable method. Still much, much faster than all the LINQ.
Interesting to note, though, that the fastest way to get distinct values from a particular field from a SPListItemCollection seems to be the DataTable/DataView conversion method.

You're potentially introducing a significant delay by retrieving all items first before checking for distinctness.
An alternative approach would be to perform multiple CAML queries against SharePoint; this would result in one query per unique value (plus one final query that returns no results).
Make sure your list has column indexing applied to the field whose values you want to enumerate.
In your initial CAML query, sort by the field you want to enumerate and impose a row limit of one item.
Get the value of the field from the item returned by that query and add it to your collection of unique values.
Query the list again, sorting by the field and imposing a row limit of 1, but this time add a filter condition such that it only retrieves items where the field value is greater than the field value you just detected.
Add the value of the field in the returned item to your collection of unique values.
Repeat steps 4 and 5 until the query returns an empty result set, at which point your collection of unique values should contain all current values of the field (assuming more haven't been added since you started).
Will this be any faster? That depends on your data, and how frequently duplicate values occur.
If you have 4000 items and only 5 unique values, you'll be able to gather those 5 values in only 6 lightweight CAML queries, returning a total of 5 items. This makes a lot more sense than querying for all 4000 items and enumerating through them one at a time to look for unique values.
On the other hand, if you have 4000 items and 3000 unique values, you're looking at querying the list 3001 times. This might well be slower than retrieving all the items in a single query and using post-processing to find the unique values.

var distinctItems = (from SPListItem item in items select item["EmployeeName"]).Distinct().ToArray();
Or convert your results to DataView and do something like:
SPList oList = SPContext.Current.Web.Lists["ListName"];
SPQuery query = new SPQuery();
query.Query = "<OrderBy><FieldRef Name='Name' /></OrderBy>";
DataTable dtcamltest = oList.GetItems(query).GetDataTable();
DataView dtview = new DataView(dtcamltest);
DataTable dtdistinct = dtview.ToTable(true, "Name");
Source: https://sharepoint.stackexchange.com/questions/77988/caml-query-on-sharepoint-list-without-duplicates

Duplicate maybe?
.Distinct is an O(n) call.
You can't get any faster than that.
This being said, maybe you want to check if you need the cast + select for getting uniques - I'd try a HashSet.

Which is faster between Linq to Sql And SQl Query

I have List of object like this
List<Product> _products;
Then I get productId input and search in this list like this
var target = _peoducts.Where(o => o.productid == input).FirstOrDefault();
my Question is
If This list have 100 Products (productId from 1 to 100) and an
input I get productId = 100. that mean this Method must loop for 100
time Right ? (If I ORDER BY productId ASC in Query)
Between use this Method and Query on Database with where clause like
this WHERE productId = #param
Thank you.

No. If there is an index with key productId it finds the correct row with O(log n) operations
Just implement both methods and take the time. (hint: use StopWatch() class)
Edit
To get the full performance you should not create an intermediate (unsorted) List<T> but put all your logic in a LINQ query which operates on the SQL Server.

#might be helpful to get your answer.
https://www.linqpad.net/WhyLINQBeatsSQL.aspx

If you execute that Where on a List<Product>, then:
you got all 100 rows from the database
and then looped through all products in memory until you found the one that matches or until you went through the entire list and found nothing.
If, on the other hand, you used an IQueryable<Product> that was connected to the database table, then:
You wouldn't have read anything from the database yet
When you apply the Where, you still wouldn't read anything
When you apply the FirstOrDefault a sql query is constructed to find just the one row you need. Given correct indexes on the table, this would be quite fast.

Keeping count of items in a category

New to MVC so apologies if a quick google would have answered this, links to articles explaining this or examples where this is used would be appreciated!
Basically, I have a list of products in categories/sub categories, and want to be able to output the a list of them with the number of products in each sub category such as:
Computers
- Laptops (4)
- Tablets (9)
- Netbooks (3)
I am using EF Code First, and I'm not sure how best to organise this.
I could iterate through my list of products and get a count for each one, but this seems inefficient. I could also store a count variable in the model for each subcategory, and increment it each time a new product is added.
Basically, I'm after a 'best-practice' approach to knowing how to do this, so I can learn to do it right from the start.

You could have the database perform the querying:
var model = db
.Categories
.Select(c => new MyViewModel
{
Name = c.Name,
ProductsCount = c.Products.Count()
})
.ToList();
Then inside your view you will have the name and total products count in the view model so that when you are displaying the list you will be able to show this information.

What is the best C# Data Structure(s) For the Following Situation

The requirements for my application are as follows. I need to store orders which look like this:
Each order pertains to a specific stockcode(string) and has a price, volume and whether or not it is being bought or sold(boolean) associated with it.
I need to do several operations on all orders that pertain to a specific stock, for example get the sum of the volume of orders for stockcode "abc".
I need to be able to add an order to the data structure
I need to be able to remove an order from the data structure
I need to be able to find out which order is offering the best price after an order is added or removed.
Here is what I am thinking so far:
public class Order : IComparable
{
private string _StockCode;
private bool _BidSide;
private int _Volume;
private decimal _Price;
private int _ExchangeOrderId;
public int CompareTo(Order other)
{
if (_BidSide != other.BidSide)
{
return _BidSide ? 1 : -1;
}
return decimal.Compare(_Price, other.Price);
}
}
And then I would store the orders in a Dictionary<string, List<Order>>. Where each stock code would be a key in the dictionary pointing to a list of orders for that stock. I would also maintain Dictionary matching an order id to a stock code.
For adding a new order, I simply find the appropriate list of orders in the dictionary based on the current stock code, and insert the order. I would also add an entry in the orderstock dictionary matching the current order with the approrpriate list.
For finding the best price, I look up the order list in the dictionary for the current stock code, sort the list and print out the highest order.
Removing is tricky. I would first need to look up the appropriate list by stock code. I would then need to iterate through all the orders for that stock code and find the one that matches the current order id and remove it. This is obviously inefficient if there are a lot of orders for the current stock code. Is this the best way of storing this information?

If you're going to do this with a lot of data, put it in a database. This is not something you want to do in a class.
However, if you are using a small set of data, you could do this in code using LINQ.
I think you should make Order implement IEnumerable and then use a List<Order> to store your orders. Make StockCode a public property on the Order and then you can retrieve orders by using Linq:
List<Order> orders = GetOrderList();
var ibmOrders = from o in orders
where o.StockCode == "IBM"
select o;
Removing items from the list is quite simple:
List<Order> orders = GetOrderList();
var orderToRemove = (from o in orders
where o.ExchangeId == 1315
select o).FirstOrDefault();
if (orderToRemove != null) {
orders.Remove(orderToRemove);
}
Finding by best price using Linq is quite nice:
Order bestPricedOrder = (from o in orders
orderby Price
select o).FirstOrDefault();
For more great LINQ tricks, see 101 LINQ Samples.

I would add an additional dictionary, which consists of key = orderid, value = reference to order in the list in the initial dictionary of stock codes.
This will act like an index and give you constant time deletion. Assuming you Order ID is distinct it will map 1:1. Just make sure you delete it from both dictionaries.
As suggested in comments I would recommend an additional dictionary of the computed sums that you need accessible by stock code. This is trading off constant time access for memory. Unless memory is an issue this would seem favourable to calculating it every time you need it.
If you get a new order in you can just update the sums, averages, etc. Just keep in mind if you are doing stuff in parallel you'll need some locking to ensure you don't have issues.

I agree with the comments that a database would be the best option; they are designed for this type of thing.
If you needed to hold onto this data in memory, and there really are a lot of Orders per code, then I'd go with a Dictionary<string, SortedSet<Order>>. The SortedSet will make finding the min/max easy, as well as quick insertion/removal.

Problem merging IEnumerable<T> using Lambda expressions

I have 2 IEnumerables which are of type 'Product' - totalProducts and ProductsChosen. As the names suggest totalProducts holds all the products available and ProductsChosen holds the products chosen by the user .
The Product class has properties Id ,number, and IEnumerable<AdditonalOptions> options.
I am trying to merge these 2 IEnumerables like this
public IEnumerable<Product> MergeProducts(IEnumerable<Product> Products, IEnumerable<Product> ProductsChosen)
{
return ProductsChosen.Concat(Products.Where(r => !ProductsChosen.Any(x => x.ProductId.Equals(r.ProductId))));
}
I have used the code to merge from here
This code works fine only if I consider the product ids but I would also like to merge the IEnumerable<AdditionalOption> options of the productsChosen with that of the totalProducts if the Product Ids are same.But I am not sure how to do that.
Could someone please help me with that?
Thanks

As the chosen products have different options that the total products they must be distinct instances. Since you have said that total products are all that are available then I assume that the set of chosen product ids must be a subset of the product ids.
Also, since the chosen products are distinct instances then it is possible that there may be zero or more chosen products per each total product so my solution works for this case too. It will equally work if there are only zero or one.
Further, since they are distinct instances and the options property is an IEnumerable it made sense to me to create a new set of instances for the merged products so as to not alter the original set of total products.
public IEnumerable<Product> MergeProducts(
IEnumerable<Product> Products,
IEnumerable<Product> ProductsChosen)
{
return
from p in Products
join pc in ProductsChosen on p.Id equals pc.Id into pcs
select new Product()
{
Id = p.Id,
number = p.number,
options = p.options
.Concat(
from pc2 in pcs
from ao in pc2.options
select ao).ToArray(),
};
}
You didn't explain what number property is, so I've just done a simple assignment for this value from the Products list and ignored this value from the ProductsChosen list.
Also, you didn't say how the AdditionalOptions needed to be merged. This is no Merge method for enumerables. You need to use something like Concat or Union to do a merge. I've chosen Concat but that easy to change.
Does this work for you?