We Keep Coding

Jarray GroupBy using multiple columns

I have this issue that I am currently stuck with in C#.
I have about 31 columns of data within Jobject inside Jarray (JArray tableJson = new JArray();)
I would like to group them into three columns.
So far I can only group by one of the columns
eg :
var tableJsonGroup = tableJson.GroupBy(x => x["FirstColumn"]).ToList();
I want to do something like this (it does not work) :
var tableJsonGroup = tableJson.GroupBy(x => new {x["FirstColumn"], x["SecondColumn"], x["FifthColumn"]}).ToList();
How do I do this?
Thank you.

As we can see, this overload of the GroupBy extension takes a delegate which is used to enumerate the tableJson and generate a key for each item.
Items that generate the same key will be grouped together and returned as an IGrouping. There is no special treatment based on the type of the source. This doesn't do anything different, whether you have an array of ints or an array of complex objects.
So, if you want to group by a combination of columns you need to provide a function that returns a unique, repeatable, key for that combination of columns.
This can be simply achieved by compounding those columns into an anonymous type, which has a built in implementation for equality and hashing that suits our purposes, like in this answer.
var groupedTableJson = tableJson.GroupBy(x =>
new {
FirstColumn: x["FirstColumn"],
SecondColumn: x["SecondColumn"],
FifthColumn: x["FifthColumn"]
});
Your answer is almost right but, you don't provide names for properties of your anonymous type. However, since you don't explain what "does not work", it is hard to be sure.

Query ODataV4 connected service with LINQ - Get last record from table

Im trying to query my OData webservice from a C# application.
When i do the following:
var SecurityDefs = from SD in nav.ICESecurityDefinition.Take(1)
orderby SD.Entry_No descending
select SD;
i get an exception because .top() and .orderby is not supposed to be used together.
I need to get the last record in the dataset and only the last.
The purpose is to get the last used entry number in a ledger and then continue creating new entries incrementing the found entry no.
I cant seem to find anything online that explains how to do this.
Its very important that the service only returns the last record from the feed since speed is paramount in this solution.

i get an exception because .top() and .orderby is not supposed to be used together.
Where did you read that? In general .top() or .Take() should ONLY be used in conjunction WITH .orderby(), otherwise the record being retrieved is not guaranteed to be repeatable or predictable.
Probably the compounding issue here is mixing query and fluent expression syntax, which is valid, but you have to understand the order of precedence.
Your syntax is taking 1 record, then applying a sort order... you might find it easier to start with a query like this:
// build your query
var SecurityDefsQuery = from SD in nav.ICESecurityDefinition
orderby SD.Entry_No descending
select SD;
// Take the first item from the list, if it exists, will be a single record.
var SecurityDefs = SecurityDefsQuery.FirstOrDefault();
// Take an array of only the first record if it exists
var SecurityDefsDeferred = SecurityDefsQuery.Take(1);
This can be executed on a single line using brackets, but you can see how the query is the same in both cases, SecurityDefs in this case is a single ICESecurityDefinition typed record, where as SecurityDefsDeferred is an IQueryable<ICESecurityDefinition> that only has a single record.
If you only need the record itself, you this one liner:
var SecurityDefs = (from SD in nav.ICESecurityDefinition
orderby SD.Entry_No descending
select SD).FirstOrDefault();
You can execute the same query using fluent notation as well:
var SecurityDefs = nav.ICESecurityDefinition.OrderByDescending(sd => sd.Entry_No)
.FirstOrDefault();
In both cases, .Take(1) or .top() is being implemented through .FirstOrDefault(). You have indicated that speed is important, so use .First() or .FirstOrDefault() instead of .Single() or .SingleOrDefault() because the single variants will actually request .Take(2) and will throw an exception if it returns 1 or no results.
The OrDefault variants on both of these queries will not impact the performance of the query itself and should have negligble affect on your code, use the one that is appriate for your logic that uses the returned record and if you need to handle the case when there is no existing record.
If the record being returned has many columns, and you are only interested in the Entry_No column value, then perhaps you should simply query for that specific value itself:
Query expression:
var lastEntryNo = (from SD in nav.ICESecurityDefinition
orderby SD.Entry_No descending
select SD.Entry_No).FirstOrDefault();
Fluent expression:
var lastEntryNo = nav.ICESecurityDefinition.OrderByDescending(sd => sd.Entry_No)
.Select(sd => sd.Entry_No)
.FirstOrDefault();
If Speed is paramount then look at providing a specific custom endpoint on the service to either serve the record or do not process the 'Entry_No` in the client at all, make that the job of the code that receives data from the client and compute it at the time the entries are inserted.
Making the query perform faster is not the silver bullet you might be looking for though, Even if this is highly optimised, your current pattern means that X number of clients could all call the service to get the current value of Entry_No, meaning all of them would start incrementing from the same value.
If you MUST increment the Entry_No from the client then you should look at putting a custom endpoint on the service to simply return the Next Entry_No to use. This should be optimistic meaning that you don't care if the Entry_No actually gets used in the end, but you can implement the end point such that every call will increment the field in the database and return the next value.
Its getting a bit beyond the scope of your initial post, but SQL Server now has support for Sequences that formalise this type of logic from a database and schema point of view, using Sequence simplifies how we can manage these types of incrementations from the client, because we no longer rely on the outcome of data updates to be comitted to the table before the client can increment the next record. (which is what your TOP, Order By Desc solution is trying to do.

Faster way to get distinct values in LINQ?

I have a web part in SharePoint, and I am trying to populate a drop-down control with the unique/distinct values from a particular field in a list.
Unfortunately, due to the nature of the system, it is a text field, so there is no other definitive source to get the data values (i.e., if it were a choice field, I could get the field definition and just get the values from there), and I am using the chosen value of the drop-down in a subsequent CAML query, so the values must be accurate to what is present on the list items. Currently the list has arpprox. 4K items, but it is (and will continue) growing slowly.
And, it's part of a sandbox solution, so it is restricted by the user code service time limit - and it's timing out more often than not. In my dev environment I stepped through the code in debug, and it seems like the line of LINQ where I actually get the distinct values is the most time consuming, and I then commented out the call to this method entirely, and the timeouts stop, so I am fairly certain this is where the problem is.
Here's my code:
private void AddUniqueValues(SPList list, SPField filterField, DropDownList dropDownControl)
{
SPQuery query = new SPQuery();
query.ViewFields = string.Format("<FieldRef Name='{0}' />", filterField.InternalName);
query.ViewFieldsOnly = true;
SPListItemCollection results = list.GetItems(query); // retrieves ~4K items
List<string> uniqueValues = results.Cast<SPListItem>().Select(item => item[filterField.Id].ToString()).Distinct().ToList(); // this takes too long with 4K items
uniqueValues.Sort();
dropDownControl.Items.AddRange(uniqueValues.Select(itm => new ListItem(itm)).ToArray());
}
As far as I am aware, there's no way to get "distinct" values directly in a CAML query, so how can I do this more quickly? Is there a way to restructure the LINQ to run faster?
Is there an easy/fast way to do this from the client side? (REST would be preferred, but I'd do JSOM if necessary).
Thought I'd add some extra information here since I did some further testing and found some interesting results.
First, to address the questions of whether the Cast() and Select() are needed: yes, they are.
SPListItemCollection is IEnumerable but not IEnumerable<T>, so we need to cast just to be able to get to use LINQ at all.
Then after it's cast to IEnumerable<SPListItem>, SPListItem is a fairly complex object, and I am looking to find distinct values from just one property of that object. Using Distinct() directly on the IEnumerable<SPListItem> yields.. all of them. So I have to Select() just the single values I want to compare.
So yes, the Cast() and Select() are absolutely necessary.
As noted in the comments by M.kazem Akhgary, in my original line of code, calling ToString() every time (for 4K items) did add some time. But in testing some other variations:
// original
List<string> uniqueValues = results.Cast<SPListItem>().Select(item => item[filterField.Id].ToString()).Distinct().ToList();
// hash set alternative
HashSet<object> items = new HashSet<object>(results.Cast<SPListItem>().Select(itm => itm[filterField.Id]));
// don't call ToString(), just deal with base objects
List<object> obs = results.Cast<SPListItem>().Select(itm => itm[filterField.Id]).Distinct().ToList();
// alternate LINQ syntax from Pieter_Daems answer, seems to remove the Cast()
var things = (from SPListItem item in results select item[filterField.Id]).Distinct().ToList();
I found that all of those methods took multiple tens of seconds to complete. Strangely, the DataTable/DataView method from Pieter_Daems answer, to which I added a bit to extract the values I wanted:
DataTable dt = results2.GetDataTable();
DataView vw = new DataView(dt);
DataTable udt = vw.ToTable(true, filterField.InternalName);
List<string> rowValues = new List<string>();
foreach (DataRow row in udt.Rows)
{
rowValues.Add(row[filterField.InternalName].ToString());
}
rowValues.Sort();
took only 1-2 seconds!
In the end, I am going with Thriggle's answer, because it deals nicely with SharePoint's 5000 item list view threshold, which I will probably be dealing with some day, and it is only marginally slower (2-3 seconds) than the DataTable method. Still much, much faster than all the LINQ.
Interesting to note, though, that the fastest way to get distinct values from a particular field from a SPListItemCollection seems to be the DataTable/DataView conversion method.

You're potentially introducing a significant delay by retrieving all items first before checking for distinctness.
An alternative approach would be to perform multiple CAML queries against SharePoint; this would result in one query per unique value (plus one final query that returns no results).
Make sure your list has column indexing applied to the field whose values you want to enumerate.
In your initial CAML query, sort by the field you want to enumerate and impose a row limit of one item.
Get the value of the field from the item returned by that query and add it to your collection of unique values.
Query the list again, sorting by the field and imposing a row limit of 1, but this time add a filter condition such that it only retrieves items where the field value is greater than the field value you just detected.
Add the value of the field in the returned item to your collection of unique values.
Repeat steps 4 and 5 until the query returns an empty result set, at which point your collection of unique values should contain all current values of the field (assuming more haven't been added since you started).
Will this be any faster? That depends on your data, and how frequently duplicate values occur.
If you have 4000 items and only 5 unique values, you'll be able to gather those 5 values in only 6 lightweight CAML queries, returning a total of 5 items. This makes a lot more sense than querying for all 4000 items and enumerating through them one at a time to look for unique values.
On the other hand, if you have 4000 items and 3000 unique values, you're looking at querying the list 3001 times. This might well be slower than retrieving all the items in a single query and using post-processing to find the unique values.

var distinctItems = (from SPListItem item in items select item["EmployeeName"]).Distinct().ToArray();
Or convert your results to DataView and do something like:
SPList oList = SPContext.Current.Web.Lists["ListName"];
SPQuery query = new SPQuery();
query.Query = "<OrderBy><FieldRef Name='Name' /></OrderBy>";
DataTable dtcamltest = oList.GetItems(query).GetDataTable();
DataView dtview = new DataView(dtcamltest);
DataTable dtdistinct = dtview.ToTable(true, "Name");
Source: https://sharepoint.stackexchange.com/questions/77988/caml-query-on-sharepoint-list-without-duplicates

Duplicate maybe?
.Distinct is an O(n) call.
You can't get any faster than that.
This being said, maybe you want to check if you need the cast + select for getting uniques - I'd try a HashSet.

Union IQueryable and List

This is my code:
var query = context.SomeEntities.Select(e => new SomeDto
{
Name = e.Title,
})
The variable query is IQueryable.
var list = new List<SomeDto>
{
new SomeDto
{
Name = "SomeName1"
},
new SomeDto
{
Name = "SomeName2"
}
};
Now I need union these collections,I write
query = query.Union(list)
But in this place I get exception
Unable to create a constant value of type 'SomeDto'. Only primitive types or enumeration types are supported in this context
I know that I can do ToList() for first collection,but I need union collections without call to database.How can I do this?

As user3825493 pointed, there are answers regarding similar issues. The first link pretty much explains the whole thing.
Your problem is that due to the fact that Union is mapped to SQL UNION, arguments of the operation are to be converted to data types acceptable by Entity Framework. However, the only types acceptable by it for items inside IEnumerable input parameters are, as specified in the error you get, primitive types and enum's.
You should either:
var unionOfQueryWithList = query.AsEnumerable().Union(list)
or, if you like to use the SQL facilities, write a stored procedure which performs the union as part of its logic and find the way to pass your data into SQL, such as this.
The former loads all the results of the query into a memory before performing the union. It is convenient for the cases when you have all the pre-filtered data returned from EF and only want to use the output for later processing in your code.
The later takes you out of the EF zone and required for the cases when you have operations which will benefit from running on SQL engine, such as filtering or joining DB data based on a complex data you have access to in your code.

either you convert the query to IEnumerable or you can try this :
list.Union(query); // use Union() of list not query :)
it will work and you still don't need to get data from DB you can reverse the order if you want :)

Collecting metadata into table

I have tabluar data that passes through a C# program that I need to collect some metadata on before finishing. The metadata is always counts based on fields of the data. Also, I need them all grouped by one field in the data. Periodically, I need to add new counts to this collection of metadata.
I've been researching it for a little while, and I think what makes sense is to rework my program to store the data as a DataTable, then run LINQ queries on the table. The problem I'm having is being able to put the different counts into one table-like structure and then write that out.
I might run a query like this:
var query01 =
from record in records.AsEnumerable()
group record by record.Field<String>("Association Key") into associationsGroup
select new { AssociationKey = associationsGroup.Key, Count = associationsGroup.Count<DataRow>() };
To get a count of all of the records grouped by the field Association Key. I'm going to want another count, grouped in the same way:
var query02 =
from record in records.AsEnumerable()
where record.Field<String>("Number 9") == "yes"
group record by record.Field<String>("Association Key") into associationsGroup
select new { AssociationKey = associationsGroup.Key, Number9Count = associationsGroup.Count<DataRow>() };
And so on.
I thought about trying Union chain the queries but I was having trouble getting them to union since I'm projecting into anonymous types. I couldn't figure out how to do it differently to make a union work better.
So, how can I collect my metadata into one table-like structure?

Not going to union because you have different types. Add Number9Count and Count to both annonymous types and try union again.

I ended up solving the problem by creating a class that holds the set of records I need as a DataTable. A user can add queries through a method, taking an argument Func<DataRow, bool>. The method constructs the query supplying that argument as the where clause, maintaining the same grouping and properties in the resulting anonymous-typed object.
When retrieving the results, the class iterates over each query stored and enters the results into a new DataTable.