Couldn't find a group by references in ODATA V4 documentation. When we pass the group by query in URL it just returns the key, not the actual grouped results.
Any references for using Group by in Asp.Net ODATA, on extensibility
points of ODATA Web API. We're in need to take full command over how
ODATA query is parsed & transformed into LILNQ to entities query.
I am talking on the line of intercepting ODATA queries and performing manual customization e.g. LINQ to Entities
I am trying to achieve a similar extension for Case Sensitivity.
OData Case In-Sensitive filtering in Web API?
Try to approach it from SQL perspective:
by grouping you are able to get the keys, exactly, and some aggregate values
by using original filter you used for group and extending it with group keys (original filter and group keys matches) you actually load the data for the given group.
This is how our grouping grid works in angular telerik kendo (they have a nice toOdataString impl. which I also extended: https://github.com/telerik/kendo-angular/issues/2102).
This approach ensures a fixed given amount of groups in the grid (total group or 1st level group).
PRO: you see all the groups (or at least N of them)
CONS: if you unfold a group you might end up with too many items; needs lot of extra code and additional calls with special odata queries;
See: http://www.reflection.sk/#portfolios, check screenshot: Universal Plans Services (UPS) Software Bundle (.NET & Angular with KendoUI)
If you take it from UI perspective:
Then the grouping is nothing more than a list of data with priority sort over the grouped field. This is the default kendo grouping grid aproach. So they just sort the data, get a page-size of it then the grouping UX items are added (virtual items in the grid).
This approach ensures that you got a fixed items in the grid, but when you collapse all items, you might have just 1 or even pageSize count of groups (depending if/how many items are in each of the groups). See it here: https://www.telerik.com/kendo-angular-ui/components/grid/grouping/ - actually you'd need paging to be turned off to see the difference.
With items up to a fixed count this approach is the fastest. Just one call per page, but the count of groups is not know in advance (if you collapse them, it might be just 1 or even N where N is pagesize).
Regarding case sensitivity:
when filtering wrapping column name and entered value into tolower() helps:
but it's up to the DB settings how case sensitivity is handled by def.
Also a note: with grouping I was not able to do something like $groupby(tolower(columnname)) with odata, so...
Related
I'm creating a tabular grid of data and will have every column header sortable (both ascending and descending) as well as filterable by a textbox underneath the column name.
I've done some digging to find out the best approach to generating my LINQ queries for entity framework and best I found was this: https://learn.microsoft.com/en-us/aspnet/core/data/ef-mvc/sort-filter-page?view=aspnetcore-3.0
That just uses a big switch / case (for ordering only...doesn't even consider multiple filtering options beyond one). Surely there's got to be a better, more concise approach than this?
I'm half tempted to just use my old sproc and call the sproc with EF if this is really the path I'll have to go down.
Ideally, I'd want something like this:
return myContext.Where(x =>
x.%thisisadynamicfield%.Contains("%somefiltervalue%" &&
x.%anotherdynamicfield%.Contains("%someothervalue%" && ...)
.OrderBy(x => x.%someorderybydynamicfield%)
I'm ok using a conditional to determine ordering ascending or descending, but I'd really like to compact everything else as much as possible and that MS article I linked above looks like code vomit and it would only cover part of my needs.
EDIT: as is almost always the case, as soon as I post here and try one more searching effort, I find what I'm looking for. I just found this: https://github.com/harshwPro/EntityFrameworkPaginate which looks very promising. I will report back here if that solves my issue.
EDIT 2: well the github project I referenced in my first edit is certainly a cleaner solution than the MS article I posted originally but it still leaves me with writing a huge conditional to generate my filters by property. Using that solution, I'd have to write something like this:
dataPagingModel.Filters.Add(true, x => x.MyCoolProperty.Contains("blah")); but would have to conditionally add new filters based on my angular presentation layer's posted parameters - switch / case, or some huge if / else block. What I'd ideally like is a way to just loop over all of the posted filters list and dynamically add those filters in a loop. So I guess my question now boils down to, is there a way to access properties of an object dynamically (generically) in a lambda expression?
EDIT 3: I think this SO post has the answers I need: Generic method to filter a list object
Well my requirement is to do client side paging. i.e return a set of records based on the values($top, $skip) given by client. But based on my below code, am able to use only filter keyword and top or skip.
[HttpGet]
public PageResult<PersistedUser> GetAllUsers(ODataQueryOptions options)
{
TableServiceContext serviceContext = tableClient.GetDataServiceContext();
serviceContext.IgnoreResourceNotFoundException = true;
CloudTableQuery<PersistedUser> users = serviceContext
.CreateQuery<PersistedUser>(TableNames.User)
.AsTableServiceQuery();
IQueryable<PersistedUser> results = options
.ApplyTo(users.AsQueryable()) as IQueryable<PersistedUser>;
// manipulate results. Add some calculated variables to the collection etc
return new PageResult<PersistedUser>(results, null, 0);
}
I am not really sure if this is the correct way to do it as well. But my basic requirement is that I have a huge db, but i just need to return a small set of entities at a time in an efficient time. I would really appreciate if someone could provide some code snippets.
I'm using the same way and it's working fine.
Few differences:
I have a service layer that expose my entites. In my services I return IQueryable and apply the O Data filter.
[AuthKey]
[GET("api/brands/")]
public PageResult<BrandViewModel> GetBrands(ODataQueryOptions<Brand> options)
{
var brands = (IQueryable<Brand>)options.ApplyTo(_brandServices.FindBrands());
return new PageResult<BrandViewModel>(BrandViewModel.ToViewModel(brands), Request.GetNextPageLink(), Request.GetInlineCount());
}
Here's an updated version of your code that uses the generic version of ODataQueryOptions and applies the $top and $skip options for paging.
[HttpGet]
public PageResult<PersistedUser> GetAllUsers(
ODataQueryOptions<PersistedUser> options)
{
TableServiceContext serviceContext = tableClient.GetDataServiceContext();
serviceContext.IgnoreResourceNotFoundException = true;
CloudTableQuery<PersistedUser> users = serviceContext
.CreateQuery<PersistedUser>(TableNames.User)
.AsTableServiceQuery();
IQueryable<PersistedUser> results = options.ApplyTo(users);
int skip = options.Skip == null ? 0 : options.Skip.Value;
int take = options.Top == null ? 25 : options.Top.Value;
return new PageResult<PersistedUser>(
results.Skip(skip).Take(take).ToList(),
Request.GetNextPageLink(),
null);
}
When planning your OData model, separate it from the underlying storage model. In some domains it may be possible to exposed groups and then use navigation properties to access the members of the group.
For example, suppose you have a booking system. You may store your bookings in a long, long table by date-time.
But you could potentially expose the OData model by grouping into collections of year and week.
http://service.net/odata/year(2013)/week(22)/bookings
In your controller, you'd compose a Table Storage range query from the temporal parameters supplied.
If there were >1,000 bookings, but not a huge number more, you could page through them server-side, exhaust the set, and deliver all the bookings back to the OData client, or sort them and allow IQueryable over this set. See note, bottom.
This would afford the OData consumer a natural mechanism for filtering data while keeping result set sizes low. Where there are many bookings per week, it could be sub-grouped further by day of week and hour.
This is completely theoretical, but I think OData v4 and its containment feature would allow such URLs to be routed and the relationship described so as to produce correct metadata for OData consumers like Excel.
http://www.asp.net/web-api/overview/odata-support-in-aspnet-web-api/odata-v4/odata-containment-in-web-api-22
Note in the above sample code that they create an arbitrary route/URL for counting the contained items, so it seems flexible.
If nested containment is not allowed, then alternatively, consider a BookingRange entity in the OData EDM with Year and Week properties to allow:
http://service.net/odata/bookingrange(2013,22)/bookings
Another idea I've considered is calculating a page number on insert. That is, to have a PageNumber on the TS entity itself and use an algorithm to assign a page number to it. This is essentially the same as generating a good partition key but with many pages/partitions.
A table expecting to hold 200m rows could have 1m pages (generate a large psuedo random number and Mod it by 1m), each with 200 items. At first, most pages would be empty, so you'd need to write a page mapper algo that alters as the row count grows. "Page" 1 maps to page range 0000001 - 0000100, etc.
As you can see, this is getting complex, but its essentially the same system that Azure uses itself to auto-balance your data across partitions and spread those partitions across its nodes.
Ultimately, this would again need a way for the "Page" number to be specified in the URL. Finally, each page would contain varying numbers of items, depending on the distribution of the algorithm used.
Note - I think the reason that TS does not support top and skip, or skip, is that the order of the rows returned cannot be guaranteed and there is no ordering mechanism within TS (which would be a big burden). Therefore, pages collated from top and skip would contain a "random" bag each time.
This means that my suggestion above to offer paging over a subset/group of data, requires that the entire subset is brought into the service tier and a sort order applied to them, before applying the top and skip, though it could be argued that the client should understand that top/skip without an order is meaningless and the onus is on them to send the right options.
I am building an application that searches candidate's resumes. I need to use full-text search on the application as there are a lot of records and the resume field is fairly large. The issue is that for advanced searches, I have another table RelocationItems, that lists zips, states, etc. for the candidates relocation preferences and is related through a candidateID in the RelocationItems table. The problem is that sometimes a candidate will have no RelocationItems, sometimes they will have one, and sometimes they will have more than one. So, simple enough, I created a View that uses full outer join and then can select using DISTINCT on candidateID to find the candidates I need that will relocate to a certain area based on the search criteria.
The big problem with this view though as since it uses and Full Join, I can't use the full-text search now! (obviously so because my full-text index field is now not a unique not-null field)
And my stored procedure has the CONTAINS word in it so it won't even compile.
Should I :
- Create a new table based on the view? (and then create another index identity field)
- Do something to store the relocation items in the candidate table (maybe an XML field)? (I don't think you can store a table-value parameter in 2008 can you?)
- Do some sort of Union of Tables (Queries)? (Run the search against the Candidates Table and then against the RelocationTable and then merge or union)?
Thanks for any suggestions on the best way to work around this problem!!!
I created a View that uses full outer join and then can select using DISTINCT on candidateID to
find the candidates I need that will relocate to a certain area based on the search criteria.
Already a potential problem - a subselect with exists would be better.
A properly set up query would have no problem - do not use a join, go for a subselect and exists.
Im using the code from this MSDN page to create a user defined aggregate to concatenate strings with group by's in SQL server. One of my requirements is that the order of the concatenated values are the same as in the query. For example:
Value Group
1 1
2 1
3 2
4 2
Using query
SELECT
dbo.Concat(tbl.Value) As Concat,
tbl.Group
FROM
(SELECT TOP 1000
tblTest.*
FROM
tblTest
ORDER BY
tblTest.Value) As tbl
GROUP BY
tbl.Group
Would result in:
Concat Group
"1,2" 1
"3,4" 2
The result seems to always come out correct and as expected, but than I came across this page that states that the order is not guaranteed and that attribute SqlUserDefinedAggregateAttribute.IsInvariantToOrder is only reserved for future use.
So my question is: Is it correct to assume that the concatenated values in the string can end up in any order? If that is the case then why does the example code on the MSDN page use the IsInvariantToOrder attribute?
I suspect a big problem here is your statement "the same as in the query" - however, your query never defines (and cannot define) an order by the things being aggregated (you can of course order the groups, by having a ORDER BY after the GROUP BY). Beyond that, I can only say that it is based purely on a set (rather than an ordered sequence), and that technically the order is indeed undefined.
While the accepted answer is correct, I wanted to share a workaround that others may find useful. Warning: it involves not using a user-defined aggregate at all :)
The link below describes an elegant way to build a concatenated, delimited list using only a SELECT statement and a varchar variable. The upside (for this thread) is that you can specify the order in which the rows are processed. The downside is that you can't easily concatenate across many different subsets of rows without painful iteration.
Not perfect, but for my use case was a good workaround.
http://blog.sqlauthority.com/2008/06/04/sql-server-create-a-comma-delimited-list-using-select-clause-from-table-column/
I am creating a forum package for a cms and looking at caching some of the queries to help with performance, but I'm not sure if caching the below will help/do what it should on the below (BTW: Cachehelper is a simple helper class that just adds and removes from cache)
// Set cache variables
IEnumerable<ForumTopic> maintopics;
if (!CacheHelper.Get(topicCacheKey, out maintopics))
{
// Now get topics
maintopics = from t in u.ForumTopics
where t.ParentNodeId == CurrentNode.Id
orderby t.ForumTopicLastPost descending
select t;
// Add to cache
CacheHelper.Add(maintopics, topicCacheKey);
}
//End Cache
// Pass to my pager helper
var pagedResults = new PaginatedList<ForumTopic>(maintopics, p ?? 0, Convert.ToInt32(Settings.ForumTopicsPerPage));
// Now bind
rptTopicList.DataSource = pagedResults;
rptTopicList.DataBind();
Doesn't linq only execute when its enumerated? So the above won't work will it? as its only enumerated when I pass it to the paging helper which .Take()'s a certain amount of records based on a querystring value 'p'
You need to enumerate your results, for example by calling the ToList() method.
maintopics = from t in u.ForumTopics
where t.ParentNodeId == CurrentNode.Id
orderby t.ForumTopicLastPost descending
select t;
// Add to cache
CacheHelper.Add(maintopics.ToList(), topicCacheKey);
My experience with Linq-to-Sql is that it's not super performant when you start getting into complex objects and/or joins.
The first step is to set up LoadOptions on the datacontext. This will force joins so that a complete record is recalled. This was a problem in a ticket tracking system I wrote. I was displaying a list of 10 tickets and saw about 70 queries come across the wire. I had ticket->substatus->status. Due to L2S's lazy initialization, that caused each foreign key for each object that I referenced in the grid to fire off a new query.
Here's a blog post (not mine) about this subject (MSDN was weak): http://oakleafblog.blogspot.com/2007/08/linq-to-sql-query-execution-with.html
The next option is to create precompiled Linq queries. I had to do this with large joins. Here's another blog post on the subject: http://aspguy.wordpress.com/2008/08/15/speed-up-linq-to-sql-with-compiled-linq-queries/
The next option is to convert things over to using stored procedures. This makes programming and deployment harder for sure, but for complex queries where you only need a subset of data, they will be orders of magnitude faster.
The reason I bring this up is because the way you're talking about caching things (why not use the built in Cache in ASP.NET?) is going to cause you lots of headaches in the long term. I'd recommend building your system and then running SQL traces to see where your database performance problems are, then build optimizations around that. You might find that your real issues aren't in the "top 10 topics" but in other, much simpler to fix areas.
Yes, you need to enumerate your results. Linq will not evaluate your query until you enumerate the results.
If you want a general caching strategy for Linq, here is a great tutorial:
http://petemontgomery.wordpress.com/2008/08/07/caching-the-results-of-linq-queries/
The end goal is the ability to automatically generate unique cache keys for any Linq query.