We Keep Coding

LINQ Query Multiple Group and count of latest record - Oracle DB

I tried to divided Linq queries into 3 (total, success, fail) but so far "Total" Linq query is working fine. Please help me to get "Success", "Fail" columns (it has mulitple statuses and we have to check the last column of each transaction and destination)
Note: you need to group by ProcessTime, TransactionId, Destination and check last column whether it is success or Fail then apply count (we are using oracle as backend)
LINQ for Total count
var query = (from filetrans in context.FILE_TRANSACTION
join route in context.FILE_ROUTE on filetrans.FILE_TRANID equals route.FILE_TRANID
where
filetrans.PROCESS_STRT_TIME >= fromDateFilter && filetrans.PROCESS_STRT_TIME <= toDateFilter
select new { PROCESS_STRT_TIME = DbFunctions.TruncateTime((DateTime)filetrans.PROCESS_STRT_TIME), filetrans.FILE_TRANID, route.DESTINATION }).
GroupBy(p => new { p.PROCESS_STRT_TIME, p.FILE_TRANID, p.DESTINATION });
var result = query.GroupBy(x => x.Key.PROCESS_STRT_TIME).Select(x => new { x.Key, Count = x.Count() }).ToDictionary(a => a.Key, a => a.Count);

Check this solution. If it gives wrong result, then I need more details.
var fileTransQuery =
from filetrans in context.AFRS_FILE_TRANSACTION
where accountIds.Contains(filetrans.ACNT_ID) &&
filetrans.PROCESS_STRT_TIME >= fromDateFilter && filetrans.PROCESS_STRT_TIME <= toDateFilter
select filetrans;
var routesQuery =
from filetrans in fileTransQuery
join route in context.AFRS_FILE_ROUTE on filetrans.FILE_TRANID equals route.FILE_TRANID
select route;
var lastRouteQuery =
from d in routesQuery.GroupBy(route => new { route.FILE_TRANID, route.DESTINATION })
.Select(g => new
{
g.Key.FILE_TRANID,
g.Key.DESTINATION,
ROUTE_ID = g.Max(x => x.ROUTE_ID)
})
from route in routesQuery
.Where(route => d.FILE_TRANID == route.FILE_TRANID && d.DESTINATION == route.DESTINATION && d.ROUTE_ID == route.ROUTE_ID)
select route;
var recordsQuery =
from filetrans in fileTransQuery
join route in lastRouteQuery on filetrans.FILE_TRANID equals route.FILE_TRANID
select new { filetrans.PROCESS_STRT_TIME, route.CRNT_ROUTE_FILE_STATUS_ID };
var result = recordsQuery
.GroupBy(p => DbFunctions.TruncateTime((DateTime)p.PROCESS_STRT_TIME))
.Select(g => new TrendData
{
TotalCount = g.Sum(x => x.CRNT_ROUTE_FILE_STATUS_ID != 7 && x.CRNT_ROUTE_FILE_STATUS_ID != 8 ? 1 : 0)
SucccessCount = g.Sum(x => x.CRNT_ROUTE_FILE_STATUS_ID == 7 ? 1 : 0),
FailCount = g.Sum(x => failureStatus.Contains(x.CRNT_ROUTE_FILE_STATUS_ID) ? 1 : 0),
Date = g.Min(x => x.PROCESS_STRT_TIME)
})
.OrderBy(x => x.Date)
.ToList();

SQLite and LINQ: find all objects that have a sub list with all ids present in a supplied list of IDs

I have the following class:
public class Article
{
long Id;
List<Category> Categories;
}
I am using EF Core 5 and What I need is a LINQ query against SQLite that returns all the articles that have all the categories that I specify.
I tried the following code:
List<long> cIds = c.Select (x => x.Id).ToList ();
query.Where (art => cIds.All (cId => art.Categories.Select (c => c.Id).Contains (cId)));
but the compiler says
InvalidOperationException: The LINQ expression 'DbSet<Article>()
.Where(a => __cIds_0
.All(cId => DbSet<Dictionary<string, object>>("ArticleCategory")
.Where(a0 => EF.Property<Nullable<long>>(a, "Id") != null && object.Equals(
objA: (object)EF.Property<Nullable<long>>(a, "Id"),
objB: (object)EF.Property<Nullable<long>>(a0, "ArticlesId")))
.Join(
inner: DbSet<Category>(),
outerKeySelector: a0 => EF.Property<Nullable<long>>(a0, "CategoriesId"),
innerKeySelector: c => EF.Property<Nullable<long>>(c, "Id"),
resultSelector: (a0, c) => new TransparentIdentifier<Dictionary<string, object>, Category>(
Outer = a0,
Inner = c
))
.Select(ti => ti.Inner.Id)
.Any(p => p == cId)))' could not be translated. Either rewrite the query in a form that can be translated, or switch to client evaluation explicitly by inserting a call to 'AsEnumerable', 'AsAsyncEnumerable', 'ToList', or 'ToListAsync'. See https://go.microsoft.com/fwlink/?linkid=2101038 for more information.
How can I obtain it?
A possible workaround I found is the following:
List<long> cIds = c.Select (x => x.Id).ToList ();
query = query.Where (art => art.Categories.Select (c => c.Id).Any (x => cIds.Contains (x)));
query = query.Include (x => x.Categories);
result = await query.ToListAsync ();
result = result.Where (art => cIds.All (cId => art.Categories.Select (c => c.Id).Contains (cId))).ToList ();
But I was wondering if I could obtain the same result with a single LINQ query.
Thanks in advance
UPDATE:
I'll just add the function where this code will be used and make make an example to make things clearer:
This is the function where the code will be used:
public async Task<List<Article>> SearchAsync (string search, Section s, Website w,
List<Category> c)
{
List<Article> result = new List<Article> ();
if (
search == ""
&& s == null
&& w == null
&& c.Count == 0
)
return result;
IQueryable<Article> query = dbSet.AsQueryable ();
if (search != "")
query = query.Where (x => x.Title.Contains (search) || x.Summary.Contains (search));
if (s != null)
query = query.Where (x => x.SectionId == s.Id);
if (w != null)
query = query.Where (x => x.WebsiteId == w.Id);
if (c.Count > 0)
{
List<long> cIds = c.Select (x => x.Id).ToList ();
query = query.Where (art => art.Categories.Select (c => c.Id).Any (x => cIds.Contains (x)));
}
query = query.Include (x => x.Categories);
result = await query.ToListAsync ();
if (c.Count > 0)
{
List<long> cIds = c.Select (x => x.Id).ToList ();
result = result.Where (art => cIds.All (cId => art.Categories.Select (c => c.Id).Contains (cId))).ToList ();
}
return result;
}
And here is an example:
Let's say c will contain ids 9,10,11 and the articles collection is the following pseudo code:
List<article> articles = new List<Article> ()
{
new Article () {Id = 1, Categories = "12,44,55"}
new Article () {Id = 2, Categories = "7,8,9,10,11"}
new Article () {Id = 3, Categories = "9,10,11"}
}
The linq query should return Article with Id 2 and 3 because both contains all of the ids present in c.

One of the solutions using Intersect, but we have to prepare data for intersection.
// articles query
var query = ...
var cIds = c.Select(x => x.Id).ToList();
var idsCount = cIds.Count();
// translating list of IDs to IQueryable
var categoryIdsQuery = dbContext.Categories
.Where(c => cIds.Contains(c.Id))
.Select(c => c.Id);
query = query
.Where(art => art.Categories
.Select(c => c.Id)
.Intersect(categoryIdsQuery)
.Count() == idsCount
)
.Include(x => x.Categories);

What I need is a LINQ query against SQLite that returns all the articles that have all the categories that I specify.
So you have a sequence of Category Ids and you want all Articles, each Article with only the Categories that are in your sequence of Category Ids.
I'm not sure what your variable 'c' is, but it seems to me that the following statement returns the Ids of all c:
List<long> cIds = c.Select (x => x.Id).ToList ();
If c is your sequence of Categories, then you will have the Ids of all existing categories. This will mean that you will have all Articles, each with all Categories.
If you have a local sequence of Category Ids, with a limited count (say about 250), then you should use Contains:
IEnumerable<long> categoryIds = ...
var articlesWithTheseCategories = dbContext.Articles.Select(article => new
{
Id = article.Id,
Categories = article.Categories
.Where(category => categoryIds.Contains(category.Id)
.ToList(),
})
So if you have CategoryIds 2, 3, and 12, this query will give you all Articles with only the Categories with ids 2, 3, 12.
If Article 40 has only Categories 20, 21, 21, then Article 40 will be in your result, but it will have an empty Categories list.
If you don't have your Category Ids locally, but you have a predicate to select the Category Ids, then your query will be like:
IQueryable<long> categoryIds = dbContext.Categories
.Where(category => category.Status == StatusCode.Obsolete); // predicate
var articlesWithTheseCategories = dbContext.Articles.Select(article => new
{
Id = article.Id,
Categories = article.Categories
.Where(category => categoryIds.Contains(category.Id)
.ToList(),
});
Because your first query is an IQueryable<...> it is not executed yet. If you want you can make it one big statement:
var articlesWithTheseCategories = dbContext.Articles.Select(article => new
{
Id = article.Id,
Categories = article.Categories
.Where(category => dbContext.Categories
.Where(category => category.Status == StatusCode.Obsolete)
.Contains(category.Id))
.ToList(),
});
Although this will not improve efficiency, it surely deteriorates readability.

EF Core - Join on 2 IQueryable that one of them has group by not translated to SQL

I want to join two tables with EF Core with this code:
var fromDate = model.FromDate;
var toDate = model.ToDate;
var tourDate = _eContext.TourDates.OrderBy(x => x.Date).Where(x =>
(fromDate == null || x.Date >= fromDate) && (toDate == null || x.Date <= toDate))
.GroupBy(x => x.TourId).Select(g => new { TourId = g.Key, TourDateID = g.Min(y => y.TourDateId) });
var tours1 = _eContext.TourDetails;
var joinres = tours1.Join(tourDate
, tu => tu.TourDateId, td => td.TourDateId, (tu, td) => new { TourId = tu.TourId, TourDetailId = tu.Id, TourDateId = td.TourDateID });
var TourList = joinres.ToList();
When I execute the .ToList() method, I get this warning:
The LINQ expression 'join TourDate td in
value(Microsoft.EntityFrameworkCore.Query.Internal.EntityQueryable`1[ModelLayer.EuropaMundo.TourDate])
on [td].TourDateID equals [td].TourDateId' could not be translated and
will be evaluated locally.
Is there any way to translate LINQ to SQL?

LINQ query to retrieve pivoted data taking too long

I am working on a LINQ query which includes some pivot data as below
var q = data.GroupBy(x => new
{
x.Med.Name,
x.Med.GenericName,
}).ToList().Select(g =>
new SummaryDto
{
Name= g.Key.Name,
GenericName = g.Key.GenericName,
Data2012 = g.Where(z => z.ProcessDate.Year == 2012).Count(),
Data2013 = g.Where(z => z.ProcessDate.Year == 2013).Count(),
Data2014 = g.Where(z => z.ProcessDate.Year == 2014).Count(),
Data2015 = g.Where(z => z.ProcessDate.Year == 2015).Count(),
Data2016 = g.Where(z => z.ProcessDate.Year == 2016).Count(),
Data2017 = g.Where(z => z.ProcessDate.Year == 2017).Count(),
TotalCount = g.Count(),
}).AsQueryable();
return q;
The above LINQ takes too long as it queries grp q.Count()*6 times. If there are 10000 records, then it queries 60000 times
Is there a better way to make this faster?

Add year to the group key, then group again, and harvest per-group counts:
return data.GroupBy(x => new {
x.Med.Name
, x.Med.GenericName
, x.ProcessDate.Year
}).Select(g => new {
g.Key.Name
, g.Key.GenericName
, g.Key.Year
, Count = g.Count()
}).GroupBy(g => new {
g.Name
, g.GenericName
}).Select(g => new SummaryDto {
Name = g.Key.Name
, GenericName = g.Key.GenericName
, Data2012 = g.SingleOrDefault(x => x.Year == 2012)?.Count ?? 0
, Data2013 = g.SingleOrDefault(x => x.Year == 2013)?.Count ?? 0
, Data2014 = g.SingleOrDefault(x => x.Year == 2014)?.Count ?? 0
, Data2015 = g.SingleOrDefault(x => x.Year == 2015)?.Count ?? 0
, Data2016 = g.SingleOrDefault(x => x.Year == 2016)?.Count ?? 0
, Data2017 = g.SingleOrDefault(x => x.Year == 2017)?.Count ?? 0
, TotalCount = g.Sum(x => x.Count)
}).AsQueryable();
Note: This approach is problematic, because year is hard-coded in the SummaryDto class. You would be better off passing your DTO constructor an IDictionary<int,int> with counts for each year. If you make this change, the final Select(...) would look like this:
.Select(g => new SummaryDto {
Name = g.Key.Name
, GenericName = g.Key.GenericName
, TotalCount = g.Sum(x => x.Count)
, DataByYear = g.ToDictionary(i => i.Year, i => i.Count)
}).AsQueryable();

I suggest grouping inside the group by year and then converting to a dictionary to access the counts. Whether it is faster to group with year first and then count in-memory depends on the distribution of the initial grouping, but with the database it may depend on how efficiently it can group by year, so I would test to determine which seems fastest.
In any case grouping by year after the initial grouping is about 33% faster than your query in-memory, but again it is vastly dependent on the distribution. As the number of initial groups increase, the grouping by Year queries slow down to match the original query. Note that the original query without any year counts is about 1/3 the time.
Here is grouping after the database grouping:
var q = data.GroupBy(x => new {
x.Med.Name,
x.Med.GenericName,
}).ToList().Select(g => {
var gg = g.GroupBy(d => d.ProcessDate.Year).ToDictionary(d => d.Key, d => d.Count());
return new SummaryDto {
Name = g.Key.Name,
GenericName = g.Key.GenericName,
Data2012 = gg.GetValueOrDefault(2012),
Data2013 = gg.GetValueOrDefault(2013),
Data2014 = gg.GetValueOrDefault(2014),
Data2015 = gg.GetValueOrDefault(2015),
Data2016 = gg.GetValueOrDefault(2016),
Data2017 = gg.GetValueOrDefault(2017),
TotalCount = g.Count(),
};
}).AsQueryable();

Add correlated subqueries to Select statement using QueryOver

I'm attempting to convert a SQL statement to use QueryOver (in hopes of pre-fetching the entities part of the response) but I'm having trouble figuring out how to add a correlated subquery to the Select statement (all the examples I found have only shown using a subquery in the Where clause).
This is the query I'm trying to convert:
var pendingFeedbackStatus = Session.QueryOver<FeedbackStatus>().Where(fs => fs.Name == "pending");
var projectWhereClause = project != null ? "AND f1.project_id = " + project.Id : "";
var query = Session.CreateSQLQuery(string.Format(#"
SELECT
ft.id as FEEDBACK_TYPE_ID,
(SELECT COUNT(*) FROM FEEDBACK f1 WHERE ft.id = f1.feedback_type_id AND f1.archive_ind = 0 {0}) as ALL_FEEDBACK_COUNT,
(SELECT COUNT(*) FROM FEEDBACK f1 WHERE ft.id = f1.feedback_type_id AND f1.archive_ind = 0 {0} AND feedback_status_id = {1}) as PENDING_FEEDBACK_COUNT
FROM feedback f
RIGHT JOIN feedback_type ft on f.feedback_type_id = ft.id WHERE ft.RESTRICTED_IND = 0
GROUP BY ft.id, ft.sort_order
ORDER BY ft.sort_order",
projectWhereClause,
pendingFeedbackStatus.Id
))
.SetResultTransformer(Transformers.AliasToEntityMap);
var results = query.List<IDictionary>();
return results.Select(r =>
new FeedbackTypeSummary
{
Type = Get(Convert.ToInt32(r["FEEDBACK_TYPE_ID"])),
AllFeedbackCount = Convert.ToInt32(r["ALL_FEEDBACK_COUNT"]),
PendingFeedbackCount = Convert.ToInt32(r["PENDING_FEEDBACK_COUNT"])
}).ToList();
and here is what I have so far (which is mostly everything minus the correlated subqueries and some additional filtering added to the subqueries):
var pendingFeedbackStatus = Session.QueryOver<FeedbackStatus>().Where(fs => fs.Name == "pending");
Feedback feedbackAlias = null;
FeedbackType feedbackTypeAlias = null;
var allFeedback = QueryOver.Of<Feedback>()
.Where(f => f.Type.Id == feedbackTypeAlias.Id)
.Where(f => !f.IsArchived);
var pendingFeedback = QueryOver.Of<Feedback>()
.Where(f => f.Type.Id == feedbackTypeAlias.Id)
.Where(f => !f.IsArchived)
.Where(f => f.Status.Id == pendingFeedbackStatus.Id);
var foo = Session.QueryOver<Feedback>(() => feedbackAlias)
.Right.JoinAlias(f => f.Type, () => feedbackTypeAlias, ft => !ft.IsRestricted)
.SelectList(list => list
// TODO: Add correlated subqueries here?
.SelectGroup(() => feedbackTypeAlias.Id)
.SelectGroup(() => feedbackTypeAlias.SortOrder)
)
.OrderBy(() => feedbackTypeAlias.SortOrder).Asc;
var test = foo.List<object[]>();
I'd also like to find a way to return a full FeedbackType entity of from the statement, instead of returning feedbackTypeAlias.Id and then having to perform Type = Get(Convert.ToInt32(r["FEEDBACK_TYPE_ID"])) in a loop as I do in the original.

I felt like I looked for this 10 times, but I overlooked the .SelectSubQuery() method which provided the desired correlated subqueries. This answer tipped me off - https://stackoverflow.com/a/8143684/191902.
Here is the full QueryOvery version:
var pendingFeedbackStatus = Session.QueryOver<FeedbackStatus>().Where(fs => fs.Name == "pending").SingleOrDefault();
Domain.Feedback.Feedback feedbackAlias = null;
FeedbackType feedbackTypeAlias = null;
var allFeedback = QueryOver.Of<Domain.Feedback.Feedback>()
.Where(f => f.Type.Id == feedbackTypeAlias.Id)
.Where(f => !f.IsArchived);
var pendingFeedback = QueryOver.Of<Domain.Feedback.Feedback>()
.Where(f => f.Type.Id == feedbackTypeAlias.Id)
.Where(f => !f.IsArchived)
.Where(f => f.Status.Id == pendingFeedbackStatus.Id);
if (project != null)
{
allFeedback.Where(f => f.Project.Id == project.Id);
pendingFeedback.Where(f => f.Project.Id == project.Id);
}
FeedbackTypeSummary result = null;
var query = Session.QueryOver<Domain.Feedback.Feedback>(() => feedbackAlias)
.Right.JoinAlias(f => f.Type, () => feedbackTypeAlias, ft => !ft.IsRestricted)
.SelectList(list => list
.SelectSubQuery(allFeedback.ToRowCountQuery()).WithAlias(() => result.AllFeedbackCount)
.SelectSubQuery(pendingFeedback.ToRowCountQuery()).WithAlias(() => result.PendingFeedbackCount)
.SelectGroup(() => feedbackTypeAlias.Id).WithAlias(() => result.TypeId)
.SelectGroup(() => feedbackTypeAlias.Name).WithAlias(() => result.TypeName)
.SelectGroup(() => feedbackTypeAlias.NamePlural).WithAlias(() => result.TypeNamePlural)
.SelectGroup(() => feedbackTypeAlias.SortOrder)
)
.OrderBy(() => feedbackTypeAlias.SortOrder).Asc
.TransformUsing(Transformers.AliasToBean<FeedbackTypeSummary>());
var results = query.List<FeedbackTypeSummary>();
return results;
I also was able to populate my FeedbackTypeSummary DTO from a single query, although I couldn't find a way to alias an entity and ended up extracting a few of the needed properties from FeedbackType into FeedackTypeSummary (which is probably a better thing to do anyways).