We Keep Coding

LINQ query transfer huge amount of data

I am stuck on problem with optimizing LINQ query On initial load my xhr request is almost 8MB large and it took a while to take those data from server. I have list of Auctions where I display only few columns(which you can see in UserAuctionListDTO).
I uploaded image where is shown how my DTOs are connected for better understanding. Collections in CurrentsToUserAccount got only Id and Name
Goal of my query is to select all auctions which are assigned to UserContact or Organization or Region or Department. Auctions are valid when it has at least one Price.
Here is my query. It might be a bit messy but it showed me the best result with size capacity of transfered data. Each Collection of Prices in UserAuctionListDTO got around 100-1000 records, in average its about 600.
protected override IQueryable<UserAuctionListDTO> GetQueryable()
{
return AllAuctionToUser
.Where(a => a.GarancyFromDate <= _dateTimeProvider.Now
&& a.GarancyToDate >= _dateTimeProvider.Now
&& !a.IsBlocked
&& a.IsVisibleOnFe
&& a.Prices.Any(price => price.PriceMJ > 0
&& (price.Current.UserContacts.Any(x =>
x.Id == Filter.UserContactId)
|| price.Current.Departments.Any(x =>
x.Id == Filter.DepartmentId)
|| price.Current.Organizations.Any(x =>
x.Id == Filter.OrganizationId)
|| price.Current.Regions.Any(x => x.Id == Filter.RegionId))))
.Select(auction => new UserAuctionListDTO
{
Id = auction.Id,
Code = auction.Code,
Name = auction.Name,
ProductCount = auction.Prices.Count,
AttachmentsCount = auction.Attachments.Count,
Category = new CategoryBaseDTO
{
Name = auction.Category.Name
},
GarancyFromDate = auction.GarancyFromDate,
GarancyToDate = auction.GarancyToDate,
Prices = auction.Prices
.Select(x => new PriceToUserAuctionDTO
{
Id = x.Id,
Current = new CurrentsToUserAccount
{
Id = x.Current.Id,
UserContacts = x.Current.UserContacts.Where(u => u.IsBlocked == false)
.Select(z => new UserContactBaseDTO
{
Id = Filter.UserContactId
}
),
Departments = x.Current.Departments.Where(u => u.IsBlocked == false)
.Select(z => new DepartmentBaseDTO
{
Id = Filter.DepartmentId
}),
Regions = x.Current.Regions.Where(u => u.IsBlocked == false)
.Select(z => new RegionBaseDTO
{
Id = Filter.RegionId
}),
Organizations = x.Current.Organizations.Where(u => u.IsBlocked == false)
.Select(z => new OrganizationBaseDTO
{
Id = Filter.OrganizationId
}),
},
SpecificProductSuppliers = new SpecificProductSuplierToUserAccountDTO
{
Id = x.SpecificProductSuppliers.Id,
CatalogName = x.SpecificProductSuppliers.CatalogName,
SpecificProduct = new SpecificProductToUserAccountDTO
{
Id = x.SpecificProductSuppliers.SpecificProduct.Id,
Name = x.SpecificProductSuppliers.SpecificProduct.Name
}
}
})
}).OrderBy(x => x.Code);
}
Screenshot of how much data is transfered from server
I wonder if there is anything how I can improve it.

C# Predicate builder with using AND with OR

I have the following class:
public class testClass
{
public string name { get; set; }
public int id { get; set; }
public int age { get; set; }
}
and the following code:
var list = new List<testClass>();
list.Add(new testClass { name = "name", id = 1, age = 30 });
list.Add(new testClass { name = "name", id = 2, age = 22 });
list.Add(new testClass { name = "name", id = 3, age = 20 });
list.Add(new testClass { name = "name", id = 4, age = 30 });
list.Add(new testClass { name = "name", id = 5, age = 27 });
list.Add(new testClass { name = "name", id = 6, age = 30 });
var qble = list.AsQueryable();
var pred = PredicateBuilder.New<testClass>();
pred.Or(x => x.name == "name" && x.id == 1);
pred.Or(x => x.age == 30);
var predQuery = qble.AsExpandable().Where(pred);
My aim is to create a query that returns all records where:
id = 1 and name = "name"
OR
age = 30
So for the query above, it should return the items at index 0, 1, 5
For the above query it does as I want.
However, I now want to the build the predicate by combining a set of queries, rather than explicitly defining them. So I now have the following 2 queries:
var query1 = list.Where(x => x.name == "name" && x.id == 1);
var query2 = list.Where(x => x.age == 30);
and I want to build the query based on the variables query1 and query2, without explicitly defining the conditions - as these conditions will be dynamically defined and I do not know what they are,and they will be defined in different places.
My guess is I need to do something like this (continuing from above):
var qble = list.AsQueryable();
var query1 = list.Where(x => x.name == "name" && x.id == 1);
var query2 = list.Where(x => x.age == 30);
var pred = PredicateBuilder.New<testClass>();
pred.Or(query1);
pred.Or(query2);
var predQuery = qble.AsExpandable().Where(pred);
but this is not quite correct as the predicate builder will not accept the query as a parameter.
Can this be done?

You could create two Predicate<T> and invoke them in your .Where call at the end.
var qble = list.AsQueryable();
var query1 = new Predicate<testClass>(x => x.name == "name" && x.id == 1);
var query2 = new Predicate<testClass>(x => x.age == 30);
var predQuery = qble.AsExpandable().Where(x => query1(x) || query2(x));
Or you could build another Predicate<T> beforehand and use this
var query = new Predicate<testClass>(x => query1(x) || query2(x));
var predQuery = qble.AsExpandable().Where(query);

Linq return data if there are no values with corresponding entity

Example query:
var results = await campaignList
.GroupBy(x => x.Campaign)
.Select(c => new ReportModel
{
CampaignId = c.Key.Id,
CoregName = c.Key.Name,
TotalVisitors = startDate == null || endDate == null ? c.Key.LogPixelCalls.Count : c.Key.LogPixelCalls.Count(x => x.CreatedOn >= startDate && x.CreatedOn <= endDate),
TotalUsers = c.Count(),
TotalPaidUsers = c.Count(x => x.UserSignature.StripeCustomerId != null),
TotalUserConversion = !c.Any() || c.Key.LogPixelCalls.Count == 0 ? 0 : (decimal)c.Count() / (decimal)c.Key.LogPixelCalls.Count * 100,
TotalPaidUserConversion = c.Count(x => x.UserSignature.StripeCustomerId != null) == 0 || !c.Any() ? 0 : (decimal)c.Count(x => x.UserSignature.StripeCustomerId != null) / (decimal)c.Count() * 100
})
.OrderByDescending(c => c.TotalPaidUsers)
.ToListAsync();
It selects Campaigns which have data connected with corresponding CampaignId. How to select all Campaigns and if there is no data in other tables, fill 0 values in .Select ?
Example:
Campaign 1, 44, 22, 33
Campaign 2, 0, 0, 0
Campaign 2 doesn't have any data, but I still want to show it in result.
Edit:
Tried to use GroupJoin without success:
var resultinho = campaigns.GroupJoin(
results,
foo => foo.Id,
bar => bar.CampaignId,
(x, y) => new { Foo = x, Bar = y })
.SelectMany(
x => x.Bar.DefaultIfEmpty(),
(x, y) => new ReportModel
{
CampaignId = x.Bar.FirstOrDefault().CampaignId,
CoregName = x.Bar.FirstOrDefault().CoregName,
TotalVisitors = x.Bar.FirstOrDefault().TotalVisitors,
TotalUsers = x.Bar.FirstOrDefault().TotalUsers,
TotalPaidUsers = x.Bar.FirstOrDefault().TotalPaidUsers,
TotalUserConversion = x.Bar.FirstOrDefault().TotalUserConversion,
TotalPaidUserConversion = x.Bar.FirstOrDefault().TotalPaidUserConversion
})
.ToList();

I did like this:
var resultsWithEmtpyCampaigns = campaigns.GroupJoin(
results,
camp => camp.Id,
campResult => campResult.CampaignId,
(x, y) => y.Select(c => new ReportModel
{
CampaignId = y.FirstOrDefault().CampaignId,
CoregName = y.FirstOrDefault().CoregName,
TotalVisitors = y.FirstOrDefault().TotalVisitors,
TotalUsers = y.FirstOrDefault().TotalUsers,
TotalPaidUsers = y.FirstOrDefault().TotalPaidUsers,
TotalUserConversion = y.FirstOrDefault().TotalUserConversion,
TotalPaidUserConversion = y.FirstOrDefault().TotalPaidUserConversion,
ReportCountryList = y.FirstOrDefault().ReportCountryList
})
.DefaultIfEmpty(new ReportModel
{
CampaignId = x.Id,
CoregName = x.Name,
TotalVisitors = 0,
TotalUsers = 0,
TotalPaidUsers = 0,
TotalUserConversion = 0,
TotalPaidUserConversion = 0
}))
.SelectMany(g => g).ToList();
return
resultsWithEmtpyCampaigns;

group/sum campaign data first and left join the campaign with the grouped results
var cd = campaigns.GroupBy (c => new {c.ID} ).Select (cp =>
new {
cId= cp.Key.ID,
TotalUsers = cp.Count(),
TotalSum= cp.Sum( x=>x.UsersByCampaign)
...
} );
var results = campaignList.ForEach( cl => cd.Where( cdd => cdd.cId == cl.Id)
.Select(cdd => new { CampaignId = cdd.CId, TotalVisitors = cdd.TotalUsers , .. })
.DefaultIfEmpty( new { CampaignId = cdd.CId, TotalVisitors =0 , .. });

Select Last non null-able item per product?

Let's say I have,
class Product
{
public int Id {get; set;}
public string Name {get; set;}
public int Order {get; set;}
}
and my data have,
products[0] = new Product { Id = 1, Name = "P1", Order = 1 };
products[1] = new Product { Id = 1, Name = "P2", Order = 2 };
products[2] = new Product { Id = 1, Name = null, Order = 3 };
products[3] = new Product { Id = 2, Name = "P3", Order = 4 };
products[4] = new Product { Id = 2, Name = null, Order = 5 };
products[5] = new Product { Id = 2, Name = null, Order = 6 };
What I need is the last(order by Order desc) non-nullable value of Name per Product.Id. So my final output will look like,
items[0] = new { Id = 1, Name = "P2"};
items[1] = new { Id = 2, Name = "P3"};
If Id=1, I have 3 Names (P1, P2, null) and non-nullable Names (P1, P2) but last one is P3.

This should get the last products in order.
var lastOrders = products
.Where(x => x.Name != null) // Remove inapplicable data
.OrderBy(x => x.Order) // Order by the Order
.GroupBy(x => x.Id) // Group the sorted Products
.Select(x => x.Last()); // Get the last products in the groups

var result = products
.GroupBy(p => p.Id)
.Select(g => g.OrderBy(x => x.Order).Last(x => x.Name != null));

this will give you your desired output:
products.GroupBy(p => p.Id)
.Select(g => g.OrderByDescending(gg => gg.Name)
.Where(gg => gg.Name != null)
.Select(gg => new { gg.Id, gg.Name })
.First());

The task can be solved using the following Linq statement.
var Result = products.OrderBy().Where( null != iProduct.Name ).First();
This requires products to contain at least one item where Name is null, otherwise an Exception will be thrown. Alternatively,
var Result = products.OrderBy().Where( null != iProduct.Name ).FirstOrDefault();
will return null if products contains no such item.

Try with :
var expectedProduct =products.Where(p => p.Id != null).OrderByDescending(p => p.Order).GroupBy(p => p.Id).Last()

How to merge LINQ querys of lambdas which return of anonymous type?

I'm trying to combine 2 lists from these:
var quartEst = Quarterly_estimates
.OrderByDescending (q => q.Yyyy)
.ThenByDescending (q => q.Quarter)
.Where (q =>
q.Ticker.Equals("IBM")
&&
q.Eps != null
)
.Select (q => new {
ticker = q.Ticker,
Quarter = q.Quarter,
Year = q.Yyyy,
Eps = q.Eps})
.AsEnumerable()
.Where (q => Convert.ToInt32(string.Format("{0}{1}", q.Year, q.Quarter)) > Convert.ToInt32(finInfo) );
var quartAct = Quarterlies
.OrderByDescending (q => q.Yyyy)
.ThenByDescending (q => q.Quarter)
.Where (q =>
q.Ticker.Equals("IBM")
&&
Convert.ToInt16(q.Yyyy) >= DateTime.Now.Year - 3
)
.Select (q => new {
Tick = q.Ticker,
Quarter = q.Quarter,
Year = q.Yyyy,
Eps = q.Eps_adj})
.AsEnumerable()
.Where (q => Convert.ToInt32(string.Format("{0}{1}", q.Year, q.Quarter)) <= Convert.ToInt32(finInfo));
I get the error for a simple Union command:
var quartComb = quartEst.Union(quartAct);
Here's the error:
Instance argument: cannot convert from 'System.Collections.Generic.List<AnonymousType#1>' to 'System.Linq.IQueryable<AnonymousType#2>'
What do I need to do to achieve this merge?

To use the Union method, the collections must have the same type. In order for two anonymous types to be considered the same, they must have exactly the same members with exactly the same types.
Try this:
var quartEst = Quarterly_estimates
...
.Select (q => new {
Tick = q.Ticker, // renamed ticker to Tick
Quarter = q.Quarter,
Year = q.Yyyy,
Eps = q.Eps})
...
var quartAct = Quarterlies
.Select (q => new {
Tick = q.Ticker,
Quarter = q.Quarter,
Year = q.Yyyy,
Eps = q.Eps_adj})
...
var quartComb = quartEst.Union(quartAct);

To cast two different Anonymous Types you have to cast it to dynamic, Here is the below code
protected void Page_Load(object sender, EventArgs e)
{
List<AlertInfo> alert = new List<AlertInfo>();
alert.Add(new AlertInfo() { StratId = 1, GroupId = 1 });
alert.Add(new AlertInfo() { StratId = 1, GroupId = 2 });
alert.Add(new AlertInfo() { StratId = 1, GroupId = 3 });
alert.Add(new AlertInfo() { StratId = 2, GroupId = 1 });
alert.Add(new AlertInfo() { StratId = 2, GroupId = 2 });
alert.Add(new AlertInfo() { StratId = 2, GroupId = 3 });
alert.Add(new AlertInfo() { StratId = 3, GroupId = 1 });
alert.Add(new AlertInfo() { StratId = 3, GroupId = 2 });
//To know how much data will get stored
var totalDataStore = alert.Select(x => x.StratId).Distinct().ToList();
var result = alert.Where(x => x.StratId == 1).Select(x => new { Data1 = "Group" + x.StratId + x.GroupId }).Cast<dynamic>().Union(alert.Where(y => y.StratId == 2).Select(y => new { Data2 = "Group" + y.StratId + y.GroupId })).ToList();
}
}
public class AlertInfo
{
public int StratId { get; set; }
public int GroupId { get; set; }
}