We Keep Coding

Should I apply counting in DB or in Code?

I have the following code where I'm trying to get the amount of rows in the same dataset with various matches.
My question is if should I get the count in C# code with a IEnumerable or by querying a IQueryable from database?
Which one is more efficient, multiple database transactions or IEnumerable filtering and count?
public List<Tuple<string, int>> CalismaVeIzinleriHesapla(long personelId, DateTime baslangic, DateTime bitis)
{
var hesaplamalar = new List<Tuple<string, int>>();
var puantajList = puantajlar.Where(p => p.PersonelId == personelId && (p.Tarih >= baslangic && p.Tarih <= bitis));
var haftaTatili = puantajList.Where(p => p.Secenek.Deger == "Ht").Count();
var resmiTatil = puantajList.Where(p => p.Secenek.Deger == "Rt").Count();
var yillikIzin = puantajList.Where(p => p.Secenek.Deger == "Yi").Count();
var odenecekRapor = puantajList.Where(p => p.Secenek.Deger == "R+").Count();
var dogumIzni = puantajList.Where(p => p.Secenek.Deger == "Di").Count();
var olumIzni = puantajList.Where(p => p.Secenek.Deger == "Öi").Count();
var evlilikIzni = puantajList.Where(p => p.Secenek.Deger == "Ei").Count();
var odenmeyecekRapor = puantajList.Where(p => p.Secenek.Deger == "R-").Count();
var ucretsizIzin = puantajList.Where(p => p.Secenek.Deger == "Üi").Count();
var devamsizlik = puantajList.Where(p => p.Secenek.Deger == "D").Count();
return hesaplamalar;
}

As for your case, querying and counting in the db is more efficient.
something like this would be efficient.
puantajlar
.Where(p => p.PersonelId == personelId && (p.Tarih >= baslangic && p.Tarih <= bitis))
.GroupBy(x => x.Secenek.Deger)
.Select(group => new { group.Key, Count = group.Count() })

My question is if should I get the count in C# code with a IEnumarable or by querying a IQueryable in DB
If you need only the count of the rows then count must be done in database, not in memory. If you do count in memory by pulling the data list from database into memory then it will waste your server memory unnecessarily and cost performance.

Complexity and Performance, both of them depends on your situation, if there are no huge data the Performance is no matter, but sometime you have to make a decision based on your situation.
By running your code it should connect to DB and run the count query in each line of code
it is 100 per cent clear that counting same rows in DB is more efficient in one shoot so you can do something like :
select p.Secenek.Deger,
....
sum(case when p.Secenek.Deger = 'Ht' then 1 else 0 end) haftaTatili,
sum(case when p.Secenek.Deger = 'Rt' then 1 else 0 end) resmiTatil
.....
from puantajlar p
group by p.Secenek.Deger
or you can do it in more efficient way by grouping them in one shoot also like #amd mentioned:
puantajlar
.Where(p => p.PersonelId == personelId && (p.Tarih >= baslangic && p.Tarih <= bitis))
.GroupBy(x => x.Secenek.Deger)
.Select(group => new { group.Key, Count = group.Count() })

Interpreting Sql with where and having statement to linq giving different results

Please - I have this SQL statement:
SELECT FK_ClassId, LectureDays
FROM tbl_TimeTables
WHERE Term = 'First Term'
AND FK_session = 4
AND fk_classId = 1
AND (fk_subjectid <> 1)
GROUP BY FK_classId, LectureDays
HAVING (COUNT(*) < 6)
This returns this result:
Image Embedded Here, giving the right result
But when I interpret to linq, I get the a different result:
Tbl_TimeTables.GroupBy(x => new { x.FK_Session, x.Term, x.LectureDays,
x.FK_ClassId, x.FK_SubjectId })
.Where(grp => grp.Count() < 6 && grp.Key.FK_Session == 4 && grp.Key.Term ==
"First Term" && grp.Key.FK_ClassId == 1 && grp.Key.FK_SubjectId != 1)
.Select(grp => new
{
LectureDay = grp.Key.LectureDays,
ClassId = grp.Key.FK_ClassId
})
Wrong Results Picture Link here
Please look at my code, what am I doing wrong?
Thanks
Tim

This is the right way the linq query should go according to Matt Gibson's suggestion:
Tbl_TimeTables
.Where(x => x.FK_Session == 4 && x.Term == "First Term" && x.FK_ClassId == 1
&& x.FK_SubjectId != 1)
.GroupBy(x => new { x.FK_ClassId, x.LectureDays })
.Where(grp => grp.Count() < 6)
.Select(grp => new
{
ClassId = grp.Key.FK_ClassId,
LectureDay = grp.Key.LectureDays
})
This works exactly like the sql
Also to point out that this link: http://www.dotnettricks.com/learn/sqlserver/definition-use-of-group-by-and-having-clause helped me in understanding how the having statement works, which helped in seeing what Matt what saying.

Sum values in datatable using linq based on conditions

I'm having a datatable like mentioned below.
ID Percentage
1 50
1 30
2 0
2 100
Result:
ID Percentage
1 80
2 100
I tried this and it doesn't work
var n = dt.AsEnumerable()
.Where(r => (int)r["ID"] != "0" || (int)r["ID"] != "100")
.Sum(r => (int)r["Percentage"]);
I'm new to linq and pls provide some suggestions.
Now I need to sum the percentage for each ID and the percentage for each ID should be 0 or 100 percentage.
If any one of the ID in table doesn't have 0 or 100 I need to alert. Pls suggest me how I can do this in linq and I think its the best way.

var result = from row in dt.AsEnumerable()
group row by row["ID"]
into g
select new
{
ID = g.Key,
Sum = g.Sum(x => int.Parse(x["Percentage"].ToString()))
};
var errorItems = result.Where(x => x.Sum != 100 && x.Sum != 0);
if (errorItems.Any())
{
var ids = errorItems.Select(x => x.ID);
string msg = string.Format("ID(s): [{0}] don't meet condition.", string.Join(",", ids));
MessageBox.Show(msg);
}

You are not trying get the sum of "Percentage" for the whole table so directly doing a sum on it wont give you the desired result.
You're trying to find the sum of the percentage for each ID value so you need to group it by ID.
That's what GroupBy(g => g.Field<int>("ID")) does. Then you take the group(g), and for each group, you sum the "Percentage" Column of the members i.e.. .Select(g => g.Sum(p => p.Field<int>("Percentage")))
Here is the complete code.
dt.AsEnumerable().Where(r => r.Field<int>("ID") == 0 || r.Field<int>("ID") == 100).GroupBy(g => g.Field<int>("ID")).Select(g => g.Sum(p => p.Field<int>("Percentage")));
to put an alert message you can use Any instead of the where to check for the presence of the values
if(dt.AsEnumerable().Any(r => r.Field<int>("ID") != 0 && r.Field<int>("ID") != 100)
{
Console.WriteLine("Alert");
}

I guess that you want a new DataTable with the same columns as the first but with grouped percentage by ID? Then have a look at GroupBy and Sum:
var groupQuery = dt.AsEnumerable()
.Select(r => new { ID = r.Field<int>("ID"), Percentage = r.Field<int>("Percentage") })
.Where(x => x.ID != 0 && x.ID != 100)
.GroupBy(x => x.ID);
DataTable groupingTable = dt.Clone(); // empty, same columns
foreach(var grp in groupQuery)
groupingTable.Rows.Add(grp.Key, grp.Sum(x => x.Percentage));
This presumes that the type of the columns is actually int. If they are strings the best way is to change it to int, if you can't do that you have to use int.Parse.
For example:
ID = int.Parse(r.Field<int>("ID"))`
Update: Although it's not clear what you want if i reread your qustion, especially:
If any one of the ID in table doesn't have 0 or 100 I need to alert
You could use this to get all ID-groups without 0 or 100 percentage:
var without0Or100Perc = dt.AsEnumerable()
.Select(r => new { ID = r.Field<int>("ID"), Percentage = r.Field<int>("Percentage") })
.GroupBy(x => x.ID)
.Where(g => !g.Any(x => x.Percentage == 0 || x.Percentage == 100));
Now you can use Any, FirstOrDefault or a foreach loop to consume this query, so one of following approches:
bool anyWithout0Or100Perc = without0Or100Perc.Any();
var firstWithout0Or100Perc = without0Or100Perc.FirstOrDefault();
anyWithout0Or100Perc = firstWithout0Or100Perc != null;
foreach (var grp in without0Or100Perc)
{
Console.WriteLine("ID: {0} Percentages:{1}",
grp.Key,
String.Join(",", grp.Select(x => x.Percentage)));
}

Using LINQ for quering in a big database take so much time

I'm using LINQ for preparing some data in my controller and sending them to view.
My connection provided by EF6 code first migration with sql server
So in the controller and with a LINQ expression , the model of database map to proper view model as follow:
var temp = db.points.ToList().Select(pnt => new MapPointsModel()
{
pointId = pnt.pointId,
name = pnt.name,
positionX = pnt.positionX,
positionY = pnt.positionY,
road = pnt.road.id,
order = pnt.order,
signalState = pnt.signalState,
powerState = pnt.powerState,
videoState = pnt.videoState,
cameraState = pnt.cameraState,
hourTraffic = new int[]{
pnt.crossings.Where(c => DateTime.Compare(c.dateTime, lastHour) >= 0 ).Where(c => c.line == 1).Count(),
pnt.crossings.Where(c => DateTime.Compare(c.dateTime, lastHour) >= 0 ).Where(c => c.line == 2).Count(),
pnt.crossings.Where(c => DateTime.Compare(c.dateTime, lastHour) >= 0 ).Where(c => c.line == 3).Count()
},
dayTraffic = new int[]{
pnt.crossings.Where(c => DateTime.Compare(c.dateTime, lastDay) >= 0 ).Where(c => c.line == 1).Count(),
pnt.crossings.Where(c => DateTime.Compare(c.dateTime, lastDay) >= 0 ).Where(c => c.line == 2).Count(),
pnt.crossings.Where(c => DateTime.Compare(c.dateTime, lastDay) >= 0 ).Where(c => c.line == 3).Count()
},
hourViolation = new int[] {
pnt.crossings.Where(c => c.violation != null && DateTime.Compare(c.dateTime, lastHour) >= 0).Where(c => c.line == 1).Count(),
pnt.crossings.Where(c => c.violation != null && DateTime.Compare(c.dateTime, lastHour) >= 0).Where(c => c.line == 2).Count(),
pnt.crossings.Where(c => c.violation != null && DateTime.Compare(c.dateTime, lastHour) >= 0).Where(c => c.line == 3).Count()
},
dayViolation = new int[] {
pnt.crossings.Where(c => c.violation != null && DateTime.Compare(c.dateTime, lastDay) >= 0).Where(c => c.line == 1).Count(),
pnt.crossings.Where(c => c.violation != null && DateTime.Compare(c.dateTime, lastDay) >= 0).Where(c => c.line == 2).Count(),
pnt.crossings.Where(c => c.violation != null && DateTime.Compare(c.dateTime, lastDay) >= 0).Where(c => c.line == 3).Count()
},
checkedViolations = pnt.crossings.Where(c => c.violation != null).Where(c => c.violation.deliberated == true).Count(),
uncheckedViolations = pnt.crossings.Where(c => c.violation != null).Where(c => c.violation.deliberated == false).Count(),
bandAvgSpeed = new int[] {
pnt.crossings.Where(c => c.line == 1).Count() == 0 ? 0 : pnt.crossings.Where(c => c.line == 1).Sum(c => c.speed)/pnt.crossings.Where(c => c.line == 1).Count(),
pnt.crossings.Where(c => c.line == 2).Count() == 0 ? 0 : pnt.crossings.Where(c => c.line == 2).Sum(c => c.speed)/pnt.crossings.Where(c => c.line == 2).Count(),
pnt.crossings.Where(c => c.line == 3).Count() == 0 ? 0 : pnt.crossings.Where(c => c.line == 3).Sum(c => c.speed)/pnt.crossings.Where(c => c.line == 3).Count(),
},
});
return temp.ToList();
this code works for 10000 records or lower, but in 500000 record or more there is no result and in all tests timeout accurse.
I'm looking for the reason of this problem
UPDATE:
There is just 4 records in "points" table, the bigger one with more than 500000 records is "crossings" however I tried to solve the problem with removing .toList() from db.points, but it rises exception:
System.ArgumentException: Argument types do not match

Your problem is this:
db.points.ToList()....
This retrieves ALL records in their entirety from the table into memory. So, the more records you have, the longer this will take. You need to create a query that returns only the records you need.
I'm not sure what you're planning to do with 500,000 records all at once.. do you only need a subset? If so, then do something like this:
db.points.Select(....).Take(25) // or however many you need.
You're also doing numerous sub-selects and sub-counts, each of those are separate statements that get executed, so for those 500,000 you might actually have many millions of sub-queries.

You are transforming it to a List at the beginning. This is taking up all the time. Try removing the ToList() from db.points.ToList().Select. This should solve your problem.
Until you convert it using a ToList, it still is a IQueryable and will not hit the database. The moment you do a ToList, it will query the database, fetch the results in memory and then process the records in memory.

doing this
var temp = db.points.ToList()....
bring the whole table to the client, so if you have a few meg / gig of data this can take a while depending on cpu / connection / memory

You need to change the ToList() from the code and change var temp = db.points.AsNoTracking() This will save your time .Also creating pre-generated views for your code first model gives better performance.

The main problem was Average !!!
Because average function runs 4 times over the all records, it takes about 7 seconds.
I changed the linq query and instead of requesting all data at once , the average section calculated in other request from server with OrderBy GroupBy tools. and then combined two lists that retrieve from server.
Beside, I removed .ToList() from points table (as folks suggested) and changed arrays constructions to simple variables (this caused error!).
This solution caused runtime take just about 1 second instead of 7 second with the same output :)

Couting items and grouping them by levels

I am using Entity Framework 6 and I have the following Linq Query:
IDictionary<BodyMassIndexLevel, Int32> bmistats =
context.Evaluations
// Get all evaluations where Height and Weight measures were done
.Where(x => x.Height != null && x.Weight != null)
// Select the date of the evaluation, the worker id and calculate BMI
.Select(x => new { Date = x.Date, Worker = x.Worker.Id, BMI = x.Weight.Value / Math.Pow(x.Height.Value / 100, 2) })
// Group by worker
.GroupBy(x => x.Worker)
// Get the most recent evaluation for each worker and so the most recent BMI
.Select(x => x.OrderByDescending(y => y.Date).Select(y => new { BMI = y.BMI }).FirstOrDefault())
// Cache the result in memory
.ToList()
// Count the number of BMIS in each level
.With(z =>
new Dictionary<BodyMassIndexLevel, Int32> {
{ BodyMassIndexLevel.SevereThinness, z.Count(w => w.BMI < 16) },
{ BodyMassIndexLevel.MildThinness, z.Count(w => w.BMI >= 16 && w.BMI < 17) },
{ BodyMassIndexLevel.ModerateThinness, z.Count(w => w.BMI >= 17 && w.BMI < 18.5) },
{ BodyMassIndexLevel.Normal, z.Count(w => w.BMI >= 18.5 && w.BMI < 25) },
{ BodyMassIndexLevel.PreObese, z.Count(w => w.BMI >= 25 && w.BMI < 30) },
{ BodyMassIndexLevel.ObeseClassI, z.Count(w => w.BMI >= 30 && w.BMI < 35) },
{ BodyMassIndexLevel.ObeseClassII, z.Count(w => w.BMI >= 35 && w.BMI < 40) },
{ BodyMassIndexLevel.ObeseClassIII, z.Count(w => w.BMI >= 40) }
}
);
I have two questions:
Is is possible to improve the performance of this query?
Can I move the Count part in levels to the query and so having not ToList()?

For example
Make something like Truncate for BMI after
// Select the date of the evaluation, the worker id and calculate BMI
Create BMILevel table with columns (BMILevelName | BMIValues) containing rows like (BodyMassIndexLevel.ModerateThinness, 17), (BodyMassIndexLevel.PreObese, 25), (BodyMassIndexLevel.PreObese, 26), etc.
JOIN your select query with *BMILevel* table on query.BMI = BMILevel.BMIValue, than GroupBy BMILevel.BMILevelName and finally Count for all groups.
Alternatively you may define BMILevel with columns (BMILevelName | BMIValueBeginInterval, BMIValueEndInterval) containing rows like (BodyMassIndexLevel.ModerateThinness, 17, 18), (BodyMassIndexLevel.PreObese, 25, 30).
And thus perform
query JOIN BMILevel ON query.BMI BETWEEN BMILevel.BMIValueBeginInterval AND BMILevel.BMIValueEndInterval
I consider EF can transform '<', '&&', '>' within .Where() (or Join) call properly
UPDATE:
If you don't want to create another one table, you may try create in-memory list of objects of sample type
class BMILevel {
public BMILevelEnum BMILevelName {get;set;}
public double BMILevelValueBeginInterval {get;set;}
public double BMILevelValueEndInterval {get;set;}
}
than create in-memory collection:
var bmiLevels = new List<BMILevel> { new BMILevel {...}, ... }
and use it in the way I describe it above.
I don't know how good EF 6 is, but old versions were unable to handle operations with non-entities (it couldn't translate expressions to proper SQL) and thus it results in inefficient querying or errors.
The only way to perform your query faster is to delegate it to SQL server. You can use EF abilities and thus it's possible that it requires of the creation of a new table. Another way - use ADO.NET (SqlCommand, SqlConnection, etc) and do it bypassing EF.