I am new to C# (has experience in other languages before, C++, SQL AutoIT). I have a datatable with 10 columns
Name, MemberNoA, MemberNoB, DriverLicense, MobileNo, Address1, Address2, Address3, ProgramJoinned, Remark
The datatable has around 17,000 rows, what I want to do is, if the same person's records appear more than 2 times in the datatable, put a description in a remark field.
4 criteria to define "same person", any one criteria match will treat as "same person"
i Name + MemberNoA
ii Name + MemberNoB
iii Name + DriverLicense
iv Name + MobileNo
i.e. if there are 3 records with same Name and same MemberNoA, need to put description into remark field of these 3 records.
I work out result set from the above 4 criteria like this:
var resultCriteria1 = from gpr in dt.AsEnumerable()
group gpr by new {
Fld1 = gpr.Field < string > ("Name"),
Fld2 = gpr.Field < string > ("MemberNoA")
}
into grpp
where grpp.Count() > 2
select new {
Name = grpp.Key.Fld1,
MemA = grpp.Key.Fld2,
Cnt = grpp.Count()
};
after that, I loop thru all rows in dt and for each row, loop thru all result set in 4 criteria:
for (int i = 1; i < dt.Rows.Count; i++) {
foreach(var item in resultCriteria1) {
if ((item.Nam == s trName) && (item.MemA == M emberNoA)) {
dt.Rows[i].ItemArray[9] = d t.Rows[i].ItemArray[9] + "Criteria 1 match\r\n";
}
}
}
The program work but run very slow! Is there any method like simple sql statement:
update table1 where
table1.name = table2.name and
table1.MemberNoA = table2.MemberNo2
set table1.Remark = "Criteria 1 match\r\n"
Is there any way to do this in C# or any way to optimize it ? Thanks.
Regds
LAM Chi-fung
The problem is that you are making a cartesian join between the grouped results and the original datatable, without using any performant data structures such as a dictionary or hashset.
But you don't actually need to join it at all, the grouped results can actually hold the relevant data rows directly.
The following code should be performant enough
var grouped =
from gpr in dt.Rows.Cast<DataRow>()
group gpr by (
Fld1: (string)gpr["Name"],
Fld2: (string)gpr["MemberNoA"]
)
into grpp
where grpp.Count() > 2
select grpp;
foreach (var grp in grouped)
foreach (var row in grp)
row["Remark"] += "Criteria 1 match\r\n";
What you can do is to use hashtable, order your data, and then iterate comparing current row data with previous using a cursor. This should give you Log(n) time complexity.
Related
I have two different tables from a DataBase named "empleados" and "fichajes". empleados has the employees data and fichajes has the date and time from when they started working.
I want to get the total work time done by a specific employee between two dates, lets say from 20th to 29th.
I have this query which I use with Dapper on C#:
SELECT CONCAT(e.nombre, " " ,e.apellido) as fullName, tfichajes.total
FROM empleados e
INNER JOIN (
SELECT f1.nif,
SEC_TO_TIME(SUM(TIME_TO_SEC(f1.fechasalida) - TIME_TO_SEC(f1.fechaentrada))) AS total
FROM fichajes f1
WHERE f1.fechasalida <= '2019-04-29'
and f1.fechaentrada >= '2019-04-20'
GROUP BY f1.nif
) AS tfichajes
ON e.nif = tfichajes.nif
WHERE e.nif = '33333333P'
This works just fine, but I was wondering if it was possible to make it simpler.
This is the code I have in my program:
public static List<string> CalculaTotalHoras(string nif, DateTime fechaEntrada, DateTime fechaSalida)
{
var dbCon = DBConnection.Instancia();
if (dbCon.Conectado())
{
string format = "yyyy-MM-dd";
List<string> result = new List<string>();
using (IDbConnection conexion = dbCon.Conexion)
{
var output = conexion.Query($"SELECT CONCAT(e.nombre, \" \" ,e.apellido) as fullName, tfichajes.total " +
$"FROM empleados e INNER JOIN (SELECT f1.nif, SEC_TO_TIME(SUM(TIME_TO_SEC(f1.fechasalida) - TIME_TO_SEC(f1.fechaentrada))) AS total " +
$"FROM fichajes f1 where f1.fechasalida <= '{fechaSalida.ToString(format)}' and f1.fechaentrada >= '{fechaEntrada.ToString(format)}' GROUP BY f1.nif) " +
$"as tfichajes ON e.nif = tfichajes.nif where e.nif = '{ nif }';").ToList();
var i = 0;
foreach (IDictionary<string, object> row in output)
{
foreach (var pair in row)
{
if (i == 0)
{
result.Add(pair.Value.ToString());
i++;
}
else
{
result.Add(pair.Value.ToString());
}
}
}
return result;
}
}
else return null;
}
If you have problems with the readability of the code here you have a gyazo.
Workingtime fichajes table and employees empleados table.
With that exact query, the expected results are Alvaro Arguelles 00:05:00, and in the code, I want to get Alvaro Arguelles and 00:05:00 separated in the result List.
Did I made it much harder than it actually is?
Your query can be significantly simplified. Your inner query is selecting ALL employees working in the date in question, I would start with the INNER query as the main FROM table getting the data for the one employee. THEN join to the employee to grab the name.
select
CONCAT(e.nombre, " " ,e.apellido) as fullName,
OneEmp.Total
from
( select
f1.nif,
SEC_TO_TIME( SUM(TIME_TO_SEC(f1.fechasalida)
- TIME_TO_SEC(f1.fechaentrada))) AS total
from
fichajes f1
where
f1.nif = '33333333P'
AND f1.fechasalida >= '2019-04-20'
AND f1.fechasalida <= '2019-04-29' ) OneEmp
JOIN empleados e
on OneEmp.nif = e.nif
Since your query was summarizing per the one employee, it would only return a single row anyhow. So the above query will return 1 row, 2 columns... name and total. Should be able to read those two values back directly.
But as others commented, PARAMETERIZE your query calls.
I have a list of product records that I am displaying on a dataTable, that are grouped by department name / id. From the image below my returned data does not include the highlighted rows. I am trying to achieve whats shown in the image in that - for each set of products already grouped by department (deserts, stationery, household) get the totals, and name of department (and other columns which I removed to show less clutter) to append to the list
I am trying to achieve the outcome above
This is my current code/query which returns this data without the highlighted rows
List<SaleDetailViewModel> list = new List<SaleDetailViewModel>();
NpgsqlCommand query = new NpgsqlCommand
("SELECT q_department.q_name, q_saledetail.q_code, q_saledetail.q_description, " +
"SUM(q_quantity) AS qtysum, " + //running total of q_quantity for the product in that department
"FROM q_saledetail " +
"LEFT JOIN q_department ON (q_saledetail.q_departmentid = q_department.q_code ) " +
"GROUP BY " +
"q_saledetail.q_departmentid, q_department.q_name, q_saledetail.q_description, q_saledetail.q_code " +
"ORDER BY q_saledetail.q_departmentid "
, connectToDB);
NpgsqlDataReader read = query.ExecuteReader();
while(read.Read())
{
var reportData = new SaleDetailViewModel();
reportData.departmentName = read["q_name"].ToString(); //department name
reportData.q_description = read["q_description"].ToString(); //product DESCRIPTION
reportData.q_code = read["q_code"].ToString(); //product BAR CODE
reportData.sumOfQuantity = Convert.ToDecimal(read["qtysum"]); //sum of quantity sold for that product
list.Add(reportData);
}
connectToDB.Close();
return list;
My challenge now is adding the grouped department row data for each set of products, and append to the list that will be returned i.e
//--
foreach(grouped dataset by department)
{
//get the totals and heading for each set of data
reportData.q_code = //insert department id
reportData.q_description = //insert department name
reportData.sumOfQuantity = // insert total for that department
list.Add(reportData) //add that to the list that will be shown on view
}
//--
I can get department id and code for the grouped data using read.GetString().
string deptName = read.GetString(0);
How do I continue to loop through the dataReader and add totals for each overall set like the quantity columns and have that as a row of its own to be added to the list?... to get the result from the image shown.
Attempt to explain a bit better:
Within my while loop how do I aggregate the set for each department group of products and create a row to add to the list. Or there's a better way of achieving this..
Option 1: Sql and union.
SELECT
q_department.q_name,
q_saledetail.q_code,
q_saledetail.q_description,
SUM(q_quantity) AS qtysum, --running total of q_quantity for the product in that department
FROM q_saledetail
LEFT JOIN q_department ON (q_saledetail.q_departmentid = q_department.q_code )
GROUP BY
q_saledetail.q_departmentid, q_department.q_name, q_saledetail.q_description, q_saledetail.q_code
UNION
SELECT
q_department.q_name,
'',
'',
sum(q_quantity)
FROM q_saledetail
LEFT JOIN q_department ON (q_saledetail.q_departmentid = q_department.q_code )
GROUP BY
q_saledetail.q_departmentid, q_department.q_name
ORDER BY q_saledetail.q_departmentid
By Removing some of the fields that identify different q_saledetail from the group by list, the sum function will sum for all departments instead of breaking the sums up.
Sorry but I can't remember exactly how ordering works with union statements off hand, but I know you can only order
in one place, either the last statement, or the first, or with an outer query that selects from the union.
What you'll have in your data now, is rows that are the 'total' rows for each department, that you know are
'total' rows because the q_code or q_description fields are blank. You can then order the data set by department and then by code (asc or desc) so that the empty code field is at the end of the group of that department and display it like that.
Option 2: Linq and a little manual code work
List<SaleDetailViewModel> list = new List<SaleDetailViewModel>();
while(read.Read())
{
... //Your mapping code
list.Add(reportData);
}
var groupedList = list.GroupBy(x => x.departmentName);
Which will give you a IEnumerable<IGrouping<string, SaleDetailViewModel>.
What that is a your data now grouped by the name of each department which acts as a key
You'll now loop through groupedList and add the values of that IGrouping up.
foreach(var group in groupedList)
{
var total = 0;
foreach(var saledetail in group)
{
total += saledetail.sumOfQuantity;
}
}
Then you just need to add the logic to add it to a list/your datatable in the order of those groups.
Hope this helps.
Query Detail Table
Main Table
1 .PK
Detail Table
1. PK
2. Detail description.
If mysql i can group the main table PK for duplicate issue but how it implement in sql server ?Since sql server required something like aggregate
E.g
(vehicle) Main Table
Car
bikes
van
(vehicleItem) Detail Table
Car->item a,item b
bikes-> item a,item c..
van->item b ,item c
I want to filter in a query such as i want to filter item a only?Seem quite inefficient to call in sub loop.Even put all detail item it in a column consider as incorrect but when search item id(number) might be same number unless the item id are guid and distinct different.
** the main purpose is to output list not to sum crosstab :) ya.. grouping cool if you want to sum up.
var sql = "select * from vehicle ";
var command = new SqlCommand(connection,sql);
try {
var reader = command.ExecuteReader();
if (reader.HasRows) {
while (reader.Read())
if (!string.IsNullOrEmpty(Request.QueryString["ItemIdValue[]"])){
var d = Request.QueryString.GetValues("ItemIdValue[]");
if (d != null){
if (d[0].Contains("all")){
if (GetItemExist(" AND vehicleItem.itemId in (SELECT itemId from item ) ")){ }
}else{
if (GetItemExist(" AND vehicleItem.itemId IN ( " + itemFilter.Remove(itemFilter.Length - 1) + " ) ")){ }
}
}
}
}
}
}
At last thinking back, why not select main table and in subquery..
var sql = "select * from vehicle ";
if (!string.IsNullOrEmpty(Request.QueryString["itemIdValue[]"])){
var d = Request.QueryString.GetValues("itemIdValue[]");
if (d != null){
if (d[0].Contains("all")){
sql = sql + #"
AND vehicleId IN (
SELECT vehicleId
FROM vehicle
JOIN vehicleItem
ON vehicle.vehicleId = vehicleItem.vehicleId
WHERE vehicleItem.itemId in
(SELECT itemId from item)
) ";
} else{
for (var e = 0; e < d.Length; e++){
itemFilter += d[e] + ",";
_fieldVariable.Add("itemIdValue[]");
_valueVariable.Add(d[e]);
}
sql = sql + #"
AND vehicleId IN (
SELECT vehicleId
FROM vehicle
JOIN vehicleItem
ON vehicle.vehicleId = vehicleItem.vehicleId
WHERE vehicleItem.itemId in ( " + itemFilter.Remove(itemFilter.Length - 1) + " ) ) ";
}
}
}
When joining two tables t1 and t2, a result set containing all matching rows from table t1 and table t2 is created. When you only want one resulting row per t1's row, it is up to you to filter the join accordingly.
There are several possible ways to do that:
adapt the join's ON clause
group accordingly
specify a WHERE clause
As you wanted to use grouping in your request, you could use a query like this:
SELECT m.PK, min(d.Detail)
FROM Main m
LEFT JOIN Detail d on d.PK=m.pk
WHERE d.Detail = 'item a'
GROUP by m.PK
That way you will get one row per PK with the minimum value in detail. You can use other aggregate functions too. See https://msdn.microsoft.com/en-us/library/ms173454.aspx.
Edit: Added WHERE clause to filter by "item a".
From this database, I want to select two users, 321 and 102 where BOOK_TYPE is 1 or 2 and compare their BOOK_NR, if they coincide then save BOOK_COUNT only from USER_ID 102.
SqlCommand command = new SqlCommand("SELECT BOOK_NR, BOOK_COUNT, USER_ID, BOOK_TYPE " +
"FROM BOOKS " +
"WHERE (BOOK_TYPE = '1' OR" + "BOOK_TYPE = '2')" +
"AND MONTH(DATE) = '" + DateTime.Today.Month + "'" +
"GROUP BY BOOK_NR, BOOK_COUNT, USER_ID, BOOK_TYPE ", BookConn);
SqlDataReader reader = command.ExecuteReader();
while (reader.Read())
{
string BookNr = (string)reader[0];
int Count = (int)reader[1];
int User = (int)reader[2];
int Type = (int)reader[3];
if( " HERE I NEED HELP " )
{
" AND HERE :) "
}
}
reader.Close();BookConn.Close();
My only solution is as follows
List<string> User321List = new List<string>();
List<string> User102List = new List<string>();
if(User == 321 && Type == 1){ User321List.Add(BookNr.Trim());}
if(User == 102 && Type == 2){ User102List.Add(BookNr.Trim()+"\t"+Count);}
and then...
int count = 0;
foreach (string x in User321List)
{
foreach (string y in User102List)
{
List<string> part = y.Split('\t').Select(p => p.Trim()).ToList();
var z = string.Compare(part[0], x);
if (z == 0)
{
count += int.Parse(part[1]);
}
}
}
but it takes a lot of time to get result..
if someone figured something out of my nightmare please help and sorry for my bad english...
I'm going to focus on answering just the first part:
I want to select two users, 321 and 102 where BOOK_TYPE is 1 or 2 and compare their BOOK_NR, if they coincide then save BOOK_COUNT only from USER_ID 102.
You want to join the table to itself:
SELECT b1.BOOK_NR, b1.BOOK_COUNT, b1.USER_ID, b1.BOOK_TYPE
FROM BOOKS b1
INNER JOIN BOOKS b2 ON b2.BOOK_TYPE IN ('1','2')
AND b2.User_ID = 321
AND b1.BOOK_NR = b2.BOOK_NR
WHERE b1.User_ID = 102 AND b1.BOOK_TYPE IN ('1','2')
AND MONTH(b1.DATE) = MONTH(current_timestamp)
AND MONTH(b2.DATE) = MONTH(current_timestamp)
Since one of your concerns was performance -- that the query was too slow -- I also need to ask whether the BOOK_TYPE column is a string or a number. If it's a number column, you should omit the single quotes on the values: for example, just use IN (1,2) instead of IN ('1','2'). This will help performance by avoiding potential per-row conversions and by potentially better matching to indexes.
If your query is slow, consider splitting the operation in two. Create a SQL Table function so you don't repeat the SQL, passing the users, type and month in as parameters.
Do not do an IN (as suggested in comment). Keep in mind that an OR in the WHERE clause will result in no indexes being used.
Join the two instances of the table function and let sql do the matching for you.
e.g.
Select * from dbo.UserList('321',1,12) as u321
inner join dbo.UserList('102',2,12) as u102
on u321.field = u102.field
BTW... I covered quite a few advanced topics, but the idea is to let SQL do what it does best: JOINS
As a matter of interest, if your sql table function returns many rows, and you ever use them in a sp, remember to use the option (recompile) clause ...
I have bill details in list which contains the heading:
Account Numbers, Dates, Service Number, Charge category, Details, exgst, ingst, Detailsfill
Column :- Account Numbers, Dates, Service Number, Charge category, Details, exgst, ingst
are unique values but Column Detailsfill contains null. I want to allow my C# script to set values for Detailsfill where Detailsfill is null then insert data from previous row otherwise do nothing.
Note that Running total sets to 1 when detailsfill column has any text but continues to increment by 1 from 3 when detailsfill column is empty.
Below is the script I used.
var result_set2 = (from a in result_set1
join b in
(from x in result_set1a select x)
on
new {col1 = a.id, col2 = a.account, col3 = a.date, col4 = a.service, col5 = a.chargecat }
equals
new { col1 = b.id, col2 = b.account, col3 = b.date, col4 = b.service, col5 = b.chargecat } into outer
from tb in outer.DefaultIfEmpty()
where a.running_total1 != 0
where a.running_total1 != 2
where a.chargecat != ""
select new
{
running_total = a.running_total1,
account = a.account,
date = a.date,
service = a.service,
chargecat = a.chargecat,
details = a.details,
exgst = a.exgst,
ingst = a.ingst,
detailsfill = ((tb == null) ? "" : tb.details)
}).ToList();
foreach (var line in result_set2)
{
mobilenet_writerfile.WriteLine(line);
}
It sounds to me like you need a correlated subquery to populate detailsfill. This would be in the form of a select from a context with whatever parameters you determine fulfill your requirements. Example on second post of this thread: http://social.msdn.microsoft.com/forums/en-US/linqprojectgeneral/thread/18da37b0-b40e-444e-8ec7-7bd343e0c799/
Also you can find other examples here on stackoverflow: Is it possible to create a correlated subquery in LINQ?
More information would be required to write a contextual example but here is basically what it would look like:
Entities context = new Entities();
context.tbl.Where(t=>t.Id == someId)
.Select(t=> new() {
t.Id,
context.tbl2.First(tbl2=>tbl2.Id == t.Id).Value, //This is the subquery
t.whatever});
This is not a normal operation of Linq to hold onto a previous value and then use it later. That's not to say you can't make that work with Linq, but it would be more straightforward to keep that logic outside of the sequence operation. Simply iterate over the results after the query is composed.
var item = results.First();
foreach (var obj in results)
{
if (obj.detailsfill == null)
obj.detailsfill = item.detailsfill;
item = obj;
}
Note: If the first item happens to contain null in detailsfill, you will not pick up a value until the first item in the sequence that does contain a value. To get the first such item up front, simply use
var item = results.First(o => o.detailsfill != null);
// loop below