We Keep Coding

Efficient Converting DataTable to List in C#

I want to serialize the DataTable to a List in C#. My code is taking around 10 minutes for 100K data to fetch and convert to a list. Previously tried with sequential way but it three times more time to process. So, I tried Parallel and it saved 1/3 of time. But, still, it's too slow.
If anyone can help to make it fast. Any help is appreciated.
public async Task<List<DataPointModel>> GetDetails(HashParameterModel parameterModel)
{
List<DataPointModel> result = new List<DataPointModel>();
try
{
var query = "USP_GetDetailsFromMetaData";
using (var sqlConnection = new SqlConnection(connectionString))
{
using (var sqlCommand = new SqlCommand(query, sqlConnection))
{
sqlCommand.CommandType = CommandType.StoredProcedure;
sqlCommand.Parameters.AddWithValue("#GroupID", parameterModel.GroupID);
sqlCommand.CommandTimeout = 0;
sqlConnection.Open();
DataTable dataTable = new DataTable();
DataSet dataSet = new DataSet();
SqlDataAdapter da = new SqlDataAdapter
{
SelectCommand = sqlCommand
};
da.Fill(dataSet);
dataTable = dataSet.Tables[0];
DataTable dataTable1 = dataSet.Tables[1];
var questionList = dataTable1.AsEnumerable();
if (dataTable.Rows.Count > 0)
{
Parallel.ForEach(dataTable.AsEnumerable(), new ParallelOptions { MaxDegreeOfParallelism = 5 }, row =>
{
DataPointModel model = new DataPointModel();
model.ID = Convert.ToInt32(row["ID"]);
model.GroupID = Convert.ToInt32(row["GroupID"]);
model.ProviderID = Convert.ToInt32(row["SalvageProviderID"]);
model.ClaimNumber = row["ClaimNumber"].ToString();
model.PolicyNumber = row["PolicyNumber"].ToString();
model.DataPoint1 = row["DataPoint1"].ToString();
model.DataPoint2 = row["DataPoint2"].ToString();
model.DataPoint3 = row["DataPoint3"].ToString();
model.DataPoint4 = row["DataPoint4"].ToString();
model.FirstName = row["FirstName"].ToString();
model.LastName = row["LastName"].ToString();
model.PrimaryDamage = row["PrimaryDamage"].ToString();
model.Type = row["TypeCode"].ToString();
model.LossDate = row["LossDate"].ToString();
model.QuestionList = (from p in questionList
where p.Field<int>("ID") == model.ID
select new QuestionResponseModel()
{
QuestionID = p.Field<int>("QuestionID").ToString(),
Response = p.Field<string>("ResponseValue")
}).ToList();
result.Add(model);
});
}
}
}
}
catch (Exception ex)
{
throw ex;
}
return result;
}
The DataSet has two DataTable
DataTable dataTable = dataSet.Tables[0]; // Details
DataTable dataTable1 = dataSet.Tables[1]; // QUestionList
I think the time is consumed when its looping interlly for QuestionList, which may have around 120K rows. Any Suggestions

An easy and quick to implement performance improvment would be to build a look up table from your questionList, and access this to fetch the question instead of doing this piece of code
model.QuestionList = (from p in questionList
where p.Field<int>("ID") == model.ID
select new QuestionResponseModel()
{
QuestionID = p.Field<int>("QuestionID").ToString(),
Response = p.Field<string>("ResponseValue")
}).ToList();
So add the following
var questionList = dataTable1.AsEnumerable();
//maybe add .AsParallel() - questionList.AsParallel().ToLookUp(...)
var questionLookUp = questionList.ToLookUp(x => x.Field<int>("ID"), x => new QuestionResponseModel() { QuestionID = x.Field<int>("QuestionID"), Response = p.Field<string>("ResponseValue") });
And than use it like this
model.QuestionList = questionLookUp[model.ID].ToList();
https://learn.microsoft.com/en-us/dotnet/api/system.linq.lookup-2
https://learn.microsoft.com/en-us/dotnet/api/system.linq.enumerable.tolookup
Code is untested, hope I didn't make to many errors.

Are there duplicate model IDs in the Details table? If so this could help to avoid running the question list query multiple times:
model.QuestionList = getQuestions(model.ID);
Method:
public Dictionary<int, List<QuestionResponseModel>> questionBuffer = new Dictionary<int, List<QuestionResponseModel>>();
public List<QuestionResponseModel> getQuestions(int ID)
{
if (questionBuffer.ContainsKey(ID)) return questionBuffer[ID];
List<QuestionResponseModel> questions = (from p in questionList
where p.Field<int>("ID") == model.ID
select new QuestionResponseModel()
{
QuestionID = p.Field<int>("QuestionID").ToString(),
Response = p.Field<string>("ResponseValue")
}).ToList();
questionBuffer.Add(ID, questions);
return questions;
}

How to get multiple string values in return in c# windows forms

i'm trying to get multiple values from database in return, i write a code and it return me only one value but i want all values in return. i didn't know how to solve it because i'm beginner in c#. here is code
string CommandText = "SELECT * FROM SV WHERE [SVCode]=#id";
using (SQLiteCommand cmd = new SQLiteCommand(CommandText, conn))
{
cmd.Parameters.AddWithValue("#id", sve.SVID);
conn.Open();
DataTable dt = new DataTable();
SQLiteDataAdapter da = new SQLiteDataAdapter(cmd);
da.Fill(dt);
foreach (DataRow dr in dt.Rows)
{
sve.SVName = dr["SVName"].ToString();
sve.SVCity = dr["SVCity"].ToString();
sve.SVState = dr["SVState"].ToString();
sve.SVEmail = dr["SVEmail"].ToString();
sve.SVWebsite = dr["SVWebsite"].ToString();
sve.SVAddress = dr["SVAddress"].ToString();
sve.SVNote = dr["SVNote"].ToString();
}
return sve.SVName; //how to return SVCity,SVState,SVEmail,SVWebsite,SVAddress,SVNote
}
private string SVCB(){
SVE sve = new SVE();
sve.SVID = MSVD_vendorcode.SelectedItem.ToString();
sve.SVName = MSVD_vendorname.Text;
sve.SVCity = MSVD_vendorcity.Text;
sve.SVState = MSVD_vendorstate.Text;
sve.SVEmail = MSVD_vendoremail.Text;
sve.SVWebsite = MSVD_vendorwebsite.Text;
sve.SVAddress = MSVD_vendoraddress.Text;
sve.SVNote = MSVD_vendornote.Text;
SVSCB sv = new SVSCB();
return sv.SVCBSI(sve);}
private void SVcode_SelectedIndexChanged(object sender, EventArgs e)
{
MSVname.Text = SVCB();
MSVcity.Text = SVCB();
MSVstate.Text = SVCB();
MSVemail.Text = SVCB();
MSVwebsite.Text = SVCB();
MSVaddress.Text = SVCB();
MSVnote.Text = SVCB();
}
this code working but show only one value of SVName in every textbox.
Update!!
public static string SV_CBSI(SVE sve)
{
using (SQLiteConnection conn = new SQLiteConnection(DatabaseConnection.ConnectionString()))
{
string CommandText = "SELECT * FROM SV WHERE [SVCode]=#id";
using (SQLiteCommand cmd = new SQLiteCommand(CommandText, conn))
{
cmd.Parameters.AddWithValue("#id", sve.SVID);
conn.Open();
DataTable dt = new DataTable();
SQLiteDataAdapter da = new SQLiteDataAdapter(cmd);
da.Fill(dt);
foreach (DataRow dr in dt.Rows)
{
sve.SVName = dr["SVName"].ToString();
sve.SVCity = dr["SVCity"].ToString();
sve.SVState = dr["SVState"].ToString();
sve.SVEmail = dr["SVEmail"].ToString();
sve.SVWebsite = dr["SVWebsite"].ToString();
sve.SVAddress = dr["SVAddress"].ToString();
sve.SVNote = dr["SVNote"].ToString();
}
return sve.SVName; //how to return SVCity,SVState,SVEmail,SVWebsite,SVAddress,SVNote
}
}
}
public static string SVCBSI(SVE sve)
{
return SVWCB.SV_CBSI(sve);
}
here is complete code

You need to have a method that return the whole SVE retrieved from database given a particular SVCode.
Notice that I suppose that SVCode is an integer. If not change the input type for this method accordingly
public SVE SV_CBSI(int id)
{
SVE localSve = new SVE();
string CommandText = "SELECT * FROM SV WHERE [SVCode]=#id";
using (SQLiteCommand cmd = new SQLiteCommand(CommandText, conn))
{
cmd.Parameters.AddWithValue("#id", id);
conn.Open();
DataTable dt = new DataTable();
SQLiteDataAdapter da = new SQLiteDataAdapter(cmd);
da.Fill(dt);
if(dt.Rows.Count > 0)
{
localSve.SVName = dr["SVName"].ToString();
localSve.SVCity = dr["SVCity"].ToString();
localSve.SVState = dr["SVState"].ToString();
localSve.SVEmail = dr["SVEmail"].ToString();
localSve.SVWebsite = dr["SVWebsite"].ToString();
localSve.SVAddress = dr["SVAddress"].ToString();
localSve.SVNote = dr["SVNote"].ToString();
}
return localSve;
}
}
Now you can call this method and get back the reference to an SVE instance filled with your data.
private void SVcode_SelectedIndexChanged(object sender, EventArgs e)
{
SVE currentSVE = SV_CBSI(Convert.ToInt32(SVCode.SelectedItem))
MSVname.Text = currentSVE.SVName;
MSVcity.Text = currentSVE.SVCity;
MSVstate.Text = currentSVE.State;
MSVemail.Text = currentSVE.SVEmail;
MSVwebsite.Text = currentSVE.SVWebsite;
MSVaddress.Text = currentSVE.SVAddress;
MSVnote.Text = currentSVE.SVNote;
}

May be you need to return an array (or a List) of strings?

How to return count value in ADO.NET in C# function?

I have created ADO.NET in C# where I want to return count value. My procedure GetAllEmployees #UserId which returns total count of employee. But list always returns 1 eventhough the CountId shows 0. I am doing something silly. Can anyone please help me.
public List<MyEmployee> GetAll(int UserId)
{
clsClass cls = new clsClass();
DataSet ds;
List<MyEmployee> list = new List<MyEmployee>();
SqlParameter[] prms = new SqlParameter[1];
string sSQL;
sSQL = "exec GetAllEmployees #UserId";
prms[0] = new SqlParameter("#UserId", SqlDbType.Int);
prms[0].Value = UserId;
ds = cls.CommandwithParams(sSQL, prms);
DataTable dt = ds.Tables[0];
foreach (DataRow dr in dt.Rows)
{
list.Add(
new MyEmployee
{
CountId = Convert.ToInt32(dr["CountId"])
});
}
return list;
}

I suggest declaring a variable like blow then add to it list.
foreach (DataRow dr in dt.Rows)
{
var emp=new MyEmployee
{
CountId = Convert.ToInt32(dr["CountId"])
});
list.Add(emp);
}

.NET MySql batch Stored Procedure with Output Parameters

I'm trying to execute a batch call to a stored procedure to sync data between two systems. My SP has several in parameters and 1 out parameter. Everything works fine except I can't seem to get the out parameter.
I created a simplified sample that outlines my issue.
My Stored Procedure looks like this.
CREATE PROCEDURE `sampleProc`(IN i_val INT, OUT o_val INT)
BEGIN
SELECT i_val + i_val INTO o_val;
END
My .NET code
class TestingSP
{
public static void Test()
{
DataTable dt = new DataTable();
dt.Columns.Add(new DataColumn("i_val", typeof(System.Int32)));
dt.Columns.Add(new DataColumn("o_val", typeof(System.Int32)));
for (int x = 1; x <= 100; x++)
{
DataRow dr = dt.NewRow();
dr["i_val"] = x;
dr["o_val"] = 0;
dt.Rows.Add(dr);
}
MySqlCommand command = new MySqlCommand();
command.CommandText = "sampleProc";
command.CommandType = CommandType.StoredProcedure;
command.UpdatedRowSource = UpdateRowSource.OutputParameters;
command.Parameters.Add("?i_val", MySqlDbType.Int32).SourceColumn = "i_val";
MySqlParameter output = new MySqlParameter();
output.ParameterName = "?o_val";
output.MySqlDbType = MySqlDbType.Int32;
output.Direction = ParameterDirection.Output;
output.SourceColumn = "o_val";
command.Parameters.Add(output);
MySqlConnectionStringBuilder conBuilder = new MySqlConnectionStringBuilder();
conBuilder.Server = "myserver";
conBuilder.UserID = "root";
conBuilder.Password = "password";
conBuilder.Port = 3308;
conBuilder.Database = "test_db";
Console.WriteLine("Rows: " + dt.Rows.Count);
using (MySqlConnection connection = new MySqlConnection(conBuilder.ConnectionString))
{
connection.Open();
command.Connection = connection;
using (MySqlDataAdapter da = new MySqlDataAdapter())
{
da.AcceptChangesDuringUpdate = true;
da.ContinueUpdateOnError = true;
da.UpdateCommand = command;
da.UpdateBatchSize = 50;
da.Update(dt);
foreach(var c in dt.GetErrors())
{
Console.WriteLine("Err: " + c.RowError);
}
foreach(DataRow row in dt.Rows)
{
Console.WriteLine("{0}: {1}", row["i_val"], row["o_val"]);
}
}
}
command.Dispose();
Console.WriteLine("Done...");
Console.ReadLine();
}
}
I've been scratching my head for a couple of days trying to get this to work but no matter what I try o_val is always zero.
Any help would be greatly appreciated.

How to Create a Sales Invoice Report in Crystal Report(Bind data in Two Table)

I need to create a crystal report to display data in two tables(ie,Sales_Invoice and Sales_Invoice_Items).Sales_Invoice_Items has a foreign from Sales_Invoice.I need to display report like this,
I am created a Crystal report to display data like this. Data in Sales_Invoice table is properly grouped with respect to invoice_no. But Its display all the data in Sales_Invoice_Items Tables in each Sales_Invoice row.I need to display Sales_Invoice_Items data with respect to Sales_Invoice's foreign in Sales_Invoice_Items
Then I am Bind it with following code.
private DataSet getData()
{
DataSetSalesReport ds = new DataSetSalesReport();
try
{
using (SqlConnection con = new SqlConnection(DBCon.conStr))
{
con.Open();
SqlCommand cmd = new SqlCommand(AppConstraints.GET_SALES_REPORT);
cmd.Connection = con;
cmd.CommandType = CommandType.StoredProcedure;
SqlDataReader rd = cmd.ExecuteReader();
decimal slno = 1;
while (rd.Read())
{
DataRow dr = ds.Tables["DataTable1"].NewRow();
dr["Id"] = rd["Id"].ToString();
dr["invoiceno"] = rd["invoiceno"].ToString();
dr["invoice_date"] = rd["invoice_date"].ToString();
dr["delivery_date"] = rd["delivery_date"].ToString();
dr["account"] = rd["account"].ToString();
dr["customer"] = rd["customer"].ToString();
dr["gross_amount"] = rd["gross_amount"].ToString();
dr["discount"] = rd["discount"].ToString();
dr["total_tax"] = rd["total_tax"].ToString();
dr["net_amount"] = rd["net_amount"].ToString();
dr["cash_discount"] = rd["cash_discount"].ToString();
dr["freight_charge"] = rd["freight_charge"].ToString();
dr["additional"] = rd["additional"].ToString();
dr["paid"] = rd["paid"].ToString();
dr["balance"] = rd["balance"].ToString();
dr["balancepaid_date"] = rd["balancepaid_date"].ToString();
dr["note"] = rd["note"].ToString();
dr["total_amount"] = rd["total_amount"].ToString();
dr["payment_mode"] = rd["payment_mode"].ToString();
dr["ac"] = rd["ac"].ToString();
dr["reg_date"] = rd["reg_date"].ToString();
dr["last_update"] = rd["last_update"].ToString();
dr["invoice_note"] = rd["invoice_note"].ToString();
dr["slno"] = slno++.ToString();
ds.Tables["DataTable1"].Rows.Add(dr);
using (SqlConnection con1 = new SqlConnection(DBCon.conStr))
{
con1.Open();
SqlCommand cmd1 = new SqlCommand(AppConstraints.GET_SALES_ITEMS_REPORT);
cmd1.Connection = con1;
cmd1.CommandType = CommandType.StoredProcedure;
cmd1.Parameters.AddWithValue("#Id", rd["Id"].ToString());
SqlDataReader rd1 = cmd1.ExecuteReader();
decimal j = 1;
while(rd1.Read())
{
DataRow dr1 = ds.Tables["DataTable2"].NewRow();
dr1["Id"] = rd1["Id"].ToString();
dr1["invoice_id"] = rd["Id"].ToString();
dr1["item_name"] = rd1["item_name"].ToString();
dr1["barcode"] = rd1["barcode"].ToString();
dr1["rate"] = rd1["rate"].ToString();
dr1["discount"] = rd1["discount"].ToString();
dr1["qty"] = rd1["qty"].ToString();
dr1["unit"] = rd1["unit"].ToString();
dr1["tax"] = rd1["tax"].ToString();
dr1["net"] = rd1["net"].ToString();
dr1["mrp"] = rd1["mrp"].ToString();
dr1["slno"] = j++.ToString();
ds.Tables["DataTable2"].Rows.Add(dr1);
}
}
}
}
}
catch (Exception ex)
{
MessageBox.Show(ex.Message);
}
return ds;
}
The I am call above method in Window_Loaded event in WPF Window
private void Window_Loaded(object sender, RoutedEventArgs e)
{
ReportDocument report = new ReportDocument();
report.Load("rptSalesReport.rpt");
report.SetDataSource(getData());
PrintSalesReport.ViewerCore.ReportSource = report;
PrintSalesReport.ToggleSidePanel = SAPBusinessObjects.WPF.Viewer.Constants.SidePanelKind.None;
}
My DataSet is shown below
After Running it i got the following result
In above result; it display all the Sales_invoice_Items data below in each of Sales_Invoice Data.
I need to display Sales_Invoice Data with respect its Sales_Invoice_Items Data(Sales_Invoice_Items contains Sales_Invoice's foreign key).I need your help to do this.
Thank You

Use Sub Report to solve this problem