I want to serialize the DataTable to a List in C#. My code is taking around 10 minutes for 100K data to fetch and convert to a list. Previously tried with sequential way but it three times more time to process. So, I tried Parallel and it saved 1/3 of time. But, still, it's too slow.
If anyone can help to make it fast. Any help is appreciated.
public async Task<List<DataPointModel>> GetDetails(HashParameterModel parameterModel)
{
List<DataPointModel> result = new List<DataPointModel>();
try
{
var query = "USP_GetDetailsFromMetaData";
using (var sqlConnection = new SqlConnection(connectionString))
{
using (var sqlCommand = new SqlCommand(query, sqlConnection))
{
sqlCommand.CommandType = CommandType.StoredProcedure;
sqlCommand.Parameters.AddWithValue("#GroupID", parameterModel.GroupID);
sqlCommand.CommandTimeout = 0;
sqlConnection.Open();
DataTable dataTable = new DataTable();
DataSet dataSet = new DataSet();
SqlDataAdapter da = new SqlDataAdapter
{
SelectCommand = sqlCommand
};
da.Fill(dataSet);
dataTable = dataSet.Tables[0];
DataTable dataTable1 = dataSet.Tables[1];
var questionList = dataTable1.AsEnumerable();
if (dataTable.Rows.Count > 0)
{
Parallel.ForEach(dataTable.AsEnumerable(), new ParallelOptions { MaxDegreeOfParallelism = 5 }, row =>
{
DataPointModel model = new DataPointModel();
model.ID = Convert.ToInt32(row["ID"]);
model.GroupID = Convert.ToInt32(row["GroupID"]);
model.ProviderID = Convert.ToInt32(row["SalvageProviderID"]);
model.ClaimNumber = row["ClaimNumber"].ToString();
model.PolicyNumber = row["PolicyNumber"].ToString();
model.DataPoint1 = row["DataPoint1"].ToString();
model.DataPoint2 = row["DataPoint2"].ToString();
model.DataPoint3 = row["DataPoint3"].ToString();
model.DataPoint4 = row["DataPoint4"].ToString();
model.FirstName = row["FirstName"].ToString();
model.LastName = row["LastName"].ToString();
model.PrimaryDamage = row["PrimaryDamage"].ToString();
model.Type = row["TypeCode"].ToString();
model.LossDate = row["LossDate"].ToString();
model.QuestionList = (from p in questionList
where p.Field<int>("ID") == model.ID
select new QuestionResponseModel()
{
QuestionID = p.Field<int>("QuestionID").ToString(),
Response = p.Field<string>("ResponseValue")
}).ToList();
result.Add(model);
});
}
}
}
}
catch (Exception ex)
{
throw ex;
}
return result;
}
The DataSet has two DataTable
DataTable dataTable = dataSet.Tables[0]; // Details
DataTable dataTable1 = dataSet.Tables[1]; // QUestionList
I think the time is consumed when its looping interlly for QuestionList, which may have around 120K rows. Any Suggestions
An easy and quick to implement performance improvment would be to build a look up table from your questionList, and access this to fetch the question instead of doing this piece of code
model.QuestionList = (from p in questionList
where p.Field<int>("ID") == model.ID
select new QuestionResponseModel()
{
QuestionID = p.Field<int>("QuestionID").ToString(),
Response = p.Field<string>("ResponseValue")
}).ToList();
So add the following
var questionList = dataTable1.AsEnumerable();
//maybe add .AsParallel() - questionList.AsParallel().ToLookUp(...)
var questionLookUp = questionList.ToLookUp(x => x.Field<int>("ID"), x => new QuestionResponseModel() { QuestionID = x.Field<int>("QuestionID"), Response = p.Field<string>("ResponseValue") });
And than use it like this
model.QuestionList = questionLookUp[model.ID].ToList();
https://learn.microsoft.com/en-us/dotnet/api/system.linq.lookup-2
https://learn.microsoft.com/en-us/dotnet/api/system.linq.enumerable.tolookup
Code is untested, hope I didn't make to many errors.
Are there duplicate model IDs in the Details table? If so this could help to avoid running the question list query multiple times:
model.QuestionList = getQuestions(model.ID);
Method:
public Dictionary<int, List<QuestionResponseModel>> questionBuffer = new Dictionary<int, List<QuestionResponseModel>>();
public List<QuestionResponseModel> getQuestions(int ID)
{
if (questionBuffer.ContainsKey(ID)) return questionBuffer[ID];
List<QuestionResponseModel> questions = (from p in questionList
where p.Field<int>("ID") == model.ID
select new QuestionResponseModel()
{
QuestionID = p.Field<int>("QuestionID").ToString(),
Response = p.Field<string>("ResponseValue")
}).ToList();
questionBuffer.Add(ID, questions);
return questions;
}
Related
I am trying to sort a datatable into a DataSet. I want to sort by the Status Column in "DESC". But I am not aware how to go about this. I have tried the suggested solutions online but I seem not to be doing something right. Here is what I have tried, albeit, I have commented out the sorting lines of the code as they do not work for me. How can I sort my table using the Status column in Desc?
[WebMethod(EnableSession = true)]
public List < TaskListClass > getTasks() {
var userId = Session["UserId"].ToString();
List < TaskListClass > objB = new List < TaskListClass > ();
try {
using(var connection = new SqlConnection(ConfigurationManager.ConnectionStrings["DBConnString"].ToString())) {
connection.Open();
DataSet Myds = new DataSet();
// Myds.Tables[0].DefaultView.Sort = "Status desc";
SqlDataAdapter sqldr = new SqlDataAdapter();
string ProcName = "getTasks";
SqlCommand cmd = new SqlCommand(ProcName, connection);
cmd.CommandType = CommandType.StoredProcedure;
cmd.Parameters.Add("#userId", SqlDbType.VarChar, 900).Value = userId;
sqldr.SelectCommand = cmd;
sqldr.Fill(Myds);
DataTable dt = Myds.Tables[0];
// DataTable dt = Myds.Tables[0].DefaultView.ToTable();
for (int i = 0; i < dt.Rows.Count; i++) {
objB.Add(new TaskListClass() {
Id = Convert.ToString(dt.Rows[i]["Id"]),
Subject = Convert.ToString(dt.Rows[i]["Subject"]),
Customer = Convert.ToString(dt.Rows[i]["Customer"]),
Sender = Convert.ToString(dt.Rows[i]["Sender"]),
Receiver = Convert.ToString(dt.Rows[i]["Receiver"]),
Priority = Convert.ToString(dt.Rows[i]["Priority"]),
StartDate = Convert.ToString(dt.Rows[i]["StartDate"]),
EndDate = Convert.ToString(dt.Rows[i]["EndDate"]),
Status = Convert.ToString(dt.Rows[i]["Status"]),
OnProgress = Convert.ToString(dt.Rows[i]["OnProgress"]),
});
}
}
} catch (Exception e) {
msg = e.ToString();
}
return objB;
}
Ok, a few things.
first up, a dataset is a collection of tables - "many tables" possible.
But you have ONE table, so why use a dataset? I see no need. Just use a single data table for this.
And this will reduce the code.
So, I suggest this, or close to this:
using (var connection = new SqlConnection(ConfigurationManager.ConnectionStrings["DBConnString"].ToString()))
{
using (var cmd = new SqlCommand("getTasks", connection))
{
connection.Open();
cmd.CommandType = CommandType.StoredProcedure;
DataTable dt = new DataTable();
cmd.Parameters.Add("#userId", SqlDbType.VarChar).Value = userId;
dt.Load(cmd.ExecuteReader());
// now sort the datatable
dt.DefaultView.Sort = "Status DESC";
// now fill out our object with each row
foreach (DataRow OneRow in dt.Rows)
{
objB.Add(new TaskListClass()
{
Id = OneRow["Id"].ToString(),
Subject = OneRow["Subject"].ToString(),
Customer = OneRow["Customer"].ToString(),
Sender = OneRow["Sender"].ToString(),
Receiver = OneRow["Receiver"].ToString(),
Priority = OneRow["Priority"].ToString(),
StartDate = OneRow["StartDate"].ToString(),
EndDate = OneRow["EndDate"].ToString(),
Status = OneRow["Status"].ToString(),
OnProgress = OneRow["OnProgress"].ToString(),
}); ;
}
}
}
}
return objB;
The way the current code is written, you could add this after the for-loop:
objB = objB.OrderByDescending(t => t.Status).ToList();
Depending of the datatype of Status, it might be sorted alphabetically.
var dataRow = dt.AsEnumerable().OrderByDescending(x => x.Field<string>("Status")).ToList();
foreach (var item in dataRow)
{
//Enter your Code Here
}
Here dt is your datatable.
dataRow is a set of list.
After get the data list, you can asign it to your "objB".
I need to return all values from my table. How to write a code inside while?
var stringConnection = "Data Source = X; User Id = X; Password = X";
var sql = "SELECT * FROM TABLE";
OracleConnection _oracleConnection = new OracleConnection(stringConnection);
_oracleConnection.Open();
OracleCommand cmd = new OracleCommand(sql, _oracleConnection);
var dr = cmd.ExecuteReader();
var list = new List<dynamic>();
while(dr.Read())
{
// my doubt is here
}
return list;
var stringConnection = "Data Source = X; User Id = X; Password = X";
var sql = "SELECT * FROM TABLE";
OracleConnection _oracleConnection = new OracleConnection(stringConnection);
_oracleConnection.Open();
OracleCommand cmd = new OracleCommand(sql, _oracleConnection);
var dr = cmd.ExecuteReader();
var list = new List<dynamic>();
while(dr.Read())
{
// **** read column name data from table ****
string Id = (string)dr["Id"];
string company = (string)dr["company"];
string city = (string)dr["City"];
var objItem = new { Id = Id, company = company, city = "city" };
list.Add(objItem);
}
return list;
You have several options. 1 - load DataSet from OracleDataReader. There you will have all your data.
2 - you can still use select *... but you need a model. Then create List<SomeModel> instead of List<dynamic> with
while (reader.Read())
{
model.Property = reader["columnName"]; // will need convert type and take care of DB null. Can use existing extnsions
. . . .
}
3 - For arbitrary number of columns use OracleDataReader.FieldCount and some storage like List<object[]>
var data = new List<object[]>();
var fCnt = reader.FieldCount;
while (reader.Read())
{
var arr = new Object[fCnt];
for(int i = 0; i < fCnt; i++)
arr[i] = reader[i];
data.Add(arr);
}
The unfortunate part with #3 is that in the end you can get jagged array and not 2-dimentional one but you now have enough info to convert it. But I don't remember when I needed to do #3. So think about #1 and #2
And one more thing - absolutely no need for dynamic here. Stay away.
WPF Form illustration:
I would like to filter the datagrid (selected using an sql statement at initialization) using the id value in the textbox to match the id in the column in the image (somewhat similar to using an inner-join). I need help with this.
NpgsqlCommand cmd = new NpgsqlCommand("select * from central.equipment
where equipment.deleted = 1 ", con);
NpgsqlDataAdapter da = new NpgsqlDataAdapter();
da.SelectCommand = cmd;
DataTable dt = new DataTable();
da.Fill(dt);
dataGridView1.Columns[0].Name = "emp_id";
dataGridView1.Columns[0].HeaderText = "Employee ID";
dataGridView1.Columns[0].DataPropertyName = "emp_id";
You could use Linq:
For example:
//Code behind
Equipment selecteValueCodeBehind = (Equipment)dtgEquipment.SelectedItem;
using (var db = new DataContext())
{
var QueryObjects = db.Equipment.Where(x=> x.EmployeeId == selecteValueCodeBehind.EmployeeId).ToList();
dtEquipment.ItemsSource = QueryObjects;
}
//MVVM
public Equipment selecteValueMvvm {get; set; } = new Equipment();
public List<Equipment> Equipments {get; set;} = new List<Equipment>();
using (var db = new DataContext())
{
var QueryObjects = db.Equipment.Where(x=> x.EmployeeId == selecteValueMvvm.EmployeeId).ToList();
Equipments = QueryObjects;
}
By the way, I think you app is in WinForms and not WPF.
Sir I have filled my dataset with linq as
public void FillDataSet(DataSet ds1,int Id)
{
try
{
var y = from ins in cstmrDC.customers_rd(Id) select ins;
var z = from ins in cstmrDC.customersCntcts_rd(Id) select ins;
DataTable dtCst = new DataTable("dtCstmr");
dtCst.Columns.Add("cst_Id");
dtCst.Columns.Add("cst_Name");
dtCst.Columns.Add("cst_SName");
dtCst.Columns.Add("cst_AdLn1");
DataTable dtDtls = new DataTable("dtDtails");
dtDtls.Columns.Add("cst_SrlNo");
dtDtls.Columns.Add("cst_CntName");
dtDtls.Columns.Add("cst_cntDsgn");
foreach (var dtbl in y)
{
DataRow dr;
dr = dtCst.NewRow();
dr[0] = dtbl.cust_Id;
dr[1] = dtbl.cust_Name;
dr[2] = dtbl.cust_Sname;
dr[3] = dtbl.cust_Adrsln1;
dtCst.Rows.Add(dr);
}
foreach (var dtbl in z)
{
DataRow drDtls;
drDtls = dtDtls.NewRow();
drDtls[0] = dtbl.cust_Slno;
drDtls[1] = dtbl.cust_Cntctnm;
drDtls[2] = dtbl.cust_Cntctdesig;
dtDtls.Rows.Add(drDtls);
}
ds1.Tables.Add(dtCst);
ds1.Tables.Add(dtDtls);
}
catch (Exception ex)
{
MessageBox.Show(ex.Message);
}
}
and the Id is passing from another class as
intId = int.Parse(txtSearch.Text);
cstCls.FillDataSet(ds1w, intId);
from that dataset iam fillimg my textbox controllers and giving theDataSource to the dataGridView as
dgvCustInfo.DataSource = ds1w.Tables["dtDtails"];
In this way if I searched 1st time with integer id 1055 meanse the exact result is comming from database. At the same time If I gave the another integer Id as 1066 meance Iam getting exception as DataTable named 'dtCstmr' already belongs to this DataSet .. Soo how can I solve the problem.
You can check if a table is already contained in a DataSet with Contains:
if(!ds1.Tables.Contains(dtCst.TableName))
ds1.Tables.Add(dtCst);
if(!ds1.Tables.Contains(dtDtls.TableName))
ds1.Tables.Add(dtDtls);
However, as Raphael has mentioned this would not refresh the table in the DataSet. So an easy way would be to remove the old table and add the new:
if(ds1.Tables.Contains(dtCst.TableName))
ds1.Tables.Remove(dtCst.TableName);
if(ds1.Tables.Contains(dtDtls.TableName))
ds1.Tables.Remove(dtDtls.TableName);
ds1.Tables.Add(dtCst);
ds1.Tables.Add(dtDtls);
It's quite a bad idea to create and populate in the same method.
Your code is really confusing.
Create another method :
public void CreateTables(DataSet ds1) {
var dtCst = new DataTable("dtCstmr");
dtCst.Columns.Add("cst_Id");
dtCst.Columns.Add("cst_Name");
dtCst.Columns.Add("cst_SName");
dtCst.Columns.Add("cst_AdLn1");
var dtDtls = new DataTable("dtDtails");
dtDtls.Columns.Add("cst_SrlNo");
dtDtls.Columns.Add("cst_CntName");
dtDtls.Columns.Add("cst_cntDsgn");
ds1.Tables.Add(dtCst);
ds1.Tables.Add(dtDtls);
}
public void FillDataSet(DataSet ds1,int Id)
{
try
{
var y = from ins in cstmrDC.customers_rd(Id) select ins;
var z = from ins in cstmrDC.customersCntcts_rd(Id) select ins;
var dtCst = ds1.Tables["dtCstmr"];
var dtDtls = ds1.Tables["dtDtails"];
dtCst.Clear();
dtDtls.Clear();
foreach (var dtbl in y)
{
DataRow dr;
dr = dtCst.NewRow();
dr[0] = dtbl.cust_Id;
dr[1] = dtbl.cust_Name;
dr[2] = dtbl.cust_Sname;
dr[3] = dtbl.cust_Adrsln1;
dtCst.Rows.Add(dr);
}
foreach (var dtbl in z)
{
DataRow drDtls;
drDtls = dtDtls.NewRow();
drDtls[0] = dtbl.cust_Slno;
drDtls[1] = dtbl.cust_Cntctnm;
drDtls[2] = dtbl.cust_Cntctdesig;
dtDtls.Rows.Add(drDtls);
}
}
catch (Exception ex)
{
MessageBox.Show(ex.Message);
}
}
Method CreateTables should be called only one time, and FillDataSet as many times as needed.
Suppose I have this code (pseudocode)
class SomeClass
{
class Person
{
public static string Name { get; set; }
public static int Age { get; set; }
}
List<Person> person = new List<person>;
public void SelectPerson()
{
DataTable dt = new DataTable();
SqlConnection conn = GetConnection();
conn.Open();
SqlDataAdapter da = new SqlDataAdapter("SELECT name, age FROM person", conn);
da.Fill(dt);
}
}
Can I fill the List (person) based on the result of my DataAdapter?
How should I do it? Or is there any workaround? Thanks...
Probably the best way is not to read into a datatable first:
var dr = new DataReader(....) // Fill in what is needed, can't remember offhand
while(dr.Next())
{
persons.Add(
new Person() {
Name = (string) r["Name"],
Age = (int) r["Age"]
}
);
}
Caveat: You want to close the DataReader/connection quickly, don't do lots of processing. The above code is more efficient than using a DataTable as an intermediary.
But if you do want to use a data table first, you could use LINQ:
var list = dt.AsEnumerable().Select(r => new Person() {
Name = (string) r["Name"],
Age = (int) r["Age"] }
).ToList()
or just itterate of dt.Rows and create a new person and add it to the list
You should also use Using() statements around your connection and reader.
There's also Linq to DataSet that you could use to do something along these lines
var list = (from tr in dt.AsEnumerable()
select new Person() {
Name = tr.Field<string>("Name"),
Age = tr.Field<int>("Age")
}).ToList();
Robert beat me to the answer, just using slightly different syntax.
In addition to the other options presented, you could defer the execution until needed with a yield:
public static IEnumerable<Person> GetPeople()
{
using( var conn = GetConnection() )
{
conn.Open();
string sql = "SELECT name, age FROM person";
var cmd = new SqlCommand( sql, conn );
using( SqlDataReader rdr = cmd.ExecuteReader() )
{
if( rdr == null )
{
throw new NullReferenceException( "No People Available." );
}
while( rdr.Read() )
{
var person = new Person();
person.Name = rdr["name"].ToString();
person.Age = Convert.ToInt32 ( rdr["age"] );
yield return person;
}
}
}
}
First, you'll want to make your Name and Age fields non-static (so each instance of the class will have their own values for these properties.
Then you could do something like this:
foreach(DataRow row in dt.Rows){
Person p = new Person(){
Name = Convert.ToString(row["name"]),
Age = Convert.ToInt32(row["age"])
}
person.Add(p);
}
Hope this helps!
Yes you can. Iterate through the items in dt.Rows and convert them manually to Person objects.
A user recently asked me a question on converting a DataTable to a List<> in .NET 2.0. Here’s the code to do so:
C#
// Assuming there is a DataTable called dt
List<DataRow> drlist = new List<DataRow>();
foreach (DataRow row in dt.Rows)
{
drlist.Add((DataRow)row);
}