We Keep Coding

C# retrieve data from SQL Server using SqlDataReader to Master Detail List

I have Master and Detail classes:
class Master
{
public int ID { get; set; }
public string Name { get; set; }
public List<Detail> Details { get; set; }
}
class Detail
{
public Description { get; set; }
public Amount { get; set; }
}
I use below approach and working fine now.
List<Master> result = new List<Master>();
// SQL Connection
string sqlCommand = "SELECT * FROM Master LEFT JOIN Detail on Master.ID = Detail.ID";
using (System.Data.SqlClient.SqlDataReader dr = db.DbDataReader as System.Data.SqlClient.SqlDataReader)
{
if (dr.HasRows)
{
Master LastMaster = null;
while (dr.Read())
{
if (LastMaster == null || Convert.ToInt(dr["ID"]) != LastMaster.ID)
{
Master h = new Master();
h.ID = Convert.ToInt(dr["ID"]);
h.Name = Convert.ToString(dr["Name"]);
result.Add(h);
LastMaster = h;
}
if (dr["Description"] == DBNull.Value)
continue;
if (h.Detail == null)
h.Detail = new List<Detail>();
Detail d = new Detail();
d.Description = dr["Description"] as string;
d.Amount = Convert.ToDouble(dr["Amount"]);
LastMaster.Detail.Add(d);
......
}
}
.....
}
Is there any better approach to fill list of list objects in C# ? I appreciate any suggestion. Thanks.

You can use Dapper (a micro ORM) for your scenario. Below is a sample code
const string createSql = #"
create table #Users (Id int, Name varchar(20))
create table #Posts (Id int, OwnerId int, Content varchar(20))
insert #Users values(99, 'Sam')
insert #Users values(2, 'I am')
insert #Posts values(1, 99, 'Sams Post1')
insert #Posts values(2, 99, 'Sams Post2')
insert #Posts values(3, null, 'no ones post')";
using(var connection = new SqlConnection("database connection string"))
{
connection.Execute(createSql);
try
{
const string sql =#"select * from #Posts p
left join #Users u on u.Id = p.OwnerId
Order by p.Id";
var data = connection.Query<Post, User, Post>(sql, (post, user) => { post.Owner = user; return post; }).ToList();
}
catch(Exception ex){}
}

Ibram commented about EF and Dapper and Abu gave an example for Dapper (but I'm not sure it demos generating a graph with a single master and multiple detail per master, as you have - dapper can do so if you want to explore it)
In EF we could do something like:
install EF core power tools - as you have a db already we will use it to generate classes from. This operation can just be done with the command line but EFCPT makes a lot of operations easier
right click your project, choose EF Core Power Tools .. Reverse Engineer
fill in a new connection string detail
choose the database objects you wish to turn into classes
set other options as appropriate (you can find out more about them later, maybe only use the pluralize one for now, if your db tables are like Orders, Customers, Companies and you want your classes called Order/Customer/Company (classes should not have plural names). Tick on "put connectionstring in code" for now- you can remove it to config file later
finish. Eventually you'll get some classes and a context that has a load of code in OnModelCreating that lays out a description of everything in the tables, the columns, keys, relationships..
Now you can run some query like:
var c = new YourContext();
var ms = c.Masters.Include(m => m.Details).ToList();
That's basically the equivalent of what you posted
You can get more trick by shaping a more involved linq query:
var q = c.Masters.Include(m => m.Details)
.Where(m => m.Name.StartsWith("Smith"))
.OrderBy(m => m.Name);
var ms = q.ToList();
It will be translated into something like
SELECT * FROM master join detail on ...
WHERE name LIKE 'Smith%'
ORDER BY m.name
You can see the generated query if you inspect the DebugView property of q
You could make changes:
ms[0].Details.Clear(); //causes delete of all details for this master
ms[1].Details.Add(new Detail { someprop = some value}); //causes insert of new details for this master
ms[2].Name = "Hello"; //causes update of this master name
c.SaveChanges(); //carries out the above, in sql, to affect the db
When you manipulate the returned objects and save, EF will delete/insert/update as appropriate to sync the db to what happened to the objects. It is important that you understand that EF tracks what happens to all the objects it creates, so that it can do this
When would you use EF and when would you use Dapper? Well, it doesn't have to be mutually exclusive; you can use them in the same project. Generally I'd say use EF (or some other ORM like it - nHibernate is another popular one, works on a similar concept of translating linq expressions to sql and tracking the data back into an object) for stuff where the sql is so simple that it's a productivity boost to not have to write it, track it, and write the changes back. What it is not intended for, is forming as hoc queries that don't map well to client side objects. For that you can use Dapper, or you could form client side objects and add them to EF's model and then run raw sql that populates them. Dapper is fast, because it doesn't do any of that tracking changes, mapping or wiring up complex object graphs; you do all that manually. Dapper makes a convenient abstraction over raw sql and creates classes, but EF goes much further; it comes at a cost - EF is highly convenient but much more heavy weight.

How can I map a "select all" query with 2 tables using SqlQuery

I'm trying to retrieve all fields from two joined tables to any kind of a c# object.
So I'm trying to run this code:
var query = #$"EXEC('select *
from persons p join students s on p.id=s.id
where p.id = 21')";
var result = _context.Database.SqlQuery<?>(query).ToList();
But I don't get what should be instead of the question mark.
I've tried List<object> and Dictionary<string,string> but since I couldn't get exactly how this is being mapped, I don't understand to what it can be mapped.
There is a somewhat similar question here but its solution only addresses two columns, and it apparently doesn't support returning nulls.

You can try creating a stored procedure or a function in the SQL level.
Then, just select then generated table / result.
So, you already have an idea of what class it is.

I frequently use the dynamic type like this :
var lst = _context.Database.SqlQuery<dynamic>(query).ToList();
foreach (var item in lst)
{
var myVar = item.myfieldName;
}
It may be preferable to name each field in your query instead of using select *.

Optimize LINQ query that runs fast in Sql server?

I want to calculate the rows of a related table:
MainTable tbl = tblInfo(id);
var count = tbl.Related_Huge_Table_Data.Count();
The problem is: this takes too long (about 20 seconds) to execute, although when I run this query in Sql Server it executes below one second. How can I optimize this query in linq? I also tried to use stored procedure but no luck.
This is the tblInfo method:
public MainTable tblInfo(int id)
{
MyDataContext context = new MyDataContext();
MainTable mt = (from c in context.MainTables
where c.Id == id
select c).SingleOrDefault();
return mt;
}
I used LinqToSql and classes was generated by LinqToSql.

By running SingleOrDefault() you execute the query and have to deal with results in memory after that. You need to stay with IQueryable until your query is fully constructed.
The easiest way to answer "how many child records this parent record has" is to approach it from the child side:
using (var dx = new MyDataContext())
{
// If you have an association between the tables defined in the context
int count = dx.Related_Huge_Table_Datas.Where(t => t.MainTable.id == 42).Count();
// If you don't
int count = dx.Related_Huge_Table_Datas.Where(t => t.parent_id == 42).Count();
}
If you insist on the parent side approach, you can do that too:
using (var dx = new MyDataContext())
{
int count = dx.MainTables.Where(t => t.id == 42).SelectMany(t => t.Related_Huge_Table_Datas).Count();
}
If you want to keep a part of this query in a function like tblInfo, you can, but you can't instantiate MyDataContext from inside such function, otherwise you will get an exception when trying to use the query with another instance of MyDataContext. So either pass MyDataContext to tblInfo or make tblInfo a member of partial class MyDataContext:
public static IQueryable<MainTable> tblInfo(MyDataContext dx, int id)
{
return dx.MainTables.Where(t => t.id == id);
}
...
using (var dx = new MyDataContext())
{
int count = tblInfo(dx, 42).SelectMany(t => t.Related_Huge_Table_Datas).Count();
}

Try this
MyDataContext context = new MyDataContext();
var count=context.Related_Huge_Table_Data.where(o=>o.Parentid==id).Count();
//or
int count=context.Database.SqlQuery<int>("select count(1) from Related_Huge_Table_Data where Parentid="+id).FirstOrDefault();

If you wish to take full advantage of your SQL Database's performance, it may make sense to query it directly rather than use Linq. Should be reasonably more performent :)
var Related_Huge_Table_Data = "TABLENAME";//Input table name here
var Id = "ID"; //Input Id name here
var connectionString = "user id=USERNAME; password=PASSWORD server=SERVERNAME; Trusted_Connection=YESORNO; database=DATABASE; connection timeout=30";
SqlCommand sCommand = new SqlCommand();
sCommand.Connection = new SqlConnection(connectionString);
sCommand.CommandType = CommandType.Text;
sCommand.CommandText = $"COUNT(*) FROM {Related_Huge_Table_Name} WHERE Id={ID}";
sCommand.Connection.Open();
SqlDataReader reader = sCommand.ExecuteReader();
var count = 0;
if (reader.HasRows)
{
reader.Read();
count = reader.GetInt32(0);
}
else
{
Debug.WriteLine("Related_Huge_Table_Data: No Rows returned in Query.");
}
sCommand.Connection.Close();

Try this:
MyDataContext context = new MyDataContext();
var count = context.MainTables.GroupBy(x => x.ID).Distict().Count();

The answer of GSerg is the correct one in many case. But when your table starts to be really huge, even a Count(1) directly in SQL Server is slow.
The best way you can get round this is to query the database stats directly, which is impossible with Linq (or I don't know of).
The best thing you can do is to create a static sub (C#) on your tables definition witch will return the result of the following query:
SELECT
SUM(st.row_count)
FROM
sys.dm_db_partition_stats st
WHERE
object_name(object_id) = '{TableName}'
AND (index_id < 2)
where {TableName} is the database name of your table.
Beware it's an answer only for the case of counting all records in a table!

Is your linq2sql returning the recordset and then doing the .Count() locally, or is it sending SQL to the server to do the count on the server? There will be a big difference in performance there.
Also, have you inspected the SQL that's being generated when you execute the query? From memory, Linq2Sql allows you to inspect SQL (maybe by setting up a logger on your class?). In Entity Framework, you can see it when debugging and inspecting the IQueryable<> object, not sure if there's an equivalent in Linq2Sql.
Way to view SQL executed by LINQ in Visual Studio?
Alternatively, use the SQL Server Profiler (if available), or somehow see what's being executed.

You may try following:-
var c = from rt in context.Related_Huge_Table_Data
join t in context.MainTables
on rt.MainTableId ==t.id where t.id=id
select new {rt.id};
var count=c.Distict().Count();

Linq performance with subqueries and LET

I have a query I'm executing in LinqPad (EF4 against a SQL2010 backend), which uses navigation properties to access related tables. I don't really have access to fiddle with indexes and so forth, and was wondering if there are any ways to avoid all the subqueries inherent in those 'firstordefault' items toward the bottom of the query. I've tried using a 'let' statement at the top as in "let accounts = i.ExpenseItemAccountings.FirstOrDefault()" and then use that reference to make it run only one subquery, but it still takes easily more than an hour to run this query for any meaningful number of records.
Is there any way I can make this more efficient?
var output = from i in ExpenseItems where
i.Er_Approved_Date >= fromDate &&
i.Er_Approved_Date <= toDate
select new {ER_Num = i.ErNum,
Line_Num = i.ItemNum,
Report_Title = i.Report_Title,
Requestor = i.Requester_Name,
Preparer = i.Preparer_Name,
ER_Total_Value = i.Er_TotalVal,
Partition = i.Org,
Transaction_Date = i.Item_Transaction_Date,
Approved_Date = i.Er_Approved_Date,
Item_Amount = i.Item_Amount,
Tips = i.Item_Tips,
GST = i.Item_Gst,
Have_Receipt = i.Item_Have_ReceiptTf,
Have_Invoice = i.Item_Have_InvoiceTf,
Vendor = i.Item_Vendor,
City = i.Item_City,
Item_Expense_Type = i.Item_Expense_Type,
Item_Description = i.Item_Expense_Description,
Misc_Item_Commodity = i.Item_Misc_Commodity_Name,
Misc_Item_SubCategory = i.Item_Misc_Specify,
Misc_Item_OtherMisc_Description = i.Item_Misc_Specify_Other_Desc,
Entity_Num = i.ExpenseItemAccountings.FirstOrDefault().Item_Entity_Num,
Entity_Name = i.ExpenseItemAccountings.FirstOrDefault().Item_Entity_Name,
Account_Num = i.ExpenseItemAccountings.FirstOrDefault().Item_Account_Num,
Account_Desc = i.ExpenseItemAccountings.FirstOrDefault().Item_Account_Name,
SubAccount_Num = i.ExpenseItemAccountings.FirstOrDefault().Item_SubAccount_Num,
SubAccount_Name = i.ExpenseItemAccountings.FirstOrDefault().Item_SubAccount_Name,
CostCentre_Num = i.ExpenseItemAccountings.FirstOrDefault().Item_CostCentre_Num,
CostCentre_Name = i.ExpenseItemAccountings.FirstOrDefault().Item_CostCentre_Name,
Project_Code = i.ExpenseItemAccountings.FirstOrDefault().Expense_Item_ProjectCode,
////Percent_Allocated = i.ExpenseItemAccountings.FirstOrDefault().Item_Percent,
ER_Comments = i.Er_Comments,
Item_First_Comment = i.ExpenseItemComments.FirstOrDefault().Comment_Content,
Violations = i.ExpenseItemViolations.Count()
};

On LinqPad you should be able to view the SQL statement generated from the LINQ statement.
In my opinion there is always a debate between using let or navigation properties in LINQ to Entities (or LINQ to SQL). Sometimes a simple JOIN might work better too. In other words it all depends on how your LINQ provider (Entity Framework) optimizes your specific query into SQL statements.
I would suggest you test your query with all let/join/navigation to see the generated SQL statement.
You can use ObjectQuery under System.Data.Objects in your code to view realtime SQL statement in case you don't have SQL profiler or intellitrace tool:
((ObjectQuery)anyLinqQuery).ToTraceString();
Also you can try using multiple from clause:
var output = from i in ExpenseItems
from exp in i.ExpenseItemAccountings
where ....
select new {...};
or
var output = from i in ExpenseItems
from exp in i.ExpenseItemAccountings.DefaultIfEmpty()
where ....
select new {...};

EF: Avoiding multiple update statements

Code like this:
var compIds = from p in packinglist.List
select p.ComponentId;
var components = from c in context.Components
where compIds.Contains(c.Id)
select c;
foreach (var item in components)
{
item.CurrentSiteId = packinglist.DestinationId;
}
context.SaveChanges();
Ends up issuing lots of SQL Statements like
update [dbo].[Components] set [CurrentSiteId] = #0 where ([Id] = #1)
Is there a way to instruct EF (Code First) to issue the following statement:
update [dbo].[Components] set [CurrentSiteId] = #0 where ([Id] in (....))
Or should I look into using the one of the SQLQuery methods available, or a seperate tool like Dapper or massive or ...?

There is not currently a way to perform bulk updates in EF 4 out of the box. There are some very long, complicated work arounds that end up generating SQL though. I suggest using a stored procedure or T-SQL. Here's a quick T-SQL snippet that I've used in the past:
using (var context = new YourEntities())
{
context.ExecuteStoreCommand(
#"UPDATE Components SET CurrentSiteId = 1 WHERE ID IN(1,2,3,4)");
}

The simplest answer for this is just to write that query and use DbContext.SQLQuery() to run it. As mentioned, there's no way to do this in EF itself.