I have this Method# 1 query below that is parameterized using dapper, problem is the query times out with this approach even after waiting 30sec and normally it takes max of 1 sec on SSMS with plain sql.
However Method # 2 query actually works where the query is built on the server side instead of parameterized one. One thing i have noticed is, it might have something to do with filter for FirstName and LastName, I have single Quote on Method #2 for those filter but not for Method #1.
What is wrong with Method # 1 ?
Method # 1
string query = "SELECT *
FROM dbo.Customer c
WHERE c.MainCustomerId = #CustomerId
AND (#IgnoreCustomerId = 1 OR c.CustomerID = #FilterCustomerId)
AND (#IgnoreFirstName = 1 OR c.FirstName = #FilterFirstName)
AND (#IgnoreLastName = 1 OR c.LastName = #FilterLastName)
AND (#IgnoreMemberStatus = 1 OR c.CustomerStatusID = #FilterMemberStatus)
AND (#IgnoreMemberType = 1 OR c.CustomerTypeID = #FilterMemberType)
AND (#IgnoreRank = 1 OR c.RankID = #FilterRank)
ORDER BY c.CustomerId
OFFSET #OffSet ROWS
FETCH NEXT 50 ROWS ONLY";
_procExecutor.ExecuteSqlAsync<Report>(query, new
{
CustomerId = customerId,
IgnoreCustomerId = ignoreCustomerId,
FilterCustomerId = filter.CustomerID,
IgnoreFirstName = ignoreFirstName,
FilterFirstName = filter.FirstName,
IgnoreLastName = ignoreLastName,
FilterLastName = filter.LastName,
IgnoreMemberStatus = ignoreMemberStatus,
FilterMemberStatus = Convert.ToInt32(filter.MemberStatus),
IgnoreMemberType = ignoreMemberType,
FilterMemberType = Convert.ToInt32(filter.MemberType),
IgnoreRank = ignoreRank,
FilterRank = Convert.ToInt32(filter.Rank),
OffSet = (page - 1) * 50
});
Method # 2
string queryThatWorks =
"SELECT *
FROM dbo.Customer c
WHERE c.MainCustomerId = #CustomerId
AND ({1} = 1 OR c.CustomerID = {2})
AND ({3} = 1 OR c.FirstName = '{4}')
AND ({5}= 1 OR c.LastName = '{6}')
AND ({7} = 1 OR c.CustomerStatusID = {8})
AND ({9} = 1 OR c.CustomerTypeID = {10})
AND ({11} = 1 OR c.RankID = {12})
ORDER BY c.CustomerId
OFFSET {13} ROWS
FETCH NEXT 50 ROWS ONLY";
_procExecutor.ExecuteSqlAsync<Report>(string.Format(queryThatWorks,
customerId,
ignoreCustomerId,
filter.CustomerID,
ignoreFirstName,
filter.FirstName,
ignoreLastName,
filter.LastName,
ignoreMemberStatus,
Convert.ToInt32(filter.MemberStatus),
ignoreMemberType,
Convert.ToInt32(filter.MemberType),
ignoreRank,
Convert.ToInt32(filter.Rank),
(page - 1) * 50
), null);
I've seen this countless times before.
I'm willing to bet that your columns are varChar, but Dapper is sending in your parameters as nVarChar. When that happens, SQL Server has to run a conversion on the value stored in each and every row. Besides being really slow, this prevents you from using indexes.
See "Ansi Strings and varchar" in https://github.com/StackExchange/dapper-dot-net
Related
I have this SP (extremply simplified for exhibit)
SELECT colA, colB, colC
INTO #MyTable
select COUNT(*) from #MyTable
On the C# side, I have
int total = 0;
var command = _context.CreateStoreCommand(
"dbo.mySP",
CommandType.StoredProcedure,
new SqlParameter("Param1", parameters.Param1),
//...
);
using (command.Connection.CreateConnectionScope())
{
using (var reader = command.ExecuteReader())
{
result = new Materializer<MyType>()
.Materialize(reader)
.ToList();
if (reader.NextResult() && reader.Read())
{
// This returns the total number of elements,
// (select COUNT(*) from #MyTable, ~15000 in my case)
total = reader.GetInt32(0);
}
}
}
Because the SP may result a huge amount of data, .Materialize(reader) only returns the 10 firsts elements into the result variable.
There's absolutely no way to change this rule for some reasons I can't explain here.
In the sample above, however, total is the total number of results of the SP, which may be greater than 10 (in my case ~15000) and it works well.
I now need to return another result, let's say
SELECT COUNT(*) from #MyTable WHERE colA = 'sample value'
Obviously, I can't do it C# side, like var x = result.Where(x => x.Prop = "sample value").Count() because it would search only on the 10 first results, whereas I want it to search on the whole dataset, which is ~15000 here.
I tried to add
var total2 = reader.GetInt32(1);
but reader.GetInt32(1) systematically fired a System.OutOfRangeException
How can I read, C# side, another result returned by a new SELECT statement on my SP ?
To receive a third resultset, you need to cal NextResult again:
if (reader.NextResult() && reader.Read())
{
total2 = reader.GetInt32(0);
}
Although I would advise you to just change your second query to use conditional aggregation (this is much more performant to do it together):
select COUNT(*), COUNT(CASE WHEN colA = 'sample value' THEN 1 END)
from #MyTable
You can then use your original code:
if (reader.NextResult() && reader.Read())
{
total = reader.GetInt32(0);
total2 = reader.GetInt32(1);
}
You could also entirely remove the temp table and just use a windowed count
SELECT colA, colB, colC,
COUNT(*) OVER (),
COUNT(CASE WHEN colA = 'sample value' THEN 1 END) OVER ()
FROM Whatever
You would now only have one resultset, where each row has the total for the whole set.
I have a stored procedure in SQL Server that gets contact persons based on multiple filters (e.g. DateOfBirth, DisplayName, ...) from multiple tables. I need to alter the stored procedure to include pagination and total count, since the pagination was done in the backend. PartyId is the unique key. The caveat is that a person can have multiple emails and phones, and let's say we search for DisplayName = "Sarah", the query will return the following :
TotalCount PartyId DisplayName EmailAddress PhoneNumber
-----------------------------------------------------------------
3 1 Sarah sarah#gmail.com 1
3 1 Sarah sarah2#gmail.com 1
3 1 Sarah sarah#gmail.com 2
This is roughly what the stored procedure does, the assigned values for CurrentPage and PageSize and the ORDER BY OFFSET on the bottom I included to test the pagination :
DECLARE #CurrentPage int = 1
DECLARE #PageSize int = 1000
SELECT
COUNT(*) OVER () as TotalCount,
p.Id AS PartyId,
e.EmailAddress,
pn.PhoneNumber
etc.....
FROM
[dbo].[Party] AS p WITH(NOLOCK)
INNER JOIN
[dbo].[Email] AS e WITH(NOLOCK) ON p.[Id] = e.[PartyID]
INNER JOIN
[dbo].[PhoneNumber] AS pn WITH(NOLOCK) ON p.[Id] = pn.[PartyID]
etc.....
WHERE
p.PartyType = 1 /*Individual*/
GROUP BY
p.Id, e.EmailAddress, pn.PhoneNumber etc...
ORDER BY
p.Id
OFFSET (#CurrentPage - 1) * #PageSize ROWS
FETCH NEXT #PageSize ROWS ONLY
This is what we do in the backend to group by PartyId and assign the corresponding emails and phones.
var responseModel = unitOfWork.PartyRepository.SearchContacts(model);
if (responseModel != null && responseModel.Count == 0)
{
return null;
}
// get multiple phones/emails for a party
var emailAddresses = responseModel.GroupBy(p => new { p.PartyId, p.EmailAddress })
.Select(x => new {
x.Key.PartyId,
x.Key.EmailAddress
});
var phoneNumbers = responseModel.GroupBy(p => new { p.PartyId, p.PhoneNumber, p.PhoneNumberCreateDate })
.Select(x => new {
x.Key.PartyId,
x.Key.PhoneNumber,
x.Key.PhoneNumberCreateDate
}).OrderByDescending(p => p.PhoneNumberCreateDate);
// group by in order to avoid multiple records with different email/phones
responseModel = responseModel.GroupBy(x => x.PartyId)
.Select(grp => grp.First())
.ToList();
var list = Mapper.Map<List<SearchContactResponseModelData>>(responseModel);
// add all phones/emails to respective party
list = list.Select(x =>
{
x.EmailAddresses = new List<string>();
x.EmailAddresses.AddRange(emailAddresses.Where(y => y.PartyId == x.PartyId).Select(y => y.EmailAddress));
x.PhoneNumbers = new List<string>();
x.PhoneNumbers.AddRange(phoneNumbers.Where(y => y.PartyId == x.PartyId).Select(y => y.PhoneNumber));
return x;
}).ToList();
var sorted = SortAndPagination(model, model.SortBy, list);
SearchContactResponseModel result = new SearchContactResponseModel()
{
Data = sorted,
TotalCount = list.Count
};
return result;
And the response will be :
{
"TotalCount": 1,
"Data": [
{
"PartyId": 1,
"DisplayName": "SARAH",
"EmailAddresses": [
"sarah#gmail.com",
"sarah2#gmail.com"
],
"PhoneNumbers": [
"1",
"2"
]
}
]
}
The TotalCount returned from the stored procedure obviously is not the real one, and after the backend code (where we assign the emails/phones and group by id) we get the real totalCount which is 1 instead of 3.
If we have 3 persons with the name Sarah, because of multiple phones/emails the totalCount in the stored procedure will be lets say 9 and the real count will be 3 and if I execute the stored procedure to get persons from 1 to 2, the pagination wont work because of the 9 records.
How can I implement pagination in the above scenario ?
You might try using a CTE to isolate the query against the Party table. This would allow you to pull the right number of rows (and the proper total row count) without having to worry about the expansion from the emails and phone numbers.
It would look something like this (rearranging your query above):
DECLARE #CurrentPage int = 1;
DECLARE #PageSize int = 1000;
WITH PartyList AS (
SELECT
COUNT(*) OVER () as TotalCount,
p.Id AS PartyId
FROM
[dbo].[Party] AS p WITH(NOLOCK)
WHERE
p.PartyType = 1 /*Individual*/
GROUP BY -- You might not need this now depending on your data
p.Id
ORDER BY
p.Id
OFFSET (#CurrentPage - 1) * #PageSize ROWS
FETCH NEXT #PageSize ROWS ONLY
)
SELECT
pl.TotalCount,
pl.PartyId,
e.EmailAddress,
pn.PhoneNumber
FROM PartyList AS pl
INNER JOIN
[dbo].[Email] AS e WITH(NOLOCK) ON pl.[PartyId] = e.[PartyID]
INNER JOIN
[dbo].[PhoneNumber] AS pn WITH(NOLOCK) ON pl.[PartyId] = pn.[PartyID];
Please be aware that the CTE will require the prior statement to end in a semicolon.
I have a clause like this:
Give me all contracts where IsDeleted is 0 AND
where UstrojstvenaJedinicaId is equal to procedure parameter (#zavodId)
OR
there is only one UstrojstvenaJedinicaId and that UstrojstvenaJedinicaId is = 'HCPHS'
Example:
Procedure parameter zavodId = 5;
So in this example, I want to get all contracts where UstrojstvenaJedinicaId = 5 and only those contracts because we met the requests in first part of where clause.
If it helps, this is my C# code which is good and works and SQL query should be like this:
.Where(x => x.UstrojstveneJedinice.Any
(y => y.UstrojstvenaJedinicaId == zavodId) ||
x.UstrojstveneJedinice.All(y => y.UstrojstvenaJedinicaId == 10))
I think this is what you are asking for:
IF ( (SELECT COUNT(table.id) FROM *Tablename* WHERE UstrojstvenaJedinicaId = 'HCPHS') = 1)
BEGIN
(SELECT * FROM *Tablename* WHERE UstrojstvenaJedinicaId = 'HCPHS')
END
ELSE
(SELECT * FROM *Tablename* WHERE isdeleted = 0 and UstrojstvenaJedinicaId = #zavodId)
From this database, I want to select two users, 321 and 102 where BOOK_TYPE is 1 or 2 and compare their BOOK_NR, if they coincide then save BOOK_COUNT only from USER_ID 102.
SqlCommand command = new SqlCommand("SELECT BOOK_NR, BOOK_COUNT, USER_ID, BOOK_TYPE " +
"FROM BOOKS " +
"WHERE (BOOK_TYPE = '1' OR" + "BOOK_TYPE = '2')" +
"AND MONTH(DATE) = '" + DateTime.Today.Month + "'" +
"GROUP BY BOOK_NR, BOOK_COUNT, USER_ID, BOOK_TYPE ", BookConn);
SqlDataReader reader = command.ExecuteReader();
while (reader.Read())
{
string BookNr = (string)reader[0];
int Count = (int)reader[1];
int User = (int)reader[2];
int Type = (int)reader[3];
if( " HERE I NEED HELP " )
{
" AND HERE :) "
}
}
reader.Close();BookConn.Close();
My only solution is as follows
List<string> User321List = new List<string>();
List<string> User102List = new List<string>();
if(User == 321 && Type == 1){ User321List.Add(BookNr.Trim());}
if(User == 102 && Type == 2){ User102List.Add(BookNr.Trim()+"\t"+Count);}
and then...
int count = 0;
foreach (string x in User321List)
{
foreach (string y in User102List)
{
List<string> part = y.Split('\t').Select(p => p.Trim()).ToList();
var z = string.Compare(part[0], x);
if (z == 0)
{
count += int.Parse(part[1]);
}
}
}
but it takes a lot of time to get result..
if someone figured something out of my nightmare please help and sorry for my bad english...
I'm going to focus on answering just the first part:
I want to select two users, 321 and 102 where BOOK_TYPE is 1 or 2 and compare their BOOK_NR, if they coincide then save BOOK_COUNT only from USER_ID 102.
You want to join the table to itself:
SELECT b1.BOOK_NR, b1.BOOK_COUNT, b1.USER_ID, b1.BOOK_TYPE
FROM BOOKS b1
INNER JOIN BOOKS b2 ON b2.BOOK_TYPE IN ('1','2')
AND b2.User_ID = 321
AND b1.BOOK_NR = b2.BOOK_NR
WHERE b1.User_ID = 102 AND b1.BOOK_TYPE IN ('1','2')
AND MONTH(b1.DATE) = MONTH(current_timestamp)
AND MONTH(b2.DATE) = MONTH(current_timestamp)
Since one of your concerns was performance -- that the query was too slow -- I also need to ask whether the BOOK_TYPE column is a string or a number. If it's a number column, you should omit the single quotes on the values: for example, just use IN (1,2) instead of IN ('1','2'). This will help performance by avoiding potential per-row conversions and by potentially better matching to indexes.
If your query is slow, consider splitting the operation in two. Create a SQL Table function so you don't repeat the SQL, passing the users, type and month in as parameters.
Do not do an IN (as suggested in comment). Keep in mind that an OR in the WHERE clause will result in no indexes being used.
Join the two instances of the table function and let sql do the matching for you.
e.g.
Select * from dbo.UserList('321',1,12) as u321
inner join dbo.UserList('102',2,12) as u102
on u321.field = u102.field
BTW... I covered quite a few advanced topics, but the idea is to let SQL do what it does best: JOINS
As a matter of interest, if your sql table function returns many rows, and you ever use them in a sp, remember to use the option (recompile) clause ...
I have used the Kendo DataSourceResult ToDataSourceResult(this IQueryable enumerable, DataSourceRequest request); extension extensively and never noticed a performance issue until now when querying a table of 40 million records.
The take 10 query I wrote as a benchmark as it is the same as the request passed in.
This is my read action:
public ActionResult ReadAll([DataSourceRequest] DataSourceRequest
{
var startTimer = DateTime.Now;
var context = Helpers.EFTools.GetCADataContext();
Debug.WriteLine(string.Format("{0} : Got Context", DateTime.Now - startTimer));
var events = from e in context.Events
select
new Models.Event()
{
Id = e.Id,
DateTime = e.EventDateTime,
HostId = e.Door.HostId,
SiteId = e.Door.Host.SiteId,
UserId = (int)e.UserId,
UserName = e.User.FirstName + " " + e.User.Surname,
DoorId = e.DoorId,
Door = e.Door.Name,
Description = e.Description,
SubDescription = e.SubDescription
};
Debug.WriteLine(string.Format("{0} : Built Query", DateTime.Now - startTimer));
var tenRecods = events.OrderByDescending(i => i.DateTime).Take(10).ToList();
Debug.WriteLine(string.Format("{0} : Taken 10", DateTime.Now - startTimer));
var result = events.ToDataSourceResult(request);
Debug.WriteLine(string.Format("{0} : Datasource Result", DateTime.Now - startTimer));
return this.Json(result);
}
The output from Debug:
00:00:00.1316569 : Got Context
00:00:00.1332584 : Built Query
00:00:00.2407656 : Taken 10
00:00:21.5013946 : Datasource Result
Although sometimes the query times out.
Using dbMonitor I captured both querys, first the manual take 10:
"Project1".id,
"Project1"."C1",
"Project1".hostid,
"Project1".siteid,
"Project1".userid,
"Project1"."C2",
"Project1".doorid,
"Project1"."name",
"Project1".description,
"Project1".subdescription
FROM ( SELECT
"Extent1".id,
"Extent1".userid,
"Extent1".description,
"Extent1".subdescription,
"Extent1".doorid,
"Extent2"."name",
"Extent2".hostid,
"Extent3".siteid,
CAST("Extent1".eventdatetime AS timestamp) AS "C1",
"Extent4".firstname || ' ' || "Extent4".surname AS "C2"
FROM public.events AS "Extent1"
INNER JOIN public.doors AS "Extent2" ON "Extent1".doorid = "Extent2".id
INNER JOIN public.hosts AS "Extent3" ON "Extent2".hostid = "Extent3".id
INNER JOIN public.users AS "Extent4" ON "Extent1".userid = "Extent4".id
) AS "Project1"
ORDER BY "Project1"."C1" DESC
LIMIT 10
And the ToDataSourceRequest query:
SELECT
"GroupBy1"."A1" AS "C1"
FROM ( SELECT Count(1) AS "A1"
FROM public.events AS "Extent1"
INNER JOIN public.doors AS "Extent2" ON "Extent1".doorid = "Extent2".id
) AS "GroupBy1"
This is the DataSourceRequest request parameter passed in:
request.Aggregates Count = 0
request.Filters Count = 0
request.Groups Count = 0
request.Page 1
request.PageSize 10
request.Sorts Count = 1
This is the result of var result = events.ToDataSourceResult(request);
result.AggregateResults null
result.Data Count = 10
result.Errors null
result.Total 43642809
How can I get a DataSourceResult from the events IQueryable using the DataSourceRequest in a more efficient and faster way?
After implementing a custom binding (suggested by Atanas Korchev) with lots of debug output time stamps, it was obvious what was causing the performance issue, the total count.
Looking at the SQL I captured backs this up, don't know why I didn't see it before.
Getting the total row count quickly is another question but will post any answers I find here.