We Keep Coding

Troubleshooting LINQ Query Performance in ASP.NET MVC

I'm having trouble with a LINQ query after joining a new table to it. Actually, it returns the data I'm expecting and it runs fast in testing. However, it seems that, as more users connect to the database, the query begins to timeout. For example, everything was working fine for the first 30 or 45 minutes in Production, but then at about 8:20 AM, it started to timeout. Again, I assume this is due to increased usage of the database on the whole.
Here is a little background on the ASP.NET MVC (5) application, in case that helps.
A user submits a referral to our clinic
The referral contains one or more orders
If the person information supplied does not match an existing person, I do several things, including inserting records in an "orders" table (one record for each order selected on the referral).
If the person information supplied does match an existing person in our system, then I "hold" the referral in a queue until it is manually resolved by either matching it to an existing person or by overriding it and creating a new person in the system. At this time, any orders selected in the referral are created in the table.
So, the two main tables to think about in this scenario are the "referral" (named "Referrals" in my code) and "order" (named "ReferralPPAs" in my code) tables. Until now, I have not needed to link the query in question from the Referrals table to the ReferralPPAs table (linking the query to the ReferralPPAs table seems to be what is slowing the query down once database/application usage increases).
Also, in case this helps, the referrals are entered by external users, while the orders I created from the referral are worked in a separate application with internal staff as the users, though it's all in the same database. The ReferralPPAs table is probably being used pretty heavily most of the day.
The query looks like this:
IQueryable<ReferralListViewModel> referrals = (from r in _context.Referrals
join cu in _context.ClinicUsers on r.ClinicId equals cu.ClinicId
/* Here is the seemingly problematic join */
from ppa in _context.ReferralPPAs
.Where(p => p.ref_id == r.seq_no.ToString())
.DefaultIfEmpty()
/* End of seemingly problematic join */
join ec in _context.EnrolledClinics on r.ClinicId equals ec.ClinicId
join pm in _context.ProviderMasters on ec.ClinicId equals pm.ClinicId
join ml in _context.MasterLists on pm.HealthSystemGuid equals ml.Id
join au in _context.Users on r.ApplicationUserId equals au.Id
where cu.UserId == userId
select new ReferralListViewModel()
{
ClinicName = pm.Description,
ClinicId = r.ClinicId,
ReferralId = r.seq_no,
EnteredBy = (au.FirstName ?? string.Empty) + " " + (au.LastName ?? string.Empty),
PatientName = (r.LastName ?? string.Empty) + ", " + (r.FirstName ?? string.Empty),
DateEntered = r.create_timestamp,
Status = ppa != null ? ppa.Status : string.Empty
});
So, without the join I make reference to above, I experience no problems and it runs quite fast. Adding the join also appears to be fast, again, until a certain number of users are on the system (at least that's my assumption).
A couple of other things I've tried to help improve performance and prevent the problem. I set the UseDatabaseNullSemantics to True, which seems to make a big difference in the overall performace.
_context.Configuration.UseDatabaseNullSemantics = true;
I also wondered if the problem was an issue of locking on the table in question, so I tried wrapping the query in a transaction to do a ReadUncommitted.
using (var transaction = _context.Database.BeginTransaction(System.Data.IsolationLevel.ReadUncommitted))
{
//query
}
Again, while this improves the overall performance a little bit, it didn't seem to ultimately resolve the problem.
If anyone has any thoughts, ideas, or suggestions on how to tackle this, I would greatly appreciate it.

Based on the additional information from the comments, looks like the Guid to String conversion in the join condition
p.ref_id == r.seq_no.ToString()
translated to
t1.ref_id = LOWER(CAST(t2.seq_no AS nvarchar(max))))
makes the query not sargable, while the implicit SqlServer conversion
t1.ref_id = t2.seq_no
works just fine.
So the question is how to remove that cast. There is no option for that and also query expression tree does not allow removing it. It would be nice if the SqlServer provider sql generator was doing that optimization, but it doesn't and there is no easy way to hook into it.
As a workaround I can offer the following solution. It uses a custom IDbCommandTreeInterceptor and DbExpressionVisitor to modify the DbCommandTree of the query.
Here is the interception code:
using System;
using System.Data.Entity.Core.Common.CommandTrees;
using System.Data.Entity.Core.Common.CommandTrees.ExpressionBuilder;
using System.Data.Entity.Core.Metadata.Edm;
using System.Data.Entity.Infrastructure.Interception;
using System.Linq.Expressions;
using System.Reflection;
namespace EFHacks
{
public class MyDbCommandTreeInterceptor : IDbCommandTreeInterceptor
{
public void TreeCreated(DbCommandTreeInterceptionContext interceptionContext)
{
if (interceptionContext.OriginalResult.DataSpace != DataSpace.SSpace) return;
var queryCommand = interceptionContext.Result as DbQueryCommandTree;
if (queryCommand != null)
{
var newQuery = queryCommand.Query.Accept(new GuidToStringComparisonRewriter());
if (newQuery != queryCommand.Query)
{
interceptionContext.Result = new DbQueryCommandTree(
queryCommand.MetadataWorkspace,
queryCommand.DataSpace,
newQuery);
}
}
}
}
class GuidToStringComparisonRewriter : DefaultExpressionVisitor
{
public override DbExpression Visit(DbComparisonExpression expression)
{
if (IsString(expression.Left.ResultType) && IsString(expression.Right.ResultType))
{
var left = expression.Left;
var right = expression.Right;
if (RemoveCast(ref right) || RemoveCast(ref left))
return CreateComparison(expression.ExpressionKind, left, right);
}
return base.Visit(expression);
}
static bool IsGuid(TypeUsage type)
{
var pt = type.EdmType as PrimitiveType;
return pt != null && pt.PrimitiveTypeKind == PrimitiveTypeKind.Guid;
}
static bool IsString(TypeUsage type)
{
var pt = type.EdmType as PrimitiveType;
return pt != null && pt.PrimitiveTypeKind == PrimitiveTypeKind.String;
}
static bool RemoveCast(ref DbExpression expr)
{
var funcExpr = expr as DbFunctionExpression;
if (funcExpr != null &&
funcExpr.Function.BuiltInTypeKind == BuiltInTypeKind.EdmFunction &&
funcExpr.Function.FullName == "Edm.ToLower" &&
funcExpr.Arguments.Count == 1)
{
var castExpr = funcExpr.Arguments[0] as DbCastExpression;
if (castExpr != null && IsGuid(castExpr.Argument.ResultType))
{
expr = castExpr.Argument;
return true;
}
}
return false;
}
static readonly Func<DbExpressionKind, DbExpression, DbExpression, DbComparisonExpression> CreateComparison = BuildCreateComparisonFunc();
static Func<DbExpressionKind, DbExpression, DbExpression, DbComparisonExpression> BuildCreateComparisonFunc()
{
var kind = Expression.Parameter(typeof(DbExpressionKind), "kind");
var booleanResultType = Expression.Field(null, typeof(DbExpressionBuilder), "_booleanType");
var left = Expression.Parameter(typeof(DbExpression), "left");
var right = Expression.Parameter(typeof(DbExpression), "right");
var result = Expression.New(
typeof(DbComparisonExpression).GetConstructor(BindingFlags.NonPublic | BindingFlags.Instance, null,
new[] { kind.Type, booleanResultType.Type, left.Type, right.Type }, null),
kind, booleanResultType, left, right);
var expr = Expression.Lambda<Func<DbExpressionKind, DbExpression, DbExpression, DbComparisonExpression>>(
result, kind, left, right);
return expr.Compile();
}
}
}
and DbConfiguration to install it:
class MyDbConfiguration : DbConfiguration
{
public MyDbConfiguration()
{
AddInterceptor(new EFHacks.MyDbCommandTreeInterceptor());
}
}
Tested and working in EF6.1.3 and EF6.2 with SqlServer database.
But use it with care.
First, it works only for SqlServer.
Second, it's hackish because I had to use internal field and internal class constructor in order to skip the check for equal types of the comparison operation operands. So some future EF6 update might break it.

passing linq anonymous result to iqueryable object for another query

I have two tables (tbPerson and tbDataLog) where I need to return Id from one table (tbPerson) after checking certain conditions on both. After this, this result should be passed to another query. My first query returns the Id (primary key of a table) successfully and I need to pass these ids to another query so that it return me data based upon these Id. I also has an IQueryable type base object to check certain conditions to fetch data.
IQueryable<tbPerson> dataset
and I cannot changes this from Iqueryable to other as it will break other part of the code)
My first linq statement:
public static IQueryable<LogResults> GetResultsForYes()
{
Databasename ents = new Databasename();
var ids = (from f in ents.tbPerson
join g in ents.tbDataLog
on f.InfoID equals g.RefId
where g.Tag == "subscribed" && g.OldValue == "No" && g.Action == "Modified"
select new LogResults { _LogID = f.Id }).OrderBy(x => x._LogID);
return ids;
}
public class LogResults
{
public int _LogID { get; set; }
}
I access my result something like this where I can see in debugger all the Ids.
IQueryable<LogResults> log = GetResultsForYes();
Problem comes, when I tried to get records from tbPerson based upon these returned Id.
dataset=log.where(x=>x._LogID != 0);
I get this error:
Cannot implicitly convert type 'System.Linq.IQueryable' to 'System.Linq.IQueryable'. An explicit conversion exists(are you missing a cast)?
Any suggestions or some other good approach is welcome.

I love this thing about stackoverflow. when we write questions we force our brain to think more deeply and after 30 mins of posting this question, I solved it in a simple way. Sometimes we overcomplicated things!
var ids = (from f in ents.tbPerson
join g in ents.tbDataLog
on f.InfoID equals g.RefId
where g.Tag == "subscribed" && g.OldValue == "No" && g.Action == "Modified"
select new { f.Id }).ToArray();
var allId = ids.Select(x => x.Id).ToArray();
dataset = dataset.Where(x => allId.Contains(x.Id));
#ankit_sharma : I have not tested yours but will give a try and come back to you. Thanks for giving time and effort.

IQueryable<tbPerson> dataset=log.where(x=>x._LogID != 0);
The result of log.where(x=>x._LogID != 0) is an IQueryable<LogResults>, and you are trying to assign this result to dataset of type IQueryable<tbPerson>, two diferent types.
EDIT:
I see you make a join to get the tbPerson ids, and then you do a second query to get the persons. You could get the persons in the first join.
I just modify your code:
IQueryable<tbPerson> persons = from person in ents.tbPerson
join g in ents.tbDataLog
on person.InfoID equals g.RefId
where g.Tag == "subscribed" && g.OldValue == "No" && g.Action == "Modified"
select person;

How do I join more than one files, in a Linq (C#) where one of the joins is a left join

My Sql below works great, but now I would like to join another file that has a common key. The file I would like to join has could have one to many records, any suggestions
here is my code:
var regPoints = (from x in db.CertPoints
join y in db.CommunityPts on x.PointId equals y.PointId into z
from t in
(from r in z
where r.CommunityId == id select r).DefaultIfEmpty()
where x.TypeId == 1
select new Points
{
pointId = x.PointId,
pointsDescription = x.PointDescription,
points = x.Points,
dateApplied = t.DateApplied,
pointsEarned = t.PointsEarned,
pointsPending = t.Pending ? true : false,
pointsApproved = t.Approved ? true : false,
notes = t.Notes
}).AsEnumerable();
the new join would be a one to many records, where the key in CommunityPts is Id, and the file I would like to join is a list of file links "CommunityPtsDocs" with a foreign key CommnunityPtId. How do i add it to the above sql statement above?

Sometimes I feel I'm a navigation properties evangelist (fortunately, I'm not the only one).
The answer you accepted is OK, it does the job. But using any ORM like Entity Framework or LINQ-to-SQL you should avoid the join statement as much as possible. It's verbose and error-prone. It causes repetitive code and it's too easy to join the wrong properties erroneously.
You class CertPoint could have a 0..1-n navigation property CommunityPts (a list) and CommunityPt could have a 1-n navigation property CommunityPtsDocs (also a list). If you're using LINQ-to-SQL, chances are that they're already there but you're not aware of them. If you use Entity Framework code-first, you should add them yourself.
Having these navigation properties, your code is going to look like this:
from cert in CertPoints
from comm in cert.CommunityPts.DefaultIfEmpty()
from doc in comm.CommunityPtsDocs
where comm.CommunityId == id && cert.TypeId == 1
select new Points
{
pointId = cert.PointId,
pointsDescription = cert.PointDescription,
points = cert.Points,
dateApplied = comm.DateApplied,
pointsEarned = comm.PointsEarned,
pointsPending = comm.Pending ? true : false,
pointsApproved = comm.Approved ? true : false,
notes = comm.Notes,
something = doc.Something
})
Now the ORM will translate this into SQL with the correct joins and your code looks much cleaner (note that I also prefer more meaningful range variable names).

Following modification shall help in achieving the Task, though I prefer Fluent syntax, as that's much cleaner in my view to achieve the same, though I have not selected any column from CommunityPtsDocs in the Select statement
var regPoints = (from x in CertPoints
join y in CommunityPts on x.PointId equals y.PointId
join s in CommunityPtsDocs on y.Id equals s.CommnunityPtId into k
from t in (from r in k where r.CommunityId == id select r).DefaultIfEmpty()
where x.TypeId == 1
select new Points
{
pointId = x.PointId,
pointsDescription = x.PointDescription,
points = x.Points,
dateApplied = t.DateApplied,
pointsEarned = t.PointsEarned,
pointsPending = t.Pending ? true : false,
pointsApproved = t.Approved ? true : false,
notes = t.Notes
}).AsEnumerable();

Join tables in NHibernate without mapping

I have the following two objects:
User
class User {
public int role;
}
Role
class Role {
public int id;
public string name;
}
be note that role property inside User is int and not Role, that's our limitations.
I want to join between all the users and each of his role. In the mapping objects there is no reference as you can understand, just a simple type (int).
How do I do that join statement?

It's called a theta join:
var a = (from u in session.Query<User>()
from r in session.Query<Role>()
where u.role == r.id
select new { u.Username, Role = r.name }).ToList();
Assuming you have a Username property on the User class.

Yes, this "theta join" (as I just learned this term) is very handy and let's us not worry about putting in pointless mapping relationships.
WARNING HOWEVER IN USING THIS!!! This tripped me up a lot.
Adding to the above example...
var list = new List<int>( { 2, 3 } ); // pretend in-memory data from something.
var a =
(from u in session.Query<User>()
from x in list
from r in session.Query<Role>()
where u.role == r.id
where r.id == x.id // pretend list wants to limit to only certain roles.
select new { u.Username, Role = r.name }).ToList();
THIS WILL BOMB with some NotSupported exception.
The trick is that anything coming from NHibernate Session must come LAST. So this alteration WILL work:
var a =
(from x in list
from u in session.Query<User>()
from r in session.Query<Role>()
where u.role == r.id
where r.id == x.id // pretend list wants to limit to only certain roles.
select new { u.Username, Role = r.name }).ToList();
And and BTW, you can use join as well, however you have to make sure if you have any nullable data types, that you use the .Value if you are joining to something not-nullable.
var a =
(from x in list
from u in session.Query<User>()
join r in session.Query<Role>() on u.role equals r.id
where r.id == x.id // pretend list wants to limit to only certain roles.
select new { u.Username, Role = r.name }).ToList();
And while we're at it, let's say you have a method that has some dynamic condition. In this example the 'list' which could be a list of roles to filter by, but don't filter at all if the list is not there. Well, if you do the .ToList() then you are causing this query to execute immediately. But instead you can add a condition and then execute it later:
var a =
from u in session.Query<User>()
join r in session.Query<Role>() on u.role equals r.id
where r.id == x.id // pretend list wants to limit to only certain roles.
select new { u.Username, Role = r.name, RoleID = r.id }; // Adding the Role ID into this output.
if (list != null) // assume if the list given is null, that means no filter.
{
a = a.Where(x => list.Contains(x.RoleID));
// WARNING. Unfortunately using the "theta" format here will not work. Not sure why.
}
var b = a.ToList(); // actually execute it.
var c = a.Select(x => new { x.Username, x.Role }).ToList() // if you insist on removing that extra RoleID in the output.
One last thing.. Sometimes some simple logic will fail when executed in the select new { .. } part. I don't have an explanation. In our case the logic was just converting a DB value of a uint to an Enumerator of a model. But to get around that, I just avoided doing that conversion while reading the data but saved the value. Then in a later step, after the data was loaded, I just did the conversion in another LINQ statement.
DISCLAIMER: While I wrote many of these things all the past several weeks, I did not put this code into my compiler to verify 100%.

LinQ to SQL query database and assign to current instance (this)

I have a public method called LoadContact() in the Contact class. I want to load the contact data using Linq and assign the values to 'this' instance.
So far I have .....
var contact = (from cont in db.Contacts
from note in db.Contacts_Notes.Where(n => n.ContactID == cont.ContactID).DefaultIfEmpty()
where cont.AccountID == this.AccountID && cont.ContactID == this.ContactID
select new Contact
{
AccountID = cont.AccountID,
CompanyName = cont.CompanyName,
ContactID = cont.ContactID,
Firstname = cont.Firstname,
JobTitle = cont.JobTitle,
Lastname = cont.Lastname,
Notes = note.Note
}).SingleOrDefault();
if(contact != null)
{
this.AccountID = contact.AccountID;
this.CompanyName = contact.CompanyName etc etc etc
}
.,.. but this seems really long winded. How can I assign the results directly to the current instance?

Is there a reason why you don't make it a static method which return a new contact? The way you wants to do it looks like bad design (depending on why you do it).
I would suggest you made it static so it could be used like:
var contact = Contact.Load(...);
instead of
var contact = new Contact();
contact.LoadContact(...);
This way you would be able to just return the contact you found with your query like:
public static Load(int contactID, int accountID) //If they are integers
{
return (from cont in db.Contacts
from note in db.Contacts_Notes.Where(n => n.ContactID == cont.ContactID).DefaultIfEmpty()
where cont.AccountID == accountID && cont.ContactID == contactID
select new Contact
{
//... stuff
}).SingleOrDefault();
}

This is exactly the problem that Automapper was built to solve. Recommended!

There is no way to do that because LINQ is made to query and get results back, so you will always need to iterate through the list or get one item from the list and do your thing with it.

Just a thought: what about:
var contact = (from cont in db.Contacts
from note in db.Contacts_Notes.Where(n => n.ContactID == cont.ContactID).DefaultIfEmpty()
where cont.AccountID == this.AccountID && cont.ContactID == this.ContactID
select this.RenewSelf(
cont.AccountID,
cont.CompanyName,
cont.ContactID,
cont.Firstname,
cont.JobTitle,
cont.Lastname,
note.Note
).SingleOrDefault();
Not too pretty, but if you implement the private method RenewSelf it is at least more readable and less "long-winded". Other options exist, think of something like AutoMapper, as somebody else already suggested.
PS: normally, the DAO layer is not built inside the POCO, which is a sign of "code smell". Just meaning to say, maybe you should change your technical design to a more robust method where POCOs and DAO are separated, in which case this problem will not even occur anymore (i.e., the Load, Save etc methods are part of the DAO and take a POCO, here a Contact object, as input or result).