I try to join two different databases tables in c# but it gives me an error how can I handle that ?
this is my query:
var list = (from h in db.database1.AsEnumerable()
join j in NV_DB.database2.AsEnumerable()
on h.Creation_Date equals j.Creation_Date
where j.Ship_Status == 3 && h.CustomerNo == CustomerNo
select new
{
shipName = h.ShipName,
creationDate = j.Creation_Date,
endingDate = j.Ending_Date
}
).ToList();
if I do like this it gives me System.OverflowException error. But when I run this in sql, it gives me just 30 records*
You need to remove °AsEnumerable`. While it does not run the sql, when you use it in the where it actually brings the entire tables in memory and then performs the job where part of your query
Your answer is basically the first comment in the accepted answer here: Am I misunderstanding LINQ to SQL .AsEnumerable()?
While AsEnumerable doesn't evaluate the query at the time that it's called , it definitely has an effect. Anything further called on the query will be evaluated using LINQ to objects, so you can't compose additional elements onto the query (another Where or an OrderBy or anything of that nature) that will become part of the SQL statement.
In depth explanation here: https://www.codeproject.com/Articles/732425/IEnumerable-Vs-IQueryable
While querying data from database, IEnumerable executes select query on server side, load data in-memory on client side and then filter data. Hence does more work and becomes slow.
While querying data from database, IQueryable executes select query on server side with all filters. Hence does less work and becomes fas
To debug this, start dividing your statements into smaller steps:
var list1 = db.database1.AsEnumerable().ToList();
var list2 = NV_DB.database2.AsEnumerable().ToList();
var joinResult = list1.Join(list2, // join list1 and list2
list1Row => list1Row.CreationDate, // from every row in list1 take the CreationDate
list2Row => list2Row.CreationDate, // from every row in list2 take the CreationDate
(list1Row, list2Row) => new // when they match, make one new object
{
// You only need the following properties:
ShipName = list1Item.ShipName,
CreationDate = list2Item.CreationDate,
EndingDate = list2Item.EndingDate,
ShipStatus = list2Item.ShipStatus,
CustomerNo = list1Item.CustomerNo,
})
.ToList();
var whereResult = joinResult
.Where(joinedRow => joinedRow.ShipStatus == 3
&& joinedRow.CustomerNo == customerNo)
.ToList();
var selectResult = whereResult.Select(whereResultRow => new
{
ShipName = whereResultRow.ShipName,
CreationDate = whereResultRow.CreationDate,
EndingDate = whereResultRow.Ending_Date,
})
.ToList();
This is executed completely as enumerable (in your local process, not by the database management system). My guess would be that this runs smoothly.
Now combine thw first few statements:
var joinResult = db.database1.AsEnumerable()
.Join(NV_DB.database2.AsEnumerable(), // join list1 and list2
list1Row => list1Row.CreationDate, // from every row in list1 take the CreationDate
list2Row => list2Row.CreationDate, // from every row in list2 take the CreationDate
(list1Row, list2Row) => new // when they match, make one new object
{
// You only need the following properties:
ShipName = list1Item.ShipName,
CreationDate = list2Item.CreationDate,
EndingDate = list2Item.EndingDate,
ShipStatus = list2Item.ShipStatus,
CustomerNo = list1Item.CustomerNo,
})
.ToList();
When this works, add the Where:
var whereResult = db.database1.AsEnumerable()
.Join(NV_DB.database2.AsEnumerable(), ...)
.Where(joinedRow => joinedRow.ShipStatus == 3
&& joinedRow.CustomerNo == customerNo)
.ToList();
Etc.
Using your debugger, you'll find the problem within a few minutes (depending on your compilation time). My guess is that it is within your join.
Related
I have a LINQ query which outputs ToList it works fine other than the fact each time it's run it updates the original record instead of creating a new one.
On every run through this code the data.EventID changes so I'd like every record to appear in the list.
The code:
foreach(var data in vehicleqry)
{
bool inUK = boundryChecker.IsLongLatInUK((double)data.declatfloat, (double)data.declongfloat);
if (inUK == true)
{
var qryevent = (from e in db.events
where e.eventID == data.EventID
select new
{
e.eventID,
e.sysdatetime,
e.vehicleID
}).ToList();
}
{
I also have a list with the eventIDs in if I can use this to query the list?
I think what you actually want is to only run a single query instead of looping around. You can do this by making use of the Contains method:
var vehicleqry = ...;
// Get all of the individual event IDs for entries that are "inUK"
var vehicleEventIds = vehicleqry
.Where(ve => boundryChecker
.IsLongLatInUK((double)ve.declatfloat, (double)ve.declongfloat)
.Select(ve => ve.EventID);
// Get all the matching events
var qryevent = (from e in db.events
where vehicleEventIds.Contains(e.eventID)
select new
{
e.eventID,
e.sysdatetime,
e.vehicleID
}).ToList();
I have a query that returns a list of currencies and joins to a lookup table. The result is then put into a class object (which works fine):
var queryforobject = from x in db.CurrencyExchangeRates.AsNoTracking()
join c in db.CurrencyTypes.AsNoTracking() on x.CurrencyTypeID equals c.ID
orderby x.ID
select new CurrencyExchangeRateObject
{
ID = x.ID,
CurrencyID = c.ID,
Currency = c.Description,
ExchangeRate = x.ExchangeRate,
LastEditedDate = x.LastEditedDate,
LastEditedBy = x.LastEditedBy,
Active = x.Active
};
I want to make this more dynamic, so if no CurrencyTypeID is supplied then it will return the full list (as it does already) - otherwise if a CurrencyTypeID is supplied it will only show where X.CurrencyTypeID = ID.
Something along the lines of an inline if?
There are a few options for filtering the query based on CurrencyTypeID if a search value (named currencyTypeID in this answer) is supplied, but return all data if no currencyTypeID is supplied.
First option: You could add a where clause to your existing query expression. The WHERE clause below will return every record in the data set if null is passed in for the currencyTypeID variable, otherwise it will filter the results.
from x in db.CurrencyExchangeRates.AsNoTracking()
join c in db.CurrencyTypes.AsNoTracking() on x.CurrencyTypeID equals c.ID
where (currencyTypeID == null || x.CurrencyTypeID == currencyTypeID)
orderby x.ID
select new CurrencyExchangeRateObject {
ID = x.ID,
CurrencyID = c.ID,
Currency = c.Description,
ExchangeRate = x.ExchangeRate,
LastEditedDate = x.LastEditedDate,
LastEditedBy = x.LastEditedBy,
Active = x.Active
};
Alternatively: Since queryforobject is of type IQueryable<T>, you can use LINQ's fluent API to append a WHERE clause to the query inside an if statement. You need to be more careful about timing on this one though as it needs to be done before you force evaluation of the IQueryable with a foreach loop, .ToList(), .Select() or other LINQ methods that force evaluation.
if(currencyTypeID != null)
queryforobject = queryforobject.Where(cerObj => cerObj.CurrencyID == currencyTypeID);
Here's my code:
var myStrings = (from x in db1.MyStrings.Where(x => homeStrings.Contains(x.Content))
join y in db2.MyStaticStringTranslations on x.Id equals y.id
select new MyStringModel()
{
Id = x.Id,
Original = x.Content,
Translation = y.translation
}).ToList();
And I get the error that the specified LINQ expression contains references to queries that are associated with different contexts. I know that the problem is that I try to access tables from both db1 and db2, but how do I fix this?
MyStrings is a small table
Load filtered MyStrings in memory, then join with MyStaticStringTranslations using LINQ:
// Read the small table into memory, and make a dictionary from it.
// The last step will use this dictionary for joining.
var byId = db1.MyStrings
.Where(x => homeStrings.Contains(x.Content))
.ToDictionary(s => s.Id);
// Extract the keys. We will need them to filter the big table
var ids = byId.Keys.ToList();
// Bring in only the relevant records
var myStrings = db2.MyStaticStringTranslations
.Where(y => ids.Contains(y.id))
.AsEnumerable() // Make sure the joining is done in memory
.Select(y => new {
Id = y.id
// Use y.id to look up the content from the dictionary
, Original = byId[y.id].Content
, Translation = y.translation
});
You are right that db1 and db2 can't be used in the same Linq expression. x and y have to be joined in this process and not by a Linq provider. Try this:
var x = db1.MyStrings.Where(xx => homeStrings.Contains(xx.Content)).ToEnumerable();
var y = db2.MyStaticStringTranslations.ToEnumerable();
var myStrings = (from a in x
join b in y on x.Id equals y.id
select new MyStringModel()
{
Id = x.Id,
Original = x.Content,
Translation = y.translation
}).ToList();
Refer to this answer for more details: The specified LINQ expression contains references to queries that are associated with different contexts
dasblinkenlight's answer has a better overall approach than this. In this answer I'm trying to minimize the diff against your original code.
I also faced the same problem:
"The specified LINQ expression contains references to queries that are associated with different contexts."
This is because it's not able to connect to two context at a time so i find the solution as below.
Here in this example I want to list the lottery cards with the owner name but the Table having the owner name is in another Database.So I made two context DB1Context and DB2Context.and write the code as follows:
var query = from lc in db1.LotteryCardMaster
from om in db2.OwnerMaster
where lc.IsActive == 1
select new
{
lc.CashCardID,
lc.CashCardNO,
om.PersonnelName,
lc.Status
};
AB.LottryList = new List<LotteryCardMaster>();
foreach (var result in query)
{
AB.LottryList.Add(new LotteryCardMaster()
{
CashCardID = result.CashCardID,
CashCardNO = result.CashCardNO,
PersonnelName =result.PersonnelName,
Status = result.Status
});
}
but this gives me the above error so i found the other way to perform joining on two tables from diffrent database.and that way is as below.
var query = from lc in db1.LotteryCardMaster
where lc.IsActive == 1
select new
{
lc.CashCardID,
lc.CashCardNO,
om.PersonnelName,
lc.Status
};
AB.LottryList = new List<LotteryCardMaster>();
foreach (var result in query)
{
AB.LottryList.Add(new LotteryCardMaster()
{
CashCardID = result.CashCardID,
CashCardNO = result.CashCardNO,
PersonnelName =db2.OwnerMaster.FirstOrDefault(x=>x.OwnerID== result.OwnerID).OwnerName,
Status = result.Status
});
}
I'm a big fan of Linq, and I have been really enjoying the power of expression trees etc. But I have found that whenever I try to get too clever with my queries, I hit some kind of limitation in the framework: while the query can take a very short time to run on the database (as shown by performance analyzer), the results take ages to materialize. When that happens I know I've been too fancy, and I start breaking the query up into smaller, bite sized chunks - so I have a solution for that, though it might not always be the most optimal.
But I'd like to understand:
What is it that pushes the Linq framework over the edge in terms of materializing the query results?
Where can I read about the mechanism of materializing query results?
Is there a certain measurable complexity limit for Linq queries that should be avoided?
What design patterns are known to cause this problem, and what patterns can remedy it?
EDIT: As requested in comments, here's an example of a query that I measured to run on SQL Server in a few seconds, but took almost 2 minutes to materialize. I'm not going to try explaining all the stuff in context; it's here just so you can view the constructs and see an example of what I'm talking about:
Expression<Func<Staff, TeacherInfo>> teacherInfo =
st => new TeacherInfo
{
ID = st.ID,
Name = st.FirstName + " " + st.LastName,
Email = st.Email,
Phone = st.TelMobile,
};
var step1 =
currentReportCards.AsExpandable()
.GroupJoin(db.ScholarReportCards,
current =>
new { current.ScholarID, current.AcademicTerm.AcademicYearID },
past => new { past.ScholarID, past.AcademicTerm.AcademicYearID },
(current, past) => new
{
Current = current,
PastCards =
past.Where(
rc =>
rc.AcademicTerm.StartDate <
current.AcademicTerm.StartDate &&
rc.AcademicTerm.Grade == current.AcademicTerm.Grade &&
rc.AcademicTerm.SchoolID == current.AcademicTerm.SchoolID)
});
// This materialization is what takes a long time:
var subjects = step1.SelectMany(x => from key in x.Current.Subjects
.Select(s => new { s.Subject.SubjectID, s.Subject.SubjectCategoryID })
.Union(x.PastCards.SelectMany(c => c.Subjects)
.Select(
s => new { s.Subject.SubjectID, s.Subject.SubjectCategoryID }))
join cur in x.Current.Subjects on key equals
new { cur.Subject.SubjectID, cur.Subject.SubjectCategoryID } into jcur
from cur in jcur.DefaultIfEmpty()
join past in x.PastCards.SelectMany(p => p.Subjects) on key equals
new { past.Subject.SubjectID, past.Subject.SubjectCategoryID } into past
select new
{
x.Current.ScholarID,
IncludeInContactSection =
// ReSharper disable ConstantNullCoalescingCondition
(bool?)cur.Subject.IncludeInContactSection ?? false,
IncludeGrades = (bool?)cur.Subject.IncludeGrades ?? true,
// ReSharper restore ConstantNullCoalescingCondition
SubjectName =
cur.Subject.Subject.Name ?? past.FirstOrDefault().Subject.Subject.Name,
SubjectCategoryName = cur.Subject.SubjectCategory.Description,
ClassInfo = (from ce in myDb.ClassEnrollments
.Where(
ce =>
ce.Class.SubjectID == cur.Subject.SubjectID
&& ce.ScholarID == x.Current.ScholarID)
.Where(enrollmentExpr)
.OrderByDescending(ce => ce.TerminationDate ?? DateTime.Today)
let teacher = ce.Class.Teacher
let secTeachers = ce.Class.SecondaryTeachers
select new
{
ce.Class.Nickname,
Primary = teacherInfo.Invoke(teacher),
Secondaries = secTeachers.AsQueryable().AsExpandable()
.Select(ti => teacherInfo.Invoke(ti))
})
.FirstOrDefault(),
Comments = cur.Comments
.Select(cc => new
{
Staff = cc.Staff.FirstName + " "
+ cc.Staff.LastName,
Comment = cc.CommentTemplate.Text ??
cc.CommentFreeText
}),
// ReSharper disable ConstantNullCoalescingCondition
DisplayOrder = (byte?)cur.Subject.DisplayOrder ?? (byte)99,
// ReSharper restore ConstantNullCoalescingCondition
cur.Percentile,
cur.Score,
cur.Symbol,
cur.MasteryLevel,
PastScores = past.Select(p => new
{
p.Score,
p.Symbol,
p.MasteryLevel,
p.ScholarReportCard
.AcademicTermID
}),
Assessments = cur.Assessments
.Select(a => new
{
a.ScholarAssessment.AssessmentID,
a.ScholarAssessment.Assessment.Description,
a.ScholarAssessment.Assessment.Type.Nickname,
a.ScholarAssessment.AssessmentDate,
a.ScoreDesc,
a.ScorePerc,
a.MasteryLevel,
a.ScholarAssessment.Assessment.Type.AssessmentFormat,
a.ScholarAssessment.PublishedStatus,
a.ScholarAssessment.FPScore,
a.ScholarAssessment.TotalScore,
a.ScholarAssessment.Assessment.Type.ScoreType,
a.ScholarAssessment.Assessment.Type.OverrideBelowLabel,
a.ScholarAssessment.Assessment.Type.OverrideApproachingLabel,
a.ScholarAssessment.Assessment.Type.OverrideMeetingLabel,
a.ScholarAssessment.Assessment.Type.OverrideExceedingLabel,
})
})
.ToList();
Linq uses deferred execution for some tasks, for example while iterating through an IEnumerable<>, so what you call materialization includes some actual data fetching.
var reportCards = db.ScholarReportCards.Where(cr => ...); // this prepares the query
foreach (var rc in reportCards) {} // this executes your query and calls the DB
I think that if you trace/time queries on your SQL server you may see some queries arriving during the "materialization" step. This problem may even be exacerbated by anti-patterns such as the "Select N+1" problem : for example it looks like you're not including the AcademicTerm objects in your request; if you don't resolving these will result in a select N+1, that is for every ScholarReportCard there will be a call to the DB to lazily resolve the AcademicTerm attached.
If we focus on the Linq to DB aspect, at least try not to :
select n+1: Include the related datatables you will need
select too much data: include only the columns you need in your selection (Include on the table you need)
What's the equivalent LINQ instruction for a Datatable of the following SQL query:
SELECT code_direction, count(TP) AS CN
FROM table1
WHERE cod_time = 'A011'
GROUP BY TP,code_direction;
and how to get the result into a new datatable?
I tried to convert it but I there're some errors. Someone could take a look on this:
var query = from t in table1.AsEnumerable()
group t by new { t.TP, t.code_direction }
into grp
select new
{
grp.Key.code_direction,
CN = grp.Count(t.TP)
};
foreach (var x in query)
{
Console.Write(x.code_direction);
Console.Write(x.CN);
}
As far as your first question goes. The LINQ equivalent of the SQL query is:
var query = from t in table1.AsEnumerable()
where t.cod_time == "A011"
group t by new { t.TP, t.code_direction }
into grp
select new
{
grp.Key.code_direction,
CN = grp.Count()
};
Note that you don't have to pass any argument to grp.Count(). (For the obvious reason that in SQL COUNT(TP) is the same as COUNT(*), i.e. just count the number of rows. The story would be different if you'd use COUNT(DISTINCT TP) or similar.)
As far as the second question goes, if your query just returned an IEnumerable<T> where T is DataRow (i.e. a query like table1.AsEnumerable().Where(r => r.cod_time == "A011")) then you could just the DataTableExtensions.CopyToDataTable extension method. As your query returns an anonymous type however, you will have to follow these instructions found on MSDN.
I Have been using LINQ to work on a JSON object returned from a remote sharepoint web service. I have posted this because most of the answers I found online were slightly different from what I needed.
a json list of daily activities is returned from a remote sharepoint list & is then summarised using LINQ
The simplified version of a custom object definition is shown below( & which is defined in the models area of an MVC application)
public class MyCustomObjectList
{
public string eventdate { get; set; }
public string userid { get; set; }
public string action { get; set; }
}
The JSON object is serialised into a MyCustomObjectList array.
var customobject = serializer.Deserialize<MyCustomObjectList>(jsonobject);
I wanted to work out how many actions of each type happened on a given day. NB eventdate is stored as a string in format yyyy-mm-dd hh:MM:ss. This was to simplify conversions between c#, JSON & Jquery ( where required I create DateTime objects elsewhere in the code using the
eventdate.
Some will argue this is inefficient, but I prefer to split processes into a sequential set of really simple operations, for the sake of easier debugging & to help other people follow my code. Thats why there are 2 Linq queries .
querya strips out the time component from the eventdate This ensures our later grouping happens by day, & not by second. To be doubly sure that there is no caching, I create it in a new field called actionday. I also rename action to activity, because intellisense was getting confused!! The other columns are copied as is.
var querya =
from c in customobject.rows
select new { actionday = c.eventdate.Substring(0, 10), activity = c.action, c.userid,
c.eventdate };
/* queryb produces a grouped count of querya, grouped on actionday & activity, creating new columns actionkey,ActionCount,Dte,action & DetailList ( which is a summary for debugging purposes)
*/
var queryb=
from p in querya group p by new { p.actionday, p.activity} into idGroup
actionkey = idGroup.Key,
ActionCount = idGroup.Count(),
Dte = idGroup.Key.actionday,
action = idGroup.Key.activity,
DetailList = idGroup
};
Here’s a version that sumarises by 3 columns
var queryc = from p in querya
group p by new { p.actionday, p.userid, p.activity} into idGroup
select new
{
actionday = idGroup.Key,
ActionCount = idGroup.Count(),
userid = idGroup.Key.userid,
Dte = idGroup.Key.actionday,
action = idGroup.Key.activity,
DetailList = idGroup
};