LinqProvider that gets data from from two places and merges them - c#

I have a Linq query that looks something like the following
var query3 = from c in Session.CreateLinq<AccountTransaction>()
join a in Session.CreateLinq<Account>() on c.Account equals a
where c.DebitAmount >= 0
select new { a.Name, c.DebitAmount }
;
The Session object interacts with a datasource behind the scenes but it also has an internal cached state which may have changes. When I run a query I would like to query the both the internal cached state AND the datasource and then merge the results together, with the internal cached state taking precendence.
I am using re-linq for the generation of the query against the datasource which is working fine. What I am not sure about is how to also do the query against the internal state using the same Linq query.
There's a call GetAllCached() on Session that I can use instead of Session.CreateLinq if I just wanted to query the internal state. But I'm not sure at which point in my custom provider I can handle handing off to the datasource AND the internal state using GetAllCached().
Any suggestions appreciated from any Linq gurus.

// From Database
var query1 = from c in Session.CreateLinq<AcccountTransaction>()
join a in Session.CreateLinq<Account>()
on c.Account equals a
where c.DebitAmount >= 0
select new { Account = a, AccountTrans = c };
//select new { a.Name, c.DebitAmount };
// From Cache
var query2 = from c in Session.GetAllCached<AcccountTransaction>()
join a in Session.GetAllCached<Account>()
on c.Account equals a
where c.DebitAmount >= 0
select new { Account = a, AccountTrans = c };
//select new { a.Name, c.DebitAmount };
//var query3 = query2.Union(query1.Except(query2));
var query4 = query2.Union(query1);
Modified: 04:51 AM Singapore Time

If I understand correctly, you have a single custom LINQ provider for your datasource, and a (presumably type-safe) way of getting cached results as well.
In this case, I recommend just using LINQ to Objects to access your cached set. You can use AsEnumerable to "step out" of your custom LINQ provider into LINQ to Objects.
The join brings up a problem, though. Since either of these types may exist in the cache, it's not possible to push logic to the DB. For example, is it possible to have an AccountTransaction in the cache without its Account also being in the cache?
If you allow any situation in the cache (e.g., AccountTransaction without associated Account records), then you have to do the join in memory and not in the db:
var allDebitAccountTransactions = Session.GetAllCached<AccountTransaction>()
.Where(x => x.DebitAmount >= 0)
.Union(Session.CreateLinq<AccountTransaction>()
.Where(x => x.DebitAmount >= 0));
var allAccounts = Session.GetAllCached<Account>()
.Union(Session.CreateLinq<Account>());
var query3 = from c in allDebitAccountTransactions
join a in allAccounts where c.Account equals a
select new { a.Name, c.DebitAmount };
However, if you have more control over your cache, and only allow AccountTransaction objects to be present if their associated Account objects are present, then you can push the join operation to the datasource and do another one in memory, merging the results:
var datasourceResults = from c in Session.CreateLinq<AccountTransaction>()
join a in Session.CreateLinq<Account>() on c.Account equals a
where c.DebitAmount >= 0
select new { a.Name, c.DebitAmount, c.Id };
var cacheResults = from c in Session.GetAllCached<AccountTransaction>()
join a in Session.GetAllCached<Account>() on c.Account equals a
where c.DebitAmount >= 0
select new { a.Name, c.DebitAmount, c.Id };
var query3 = cacheResults.Union(datasourceResults)
.Select(x => new { x.Name, x.DebitAmount });
I think. I am not an expert in LINQ, so I'm curious to see other responses.

Related

MVC - Joining multiple table using LINQ / lambda - Using EF

I am implementing a controller and I need to get all staff members which have a certain RiskTypeID, which will be selected by the user when they click on Navigation Item.
Here is how I would create the joins in SQL
SQL
Select
RTHG.RiskTypeID,
SM.FullName
From RiskTypeHasGroup RTHG
Inner join RiskGroup RG On RTHG.RiskGroupID = RG.ID
Inner join RiskGroupHasGroupMembers RGHGM ON RG.ID = RGHGM.RiskGroupID
Inner Join GroupMember GM ON RGHGM.GroupMemberID = GM.ID
Inner Join GroupMemberHasStaffMember GMHSM ON GM.ID = GMHSM.GroupMemberID
Inner Join StaffMember SM ON GMHSM.StaffMemberID = SM.ID
Where RTHG.RiskTypeID = 1
I’ve pulled back data before using Linq and lambda but only using simple expressions, I now need to be able to make a call which will bring back the same data as the sql outlined above, I’ve searched online but can’t find anything similar to my requirement.
Here is my Controller, I placed comments inside as guidance
Controller
public ActionResult ViewRiskTypes(int SelectedRiskTypeID)
{
var RiskTypes = _DBContext.RiskTypes.ToList(); // Get all of the current items held in RiskTypes tables, store them as a List in Var RiskTypes
var ViewModel = new List<RiskTypeWithDetails>(); // Create colletion which holds instances of RiskTypeWithDetails and pass them to the ViewModel
var Details = new RiskTypeWithDetails(); // Create a new instance of RiskType with details and store the instance in var Details
foreach (var RiskType in RiskTypes) // Loop through each Item held in var RiskTypes
{
Details.RiskTypes.Add(new RiskTypesItem { ID = RiskType.ID, Description = RiskType.Description }); // assign each items ID & Description to the same feilds in a new
// instance of RiskTypeItems (which is a property of RiskTypeWithDetails)
}
foreach (var RiskType in RiskTypes) // Loop through each item in RiskTypes
{
if (RiskType.ID == SelectedRiskTypeID) // Check Item ID matches SelectedRiskTypeID value
{
//var Details = new RiskTypeWithDetails();
Details.RiskTypeDescription = RiskType.Description; //assign the Risk type Descripton to RiskTypeWithDetails RiskTypeDescription Property
Details.RiskDetails = _DBContext
.RiskTypeHasGroups
//.GroupMemberTypeHasGroupMembers
.Where(r => r.RiskTypeID == SelectedRiskTypeID) // Where RiskTypeId matches Selected ID bring back following data from Db
.Select(r => new RiskDetails
{
RiskGroupDescription = r.RiskGroup.Description,
GroupMembers = r.RiskGroup.RiskGroupHasGroupMembers
.Select(v => v.GroupMember).ToList(),
//StaffMembers = r.RiskGroup.RiskTypeHasGroups
// .Join(r.RiskGroup.RiskTypeHasGroups,
// a => a.RiskGroupID , b => b.RiskGroup.ID,
// (a, b) => new {a, b})
// .Join(r.RiskGroup.RiskGroupHasGroupMembers,
// c => c.) // Dosent join as I would expect... no idea what to do here
}).ToList();
ViewModel.Add(Details); //Add all data retrieved to the ViewModel (This creates one item in the collection)
}
}
return View(ViewModel);
}
As you will see I want to get all Staff Members with a match for the selected RiskTypeID. I need some assistance in converting the above SQL to work within my controller as a lambda expression
Thanks in advance
You were on the right track with your commented out code! For starters, LINQ has two different sytaxes: query and method chain. You were using the method chain syntax and it can get really unmaintainable really quickly.
For an instance like this, query syntax is where it's at.
Here's the result:
from rhtg in _dbContext.RiskTypeHasGroup
where rhtg.RiskTypeID == 1
join rg in _dbContext.RiskGroup
on rhtg.RiskGroupID equals rg.ID
join rghgm in _dbContext.RiskGroupHasGroupMembers
on rg.ID equals rhtg.ID
join gm in _dbContext.GroupMember
on rg.ID equals gm.ID
join gmhsm in _dbContext.GroupMemberHasStaffMember
on gm.ID equals gmhsm.GroupMemberID
join sm in _dbContext.StaffMember
on gmhsm.StaffMemberID equals sm.ID
select new
{
rhtg.RiskTypeId,
sm.FullName
};
Do note, that I used .Net conventions for the different variables.
Here's some documentation on the query syntax:
https://msdn.microsoft.com/en-us/library/gg509017.aspx
You can write the exact same query in linq as follows:
var query = (from RTHG in _DBContext.RiskTypeHasGroup RTHG
join RG in _DBContext.RiskGroup on RTHG.RiskGroupID equals RG.ID
join RGHGM in _DBContext.RiskGroupHasGroupMembers on RG.ID equals RGHGM.RiskGroupID
join GM in _DBContext.GroupMember on RGHGM.GroupMemberID = GM.ID
join GMHSM in _DBContext.GroupMemberHasStaffMember on GM.ID equals GMHSM.GroupMemberID
join SM in _DBContext.StaffMember on GMHSM.StaffMemberID equals SM.ID
where RTHG.RiskTypeID == 1
select new {RTHG.RiskTypeID,SM.FullName});

AsEnumerable and Query Syntax

I have a rather complex Linq expression written using query syntax. Simplified, it looks like:
var query =
from a in ctx.A
from b in ctx.B
where a.Bid == b.Id
select new MyType()
{
Foo = a.Foo,
Bar = b.Bar
}
I have to modify the query to set a property on the new instance of MyType in a manner that cannot be evaluated by the query provider. The result set fairly small, so it is reasonable to use .AsEnumerable(). I found an example on MSDN that shows how to do this in a simple case
IEnumerable<DataRow> query =
from product in products.AsEnumerable()
select product;
How do I use AsEnumerable() in my more complex case? That is, how do I achieve something like
var query =
from a in ctx.A
from b in ctx.B
where a.Bid == b.Id
AsEnumerableGoesHereSomehow
select new MyType()
{
Foo = a.SomeClientSideCalculation(),
Bar = b.Bar
}
I don't want to perform the join and filtering client side.
Feel free to reduce {a=a,b=b} to just the properties you actually need, but you can use this as a start:
var query =(
from a in ctx.A
from b in ctx.B
where a.Bid == b.Id
select new
{
a = a,
b = b
})
.AsEnumerable()
.Select(x=>new MyType {
Foo=Something(x.a),
Bar=x.b
});

Join tables in NHibernate without mapping

I have the following two objects:
User
class User {
public int role;
}
Role
class Role {
public int id;
public string name;
}
be note that role property inside User is int and not Role, that's our limitations.
I want to join between all the users and each of his role. In the mapping objects there is no reference as you can understand, just a simple type (int).
How do I do that join statement?
It's called a theta join:
var a = (from u in session.Query<User>()
from r in session.Query<Role>()
where u.role == r.id
select new { u.Username, Role = r.name }).ToList();
Assuming you have a Username property on the User class.
Yes, this "theta join" (as I just learned this term) is very handy and let's us not worry about putting in pointless mapping relationships.
WARNING HOWEVER IN USING THIS!!! This tripped me up a lot.
Adding to the above example...
var list = new List<int>( { 2, 3 } ); // pretend in-memory data from something.
var a =
(from u in session.Query<User>()
from x in list
from r in session.Query<Role>()
where u.role == r.id
where r.id == x.id // pretend list wants to limit to only certain roles.
select new { u.Username, Role = r.name }).ToList();
THIS WILL BOMB with some NotSupported exception.
The trick is that anything coming from NHibernate Session must come LAST. So this alteration WILL work:
var a =
(from x in list
from u in session.Query<User>()
from r in session.Query<Role>()
where u.role == r.id
where r.id == x.id // pretend list wants to limit to only certain roles.
select new { u.Username, Role = r.name }).ToList();
And and BTW, you can use join as well, however you have to make sure if you have any nullable data types, that you use the .Value if you are joining to something not-nullable.
var a =
(from x in list
from u in session.Query<User>()
join r in session.Query<Role>() on u.role equals r.id
where r.id == x.id // pretend list wants to limit to only certain roles.
select new { u.Username, Role = r.name }).ToList();
And while we're at it, let's say you have a method that has some dynamic condition. In this example the 'list' which could be a list of roles to filter by, but don't filter at all if the list is not there. Well, if you do the .ToList() then you are causing this query to execute immediately. But instead you can add a condition and then execute it later:
var a =
from u in session.Query<User>()
join r in session.Query<Role>() on u.role equals r.id
where r.id == x.id // pretend list wants to limit to only certain roles.
select new { u.Username, Role = r.name, RoleID = r.id }; // Adding the Role ID into this output.
if (list != null) // assume if the list given is null, that means no filter.
{
a = a.Where(x => list.Contains(x.RoleID));
// WARNING. Unfortunately using the "theta" format here will not work. Not sure why.
}
var b = a.ToList(); // actually execute it.
var c = a.Select(x => new { x.Username, x.Role }).ToList() // if you insist on removing that extra RoleID in the output.
One last thing.. Sometimes some simple logic will fail when executed in the select new { .. } part. I don't have an explanation. In our case the logic was just converting a DB value of a uint to an Enumerator of a model. But to get around that, I just avoided doing that conversion while reading the data but saved the value. Then in a later step, after the data was loaded, I just did the conversion in another LINQ statement.
DISCLAIMER: While I wrote many of these things all the past several weeks, I did not put this code into my compiler to verify 100%.

More efficient way of loading children of entity objects in linq to entity query

I have a rather complex linq to entity query that I'm performing, in the end, I have a result set. I loop through that result set, build business objects and return that list of business objects. it's pretty quick, the problem is that 2 of the child properties are complex objects with their own child objects. for every business object in my loop, I then have to make 2 DB calls to fill its child object. Those 2 calls slow down the overall process, is there a better way to do this? noob to EF here. (EF 4,SQL Server 2008,c#)
Get a result set:
var newresult = from r in result // result is another complex query
join subedit in
(from sa in context.Security_Access
join g in context.Security_UserGroup on sa.EntityID equals g.GroupID
where (sa.PrivledgeID == xx) && g.UserID == userId
select new { user = g.UserID, linkid = sa.LinkID }).Distinct() on new { aid = r.AssetId } equals new { aid = subedit.linkid } into theSubEdit
from subEditAccess in theSubEdit.DefaultIfEmpty()
join subdownload in
(from sa in context.Security_Access
join g in context.Security_UserGroup on sa.EntityID equals g.GroupID
where (sa.PrivledgeID == xx|| sa.PrivledgeID == yy) && g.UserID == userId
select new { user = g.UserID, linkid = sa.LinkID }).Distinct() on new { aid = r.AssetId } equals new { aid = subdownload.linkid } into theSubDownload
from subDownloadAccess in theSubDownload.DefaultIfEmpty()
join subView in
(from sa in context.Security_Access
join g in context.Security_UserGroup on sa.EntityID equals g.GroupID
where (sa.PrivledgeID == xx|| sa.PrivledgeID == yy|| sa.PrivledgeID == 101) && g.UserID == userId
select new { user = g.UserID, linkid = sa.LinkID }).Distinct() on new { aid = r.AssetId } equals new { aid = subView.linkid } into theSubView
from subViewAccess in theSubView.DefaultIfEmpty()
select new { r, EditAccess = (int?)subEditAccess.user, DownloadAccess = (int?)subDownloadAccess.user, ViewAccess = (int?)subViewAccess.user };
I then loop through that result set:
foreach (var asset in newresult)
{
// and build a new business object, set its properties
BoAsset boAsset = new BoAsset();
boAsset.HasEditRights = (asset.EditAccess > 0);
boAsset.HasDownloadRights = (asset.DownloadAccess > 0);
boAsset.HasViewRights = (asset.ViewAccess > 0);
boAsset.Description = asset.r.Description;
boAsset.DetailedDescription = asset.r.DetailedDescription;
boAsset.Keywords = asset.r.Keywords;
boAsset.Notes = asset.r.Notes;
boAsset.Photographer = asset.r.Photographer;
boAsset.PhotographerEmail = asset.r.PhotographerEmail;
boAsset.Notes = asset.r.Notes;
boAsset.Author = asset.r.Author;
// these 2 properties i've commented out are
// complex objects/entities, setting them the way I am
// requires me to call 2 separate methods which make 2 DB trips
// per business object.
//boAsset.Domains = GetAssetDomains(asset.r.AssetId);
//boAsset.DomainEntries = GetAssetCustomDomains(asset.r.AssetId);
myListofObjects.Add(boAsset);
}
return myListofObjects;
Is there a better way?
Just add this .Include("Domains").Include("DomainEntries") to your Linq in in context.Security_Access That should get rows from those tables all in one go.
So your "inner" queries would look like:
from sa in context.Security_Access.Include("Domains").Include("DomainEntries")
join g in context.Security_UserGroup on sa.EntityID equals g.GroupID
where (sa.PrivledgeID == xx) && g.UserID == userId
select new { ...
Here is the documentation from MS: http://msdn.microsoft.com/en-us/library/bb738708.aspx
If you want to improve your performance use compile queries !
You can check the example here.
static readonly Func<AdventureWorksEntities, Decimal,
IQueryable<SalesOrderHeader>> s_compiledQuery2 =
CompiledQuery.Compile<AdventureWorksEntities, Decimal, IQueryable<SalesOrderHeader>>((ctx, total) =>
from order in ctx.SalesOrderHeaders.Include("Orders") where order.TotalDue >= total select order);
MSDN
AND
You can Introduce Include suppose to select all the employees along with their departments . If you have a navigational property, you won't need a join at all. You can use Include like this:
List<Employee> employeesWithDepartments = CreateObjectSet<Employee>().
Include(e => e.Department).
ToList();

LINQ using dictionary in where clause

er have the following query in linq...
Whenever I try to run it I get a No comparison operator for type System.Int[] exception.
It's got something to do with the dictionary I am sure, but I don't understand why this isn't valid and was wondering if someone could explain?
// As requested... not sure it will help though.
var per = (
from p in OtherContext.tblPeriod
where activeContractList.Select(c => c.DomainSetExtensionCode).Contains(p.DomainSetExtensionCode)
select p).ToArray();
var com = (
from c in MyContext.tblService
join sce in MyContext.tblServiceExtension
on c.ServiceExtensionCode equals sce.ServiceExtensionCode
join sc in MyContext.tblServiceContract
on sce.ServiceContractCode equals sc.ContractCode
group sc by c.Period into comG
select new
{
PeriodNumber = comG.Key,
Group = comG,
}).ToArray();
var code =
(from c in com
join p in per on c.PeriodNumber equals p.PeriodNumber
select new
{
p.Code,
c.Group
}).ToArray();
var payDictionary = new Dictionary<int, int[]>();
// This is another linq query that returns an anonymous type with
// two properties, and int and an array.
code.ForEach(c => payDictionary.Add(c.Code, c.Group.Select(g => g.Code).ToArray()));
// MyContext is a LINQ to SQL DataContext
var stuff = (
from
p in MyContext.tblPaySomething
join cae in MyContext.tblSomethingElse
on p.PaymentCode equals cae.PaymentCode
join ca in MyContext.tblAnotherThing
on cae.SomeCode equals ca.SomeCode
where
// ca.ContractCode.Value in an int?, that should always have a value.
payDictionary[p.Code].Contains(ca.ContractCode.Value)
select new
{
p.Code,
p.ExtensionCode,
p.IsFlagged,
p.Narrative,
p.PayCode,
ca.BookCode,
cae.Status
}).ToList();
You won't be able to do this with a dictionary. The alternative is to join the three linq queries into one. You can do this with minimal impact to your code by not materializing the queries with ToArray. This will leave com and code as IQueryable<T> and allow for you compose other queries with them.
You will also need to use a group rather than constructing a dictionary. Something like this should work:
var per = (
from p in OtherContext.tblPeriod
where activeContractList.Select(c => c.DomainSetExtensionCode).Contains(p.DomainSetExtensionCode)
select p.PeriodNumber).ToArray(); // Leave this ToArray because it's materialized from OtherContext
var com =
from c in MyContext.tblService
join sce in MyContext.tblServiceExtension on c.ServiceExtensionCode equals sce.ServiceExtensionCode
join sc in MyContext.tblServiceContract on sce.ServiceContractCode equals sc.ContractCode
group sc by c.Period into comG
select new
{
PeriodNumber = comG.Key,
Group = comG,
}; // no ToArray
var code =
from c in com
where per.Contains(c.PeriodNumber) // have to change this line because per comes from OtherContext
select new
{
Code = c.PeriodNumber,
c.Group
}; // no ToArray
var results =
(from p in MyContext.tblPaySomething
join cae in MyContext.tblSomethingElse on p.PaymentCode equals cae.PaymentCode
join ca in MyContext.tblAnothThing on cae.SomeCode equals ca.SomeCode
join cg in MyContext.Codes.GroupBy(c => c.Code, c => c.Code) on cg.Key equals p.Code
where cg.Contains(ca.ContractCode.Value)
select new
{
p.ContractPeriodCode,
p.DomainSetExtensionCode,
p.IsFlagged,
p.Narrative,
p.PaymentCode,
ca.BookingCode,
cae.Status
})
.ToList();
Side Note: I also suggest using navigation properties where possible instead of joins. It makes it much easier to read and understand how objects are related and create complex queries.

Categories