I need to join 3 collections in aggregation with multiple $lookup
I tried in C# driver it allow me to $lookup User collection but can't perform second $lookup for Setting collection.
Anyone can help?
db.Transactions.aggregate([
{
$lookup:
{
from: "Account",
localField: "AccountId",
foreignField: "_id",
as: "Account"
}
},
{
$lookup:
{
from: "User",
localField: "UserId",
foreignField: "_id",
as: "User"
}
}
])
.match({
})
.project({})
here is the C# code:
var account = _dbClient.GetDatabase(_dbName).GetCollection<Account>("Accounts");
var user = _dbClient.GetDatabase(_dbName).GetCollection<User>("Users");
var transaction = _dbClient.GetDatabase(_dbName).GetCollection<Transaction>("Transactions");
var result = (from t in transaction.AsQueryable()
join a in account.AsQueryable() on t.AccountId equals a.Id
join u in user.AsQueryable() on t.UserId equals u.Id into userList
from acc in userList.DefaultIfEmpty()
where acc.CompanyName.ToLower().Contains(companyName) && c.CreatedDate >= fromDate && c.CreatedDate <= toDate
select new TransactionHistory
{
Id = t.Id,
CompanyName = acc.CompanyName,
UserId = u.UserId
FirstName = u.FirstName
}).ToList();
I got the error $project or $group does not support {document}. using Linq.
I need to join 3 collections in aggregation with multiple $lookup
Given the following classes:
public class Transactions
{
public ObjectId Id { get; set; }
public int UserId { get; set; }
public int AccountId { get; set; }
public int SettingId { get; set; }
}
public class Account
{
public int Id {get; set;}
public int Name {get; set;}
}
public class User
{
public int Id {get; set;}
public int Name {get; set;}
}
public class Setting
{
public int Id {get; set;}
public int Name {get; set;}
}
You can perform multiple $lookup stage as below using MongoDB .NET/C# driver (currently v2.9):
var collection = database.GetCollection<Transactions>("transactions");
var docs = collection.Aggregate()
.Lookup("account", "AccountId", "_id", "asAccounts")
.Lookup("user", "UserId", "_id", "asUsers")
.Lookup("setting", "SettingId", "_id", "asSettings")
.As<BsonDocument>()
.ToList();
foreach (var doc in docs) {
Console.WriteLine(doc.ToJson());
}
You can add a Match, in between/before/after if you would like to filter for specific values. Just keep in mind that the documents after altered after each Lookup stage.
Worth mentioning that if you need to join multiple collections as part of your common operations, you should reconsider the database data model. Please see Schema Design: Summary for more information.
Related
I have many to many relationship between entities user and group, I also have joining table GroupParticipants.
public class User
{
public string Id {get; set;}
public ICollection<GroupParticipant> Group { get; set;}
}
public class Group
{
public int Id { get; set; }
public ICollection<GroupParticipant> Participants { get; set; }
}
public class GroupParticipant
{
public int GroupId { get; set; }
public string ParticipantId { get; set; }
public User Participant { get; set; }
public Group Group { get; set; }
}
I need to select groups which user specified user did not join. I want to do something like:
string userId = 5;
var groupsAvailableToJoin = await _context.Groups
.Where(group => group.Participants.Id != userId);
Thanks!
A query like:
_context.Groups.Where(g =>
!_context.GroupParticipants.Any(gp => gp.UserId == userId && gp.GroupId == g.I'd
);
Should translate to:
SELECT * FROM Groups g
WHERE NOT EXISTS(SELECT null FROM groupParticipants gp WHERE gp.UserId = 5 AND gp.GroupId = g.Id)
Which should be a reasonably performant way of getting you what you're looking for.. I'm sure that the GroupParticipants columns are indexed..
There are various ways to write this - if you find a two step approach easier to understand, it's effectively the same as:
var joined = _context.GroupParticipants.Where(gp => gp.UserId == 5).Select(gp => gp.GroupId).ToList();
var notJoined = _context.Groups.Where(g => !joined.Contains(g.Id));
This one translates as a NOT IN (list,of,groups,they,are,in) for a similar effect
I have referenced numerous questions on this site related to calculated fields and ViewModels, but I can't seem to extrapolate from examples given. I hope that laying out a specific scenario would allow someone to pin point what I can't see. I am new to WebApp design in general. Please take that into consideration. Also, if I've left off any relevant information, please let me know and I will update the question.
Here is the scenario:
I have a complex query that is spanning multiple tables to return data used in calculations. Specifically, I store units for a recipe converted to a base unit and then convert the quantity to the units specified by the user.
I am using AutoMapper to map from entities to ViewModels and vice versa, but I am not sure how to handle the calculated values. Especially with the nested ViewModel Collection thrown into the mix.
Option 1
Do I return an autonomous set of data? Like the following... and then somehow use AutoMapper to do the mapping? Perhaps I would need to do the mapping manually, which I haven't found a solid example which includes nested ViewModels. At this point, I'm not even sure if the following code handles the nested collection correctly for the autonomous data.
var userId = User.Identity.GetUserId();
var recipes = from u in db.Users.Where(u => u.Id == userId)
from c in db.Categories
from r in db.Recipes
join ur in db.UserRecipes.Where(u => u.UserId == userId) on r.Id equals ur.RecipeId
join mus in db.MeasUnitSystems on ur.RecipeYieldUnitSysId equals mus.Id
join muc in db.MeasUnitConvs on mus.Id equals muc.UnitSysId
join mu in db.MeasUnits on mus.UnitId equals mu.Id
join msy in db.MeasUnitSymbols on mu.Id equals msy.UnitId
select new
{
Id = c.Id,
ParentId = c.ParentId,
Name = c.Name,
Descr = c.Descr,
Category1 = c.Category1,
Category2 = c.Category2,
Recipes = new
{
Id = r.Id,
Title = r.Title,
Descr = r.Descr,
Yield = String.Format("{0} {1}", ((r.Yield * muc.UnitBaseConvDiv / muc.UnitBaseConvMult) - muc.UnitBaseConvOffset), msy.Symbol)
}
};
Option 2
Another option that crossed my mind was to return the entities and use AutoMapper as I normally would. Then iterate through the collections and perform the calculations there. I feel like I could make this work, but it seems inefficient to me because it would result in many queries back to the database.
Option 3
???? I can't think of any other method to do this. But, please, if you have suggestions, I am more than willing to hear them.
Relevant Data
Here is the query returning the data I want in SQL Server (more or less).
declare #uid as nvarchar(128) = 'da5435ae-5198-4690-b502-ea3723a9b217'
SELECT c.[Name] as [Category]
,r.Title
,r.Descr
,(r.Yield*rmuc.UnitBaseConvDiv/rmuc.UnitBaseConvMult)-rmuc.UnitBaseConvOffset as [Yield]
,rmsy.Symbol
FROM Category as c
inner join RecipeCat as rc on c.Id = rc.CategoryId
inner join Recipe as r on rc.RecipeId = r.Id
inner join UserRecipe as ur on r.Id = ur.RecipeId and ur.UserId = #uid
inner join MeasUnitSystem as rmus on ur.RecipeYieldUnitSysId = rmus.Id
inner join MeasUnitConv as rmuc on rmus.Id = rmuc.UnitSysId
inner join MeasUnit as rmu on rmus.UnitId = rmu.Id
inner join MeasUnitSymbol as rmsy on rmu.Id = rmsy.UnitId
inner join UserUnitSymbol as ruus on rmsy.UnitId = ruus.UnitId and rmsy.SymIndex = ruus.UnitSymIndex and ruus.UserId = #uid
ViewModels
public class CategoryRecipeIndexViewModel
{
public int Id { get; set; }
public int ParentId { get; set; }
[Display(Name = "Category")]
public string Name { get; set; }
[Display(Name = "Description")]
public string Descr { get; set; }
public ICollection<CategoryRecipeIndexViewModel> Category1 { get; set; }
public CategoryRecipeIndexViewModel Category2 { get; set; }
public ICollection<RecipeIndexViewModel> Recipes { get; set; }
}
public class RecipeIndexViewModel
{
public int Id { get; set; }
[Display(Name = "Recipe")]
public string Title { get; set; }
[Display(Name = "Description")]
public string Descr { get; set; }
[Display(Name = "YieldUnit")]
public string Yield { get; set; }
}
UPDATE 2/10/2018
I found an answer here that does a very good job of explaining exactly what I'm looking at. Particularly under the A Better solution ? section. Mapping queries directly to my ViewModels looks like it would allow me to get my calculated values as well. Problem is, the example given is once again too simplistic.
He gives the following DTO's
public class UserDto
{
public int Id {get;set;}
public string Name {get;set;}
public UserTypeDto UserType { set; get; }
}
public class UserTypeDto
{
public int Id { set; get; }
public string Name { set; get; }
}
And does the following for mapping:
var users = dbContext.Users.Select(s => new UserDto
{
Id = s.Id,
Name = s.Name,
UserType = new UserTypeDto
{
Id = s.UserType.Id,
Name = s.UserType.Name
}
});
Now what if the UserDTO looked like this:
public class UserDto
{
public int Id {get;set;}
public string Name {get;set;}
public ICollection<UserTypeDto> UserTypes { set; get; }
}
How would the mapping be done if the UserTypes were a collection?
Update 2/13/2018
I feel I am making progress, but am currently headed in the wrong direction. I found this and came up with the following (which currently errors because of the method call in the linq query):
*Note: I removed Category2 from the ViewModel as I found it was not needed and only complicated this further.
query inside index controller method
IEnumerable<CategoryRecipeIndexViewModel> recipesVM = db.Categories
.Where(x => x.ParentId == null)
.Select(x => new CategoryRecipeIndexViewModel()
{
Id = x.Id,
ParentId = x.ParentId,
Name = x.Name,
Descr = x.Descr,
Category1 = MapCategoryRecipeIndexViewModelChildren(x.Category1),
Recipes = x.Recipes.Select(y => new RecipeIndexViewModel()
{
Id = y.Id,
Title = y.Title,
Descr = y.Descr
})
});
Recursive Method
private static IEnumerable<CategoryRecipeIndexViewModel> MapCategoryRecipeIndexViewModelChildren(ICollection<Category> categories)
{
return categories
.Select(c => new CategoryRecipeIndexViewModel
{
Id = c.Id,
ParentId = c.ParentId,
Name = c.Name,
Descr = c.Descr,
Category1 = MapCategoryRecipeIndexViewModelChildren(c.Category1),
Recipes = c.Recipes.Select(r => new RecipeIndexViewModel()
{
Id = r.Id,
Title = r.Title,
Descr = r.Descr
})
});
}
At this point, I don't even have the calculations I require, but that doesn't matter until I get this working (small steps). I quickly discovered you can't really call a method inside a Linq Query. Then a thought occurs to me, if I need to force the Linq Query to execute and then perform all the mapping on the in memory data, then I would essentially be doing the same thing as Option 2 (above), but I could perform the calculations within the ViewModel. This is the solution I will pursue and will keep everyone posted.
You have to iterate over UserType Collection and map the value to UserType dto's collection.
Use this code.
var users = dbContext.Users.Select(s => new UserDto
Id = s.Id,
Name = s.FullName,
UserType = s.UserType.Select(t => new UserTypeDto
{
Id = t.Id,
Name = t.Name
}).ToList()
Hope this will help.
I got it working! ...I think. ...Maybe. If anything, I'm querying the data, mapping it to my ViewModels and I have the calculations too. I do have additional questions, but they are a lot more specific. I will layout the solution I followed and where I think it requires work below.
I basically implemented my Option 2 from above, but instead of iterating through the collections, I just performed the calculations within the ViewModels.
Controller Method
public ActionResult Index()
{
var userId = User.Identity.GetUserId();
var recipes = db.Categories.Where(u => u.Users.Any(x => x.Id == userId))
.Include(c => c.Category1)
.Include(r => r.Recipes
.Select(u => u.UserRecipes
.Select(s => s.MeasUnitSystem.MeasUnitConv)))
.Include(r => r.Recipes
.Select(u => u.UserRecipes
.Select(s => s.MeasUnitSystem.MeasUnit.MeasUnitSymbols)));
IEnumerable<CategoryRecipeIndexViewModel> recipesVM = Mapper.Map<IEnumerable<Category>, IEnumerable<CategoryRecipeIndexViewModel>>(recipes.ToList());
return View(recipesVM);
}
View Models
public class CategoryRecipeIndexViewModel
{
public int Id { get; set; }
public int ParentId { get; set; }
[Display(Name = "Category")]
public string Name { get; set; }
[Display(Name = "Description")]
public string Descr { get; set; }
public ICollection<CategoryRecipeIndexViewModel> Category1 { get; set; }
public ICollection<RecipeIndexViewModel> Recipes { get; set; }
}
public class RecipeIndexViewModel
{
public int Id { get; set; }
[Display(Name = "Recipe")]
public string Title { get; set; }
[Display(Name = "Description")]
public string Descr { get; set; }
public double Yield { get; set; }
public ICollection<UserRecipeIndexViewModel> UserRecipes { get; set; }
[Display(Name = "Yield")]
public string UserYieldUnit
{
get
{
return System.String.Format("{0} {1}", ((Yield *
UserRecipes.FirstOrDefault().MeasUnitSystem.MeasUnitConv.UnitBaseConvDiv /
UserRecipes.FirstOrDefault().MeasUnitSystem.MeasUnitConv.UnitBaseConvMult) -
UserRecipes.FirstOrDefault().MeasUnitSystem.MeasUnitConv.UnitBaseConvOffset).ToString("n1"),
UserRecipes.FirstOrDefault().MeasUnitSystem.MeasUnit.MeasUnitSymbols.FirstOrDefault().Symbol);
}
}
}
public class UserRecipeIndexViewModel
{
public MeasUnitSystemIndexViewModel MeasUnitSystem { get; set; }
}
public class MeasUnitSystemIndexViewModel
{
public MeasUnitIndexViewModel MeasUnit { get; set; }
public MeasUnitConvIndexViewModel MeasUnitConv { get; set; }
}
public class MeasUnitIndexViewModel
{
public ICollection<MeasUnitSymbolIndexViewModel> MeasUnitSymbols { get; set; }
}
public class MeasUnitConvIndexViewModel
{
public double UnitBaseConvMult { get; set; }
public double UnitBaseConvDiv { get; set; }
public double UnitBaseConvOffset { get; set; }
}
public class MeasUnitSymbolIndexViewModel
{
public string Symbol { get; set; }
}
This appears to be working, but I know it needs some work.
For instance, the relation shown between the Recipe and UserRecipe shows one to many. In reality, if the UserRecipe were filtered by the current user, the relationship would be one to one. Also, the same goes for the MeasUnit and the MeasUnitSymbol entities. Currently, I'm relying on the FirstOrDefault of those collections to actually perform the calculations.
Also, I have seen numerous posts that state that calculations should not be done in the View Models. Except for some who say it's okay if it is only a requirement of the View.
Last I will say that paying attention to variable names within the ViewModels would have saved me some headaches. And I thought I knew how to utilize Linq Queries, but had issues with the data returned. It was easier to rely on the eager loading provided by Entity Framework to bring back the hierarchical data structure needed, versus the flat table structures I'm used to working with.
I'm still new to a lot of this and wrapping my head around some of the quirks of MVC and Entity Framework leaves me brain dead after a few hours, but I will continue to optimize and adopt better programming methods as I go.
I have the following classes:
public class User
{
public int Id { get; set; }
public string Name { get; set; }
public string Department { get; set; }
public List<Event> Events { get; set; }
}
public class Event
{
public int Id { get; set; }
public int UserId { get; set; }
public string EventText { get; set; }
public string StartTime { get; set; }
public string EndTime { get; set; }
public string Day { get; set; }
public string ColorIdentifier { get; set; }
public int Week { get; set; }
}
I'm trying to get all the users and their events with Dapper like this:
var sql = "SELECT u.Id, e.UserId, e.EventText FROM cpc.PLANNING_TOOL_USERS u LEFT JOIN cpc.PLANNING_TOOL_EVENTS e ON u.Id=e.UserId";
var result = SqlMapper.Query<User, Event, User>(connection, sql, (u, e) =>
{
if (u.Events == null)
u.Events = new List<Event>();
u.Events.Add(e);
return u;
}, splitOn: "Id, UserId");
The Id for the user is returned back, but the list of events is not populated. I have looked at many examples here on Stack Overflow regarding this, but I can't see what I'm doing wrong.
To omit the situation that SQL returns no data I have just mocked two user rows with SQL union.
User with Id=1 and one Event, and User with Id=2 and two Events.
SqlMapper.Query returns flat results that are best for 1 to 1 relation. You have one user to many events relation, so some helper storage needed to maintain that relation as a mapping thru the results. I have used .NET dictionary for that.
My code sample below:
// introducing temporary storage
var usersDictionary = new Dictionary<int, User>();
var sql = #"SELECT 1 Id, 1 UserId, 'EventText1' EventText
union SELECT 2 Id, 2 UserId, 'EventText2' EventText
union SELECT 2 Id, 2 UserId, 'Another EventText2' EventText";
var result = SqlMapper.Query<User, Event, User>(connection, sql, (u, e) =>
{
if (!usersDictionary.ContainsKey(u.Id))
usersDictionary.Add(u.Id, u);
var cachedUser = usersDictionary[u.Id];
if (cachedUser.Events == null)
cachedUser.Events = new List<Event>();
cachedUser.Events.Add(e);
return cachedUser;
}, splitOn: "UserId");
// we are not really interested in `result` here
// we are more interested in the `usersDictionary`
var users = usersDictionary.Values.AsList();
Assert.AreEqual(2, users.Count);
Assert.AreEqual(1, users[0].Id);
CollectionAssert.IsNotEmpty(users[0].Events);
Assert.AreEqual(1, users[0].Events.Count);
Assert.AreEqual("EventText1", users[0].Events[0].EventText);
Assert.AreEqual(2, users[1].Events.Count);
I hope that helped you solving your mapping issue and events being null.
I have a big problem of querying diverse types of inherited subentities in a single query in Entity Framework. My essential aim is providing all of my data model structure in a single JSON string by eager loading. And the tricky point is "the inherited subclasses may contain another inherited subclass". The example seen below will clearly explain the situation.
Assume that I have a simple class structure like this:
public class Teacher
{
public int id { get; set; }
public string fullname{ get; set; }
//navigation properties
public virtual HashSet<Course> courses{ get; set; }
}
public class Course
{
public int id { get; set; }
public string coursename{ get; set; }
//foreign keys
public int TeacherId{ get; set; }
//navigation properties
public virtual Teacher teacher{ get; set; }
public virtual HashSet<Course> prerequisites{ get; set; }
}
Course has some subclasses GradedCourse and UngradedCourse
B1 or B2 may have a list of subentities consists of entities of types B1 or B2.
public class GradedCourse : Course
{
public string gradeType{ get; set; }
}
public class UngradedCourse: Course
{
public string successMetric { get; set; }
}
Now by this structure I want to provide a JSON structure from my WEBApi yielding list of Teacher objects including both GradedCourse and UngradedCourse with their subentities and specific fields. I have a query like this but it does not compile
db.Teachers.Select(t => new
{
t.id,
t.fullName
courses = t.courses.OfType<GradedCourses>()
.Select(g => new
{
id = g.id,
coursename = g.coursename,
prerequisites = g.prerequisites, // this is the list of other subentities
gradeType = g.gradeType
}
).Concat(t.courses.OfType<UngradedCourses>()
.Select(u => new
{
id = u.id,
coursename = u.coursename,
prerequisites = g.prerequisites, // this is the list of other subentities
successMetric= u.successMetric // subclass specific field
}
)
)
}
)
The problem is concating two different types of objects (they have different fields which is not possible for SQL UNION)
How can I handle this? Any help will open my mind. Thanks in advance for the professionals :)
It does not compile because the element type of 2 sets is not the same. So you just need to make them the same before being able to do anything:
db.Teachers.Select(t => new
{
t.id,
t.fullName
courses = t.courses.OfType<GradedCourses>()
.Select(g => new
{
id = g.id,
coursename = g.coursename,
prerequisites = g.prerequisites, // this is the list of other subentities
isGradedCourse = true,
gradeTypeOrMetric = g.gradeType
}).Concat(t.courses.OfType<UngradedCourses>()
.Select(u => new
{
id = u.id,
coursename = u.coursename,
prerequisites = g.prerequisites, // this is the list of other subentities
isGradedCourse = false,
gradeTypeOrMetric= u.successMetric // subclass specific field
}))
//finally select what of your choice
.Select(e => new {
id = e.id,
coursename = e.coursename,
prerequisites = e.prerequisites,
gradeType = e.isGradedCourse ? e.gradeTypeOrMetric : "",
successMetric = e.isGradedCourse ? "" : e.gradeTypeOrMetric
})
});
You still benefit the query being executed on server side without having to pull all teachers to local (and then being able to cast the entities - which is not supported in LinqToEntity query).
I have 4 entities which I've defined Navigation properties on like the following:
internal class ShipSet
{
[Key]
[Column("SHIPSET_ID")]
public decimal ID { get; set; }
public virtual ICollection<InstallLocation> InstallLocations { get; set; }
}
internal class InstallLocation
{
[Key]
[Column("INSTLOC_ID")]
public decimal ID { get; set; }
[Column("SHIPSET_ID")]
public decimal ShipSetID { get; set; }
public virtual ICollection<ShipSetPart> ShipSetParts { get; set; }
public virtual ShipSet ShipSet { get; set; }
}
internal class ShipSetPart
{
[Key]
[Column("PARTS_ID")]
public decimal ID { get; set; }
[Column("INSTLOC_ID")]
public decimal InstallLocationID { get; set; }
public virtual CmQueueItem CmQueueItem { get; set; }
public virtual InstallLocation InstallLocation { get; set; }
}
internal class CmQueueItem
{
[Key]
[Column("INVENTORY_ITEM_ID")]
public decimal ID { get; set; }
public virtual ICollection<ShipSetPart> ShipSetParts { get; set; }
}
I have the following fluent config:
modelBuilder.Entity<CmQueueItem>().HasMany(p => p.ShipSetParts).
WithRequired(s=>s.CmQueueItem).Map(m=>m.MapKey("INVENTORY_ITEM_ID"));
modelBuilder.Entity<ShipSetPart>().HasRequired(p => p.InstallLocation);
modelBuilder.Entity<InstallLocation>().HasRequired(p => p.ShipSet);
modelBuilder.Entity<ShipSet>().HasRequired(p => p.Program);
modelBuilder.Entity<CmQueueItem>().Property(p => p.LastUpdateDate).IsConcurrencyToken();
So in a nutshell, I have
ShipSet -> InstallLocation (1 to many)
InstallLocation -> ShipSetPart (1 to many)
CmQueueItem -> ShipSetPart (1 to many via INVENTORY_ITEM_ID)
I am trying to figure out how to write a LINQ query where I can create an anonymous object which includes the count of ShipSets for each CmQueueItem.
var queueItems = from c in dbContext.CmQueueItems
select new
{
InventoryItemID = c.ID,
ShipSets = 0 //[magical LINQ goes here]
};
It should generate a SQL statement similar to the following:
select d.inventory_item_id, count(a.shipset_id) as ShipSets
from shipsets a,
instlocs b,
ss_parts c,
cm_queue d
where a.shipset_id = b.shipset_id
and b.instloc_id = c.instloc_id
and c.inventory_item_id = d.inventory_item_id
group by d.inventory_item_id;
I'm new to LINQ and am having a hard time understanding how to perform aggregates and groupings like this. Any ideas?
The answer as provided below is to use the "let" keyword in LINQ:
var query = from c in dbContext.CmQueueItems
let shipSets = (from s in c.ShipSetParts
select s.InstallLocation.ShipSet)
select new
{
InventoryItemId = c.ID,
ShipSets = shipSets.Count(),
};
I haven't got an EF model to test this against, but give this a try...
UPDATE: Here's how to perform the join, see if that works. Something isn't right though because we shouldn't have to perform the join manually, that's what your property mapping is for.
var queueItems = from c in dbContext.CmQueueItems
join s in dbContext.ShipSetParts
on c.ID equals s.CmQueueItem.ID
group s by s.CmQueueItem.ID into grouped
select new
{
InventoryItemID = grouped.Key,
//this may need a distinct before the Count
ShipSets = grouped.Select(g => g.InstallLocation.ShipSetID).Count()
};
Here's an alternate approach that's much cleaner, but I'm unsure if it will work in EF. Give it a try and see what you think.
var queueItems = from c in dbContext.CmQueueItems
let shipSets = (from s in c.ShipSetParts
select s.InstallLocation.ShipSet)
select new
{
InventoryItemID = c.ID,
ShipSets = shipSets.Count()
};