I'm working on building a repository system on top of a system that is a bit harder to work on than usual (ref. a previous question by me).
Anyway.
My data model is fairly simple at this point: I have several countries, and each country has 0 or more airports. Here's my base Repository:
public abstract class Repository<T> : IRepository<T> where T : Entity, new()
{
protected SimpleSQLManager SQLManager = DatabaseManager.Instance.SQLManager;
public virtual IQueryable<T> GetAll()
{
IQueryable<T> all = SQLManager.Table<T>().AsQueryable();
return all;
}
public virtual IQueryable<T> GetAll(Expression<Func<T, bool>> predicate)
{
IQueryable<T> all = SQLManager.Table<T>().Where(predicate).AsQueryable();
return all;
}
public T GetById(string tableName, int id)
{
return SQLManager.Query<T>( "SELECT * FROM " + tableName + " WHERE Id = ?", id )[0];
}
}
Please ignore the ugly GetById() implementation; I'm running this on Unity3D's (Mono's) .NET libraries, and there's seemingly a bug in there which makes it impossible at the moment to do it properly. Either way, that's not the problem. :)
Now, a normal EntityRepository looks like this (CountryRepository in this case):
public class CountryRepository : Repository<Country>
{
public override IQueryable<Country> GetAll()
{
return base.GetAll().OrderBy( c => c.Name );
}
public Country GetById(int id)
{
return base.GetById( "Country", id );
}
}
The Country entity looks like this:
public class Country : Entity
{
public IQueryable<Airport> Airports()
{
return RepositoryFactory.AirportRepository.GetByCountry( this );
}
}
Then, in my application I can do something like this:
foreach ( Country c in RepositoryFactory.CountryRepository.GetAll() )
{
foreach ( Airport a in c.Airports() )
{
// ...
}
}
...and this works just fine; I'm happy with how everything is abstracted away etc. etc. :)
The problem is that the above code creates one database SELECT per country, which is highly ineffective. This is where I'm not sure where to go forward. I know how to do this with plain old SQL, but I want to go the Linq (or otherwise "non-SQL") way.
Can someone point me in the right/correct direction?
Thanks!
I didn't see anything in SimpleSQL's documentation that looked like it would make sql lite generate a join--something like entity framework's Include method.
That said, you could just bring in all airports and countries into memory with 2 queries and hook them to each other manually, like so:
var airports = RepositoryFactory.AirportRepository.GetAll().ToList();
var countries = RepositoryFactory.CountryRepository.GetAll().ToList();
countries.ForEach(c => c.Airports = airports.Where(a => a.CountryId == c.Id));
Note that you'll need to add a property to your country class:
public IEnumerable<Airport> Airports {get;set;}
I don't like it, but that might be the only way given your environment. You could further abstract the join/mapping logic with generics, but that's the basic idea.
Related
I'm getting the following content when I invoke my API. It kind of breaks up in the middle when the tenant entity that member is linked to, will start listing its member entities.
{
"id":"00000000-7357-000b-0001-000000000000",
"tenantId":"00000000-7357-000a-0001-000000000000",
"userName":"user1",
"tenant":{
"id":"00000000-7357-000a-0001-000000000000",
"name":"First Fake Org",
"members":[
I configured the lazy loading like this.
services.AddDbContext<Context>(config => config
.UseLazyLoadingProxies()
.UseSqlServer(Configuration.GetConnectionString("Register")));
How should I change the code so that the lazily loaded entities don't get served? I was hoping that it would simply return an empty list to the client. Should I use a DTO for that purpose and not return from the DB like this? There's talk about not using lazy loading for APIs at all here.
[HttpGet("test1/{username}"), AllowAnonymous]
public IActionResult GetStuff(string userName)
{
Member output;
output = Context.Members
.Include(e => e.Tenant)
.Single(e => e.UserName == userName);
return Ok(output);
}
I'm not sure what to google for and all the hits I got were pointing to the UseLazyLoadingProxies() invokation.
This will probably be somewhat long winded: But here goes.
It sounds like you have Entities which look something like:
public partial class Member
{
public virtual long Id { get; set; }
public virtual List<Tenant> Tenants { get; set; } //tables have fk relationship
}
public partial class Tenant
{
public virtual long Id { get; set; }
public virtual List<Member> Members{ get; set; } //tables have another fk relationship?
}
And then for this method:
[HttpGet("test1/{username}"), AllowAnonymous]
public IActionResult GetStuff(string userName)
{
Member output;
output = Context.Members
.Include(e => e.Tenant)
.Single(e => e.UserName == userName);
return Ok(output);
}
I see a few issues, but I'll try to keep it short:
I wouldn't have the controller do this directly. But it should work.
What I think you over looked is exactly what the .Include statement does. When the object is instantiated, it will get all of those related entities. Includes essentially converts your where statement to a left join, where the foreign keys match (EF calls these navigation properties).
If you don't want the Tenant property, then you can omit the .Include statement. Unless this is meant to be more generic (In which case, an even stronger reason to use a different pattern and auto mapper).
Hopefully your database doesn't truly have a FK relationship both ways, if it does, fix that ASAP.
The next issue is that you might not want a list of child properties, but it is in the model so they will be "there". Although your List Tenants might be null. And while this might be fine to you, right now. As a general rule when I see an API returning a property, I expect something to be either not there (This Member doesn't have tenants) or something is wrong, like perhaps there is a second parameter I missed. This probably isn't a problem 93.284% of the time, but it is something to be mindful of.
This starts to get into why an AutoMapper is great. Your Database Models, Business Models and views are likely different. And as much as you shouldn't return the database models directly. Taking control of how the data is represented for each part of the application is a great idea.
You could reduce the code easily, and remove the navitation properties:
[HttpGet("test1/{username}"), AllowAnonymous]
public IActionResult GetStuff(string userName)
{
return Ok(Context.Members
.Include(e => e.Tenant)
.Single(e => e.UserName == userName));
}
But again, a business layer would be better:
[HttpGet("test1/{username}"), AllowAnonymous]
public IActionResult GetStuff(string userName)
{
return Ok(MemberRepository.GetMember(userName));
}
The main point I'd stress though, is creating a view model.
For example, Let's say a user Detail:
public class MemberDetail
{
public string UserName {get; set;}
public long UserId { get; set; }
public string FullName { get; set; }
}
This way the view always receives exactly what you want to see, and not the extra data. Add this with the fact that you can know exactly how every use of Member to MemberDetail will map.
I use LINQ-to-SQL to extract and save data from/to SQL Server.
Suppose I have the proverbial Products table, which has a field named Description. The field is free-text and as such it may contain newlines. To avoid Windows/Unix newline issues, I may decide to replace "\r\n" to "\n". However, I want to perform this substitution as early as possible, ideally right as the data is received from SQL Server. This way, myDataContext.Products would return Product objects whose description only contains "\n".
How can I do that?
EDIT
I know I can do this by calling Select(). However, I'd have to call the Select() every time I use the Products table.
Let me explain by showing some code. I have a DataManager class that wraps the reading/writing logic. It has a few methods like these:
public Product GetProduct(int i_id)
{
return m_database.Products.Where(p => p.Id == i_id).FirstOrDefault();
}
public Product GetProductByName(string i_name)
{
return m_database.Products.Where(p => p.Name == i_name).FirstOrDefault();
}
Here, m_database is the data context, and Products is a System.Data.Linq.Table. Surely, I could call Select() on each of these lines, but it would be against DRY (Don't Repeat Yourself), and would actually be WET (Write Everything Twice). That's why I'm looking for a way to include the transformation "inside" Products, so that just calling m_database.Products returns the transformed data.
Something like this maybe:
products.Select(p => new Product
{
Id = p.Id,
Description = p.Description.Replace("\r\n", "\n")
});
Or possibly a custom getter...
public class Product
{
private string _description;
public int Id { get; set; }
public string Description
{
get
{
return _description.Replace("\r\n", "\n");
}
set
{
_description = value;
}
}
}
I would probably extend some of the core Linq features you use.
Ex:
public static class ProductQueryExtensions
{
public static List<Product> CleanSelect(this IQueryable<Product> q)
{
return q.Select(p => new Product
{
Id = p.Id,
Description = p.Description.Replace("\r\n", "\n")
}).ToList();
}
public static Product CleanFirstOrDefault(this IQueryable<Product> q)
{
return q.CleanSelect().FirstOrDefault();
}
}
Then your example code would become:
return m_database.Products.Where(p => p.Id == i_id).CleanFirstOrDefault();
I am building a relatively simple webapp, but have run into a bit of a problem. After having searched low and high, I can't seem to find anyone with similar issues.
So the situation:
I have an entity:
public class Entity
{
[Key]
public int EntityId { get; set;}
public string EntityName { get; set; }
public virtual ICollection<OtherEntity> OtherEntities { get; set; }
}
Reason for the virtual ICollection<OtherEntity> is a many to many relationship between the two.
My DbContext:
public class WebAppDB : IdentityDbContext<ApplicationUser>
{
public DbSet<Entity> Entities{ get; set; }
public WebAppDB () : base("DefaultConnection")
{
}
public static WebAppDB Create()
{
return new WebAppDB();
}
}
}
The default connection goes to an Amazon RDS SQL Server Express.
Around this I have created a repository:
public interface IEntityRepository
{
IQueryable<Entity> AlEntitiss { get; }
IQueryable<Entity> AllEntitissIncluding(params Expression<Func<Entity, object>>[] includeProperties);
Entity FindEntity(int id);
void InsertOrUpdateEntity(Entity entity);
void DeleteEntity(int id);
void Save();
}
public class EntityRepository : IEntityRepository
{
// handle to the database through the O.R.M. system.
private WebAppDB context = new WebAppDB();
public IQueryable<Entity> AllCEntitys
{
get { return context.Entitiss; }
}
public IQueryable<Entity> AllClassesIncluding(params Expression<Func<Entity, object>>[] includeProperties)
{
IQueryable<Entity> query = context.Entitiss;
foreach (var includeProperty in includeProperties)
{
query = query.Include(includeProperty);
}
return query;
}
public Entity FindEntity(int id)
{
return context.Entitiss.Find(id);
}
public void InsertOrUpdateEntity(Entity Entity)
{
if (Entity.EntityId == 0)
{
context.Entitiss.Add(Entity);
}
else
{
context.Entry(Entity).State = EntityState.Modified;
}
}
public void DeleteEntity(int id)
{
Entity Entity = FindEntity(id);
context.Entitys.Remove(Entity);
}
public void Save()
{
context.SaveChanges();
}
}
Now, the problem lies in the fact that I can get a list of the entities I have in the database and load them just fine into Selectlists or similar structures.
I can also get specific Entities based on other properties than the primary key.
But I can't get a single entity from the database using the primary key as my entry point. I have tried using DbSet.Entities.SingleOrDefault(), .Single(), .Find()
I have also tried to retrieve it using
DbSet.Entities.ToList().Where(x => x.EntityId == id)
I have moved from 2 data contexts down to 1, I have checked that I am referencing the same version of the EF in all projects in the solution (have a separate project for the entities).
After having read This, I fiddled with the Lazy loading, but no effect.
So I am running out of ideas and places to look.
If you need more information I will provide it to the best of my ability.
Thank you for reading and helping.
Update1:
I have tried moving to a local database with no luck, and it seems what I earlier described with loading lists of Entities working was exaggerated. They too don't load consistently when called.
Update 2:
Having used the SQL profiler to determine that not all calls from the controllers where executed on the database, I tried transplanting the entities to a fresh MVC project. Unfortunately the problem persists and I am out of ideas as this point.
Update 3:
After some more investigation and some help from Gert Arnold it has turned out that I misdiagnosed the initial problem. Due to poor coding practices it was not 1 central problem, but several separate problems all exhibiting similar characteristics that led me to the wrong conclusion that the EF was at fault. I am sorry to have wasted anyones time with this.
public Class FindEntity(int id)
{
//Want to find a specific student
return context.Entitys.Find(id);
}
This one seems a bit weird to me, what's the Class? I think it should be Student. Anyway, check out the code sample below:
public Entity FindEntity(int id)
{
return context.Entities.FirstOrDefault(e => e.EntityId == id); // note that it will return null if not found.
}
I am new to WebApi, so please excuse if the question is amateurish: I use AngularJS's "$resource" to communicate with the WebApi-Controller "BondController". This works great.
My problem: The entity "Bond" has a reference to a list of entity "Price":
public class Bond
{
public int ID { get; set; }
...
public virtual List<Price> Prices { get; set; }
}
What I am looking for is a way to exclude the nested list "Prices" such as
[JsonIgnore]
BUT, in some other situation, I still need a way to retrieve Bonds including this nested list, e.g. via a second controller "Bond2".
What can I do?
Will I need some ViewModel on top of the entity Bond?
Can I somehow exclude the List of Prices in the controller itself:
public IQueryable<Bond> GetBonds()
{
return db.Bonds [ + *some Linq-Magic that excludes the list of Prices*]
}
Background: the list of Prices might become rather long and the Get-Requests would easily become > 1MB. In most cases, the prices don't even need to be displayed to the user, so I'd like to exclude them from the response. But in one case, they do... Thank you for your input!
EDIT:
I see that, for some sort of Linq Magic, I would need a new type "PricelessBond"
EDIT2
Found a nice example of using DTO here and will use that.
The solution is to create a non-persistent BondDTO class that acts as a "shell" and that has only those properties you desire to be visible in a certain use-case and then, in the BondDTOController, transform the selection of Bond => BondDTO via means of a Linq Lambda Select expression.
I am no expert in WebApi but it seems that you have more than one problem.
Why won't you create a class hierarchy?
public class PricelessBond // :)
{
public int ID {get; set;}
}
public class Bond : PricelessBond
{
public List<Price> Prices {get; set;}
}
Then you can expose data via two different methods:
public class BondsController : ApiController
{
[Route("api/bonds/get-bond-without-price/{id}")]
public PricelessBond GetBondWithoutPrice(int id)
{
return DataAccess.GetBondWithoutPrice(id);
}
[Route("api/bonds/get-bond/{id}")]
public Bond GetBond()
{
return DataAccess.GetBond(id);
}
}
And in your DataAccess class:
public class DataAccess
{
public PricelessBond GetBondWithoutPrice(int id)
{
return db.Bonds
.Select(b => new PricelessBond
{
ID = b.ID
})
.Single(b => b.ID == id);
}
public Bond GetBond(int id)
{
return db.Bonds
.Select(b => new Bond
{
ID = b.ID,
Prices = b.Prices.Select(p => new Price{}).ToArray()
})
.Single(b => b.ID == id);
}
}
Of course, having two data access methods implies some code overhead but since you say the response could get greater than 1MB this also means that you should spare your database server and not fetch data that you don't need.
So, in your data access layer load only required data for each operation.
I have tested this in a scratch project and it worked.
Using Entity Framework 5, I have a generic method that retrieves entities from my context with optional parameters to filter, include related entities, and set the order of results. When I pass the method a set of include properties, however, it never modifies the query with joins to include the related entities. Any ideas why my query isn't updating?
Method:
public virtual IQueryable<TEntity> Get(
Expression<Func<TEntity, bool>> filter = null,
string includeProperties = "",
Func<IQueryable<TEntity>, IOrderedQueryable<TEntity>> orderBy = null)
{
IQueryable<TEntity> query = dbSet.AsExpandable();
if (filter != null)
{
query = query.Where(filter);
}
foreach (var includeProperty in includeProperties.Split
(new char[] { ',' }, StringSplitOptions.RemoveEmptyEntries))
{
query = query.Include(includeProperty);
}
if (orderBy != null)
{
return orderBy(query);
}
else
{
return query;
}
}
Query before foreach (var includeProperty... that remains unchanged afterwards
{SELECT
[Extent1].[Pid] AS [Pid],
[Extent1].[Created] AS [Created],
[Extent1].[Creator] AS [Creator]
FROM [Administrator] AS [Extent1]}
EDIT...More Info
Originally I was using the following method call: AdministratorDTO admin = unitOfWork.AdministratorComponent.GetByID(userPid); on the following POCO class:
public virtual TEntity GetByID(object id)
{
return dbSet.Find(id);
}
public class Administrator : IPrincipal
{
[Key]
[Required]
[StringLength(1024)]
[Display(Name = "PID")]
public string Pid { get; set; }
public DateTime Created { get; set; }
public string Creator { get; set; }
...
public ICollection<Role> Roles { get; set; }
public ICollection<Area> Areas { get; set; }
...
}
And using AutoMapper to map to a DTO with the following configuration:
public class AdministratorDTO
{
[Key]
[Required]
[StringLength(1024)]
[Display(Name = "PID")]
public string Pid { get; set; }
...
public string[] Roles { get; set; }
public string[] Areas { get; set; }
}
public class WebApiApplication : System.Web.HttpApplication
{
...
AutoMapperConfiguration.Configure();
}
public static class AutoMapperConfiguration
{
public static void Configure()
{
ConfigureAdministratorMapping();
...
}
private static void ConfigureAdministratorMapping()
{
Mapper.CreateMap<Administrator, AdministratorDTO>()
.ForMember(dest => dest.Roles,
opt => opt.MapFrom(src => src.Roles == null ?
null : src.Roles.Select(r => r.Name).ToArray()))
.ForMember(dest => dest.Areas,
opt => opt.MapFrom(src => src.Areas == null ?
null : src.Areas.Select(a => a.Id).ToArray()));
...
}
}
public class BusinessComponent<TEntity, TDto>
: IBusinessComponent<TEntity, TDto>
where TEntity : class
where TDto : class
{
...
protected TDto Flatten(TEntity entity)
{
return Mapper.Map<TEntity, TDto>(entity);
}
}
My understanding was that if I didn't mark the Administrator's navigation properties (Areas and Roles) as virtual they would be eagerly loaded, but I kept getting an empty string[] in my DTO.
I looked at the TEntity parameter going into my Flatten method and Areas/Roles were null before I called Map, so I don't think it was something to do with AutoMapper.
Next I tried using the following method call:
AdministratorDTO admin = unitOfWork.AdministratorComponent
.Get(filter: a => a.Pid == "csherman", includeProperties: "Roles, Areas")
.SingleOrDefault();
Finally, just in case the Include was being ignored because the navigation properties were not virtual, I added the virtual keyword to both Areas and Roles on my Administrator class. When I did this, both the GetByID and the Get(filter: ..., includeProperties: ...) methods worked, thereby including the Areas/Roles in my TEntity Flatten parameter and populating the string arrays in my DTO.
Problem solved I suppose, but...
Question is, especially for the GetById method, why did it work with the virtual keyword but not without?
If EF actually factors in the projection from the time of the original method call, why would these entities be included?
Includes don't work if you do any sort of projection after the include.
Here's a post that deals with this and how to work around it.
And here's an SO question that deals with it as well.
EDIT: In response to your edit I tried looking around to see if I could find a reason why the virtual keyword would make your Include work but not work without it. I couldn't find anything in passing that would directly answer that.
Here's what I think is happening, though: The virtual keyword, on navigation properties, tells Entity Framework that this property should employ lazy loading. If there are times that you want to load them eagerly is when you would use Include. I think that that's what Include is built for. I think it tries to look specifically for virtual properties with that name, and when it can't find it, dies gracefully without exception. If you don't mark it as virtual I think the implementation of Include misses it completely. I base this guess on the fact that none of the articles that I've seen on Include mentioned it outside the context of lazy loading--which would mean using the virtual keyword.
There's a serious problem with IQueryable.Include: the underlying object must be of type ObjectQuery or DbQuery, otherwise this method will not work!
Include is leaky abstraction and it works only with Entity framework. EF 4.1 already contains Include over generic IQueryable but it internally only converts passed generic IQueryable to generic ObjectQuery or DbQuery and calls their Include.
https://stackoverflow.com/a/6791874/2444725
When I pass the method a set of include properties, however, it never modifies the query with joins to include the related entities. Any ideas why my query isn't updating?
You seem to be using LinqKit:
IQueryable<TEntity> query = dbSet.AsExpandable();
Extension method AsExpandable gets IQueryable parameter and returns a new object of type ExpandableQuery, that decorates the original ObjectQuery or DbQuery object. Extension method IQueryable.Include, applied to ExpandableQuery, can't make the conversion and silently skips.
why did it work with the virtual keyword but not without?
Virtual method has nothing to do with include. Instead it turns on lazy loading. It means that navigation properties are not loaded on the first request, but are loaded on separate requests when these properties are actually used.