Scenario
I am working on updating my .NET API to encode all database key fields so that the sequential key is not exposed to the end user. I'm using hashids.org for this and have built helper methods to quickly decode/encode properties in my automapper mapping. However, there are multiple versions of the API and only the most current version should be updated with this functionality, which means that I can't simply overwrite my existing classes. I've implemented a few solutions that work, but they all have a bad code smell that I'm hoping to clear up.
Solutions
I am currently performing the encoding at the controller layer. I can see the merits of doing this at the data access layer as well, but feel there is more risk of leaks/missed conversions at that layer, especially since the API has many different data sources. Plus, hiding keys is an issue with the outside world, for which the controller is the gatekeeper, so it feels appropriate there.
The application currently has the following model pattern, which cannot be changed: Model (model that exists in DB) > ValueObject (service model, VO) > DTO (API model).
(1) Initial attempt
Below is an example of a class that needs to support an encoded and decoded state, where Utils.Encode() and Utils.Decode() are helper methods that will convert the field between int and string using Hashids.
//EquipmentDTO.cs
public class EquipmentDTO //encoded class
{
public string Id {get; set;}
public string Name {get; set;}
}
public class EquipmentUnencodedDTO //decoded class
{
public int Id {get; set;}
public string Name {get; set;}
}
//Automapper.cs
CreateMap<EquipmentUnencodedDTO, EquipmentDTO>()
.ForMember(dst => dst.Id, opt => opt.MapFrom(src => Utils.Encode(src.Id)));
CreateMap<EquipmentDTO, EquipmentUnencodedDTO>()
.ForMember(dst => dst.Id, opt => opt.MapFrom(src => Utils.Decode(src.Id)));
CreateMap<EquipmentVO, EquipmentDTO>() //mapping from service model to controller model
.ForMember(dst => dst.Id, opt => opt.MapFrom(src => Utils.Encode(src.Id)));
CreateMap<EquipmentDTO, EquipmentVO>()
.ForMember(dst => dst.Id, opt => opt.MapFrom(src => Utils.Decode(src.Id)));
CreateMap<Equipment, EquipmentVO>() //mapping from DB model to service model
.ForMember(dst => dst.Id, opt => opt.MapFrom(src => src.Id));
I chose to make the existing EquipmentDTO the encoded version
because I want this to become the new standard, which would
eventually lead to the deprecation and removal of
EquipmentUnencodedDTO as the old controllers eventually get
updated.
I chose to not copy CreateMap<EquipmentVO, EquipmentDTO> for CreateMap<EquipmentVO, EquipmentUnencodedDTO> (and the reverse) because
it would lead to a lot of duplication in the AutoMapper file, which
is already huge (though maybe this isn't a real problem?)
I do not like this solution because in my old controllers, the mapping is now confusing. In a POST, for example, the unencoded input DTO has to be converted to the service model via: Mapper.Map<EquipmentVO>(Mapper.Map<EquipmentDTO>(unencodedEquipmentInput)) which is super ugly.
That being said, this is supposedly a temporary problem, so is this a real problem?
This problem would go away if I created CreateMap<EquipmentVO, EquipmentUnencodedDTO>
I do not like this solution because my classes have a lot of duplicated fields that are not changing between the encoded and decoded versions
(2) Second Attempt
The two bullet points above led me to refactor to this:
public class EquipmentDTO
{
public string Id {get; set;}
public string Name {get; set;}
public Decoded Decode(){
return Mapper.Map<Decoded>(this);
}
public class Decoded: EquipmentDTO {
public new int Id {get; set;}
public EquipmentDTO Encode(){
return Mapper.Map<EquipmentDTO>(this);
}
}
}
// Automappers are the same, except EquipmentUnencodedDTO is now EquipmentDTO.Decoded
I like how simple it is to switch between encoded and decoded states now, reducing my double mapping above to: Mapper.Map<EquipmentVO>(unencodedEquipmentInput.Encode());
I like the nested class because it codifies the relationship between the two classes and also does a better job at identifying which fields get encoded/decoded
I think this smells a lot worse
(3) Next Attempt
My next attempt was to add in the missing mappings for the decoded class to the service model and to undo the changes from attempt #2. This created a ton of duplicated mapping code, I'm still stuck with duplicated properties in both classes without a clear indication to which fields get decoded/encoded, and it all feels much more cumbersome than necessary.
Thanks for any advice!
This is one of those answers that doesn't really answer your question directly, but is a different kind of approach to the problem at hand. Based on my comment above.
I would not try to bake in a "hardcoded" transformation, or make the aliasing some intrinsic part of object lifecycle. The idea here is that the transformation of identifiers should be obvious, explicit, and pluggable.
Let's start with an interface:
public interface IObscuredIDProvider
{
public string GetObscuredID(int id);
public void SetObscuredID(int id, string obscuredID);
}
Then, for our testing, a very simple mapper that just returns the int as a string. Your production version can be backed by the hashids.org project or whatever you like:
public class NonObscuredIDProvider : IObscuredIDProvider
{
public string GetObscuredID(int id)
{
return id.ToString();
}
public void SetObscuredID(int id, string obscuredID)
{
// noop
}
}
You'll need to inject the instance of IObscuredIDProvider into whatever layer transforms your "outside/untrusted" data into "trusted/domain" data. This is the place where you will assign the entity IDs from the obscured version to the internal version, and vice versa.
Does that make sense? Hopefully, this is a much more simple to understand and implement solution than baking in a complex, nested transformation....
After a lot of playing around, I ended up going the route of not using automapper and only having a single DTO for both the encoded/unencoded states by using custom getters/setters to control what value would be returned based on a readonly property isEncoded.
My problem with automapper and having multiple DTOs was that there was too much duplication and way too much code to write to add a new decodable DTO. Also, there were too many ways to break the relationship between encodedDTO and unencodedDTO, especially since there are other developers on the team (not to mention future hires) who could forget to create the encoded DTO or to create a mapping to properly encode or decode the ID values.
While I still have separate util methods to perform the encoding of a value, I moved all of the automapper "logic" into a base class EncodableDTO, which would allow a user to run Decode() or Encode() on a DTO to toggle its encoded state, including the encoded state for all of its encodable properties via reflection. Having a DTO inherit EncodableDTO also serves as a clear indicator to developers to what's going on, while custom getters/setters clearly indicate what I'm trying to do for specific fields.
Here's a sample:
public class EquipmentDTO: EncodableDTO
{
private int id;
public string Id {
get
{
return GetIdValue(id);
}
set
{
id = SetIdValue(value);
}
}
public List<PartDTO> Parts {get; set;}
public string Name {get; set;}
}
public class PartDTO: EncodableDTO
{
private int id;
public string Id {
get
{
return GetIdValue(id);
}
set
{
id = SetIdValue(value);
}
}
public string Name {get; set;}
}
public class EncodableDTO
{
public EncodableDTO()
{
// encode models by default
isEncoded = true;
}
public bool isEncoded { get; private set; }
public void Decode()
{
isEncoded = false;
RunEncodableMethodOnProperties(MethodBase.GetCurrentMethod().Name);
}
public void Encode()
{
isEncoded = true;
RunEncodableMethodOnProperties(MethodBase.GetCurrentMethod().Name);
}
protected string GetIdValue(int id)
{
return isEncoded ? Utils.EncodeParam(id) : id.ToString();
}
// TryParseInt() is a custom string extension method that does an int.TryParse and outputs the parameter if the string is not an int
protected int SetIdValue(string id)
{
// check to see if the input is an encoded value, otherwise try to parse it.
// the added logic to test if the 'id' is an encoded value allows the inheriting DTO to be received both in
// unencoded and encoded forms (unencoded/encoded http request) and still populate the correct numerical value for the ID
return id.TryParseInt(-1) == -1 ? Utils.DecodeParam(id) : id.TryParseInt(-1);
}
private void RunEncodableMethodOnProperties(string methodName)
{
var self = this;
var selfType = self.GetType();
// Loop through properties and check to see if any of them should be encoded/decoded
foreach (PropertyInfo property in selfType.GetProperties())
{
var test = property;
// if the property is a list, check the children to see if they are decodable
if (property is IList || (
property.PropertyType.IsGenericType
&& (property.PropertyType.GetGenericTypeDefinition() == typeof(List<>)
|| property.PropertyType.GetGenericTypeDefinition() == typeof(IList<>))
)
)
{
var propertyInstance = (IList)property.GetValue(self);
if (propertyInstance == null || propertyInstance.Count == 0)
{
continue;
}
foreach (object childInstance in propertyInstance)
{
CheckIfObjectEncodable(childInstance, methodName);
}
continue;
}
CheckIfObjectEncodable(property.GetValue(self), methodName);
}
}
private void CheckIfObjectEncodable(object instance, string methodName)
{
if (instance != null && instance.GetType().BaseType == typeof(EncodableDTO))
{
// child instance is encodable. Run the same decode/encode method we're running now on the child
var method = instance.GetType().GetMethod(methodName);
method.Invoke(instance, new object[] { });
}
}
}
An alternative to RunEncodableMethodOnProperties() was the explicitly decode/encode child properties in the inheriting class:
public class EquipmentDTO: EncodableDTO
{
private int id;
public string Id {
get
{
return GetIdValue(id);
}
set
{
id = SetIdValue(value);
}
}
public List<PartDTO> Parts {get; set;}
public string Name {get; set;}
public new void Decode() {
base.Decode();
// explicitly decode child properties
Parts.ForEach(p => p.Decode());
}
}
I chose not to do the above because it created more work for DTO creators to have to remember to explicitly add (1) the override method, and (2) any new decodable properties to the override method. That being said, I'm sure I'm taking some sort of a performance hit by looping through every class of my class' properties and its children, so in time I may have to migrate towards this solution instead.
Regardless of the method I chose to decode/encode properties, here was the end result in the controllers:
// Sample controller method that does not support encoded output
[HttpPost]
public async Task<IHttpActionResult> AddEquipment([FromBody] EquipmentDTO equipment)
{
// EquipmentDTO is 'isEncoded=true' by default
equipment.Decode();
// send automapper the interger IDs (stored in a string)
var serviceModel = Mapper.Map<EquipmentVO>(equipment);
var addedServiceModel = myService.AddEquipment(serviceModel);
var resultValue = Mapper.Map<EquipmentDTO>(addedServiceModel);
resultValue.Decode();
return Created("", resultValue);
}
// automapper
CreateMap<EquipmentVO, EquipmentDTO>().ReverseMap();
CreateMap<Equipment, EquipmentVO>();
While I don't think its the cleanest solution, it hides a lot of the necessary logic to make encoding/decoding work with the least amount of work for future developers
Related
As an example let's say my database has a table with thousands of ships with every ship potentially having thousands of passengers as a navigation property:
public DbSet<Ship> Ship { get; set; }
public DbSet<Passenger> Passenger { get; set; }
public class Ship
{
public List<Passenger> passengers { get; set; }
//properties omitted for example
}
public class Passenger
{
//properties omitted for example
}
The example use case is that someone is fetching all ships per API and would like to know for each ship whether it is empty (0 passengers), so the returned JSON will contain a list of ships each with a bool whether it is empty.
My current code seems very inefficient (including all passengers just to determine if a ship is empty):
List<Ship> ships = dbContext.Ship
.Include(x => x.passengers)
.ToList();
and later when the ships are serialized to JSON:
jsonShip.isEmpty = !ship.passengers.Any();
I would like a more performant (and not bloated) alternative to including all passengers. What options do I have?
I have looked at computed columns but they only seem to support sql as string. If possible I would like to stay in the C# code world, so for example having a property which is set correctly by being automatically woven in the SQL query would be optimal.
Create a Data Transfer Object for Ship that reflects the shape of your JSON result, like -
public class ShipDto
{
public int Id { get; set; }
public string Name { get; set; }
public bool IsEmpty { get; set; }
}
Then use projection in your query -
var ships = dbCtx.Ships
.Select(p => new ShipDto
{
Id = p.Id,
Name = p.Name,
IsEmpty = !p.Passengers.Any()
})
.ToList();
Usually, APIs need to produce responses of various shapes and DTOs give you well defined models to represent the shape of your API response. Domain entities are not always suitable for this.
If your domain entity (Ship) has a lot of properties, then copying all those properties in the .Select() method might be cumbersome. You can use AutoMapper to map them for you. AutoMapper has a ProjectTo<T>() method that can generate the SQL and return the projected result. For example, you can achieve the above result with a mapping configuration -
CreateMap<Ship, ShipDto>()
.ForMember(d => d.IsEmpty, opt => opt.MapFrom(s => !s.Passengers.Any()));
and a query -
var ships = Mapper.ProjectTo<ShipDto>(dbCtx.Ships).ToList();
assuming all other properties in ShipDto are named similar as in Ship entity.
EDIT :
If you don't want a DTO model -
you can add a NotMapped property in Ship model -
public class Ship
{
public int Id { get; set; }
public string Name { get; set; }
[NotMapped]
public bool IsEmpty { get; set; }
public List<Passenger> passengers { get; set; }
}
and then do the query like -
var ships = dbCtx.Ships
.Select(p => new Ship
{
Id = p.Id,
Name = p.Name,
IsEmpty = !p.Passengers.Any()
})
.ToList();
You can return an anonymous type -
var ships = dbCtx.Ships
.Select(p => new
{
Id = p.Id,
Name = p.Name,
IsEmpty = !p.Passengers.Any()
})
.ToList();
If I understand your intention correctly...
One way is to store the number of passengers inside each Ship entity. This can work well if you use Domain Driven Design, treat the Ship as an aggregate root, and only add or remove passengers through methods exposed on the given Ship entity, e.g. RegisterPassenger() / RemovePassenger(). Inside these methods, increment or decrement the passenger number along with adding or removing the passenger.
Then, obviously, you can query the Ships dbset with a PassengerCount < 0 projection to the bool you need. And, again obviously, it won't even touch the Passengers table.
In traditional anemic domain ASP.NET systems this sort of data redundancy might be a bit more risky, because properties are usually publicly mutable, and you have multiple services that 'massage' the entities, which is a potential source of data integrity loss.
I'm getting the following content when I invoke my API. It kind of breaks up in the middle when the tenant entity that member is linked to, will start listing its member entities.
{
"id":"00000000-7357-000b-0001-000000000000",
"tenantId":"00000000-7357-000a-0001-000000000000",
"userName":"user1",
"tenant":{
"id":"00000000-7357-000a-0001-000000000000",
"name":"First Fake Org",
"members":[
I configured the lazy loading like this.
services.AddDbContext<Context>(config => config
.UseLazyLoadingProxies()
.UseSqlServer(Configuration.GetConnectionString("Register")));
How should I change the code so that the lazily loaded entities don't get served? I was hoping that it would simply return an empty list to the client. Should I use a DTO for that purpose and not return from the DB like this? There's talk about not using lazy loading for APIs at all here.
[HttpGet("test1/{username}"), AllowAnonymous]
public IActionResult GetStuff(string userName)
{
Member output;
output = Context.Members
.Include(e => e.Tenant)
.Single(e => e.UserName == userName);
return Ok(output);
}
I'm not sure what to google for and all the hits I got were pointing to the UseLazyLoadingProxies() invokation.
This will probably be somewhat long winded: But here goes.
It sounds like you have Entities which look something like:
public partial class Member
{
public virtual long Id { get; set; }
public virtual List<Tenant> Tenants { get; set; } //tables have fk relationship
}
public partial class Tenant
{
public virtual long Id { get; set; }
public virtual List<Member> Members{ get; set; } //tables have another fk relationship?
}
And then for this method:
[HttpGet("test1/{username}"), AllowAnonymous]
public IActionResult GetStuff(string userName)
{
Member output;
output = Context.Members
.Include(e => e.Tenant)
.Single(e => e.UserName == userName);
return Ok(output);
}
I see a few issues, but I'll try to keep it short:
I wouldn't have the controller do this directly. But it should work.
What I think you over looked is exactly what the .Include statement does. When the object is instantiated, it will get all of those related entities. Includes essentially converts your where statement to a left join, where the foreign keys match (EF calls these navigation properties).
If you don't want the Tenant property, then you can omit the .Include statement. Unless this is meant to be more generic (In which case, an even stronger reason to use a different pattern and auto mapper).
Hopefully your database doesn't truly have a FK relationship both ways, if it does, fix that ASAP.
The next issue is that you might not want a list of child properties, but it is in the model so they will be "there". Although your List Tenants might be null. And while this might be fine to you, right now. As a general rule when I see an API returning a property, I expect something to be either not there (This Member doesn't have tenants) or something is wrong, like perhaps there is a second parameter I missed. This probably isn't a problem 93.284% of the time, but it is something to be mindful of.
This starts to get into why an AutoMapper is great. Your Database Models, Business Models and views are likely different. And as much as you shouldn't return the database models directly. Taking control of how the data is represented for each part of the application is a great idea.
You could reduce the code easily, and remove the navitation properties:
[HttpGet("test1/{username}"), AllowAnonymous]
public IActionResult GetStuff(string userName)
{
return Ok(Context.Members
.Include(e => e.Tenant)
.Single(e => e.UserName == userName));
}
But again, a business layer would be better:
[HttpGet("test1/{username}"), AllowAnonymous]
public IActionResult GetStuff(string userName)
{
return Ok(MemberRepository.GetMember(userName));
}
The main point I'd stress though, is creating a view model.
For example, Let's say a user Detail:
public class MemberDetail
{
public string UserName {get; set;}
public long UserId { get; set; }
public string FullName { get; set; }
}
This way the view always receives exactly what you want to see, and not the extra data. Add this with the fact that you can know exactly how every use of Member to MemberDetail will map.
I am new to WebApi, so please excuse if the question is amateurish: I use AngularJS's "$resource" to communicate with the WebApi-Controller "BondController". This works great.
My problem: The entity "Bond" has a reference to a list of entity "Price":
public class Bond
{
public int ID { get; set; }
...
public virtual List<Price> Prices { get; set; }
}
What I am looking for is a way to exclude the nested list "Prices" such as
[JsonIgnore]
BUT, in some other situation, I still need a way to retrieve Bonds including this nested list, e.g. via a second controller "Bond2".
What can I do?
Will I need some ViewModel on top of the entity Bond?
Can I somehow exclude the List of Prices in the controller itself:
public IQueryable<Bond> GetBonds()
{
return db.Bonds [ + *some Linq-Magic that excludes the list of Prices*]
}
Background: the list of Prices might become rather long and the Get-Requests would easily become > 1MB. In most cases, the prices don't even need to be displayed to the user, so I'd like to exclude them from the response. But in one case, they do... Thank you for your input!
EDIT:
I see that, for some sort of Linq Magic, I would need a new type "PricelessBond"
EDIT2
Found a nice example of using DTO here and will use that.
The solution is to create a non-persistent BondDTO class that acts as a "shell" and that has only those properties you desire to be visible in a certain use-case and then, in the BondDTOController, transform the selection of Bond => BondDTO via means of a Linq Lambda Select expression.
I am no expert in WebApi but it seems that you have more than one problem.
Why won't you create a class hierarchy?
public class PricelessBond // :)
{
public int ID {get; set;}
}
public class Bond : PricelessBond
{
public List<Price> Prices {get; set;}
}
Then you can expose data via two different methods:
public class BondsController : ApiController
{
[Route("api/bonds/get-bond-without-price/{id}")]
public PricelessBond GetBondWithoutPrice(int id)
{
return DataAccess.GetBondWithoutPrice(id);
}
[Route("api/bonds/get-bond/{id}")]
public Bond GetBond()
{
return DataAccess.GetBond(id);
}
}
And in your DataAccess class:
public class DataAccess
{
public PricelessBond GetBondWithoutPrice(int id)
{
return db.Bonds
.Select(b => new PricelessBond
{
ID = b.ID
})
.Single(b => b.ID == id);
}
public Bond GetBond(int id)
{
return db.Bonds
.Select(b => new Bond
{
ID = b.ID,
Prices = b.Prices.Select(p => new Price{}).ToArray()
})
.Single(b => b.ID == id);
}
}
Of course, having two data access methods implies some code overhead but since you say the response could get greater than 1MB this also means that you should spare your database server and not fetch data that you don't need.
So, in your data access layer load only required data for each operation.
I have tested this in a scratch project and it worked.
I've run into an annoying problem when trying to update an entity that has a relationship which I don't care about in the frontend.
The problem is that when automapper creates my Item the default constructor sets the relationship to an empty list (should be null) which causes problems when EF is trying to save it, because now it thinks that it should delete the relationship the existing entity has.
I was thinking that this is a problem many have had, but google didn't seem to think so. Am I approaching this in a wrong way? How do you maintain your entity relationships when mapping from DTO to entity model?
Classes
// normal code first POCO class
public class Item
{
public Item()
{
Others = new List<Other>();
}
public int Id {get; set;}
public virtual ICollection<Other> Others {get; set;}
}
// my DTO
public class ItemDTO
{
public int Id {get; set;}
}
Controller action
[HttpPost]
public void PostAction(ItemDTO dto)
{
var item = Mapper.Map<Item>(dto);
// The problem here is that item.Others.Count is 0, should be null
// so EF thinks it needs to delete the relationships
_repo.Update(item);
}
There's two ways you can go about this:
Either you could just set the relation to null after AutoMapper has mapped it:
[HttpPost]
public void PostAction(ItemDTO dto)
{
var poco = Mapper.Map<Item>(dto);
poco.Others = null;
_repo.Update(poco);
}
Or you could create a custom automapper profile that always sets it to null:
public class ItemMap : Profile
{
protected override void Configure()
{
CreateMap<Item, Listing>().ForMember(d => d.Others, o => o.UseValue(null));
}
}
You'd use the first method if that's the only place you want it. Use the second one if it's always the case
Can anyone provide an easier more automatic way of doing this?
I have the following save method for a FilterComboTemplate model. The data has been converted from json to a c# model entity by the webapi.
So I don't create duplicate entries in the DeviceProperty table I have to go through each filter in turn and retrieve the assigned DeviceFilterProperty from the context and override the object in the filter. See the code below.
I have all the object Id's if they already exist so it seems like this should be handled automatically but perhaps that's just wishful thinking.
public void Save(FilterComboTemplate comboTemplate)
{
// Set the Device Properties so we don't create dupes
foreach (var filter in comboTemplate.Filters)
{
filter.DeviceProperty = context.DeviceFilterProperties.Find(filter.DeviceFilterProperty.DeviceFilterPropertyId);
}
context.FilterComboTemplates.Add(comboTemplate);
context.SaveChanges();
}
From here I'm going to have to check whether any of the filters exist too and then manually update them if they are different to what's in the database so as not to keep creating a whole new set after an edit of a FilterComboTemplate.
I'm finding myself writing a lot of this type of code. I've included the other model classes below for a bit of context.
public class FilterComboTemplate
{
public FilterComboTemplate()
{
Filters = new Collection<Filter>();
}
[Key]
public int FilterComboTemplateId { get; set; }
[Required]
public string Name { get; set; }
[Required]
public ICollection<Filter> Filters { get; set; }
}
public class Filter
{
[Key]
public int FilterId { get; set; }
[Required]
public DeviceFilterProperty DeviceFilterProperty { get; set; }
[Required]
public bool Exclude { get; set; }
[Required]
public string Data1 { get; set; }
}
public class DeviceFilterProperty
{
[Key]
public int DeviceFilterPropertyId { get; set; }
[Required]
public string Name { get; set; }
}
Judging from some similar questions on SO, it does not seem something EF does automatically...
It's probably not a massive cut on code but you could do something like this, an extension method on DbContext (or on your particular dataContext):
public static bool Exists<TEntity>(this MyDataContext context, int id)
{
// your code here, something similar to
return context.Set<TEntity>().Any(x => x.Id == id);
// or with reflection:
return context.Set<TEntity>().Any(x => {
var props = typeof(TEntity).GetProperties();
var myProp = props.First(y => y.GetCustomAttributes(typeof(Key), true).length > 0)
var objectId = myProp.GetValue(x)
return objectId == id;
});
}
This will check if an object with that key exists in the DbContext. Naturally a similar method can be created to actually return that entity as well.
There are two "returns" in the code, just use the one you prefer. The former will force you to have all entities inherit from an "Entity" object with an Id Property (which is not necessarily a bad thing, but I can see the pain in this... you will also need to force the TEntity param: where TEntity : Entity or similar).
Take the "reflection" solution with a pinch of salt, first of all the performance may be a problem, second of all I don't have VS running up now, so I don't even know if it compiles ok, let alone work!
Let me know if that works :)
It seems that you have some common operations for parameters after it's bound from request.
You may consider to write custom parameter bindings to reuse the code. HongMei's blog is a good start point: http://blogs.msdn.com/b/hongmeig1/archive/2012/09/28/how-to-customize-parameter-binding.aspx
You may use the code in Scenario 2 to get the formatter binding to deserialize the model from body and perform the operations your want after that.
See the final step in the blog to specify the parameter type you want customize.