We Keep Coding

Exclude a model property from loading in MVC5 EF6 [duplicate]

I'm using entity framework to connect with the database. I've one little problem:
I've one table which have one varbinary(MAX) column(with filestream).
I'm using SQL request to manage the "Data" part, but EF for the rest(metadata of the file).
I've one code which has to get all files id, filename, guid, modification date, ... of a file. This doesn't need at all the "Data" field.
Is there a way to retrieve a List but without this column filled?
Something like
context.Files.Where(f=>f.xyz).Exclude(f=>f.Data).ToList();
??
I know I can create anonymous objects, but I need to transmit the result to a method, so no anonymous methods. And I don't want to put this in a list of anonymous type, and then create a list of my non-anonymous type(File).
The goal is to avoid this:
using(RsSolutionsEntities context = new RsSolutionsEntities())
{
var file = context.Files
.Where(f => f.Id == idFile)
.Select(f => new {
f.Id, f.MimeType, f.Size, f.FileName, f.DataType,
f.DateModification, f.FileId
}).FirstOrDefault();
return new File() {
DataType = file.DataType, DateModification = file.DateModification,
FileId = file.FileId, FileName = file.FileName, Id = file.Id,
MimeType = file.MimeType, Size = file.Size
};
}
(I'm using here the anonymous type because otherwise you will get a NotSupportedException: The entity or complex type 'ProjectName.File' cannot be constructed in a LINQ to Entities query.)
(e.g. this code throw the previous exception:
File file2 = context.Files.Where(f => f.Id == idFile)
.Select(f => new File() {Id = f.Id, DataType = f.DataType}).FirstOrDefault();
and "File" is the type I get with a context.Files.ToList(). This is the good class:
using File = MyProjectNamespace.Common.Data.DataModel.File;
File is a known class of my EF datacontext:
public ObjectSet<File> Files
{
get { return _files ?? (_files = CreateObjectSet<File>("Files")); }
}
private ObjectSet<File> _files;

Is there a way to retrieve a List but without this column filled?
Not without projection which you want to avoid. If the column is mapped it is natural part of your entity. Entity without this column is not complete - it is different data set = projection.
I'm using here the anonymous type because otherwise you will get a
NotSupportedException: The entity or complex type 'ProjectName.File'
cannot be constructed in a LINQ to Entities query.
As exception says you cannot project to mapped entity. I mentioned reason above - projection make different data set and EF don't like "partial entities".
Error 16 Error 3023: Problem in mapping fragments starting at line
2717:Column Files.Data in table Files must be mapped: It has no
default value and is not nullable.
It is not enough to delete property from designer. You must open EDMX as XML and delete column from SSDL as well which will make your model very fragile (each update from database will put your column back). If you don't want to map the column you should use database view without the column and map the view instead of the table but you will not be able to insert data.
As a workaround to all your problems use table splitting and separate the problematic binary column to another entity with 1 : 1 relation to your main File entity.

I'd do something like this:
var result = from thing in dbContext.Things
select new Thing {
PropertyA = thing.PropertyA,
Another = thing.Another
// and so on, skipping the VarBinary(MAX) property
};
Where Thing is your entity that EF knows how to materialize. The resulting SQL statement shouldn't include the large column in its result set, since it's not needed in the query.
EDIT: From your edits, you get the error NotSupportedException: The entity or complex type 'ProjectName.File' cannot be constructed in a LINQ to Entities query. because you haven't mapped that class as an entity. You can't include objects in LINQ to Entities queries that EF doesn't know about and expect it to generate appropriate SQL statements.
You can map another type that excludes the VarBinary(MAX) column in its definition or use the code above.

you can do this:
var files = dbContext.Database.SqlQuery<File>("select FileId, DataType, MimeType from Files");
or this:
var files = objectContext.ExecuteStoreQuery<File>("select FileId, DataType, MimeType from Files");
depending on your version of EF

I had this requirement because I have a Document entity which has a Content field with the content of the file, i.e. some 100MB in size, and I have a Search function that I wanted to return the rest of the columns.
I chose to use projection:
IQueryable<Document> results = dbContext.Documents.Include(o => o.UploadedBy).Select(o => new {
Content = (string)null,
ContentType = o.ContentType,
DocumentTypeId = o.DocumentTypeId,
FileName = o.FileName,
Id = o.Id,
// etc. even with related entities here like:
UploadedBy = o.UploadedBy
});
Then my WebApi controller passes this results object to a common Pagination function, which applies a .Skip, .Take and a .ToList.
This means that when the query gets executed, it doesn't access the Content column, so the 100MB data is not being touched, and the query is as fast as you'd want/expect it to be.
Next, I cast it back to my DTO class, which in this case is pretty much exactly the same as the entity class, so this might not be a step you need to implement, but it's follows my typical WebApi coding pattern, so:
var dtos = paginated.Select(o => new DocumentDTO
{
Content = o.Content,
ContentType = o.ContentType,
DocumentTypeId = o.DocumentTypeId,
FileName = o.FileName,
Id = o.Id,
UploadedBy = o.UploadedBy == null ? null : ModelFactory.Create(o.UploadedBy)
});
Then I return the DTO list:
return Ok(dtos);
So it uses projection, which might not fit the original poster's requirements, but if you're using DTO classes, you're converting anyway. You could just as easily do the following to return them as your actual entities:
var dtos = paginated.Select(o => new Document
{
Content = o.Content,
ContentType = o.ContentType,
DocumentTypeId = o.DocumentTypeId,
//...
Just a few extra steps but this is working nicely for me.

For EF Core 2
I implemented a solution like this:
var files = context.Files.AsNoTracking()
.IgnoreProperty(f => f.Report)
.ToList();
The base idea is to turn for example this query:
SELECT [f].[Id], [f].[Report], [f].[CreationDate]
FROM [File] AS [f]
into this:
SELECT [f].[Id], '' as [Report], [f].[CreationDate]
FROM [File] AS [f]
you can see the full source code in here:
https://github.com/aspnet/EntityFrameworkCore/issues/1387#issuecomment-495630292

I'd like to share my attempts to workaround this problem in case somebody else is in the same situation.
I started with what Jeremy Danyow suggested, which to me is the less painful option.
// You need to include all fields in the query, just make null the ones you don't want.
var results = context.Database.SqlQuery<myEntity>("SELECT Field1, Field2, Field3, HugeField4 = NULL, Field5 FROM TableName");
In my case, I needed a IQueryable<> result object so I added AsQueryable() at the end. This of course let me add calls to .Where, .Take, and the other commands we all know, and they worked fine. But there's a caveat:
The normal code (basically context.myEntity.AsQueryable()) returned a System.Data.Entity.DbSet<Data.DataModel.myEntity>, while this approach returned System.Linq.EnumerableQuery<Data.DataModel.myEntity>.
Apparently this means that my custom query gets executed "as is" as soon as needed and the filtering I added later is done afterwards and not in the database.
Therefore I tried to mimic Entity Framework's object by using the exact query EF creates, even with those [Extent1] aliases, but it didn't work. When analyzing the resulting object, its query ended like
FROM [dbo].[TableName] AS [Extent1].Where(c => ...
instead of the expected
FROM [dbo].[TableName] AS [Extent1] WHERE ([Extent1]...
Anyway, this works, and as long as the table is not huge, this method will be fast enough. Otherwise you have no option than to manually add the conditions by concatenating strings, like classic dynamic SQL. A very basic example in case you don't know what I'm talking about:
string query = "SELECT Field1, Field2, Field3, HugeField4 = NULL, Field5 FROM TableName";
if (parameterId.HasValue)
query += " WHERE Field1 = " + parameterId.Value.ToString();
var results = context.Database.SqlQuery<myEntity>(query);
In case your method sometimes needs this field, you can add a bool parameter and then do something like this:
IQueryable<myEntity> results;
if (excludeBigData)
results = context.Database.SqlQuery<myEntity>("SELECT Field1, Field2, Field3, HugeField4 = NULL, Field5 FROM TableName").AsQueryable();
else
results = context.myEntity.AsQueryable();
If anyone manages to make the Linq extensions work properly like if it was the original EF object, please comment so I can update the answer.

I'm using here the anonymous type because otherwise you will get a
NotSupportedException: The entity or complex type 'ProjectName.File'
cannot be constructed in a LINQ to Entities query.
var file = context.Files
.Where(f => f.Id == idFile)
.FirstOrDefault() // You need to exeucte the query if you want to reuse the type
.Select(f => new {
f.Id, f.MimeType, f.Size, f.FileName, f.DataType,
f.DateModification, f.FileId
}).FirstOrDefault();
And also its not a bad practice to de-normalize the table into further, i.e one with metadata and one with payload to avoid projection. Projection would work, the only issue is, need to edit any time a new column is added to the table.

I tried this:
From the edmx diagram (EF 6), I clicked the column I wanted to hide from EF and on their properties you can set their getter and setter to private. That way, for me it works.
I return some data which includes a User reference, so I wanted to hide the Password field even though it's encrypted and salted, I just didn't want it on my json, and I didn't want to do a:
Select(col => new {})
because that's a pain to create and maintain, especially for big tables with a lot of relationships.
The downside with this method is that if you ever regenerate your model, you would need to modify their getter and setter again.

Using Entity Framework Power Tools you can do the following in efpt.config.json:
"Tables": [
{
"ExcludedColumns": [
"FileData"
],
"Name": "[dbo].[Attachment]",
"ObjectType": 0
}
]

A "Composable" Full Text Search with a Code First Model

UPDATE 18 Sep 2013
It looks like there isn't an easy way to do this. I'm holding out for a solution that involves some extension to Entity Framework.
If you'd like to see these features in Entity Framework, vote for them on the user voice site, perhaps here and here
There are several similar questions on SO but I can't find a question new and similar enough to have the answer I'm looking for.
If this looks like information overload, jump down to In Summary.
Background
I'm writing a WebApi REST service to expose some pre-existing data through an OData end point. I'm using the EntitySetContoller<TEntity, TKey> to do all the grunt work for me. As well as the standard OData parameters, that are routed and translated by the base class, I've added some custom parameters, to allow specific functionality for my controller.
My database server is MS SQL Server with a full text index on the [BigText] NVarChar[4000] column of the [SomeEntity] table.
I have one limitation, I must use a Code First model.
// Model POCO
public class SomeEntity
{
public int Id { get; set; }
public string BigText { get; set; }
}
// Simple Controller
public class SomeEntityController : EntitySetController<SomeEntity, int>
{
private readonly SomeDbContext context = new SomeDbContext();
public override IQueryable<SomeEntity> Get()
{
var parameters = Request.GetQueryNameValuePairs()
.ToDictionary(p => p.Key, p => p.Value);
if (parameters.ContainsKey("BigTextContains")
(
var searchTerms = parameters["BigTextContains"];
// return something special ...
)
return this.context.SomeEntities;
}
// ... The rest is omitted for brevity.
}
The Problem
How to implement the // return something special ... part of my example?
Obviously, the niave
return this.context.SomeEntities.Where(e =>
e.BigText.Contains(searchTerm));
is completely wrong, it composes to a WHERE clause like
[BigText] LIKE '%' + #searchTerm + '%'
This doesn't use Full Text Searching so, doesn't support complex search terms and otherwise, performs terribley.
This approach,
return this.context.SomeEntities.SqlQuery(
"SELECT E.* FROM [dbo].[SomeEntity] E " +
"JOIN CONTAINSTABLE([SomeEntity], [BigText], #searchTerm) FTS " +
" ON FTS.[Key] = E.[Id]",
new object[] { new SqlParameter("#searchTerm", searchTerm) })
.AsQueryable();
Looks promising, it actually uses Full Text Searching, and is quite functional. However, you'll note that DbSqlQuery, the type returned from the SqlQuery function does not implement IQueryable. Here, it is coerced to the right return type with the AsQueryable() extension but, this breaks the "chain of composition". The only statement that will be performed on the server is the one specified in the code above. Any additional clauses, specified on the OData URL will be serviced on the API hosting web server, without benefitting from the indices and specialised set based functionality of the database engine.
In Summary
What is the most expedient way of accessing MS SQL Server's Full Text Search CONTAINSTABLE function with an Entity Framework 5 Code First model and acquiring a "composable" result?
Do I need to write my own IQueryProvider? Can I extend EF in some way?
I don't want to use Lucene.Net, I don't want to use a Database Generated Model. Perhaps I could add extra packages or wait for EF6, would that help?

It is not perfect, but you can accomplish what you are after with 2 calls to the database.
The first call would retrieve a list of matching key's from CONTAINSTABLE and then the second call would be your composable query utilizing the IDs that you returned from the first call.
//Get the Keys from the FTS
var ids = context.Database.SqlQuery<int>(
"Select [KEY] from CONTAINSTABLE([SomeEntity], [BigText], #searchTerm)",
new object[] { new SqlParameter("#searchTerm", searchTerm) });
//Use the IDs as an initial filter on the query
var composablequery = context.SomeEntities.Where(d => ids.Contains(d.Id));
//add on whatever other parameters were captured to the 'composablequery' variable
composablequery = composablequery.Where(.....)

I had this same issue recently:
EF 5 Code First FTS Queriable
Let me extend that post.
Your first option was mine first as well - using SqlQuery
I also needed to do more filtering, so instead of always writing full sql I used QueryBuilder, to which I made some changes and added more functions to fit my needs(I could upload it somewhere if needed):
QueryBuilder
After I have found another idea which I implemented.
Someone already mention it here, and that is to use SqlQuery that will return HashSet of Ids and that you can use it in EF queries with Contains.
This is better but not most optimal since you need 2 queries and Id list in memory.
Example:
public IQueryable<Company> FullTextSearchCompaniesByName(int limit, int offset, string input, Guid accountingBureauId, string orderByColumn)
{
FtsQueryBuilder ftsQueryBuilder = new FtsQueryBuilder();
ftsQueryBuilder.Input = FtsQueryBuilder.FormatQuery(input);
ftsQueryBuilder.TableName = FtsQueryBuilder.GetTableName<Company>();
ftsQueryBuilder.OrderByTable = ftsQueryBuilder.TableName;
ftsQueryBuilder.OrderByColumn = orderByColumn;
ftsQueryBuilder.Columns.Add("CompanyId");
if (accountingBureauId != null && accountingBureauId != Guid.Empty)
ftsQueryBuilder.AddConditionQuery<Guid>(Condition.And, "" , #"dbo.""Company"".""AccountingBureauId""", Operator.Equals, accountingBureauId, "AccountingBureauId", "");
ftsQueryBuilder.AddConditionQuery<bool>(Condition.And, "", #"dbo.""Company"".""Deleted""", Operator.Equals, false, "Deleted", "");
var companiesQuery = ftsQueryBuilder.BuildAndExecuteFtsQuery<Guid>(Context, limit, offset, "Name");
TotalCountQuery = ftsQueryBuilder.Total;
HashSet<Guid> companiesIdSet = new HashSet<Guid>(companiesQuery);
var q = Query().Where(a => companiesIdSet.Contains(a.CompanyId));
return q;
}
However EF 6 now has something called Interceptors that can be used to implement queriable FTS, and it is pretty simple and generic(last post):
EF 6 Interceptors for FTS.
I have tested this and it works fine.
!! REMARK: EF Code First, even with version 6, does not support Custom Stored Procedures.
There are only some for predefined CUD operations if I understood it well:
Code First Insert/Update/Delete Stored Procedure Mapping, so it can't be done with it.
Conclusion: if you can use EF 6 go for third options, is gives all you need.
If you are stucked with EF 5 or less, second option is better then first but not most optimal.

Using Dapper to map more than 5 types

I am currently building a SELECT query that joins 12 tables together. I've been using Dapper for all my other queries and it works great. Problem is, the generic methods only have to five generic parameters.
I've previously modified the code to support up to 6 for another query, but now I really don't think I should be hacking 6 more levels of generics.
Is there a way to pass dapper an array of types, and it returns the results as an array of objects, which I can cast manually if I have to?
I also might be approaching the problem the wrong way! Any help will be appreciated!

In a project I worked on I saw something like this to get more than 7 types mapped. We used Dapper 1.38:
connection.Query<TypeOfYourResult>
(
queryString,
new[]
{
typeof(TypeOfArgument1),
typeof(TypeOfArgument2),
...,
typeof(TypeOfArgumentN)
},
objects =>
{
TypeOfArgument1 arg1 = objects[0] as TypeOfArgument1;
TypeOfArgument2 arg2 = objects[1] as TypeOfArgument2;
...
TypeOfArgumentN argN = objects[N] as TypeOfArgumentN;
// do your processing here, e.g. arg1.SomeField = arg2, etc.
// also initialize your result
var result = new TypeOfYourResult(...)
return result;
},
parameters,
splitOn: "arg1_ID,arg2_ID, ... ,argN_ID"
);
The queryString is self-explanatory. The splitOn parameter says how Dapper should split the columns from the SELECT statement so that everything can be mapped properly to the objects,
you can read about it here.

You could use a dynamic query and map it afterwards. Something like this
var result = conn.Query<dynamic>(query).Select(x => new Tuple<Type1, Type2, Type3, Type4, Type5>(
// type initialization here
new Type1(x.Property1,x.Property2),
new Type2(x.Property3,x.Property4),
new Type3(x.Property5,x.Property6) etc....));
Edit: With a rather huge result set, another option might be to use multiple querys and then use a Grid Reader. That might work for you.
There's the example taken from the dapper age:
var sql =
#"
select * from Customers where CustomerId = #id
select * from Orders where CustomerId = #id
select * from Returns where CustomerId = #id";
using (var multi = connection.QueryMultiple(sql, new {id=selectedId}))
{
var customer = multi.Read<Customer>().Single();
var orders = multi.Read<Order>().ToList();
var returns = multi.Read<Return>().ToList();
...
}

This has been answered long time ago, but I would like to add my two cents here. Instead of manually modify Dapper's source code, why don't you just create a poco class with those fields and use your query like a table?
The mapping would work fine, I know it is a pain also to do that class definition, but seems easier than dealing with later Dapper's updates.

C# : Passing Parameters to a function via a 2d object array

I'm trying to write a neat little generic Sql method that will accept a SQL Query and a list of parameters, and return a result. I want to keep it neat enough that I can call it using one line from any other code.
Is there any really awesomely neat way of doing this? I don't want to create all the SqlParameters in the calling code, and I don't want to have to pass and split a string. In the past I've used a string[] array and accepted every odd member as a parameter name and every even as a param value but that's too easy to screw up when calling the method.
Ideally I'd love to do just this:
Data.SQL("Select * from Table where my_id = #my_id", { my_id = 1 });
I know that's a little unrealistic, So I tried this:
Data.SQL("Select * from Table where my_id = #my_id", new Object[,]{ { "my_id", 1 } });
However when I try and handle that on the other end, I get nothing but trouble:
public static Object SQL(String command, Object[,] parameters = null){
[ ... reusable SQL code here... ]
foreach(Object[] p in parameters){
cmd.Parameters.Add(new SqlParameter(p[0].ToString(), p[1].ToString());
}
}
Looks fine, but throws an error on the foreach statement
foreach (Object[] p in parameters)
Unable to cast object of type 'System.String' to type 'System.Object[]'
But I didn't pass it an array of System.String. What I passed was a 2D System.Object[]! Wasn't it?
Maybe this is just some small code problem, something stupid I'm doing wrong. It usually is. But I'm figuring you guys know some even neater way to do the above.

Ideally I'd love to do just this:
Data.SQL("Select * from Table where my_id = #my_id", { my_id = 1 });
I know that's a little unrealistic,
Well, in exactly that form, yes... but try this instead:
Data.SQL("Select * from Table where my_id = #my_id", new { my_id = 1 });
That will use an anonymous type for the argument, which you can examine by reflection. You probably only need a single parameter (i.e. it would be SQL(string sql, object parameters)) because you would pass multiple parameters in a single object:
Data.SQL("Select * from Table where my_id = #my_id and name = #name",
new { my_id = 1, name = "Jon" });
More alternatives:
If you're using C# 4, you might find dynamic typing useful; look at what Massive does for example.
As mentioned by Ray, you could pass in a Dictionary<string, object>; again, C# 3 makes this easier than otherwise:
Data.SQL("...", new Dictionary<string, object> {
{ "my_id", 1 },
{ "name", "Jon" }});
EDIT: As for the exact problem you're running into: you need to understand the difference between a rectangular array (e.g. Object[,]) and a jagged array (e.g. Object[][]). The latter is an array of arrays, which is how you're trying to use the parameter, but it's really only a rectangular array. Changing your parameter type to Object[][] may well fix that immediate problem - but personally I'd move to one of the approaches above. I'd also try to avoid making everything into a string, by the way.

RavenDb store and retrieve class-less objects (json)

I'm writing a application where the user can write json-code and store that json code with an Id and a Collection. In other words, they specify an Id, a Collection (string; [a-zA-Z0-9]) and a Data (json, can be anything that is valid json).
Up til now I've been using RavenDb for this, cause I thought a document-database would be perfect, but I've had some problems with querying.
One of the objects that needs to be stored and queried is the following:
{
"network": "some_network",
"names": ["name1","name2"],
"data": {"values":[],"keys":[]}
}
This object should be stored with some Id that is either specified, or auto-generated (if null is given), and a Collection (must always be specified), and then I need to be able to query it based on Collection, network and a single name.
For instance, I have the code query('users', '{"network":"some_network","names":"name1"}'), and I need that code to return this object (and any other object that matches it).
Also, I'm ok with changing database, but the database needs to be able to run in-process (self-hosted), and to be able to run without admin-rights without installation (in other words, it can't bind to hostname/ip like wcf does).
How can I achieve something like this?

I found the answer to this:
public string Query(string dynIndexName, string query)
{
using (var session = store.OpenSession())
{
var q = new IndexQuery();
q.Query = query;
var result = session.Advanced.DatabaseCommands.Query("dynamic/" + dynIndexName, q, new string[0]);
return "[" + String.Join(",", result.Results.Select(r => r.ToString())) + "]";
}
}
Before calling the Query-method I convert the json-query-object into a Lucene-query that looks like this: (network:"some_network" AND names:"name1"), and use that as a query-parameter to the database. The whole class for storing and retrieving can be found here: https://github.com/Alxandr/RunJS/blob/master/src/AddIns/Storage/StorageMbro.cs