My Xamarin app pulls data from an API and inserts that data into a SQLite table. The API currently has 9 tables defined, and as such there are 9 classes in my app that match those tables. I used the code snippet from this question's accepted answer:
Getting all types in a namespace via reflection
Below is my code, using the snippet from the answer and the foreach loop I'm trying to build that'll insert the data.
string nspace = "App.Tables";
var q = from t in Assembly.GetExecutingAssembly().GetTypes()
where t.IsClass && t.Namespace == nspace
select t.Name; // Getting list of classes as IEnumerable
var L = q.ToList(); // Converting to List
foreach (var name in L) // Inserts data for every class found
{
var response = await httpClient.Value.GetStringAsync("http://website.com/api/" + name + "s"); // Substitutes each class name into API url
var data = JsonConvert.DeserializeObject<List<TableName>>(response); // Deserializes into List
using (SQLiteConnection conn = new SQLiteConnection(App.DatabaseLocation)) // Opens database connection
{
conn.CreateTable<TableName>(); // Create table
conn.DeleteAll<TableName>(); // Delete all old data
conn.InsertAll(data); // Inserts new data
}
}
I don't know what TableName should be in order to get the correct class for each item on the list. For example: say the list contained the strings Table1, Table2, and Table3 - it got those strings from the App.Tables namespace which contains three separate classes called Table1, Table2, and Table3. When the first part of the code gets the list and stores it as the variable L, it should get the data for each "name" variable in L and then insert it into the matching table. How do I refer it to the table?
Before I would give my answer, I would like to tell you that I do not
recommend updating tables via reflection - tables should contain
logically distinct entities, so batch deleting and updating them is
kinda weird. This is one of those few occurrences where I would never
work around typing the updates of the tables one by one. But those are
just my two cents... Of course if I had a thousand tables my opinion
would not stand.
This kind of reflection also makes your code hard to follow and trace - think about how you will search for usages of the CreateTable<ExampleClass>() method? You will never trace it back - or only via great efforts - that you called it in this piece of code.
So to answer your question...
You first get the method group, then create a generic version of it based on the type. I think converting from Type to string is unnecessary for the part you're looking for, since you need to convert back to Type
using (SQLiteConnection conn = new SQLiteConnection(App.DatabaseLocation)) {
MethodInfo nonGenMethodCreate = typeof(SQLiteConnection).GetMethod("CreateTable");
MethodInfo nonGenMethodDeleteAll = typeof(SQLiteConnection).GetMethod("DeleteAll");
MethodInfo nonGenMethodInsertAll = typeof(SQLiteConnection).GetMethod("InsertAll");
foreach(Type t in Assembly.GetExecutingAssembly.GetTypes()) {
MethodInfo genMethodCreate = nonGenMethodCreate.MakeGenericMethod(t);
MethodInfo genMethodDeleteAll = nonGenMethodDeleteAll.MakeGenericMethod(t);
MethodInfo genMethodInsertAll = nonGenMethodInsertAll.MakeGenericMethod(t);
genMethodCreate.Invoke(conn, null);
genMethodDeleteAll.Invoke(conn, null);
genMethodInsertAll.Invoke(conn, new[] { data });
}
}
Related
I'm having some troubles selecting a Tuple of Objects from my custom SQL query with ORMLite.
I have the following code:
var query = "select definition.*, timeslot.*, type.* from <blah blah>";
var defs = dbConnection.SqlList<Tuple<Definition, Timeslot, Type>>(query, new
{
/* query parameters */
});
The query itself is fine (I've tested it in SQL Management Studio).
The code above sets attributes only for the first item of the Tuple, leaving to the default state the others.
I've selected singularly each object and the result is correct (so no trouble during the conversion to POCO I guess).
Same thing goes if I use Select<Tuple<Definition, Timeslot, Type>> instead of SqlList.
I couldn't manage to try with MultiSelect since it appears to not take a string.
What is the correct way to select a Tuple in this manner?
I am working in C#.
Thanks in advance!
SelectMulti seems to be what you're looking for here.
From the documentation under the Selecting multiple columns across joined tables heading:
// Note: I'm making an assumption on your query here.
// Build the `q` object however it needs to be.
var q = db.From<Definition>()
.Join<Definition, Timeslot>()
.Join<Definition, Type>();
var results = db.SelectMulti<Definition, Timeslot, Type>(q);
foreach (var tuple in results)
{
var definition = tuple.Item1;
var timeslot = tuple.Item2;
var type = tuple.Item3;
}
I'm using entity framework to connect with the database. I've one little problem:
I've one table which have one varbinary(MAX) column(with filestream).
I'm using SQL request to manage the "Data" part, but EF for the rest(metadata of the file).
I've one code which has to get all files id, filename, guid, modification date, ... of a file. This doesn't need at all the "Data" field.
Is there a way to retrieve a List but without this column filled?
Something like
context.Files.Where(f=>f.xyz).Exclude(f=>f.Data).ToList();
??
I know I can create anonymous objects, but I need to transmit the result to a method, so no anonymous methods. And I don't want to put this in a list of anonymous type, and then create a list of my non-anonymous type(File).
The goal is to avoid this:
using(RsSolutionsEntities context = new RsSolutionsEntities())
{
var file = context.Files
.Where(f => f.Id == idFile)
.Select(f => new {
f.Id, f.MimeType, f.Size, f.FileName, f.DataType,
f.DateModification, f.FileId
}).FirstOrDefault();
return new File() {
DataType = file.DataType, DateModification = file.DateModification,
FileId = file.FileId, FileName = file.FileName, Id = file.Id,
MimeType = file.MimeType, Size = file.Size
};
}
(I'm using here the anonymous type because otherwise you will get a NotSupportedException: The entity or complex type 'ProjectName.File' cannot be constructed in a LINQ to Entities query.)
(e.g. this code throw the previous exception:
File file2 = context.Files.Where(f => f.Id == idFile)
.Select(f => new File() {Id = f.Id, DataType = f.DataType}).FirstOrDefault();
and "File" is the type I get with a context.Files.ToList(). This is the good class:
using File = MyProjectNamespace.Common.Data.DataModel.File;
File is a known class of my EF datacontext:
public ObjectSet<File> Files
{
get { return _files ?? (_files = CreateObjectSet<File>("Files")); }
}
private ObjectSet<File> _files;
Is there a way to retrieve a List but without this column filled?
Not without projection which you want to avoid. If the column is mapped it is natural part of your entity. Entity without this column is not complete - it is different data set = projection.
I'm using here the anonymous type because otherwise you will get a
NotSupportedException: The entity or complex type 'ProjectName.File'
cannot be constructed in a LINQ to Entities query.
As exception says you cannot project to mapped entity. I mentioned reason above - projection make different data set and EF don't like "partial entities".
Error 16 Error 3023: Problem in mapping fragments starting at line
2717:Column Files.Data in table Files must be mapped: It has no
default value and is not nullable.
It is not enough to delete property from designer. You must open EDMX as XML and delete column from SSDL as well which will make your model very fragile (each update from database will put your column back). If you don't want to map the column you should use database view without the column and map the view instead of the table but you will not be able to insert data.
As a workaround to all your problems use table splitting and separate the problematic binary column to another entity with 1 : 1 relation to your main File entity.
I'd do something like this:
var result = from thing in dbContext.Things
select new Thing {
PropertyA = thing.PropertyA,
Another = thing.Another
// and so on, skipping the VarBinary(MAX) property
};
Where Thing is your entity that EF knows how to materialize. The resulting SQL statement shouldn't include the large column in its result set, since it's not needed in the query.
EDIT: From your edits, you get the error NotSupportedException: The entity or complex type 'ProjectName.File' cannot be constructed in a LINQ to Entities query. because you haven't mapped that class as an entity. You can't include objects in LINQ to Entities queries that EF doesn't know about and expect it to generate appropriate SQL statements.
You can map another type that excludes the VarBinary(MAX) column in its definition or use the code above.
you can do this:
var files = dbContext.Database.SqlQuery<File>("select FileId, DataType, MimeType from Files");
or this:
var files = objectContext.ExecuteStoreQuery<File>("select FileId, DataType, MimeType from Files");
depending on your version of EF
I had this requirement because I have a Document entity which has a Content field with the content of the file, i.e. some 100MB in size, and I have a Search function that I wanted to return the rest of the columns.
I chose to use projection:
IQueryable<Document> results = dbContext.Documents.Include(o => o.UploadedBy).Select(o => new {
Content = (string)null,
ContentType = o.ContentType,
DocumentTypeId = o.DocumentTypeId,
FileName = o.FileName,
Id = o.Id,
// etc. even with related entities here like:
UploadedBy = o.UploadedBy
});
Then my WebApi controller passes this results object to a common Pagination function, which applies a .Skip, .Take and a .ToList.
This means that when the query gets executed, it doesn't access the Content column, so the 100MB data is not being touched, and the query is as fast as you'd want/expect it to be.
Next, I cast it back to my DTO class, which in this case is pretty much exactly the same as the entity class, so this might not be a step you need to implement, but it's follows my typical WebApi coding pattern, so:
var dtos = paginated.Select(o => new DocumentDTO
{
Content = o.Content,
ContentType = o.ContentType,
DocumentTypeId = o.DocumentTypeId,
FileName = o.FileName,
Id = o.Id,
UploadedBy = o.UploadedBy == null ? null : ModelFactory.Create(o.UploadedBy)
});
Then I return the DTO list:
return Ok(dtos);
So it uses projection, which might not fit the original poster's requirements, but if you're using DTO classes, you're converting anyway. You could just as easily do the following to return them as your actual entities:
var dtos = paginated.Select(o => new Document
{
Content = o.Content,
ContentType = o.ContentType,
DocumentTypeId = o.DocumentTypeId,
//...
Just a few extra steps but this is working nicely for me.
For EF Core 2
I implemented a solution like this:
var files = context.Files.AsNoTracking()
.IgnoreProperty(f => f.Report)
.ToList();
The base idea is to turn for example this query:
SELECT [f].[Id], [f].[Report], [f].[CreationDate]
FROM [File] AS [f]
into this:
SELECT [f].[Id], '' as [Report], [f].[CreationDate]
FROM [File] AS [f]
you can see the full source code in here:
https://github.com/aspnet/EntityFrameworkCore/issues/1387#issuecomment-495630292
I'd like to share my attempts to workaround this problem in case somebody else is in the same situation.
I started with what Jeremy Danyow suggested, which to me is the less painful option.
// You need to include all fields in the query, just make null the ones you don't want.
var results = context.Database.SqlQuery<myEntity>("SELECT Field1, Field2, Field3, HugeField4 = NULL, Field5 FROM TableName");
In my case, I needed a IQueryable<> result object so I added AsQueryable() at the end. This of course let me add calls to .Where, .Take, and the other commands we all know, and they worked fine. But there's a caveat:
The normal code (basically context.myEntity.AsQueryable()) returned a System.Data.Entity.DbSet<Data.DataModel.myEntity>, while this approach returned System.Linq.EnumerableQuery<Data.DataModel.myEntity>.
Apparently this means that my custom query gets executed "as is" as soon as needed and the filtering I added later is done afterwards and not in the database.
Therefore I tried to mimic Entity Framework's object by using the exact query EF creates, even with those [Extent1] aliases, but it didn't work. When analyzing the resulting object, its query ended like
FROM [dbo].[TableName] AS [Extent1].Where(c => ...
instead of the expected
FROM [dbo].[TableName] AS [Extent1] WHERE ([Extent1]...
Anyway, this works, and as long as the table is not huge, this method will be fast enough. Otherwise you have no option than to manually add the conditions by concatenating strings, like classic dynamic SQL. A very basic example in case you don't know what I'm talking about:
string query = "SELECT Field1, Field2, Field3, HugeField4 = NULL, Field5 FROM TableName";
if (parameterId.HasValue)
query += " WHERE Field1 = " + parameterId.Value.ToString();
var results = context.Database.SqlQuery<myEntity>(query);
In case your method sometimes needs this field, you can add a bool parameter and then do something like this:
IQueryable<myEntity> results;
if (excludeBigData)
results = context.Database.SqlQuery<myEntity>("SELECT Field1, Field2, Field3, HugeField4 = NULL, Field5 FROM TableName").AsQueryable();
else
results = context.myEntity.AsQueryable();
If anyone manages to make the Linq extensions work properly like if it was the original EF object, please comment so I can update the answer.
I'm using here the anonymous type because otherwise you will get a
NotSupportedException: The entity or complex type 'ProjectName.File'
cannot be constructed in a LINQ to Entities query.
var file = context.Files
.Where(f => f.Id == idFile)
.FirstOrDefault() // You need to exeucte the query if you want to reuse the type
.Select(f => new {
f.Id, f.MimeType, f.Size, f.FileName, f.DataType,
f.DateModification, f.FileId
}).FirstOrDefault();
And also its not a bad practice to de-normalize the table into further, i.e one with metadata and one with payload to avoid projection. Projection would work, the only issue is, need to edit any time a new column is added to the table.
I tried this:
From the edmx diagram (EF 6), I clicked the column I wanted to hide from EF and on their properties you can set their getter and setter to private. That way, for me it works.
I return some data which includes a User reference, so I wanted to hide the Password field even though it's encrypted and salted, I just didn't want it on my json, and I didn't want to do a:
Select(col => new {})
because that's a pain to create and maintain, especially for big tables with a lot of relationships.
The downside with this method is that if you ever regenerate your model, you would need to modify their getter and setter again.
Using Entity Framework Power Tools you can do the following in efpt.config.json:
"Tables": [
{
"ExcludedColumns": [
"FileData"
],
"Name": "[dbo].[Attachment]",
"ObjectType": 0
}
]
C# rookie. Below is my code, been trying for hours now to get this to update some fields in my DB and tried many different implementations without luck.
// Select all fields to update
using (var db = new Entities())
{
// dbFields are trusted values
var query = db.tblRecords
.Where("id == " + f.id)
.Select("new(" + string.Join(",", dbFields.Keys) + ")");
foreach (var item in query)
{
foreach (PropertyInfo property in query.ElementType.GetProperties())
{
if (dbFields.ContainsKey(property.Name))
{
// Set the value to view in debugger - should be dynamic cast eventually
var value = Convert.ToInt16(dbFields[property.Name]);
property.SetValue(item, value);
// Something like this throws error 'Object does not match target type'
// property.SetValue(query, item);
}
}
}
db.SaveChanges();
}
The above code when run does not result in any changes to the DB. Obviously this code needs a bit of cleanup but i'm trying to get the basic functionality working. I believe what I might need to do is to somehow reapply 'item' back into 'query' but I've had no luck getting that to work no matter what implementation I try i'm always receiving 'Object does not match target type'.
This semi similar issue reaffirms that but isn't very clear to me since i'm using a Dynamic LINQ query and cannot just reference the property names directly. https://stackoverflow.com/a/25898203/3333134
Entity Framework will perform updates for you on entities, not on custom results. Your tblRecords holds many entities, and this is what you want to manipulate if you want Entity Framework to help. Remove your projection (the call to Select) and the query will return the objects directly (with too many columns, yes, but we'll cover that later).
The dynamic update is performed the same way any other dynamic assignment in C# would be, since you got a normal object to work with. Entity Framework will track the changes you make and, upon calling SaveChanges, will generate and execute the corresponding SQL queries.
However, if you want to optimize and stop selecting and creating all the values in memory in the first place, even those that aren't needed, you could also perform the update from memory. If you create an object of the right type by yourself and assign the right ID, you can then use the Attach() method to add it to the current context. From that point on, any changes will be recorded by Entity Framework, and when you call SaveChanges, everything should be sent to the database :
// Select all fields to update
using (var db = new Entities())
{
// Assuming the entity contained in tblRecords is named "ObjRecord"
// Also assuming that the entity has a key named "id"
var objToUpdate = new ObjRecord { id = f.id };
// Any changes made to the object so far won't be considered by EF
// Attach the object to the context
db.tblRecords.Attach(objToUpdate);
// EF now tracks the object, any new changes will be applied
foreach (PropertyInfo property in typeof(ObjRecord).GetProperties())
{
if (dbFields.ContainsKey(property.Name))
{
// Set the value to view in debugger - should be dynamic cast eventually
var value = Convert.ToInt16(dbFields[property.Name]);
property.SetValue(objToUpdate, value);
}
}
// Will only perform an UPDATE query, no SELECT at all
db.SaveChanges();
}
When you do a SELECT NEW ... it selects only specific fields and won't track updates for you. I think if you change your query to be this it will work:
var query = db.tblRecords.Where(x=>x.id == id);
I don't think is possible but wanted to ask to make sure. I am currently debugging some software someone else wrote and its a bit unfinished.
One part of the software is a search function which searches by different fields in the database and the person who wrote the software wrote a great big case statement with 21 cases in it 1 for each field the user may want to search by.
Is it possible to reduce this down using a case statement within the Linq or a variable I can set with a case statement before the Linq statement?
Example of 1 of the Linq queries: (Only the Where is changing in each query)
var list = (from data in dc.MemberDetails
where data.JoinDate.ToString() == searchField
select new
{
data.MemberID,
data.FirstName,
data.Surname,
data.Street,
data.City,
data.County,
data.Postcode,
data.MembershipCategory,
data.Paid,
data.ToPay
}
).ToList();
Update / Edit:
This is what comes before the case statement:
string searchField = txt1stSearchTerm.Text;
string searchColumn = cmbFirstColumn.Text;
switch (cmbFirstColumn.SelectedIndex + 1)
{
The cases are then done by the index of the combo box which holds the list of field names.
Given that where takes a predicate, you can pass any method or function which takes MemberDetail as a parameter and returns a boolean, then migrate the switch statement inside.
private bool IsMatch(MemberDetail detail)
{
// The comparison goes here.
}
var list = (from data in dc.MemberDetails
where data => this.IsMatch(data)
select new
{
data.MemberID,
data.FirstName,
data.Surname,
data.Street,
data.City,
data.County,
data.Postcode,
data.MembershipCategory,
data.Paid,
data.ToPay
}
).ToList();
Note that:
You may look for a more object-oriented way to do the comparison, rather than using a huge switch block.
An anonymous type with ten properties that you use in your select is kinda weird. Can't you return an instance of MemberDetail? Or an instance of its base class?
How are the different where statements handled, are they mutually excluside or do they all limit the query somehow?
Here is how you can have one or more filters for a same query and materialized after all filters have been applied.
var query = (from data in dc.MemberDetails
select ....);
if (!String.IsNullOrEmpty(searchField))
query = query.Where(pr => pr.JoinDate.ToString() == searchField);
if (!String.IsNullOrEmpty(otherField))
query = query.Where(....);
return query.ToList();
UPDATE 18 Sep 2013
It looks like there isn't an easy way to do this. I'm holding out for a solution that involves some extension to Entity Framework.
If you'd like to see these features in Entity Framework, vote for them on the user voice site, perhaps here and here
There are several similar questions on SO but I can't find a question new and similar enough to have the answer I'm looking for.
If this looks like information overload, jump down to In Summary.
Background
I'm writing a WebApi REST service to expose some pre-existing data through an OData end point. I'm using the EntitySetContoller<TEntity, TKey> to do all the grunt work for me. As well as the standard OData parameters, that are routed and translated by the base class, I've added some custom parameters, to allow specific functionality for my controller.
My database server is MS SQL Server with a full text index on the [BigText] NVarChar[4000] column of the [SomeEntity] table.
I have one limitation, I must use a Code First model.
// Model POCO
public class SomeEntity
{
public int Id { get; set; }
public string BigText { get; set; }
}
// Simple Controller
public class SomeEntityController : EntitySetController<SomeEntity, int>
{
private readonly SomeDbContext context = new SomeDbContext();
public override IQueryable<SomeEntity> Get()
{
var parameters = Request.GetQueryNameValuePairs()
.ToDictionary(p => p.Key, p => p.Value);
if (parameters.ContainsKey("BigTextContains")
(
var searchTerms = parameters["BigTextContains"];
// return something special ...
)
return this.context.SomeEntities;
}
// ... The rest is omitted for brevity.
}
The Problem
How to implement the // return something special ... part of my example?
Obviously, the niave
return this.context.SomeEntities.Where(e =>
e.BigText.Contains(searchTerm));
is completely wrong, it composes to a WHERE clause like
[BigText] LIKE '%' + #searchTerm + '%'
This doesn't use Full Text Searching so, doesn't support complex search terms and otherwise, performs terribley.
This approach,
return this.context.SomeEntities.SqlQuery(
"SELECT E.* FROM [dbo].[SomeEntity] E " +
"JOIN CONTAINSTABLE([SomeEntity], [BigText], #searchTerm) FTS " +
" ON FTS.[Key] = E.[Id]",
new object[] { new SqlParameter("#searchTerm", searchTerm) })
.AsQueryable();
Looks promising, it actually uses Full Text Searching, and is quite functional. However, you'll note that DbSqlQuery, the type returned from the SqlQuery function does not implement IQueryable. Here, it is coerced to the right return type with the AsQueryable() extension but, this breaks the "chain of composition". The only statement that will be performed on the server is the one specified in the code above. Any additional clauses, specified on the OData URL will be serviced on the API hosting web server, without benefitting from the indices and specialised set based functionality of the database engine.
In Summary
What is the most expedient way of accessing MS SQL Server's Full Text Search CONTAINSTABLE function with an Entity Framework 5 Code First model and acquiring a "composable" result?
Do I need to write my own IQueryProvider? Can I extend EF in some way?
I don't want to use Lucene.Net, I don't want to use a Database Generated Model. Perhaps I could add extra packages or wait for EF6, would that help?
It is not perfect, but you can accomplish what you are after with 2 calls to the database.
The first call would retrieve a list of matching key's from CONTAINSTABLE and then the second call would be your composable query utilizing the IDs that you returned from the first call.
//Get the Keys from the FTS
var ids = context.Database.SqlQuery<int>(
"Select [KEY] from CONTAINSTABLE([SomeEntity], [BigText], #searchTerm)",
new object[] { new SqlParameter("#searchTerm", searchTerm) });
//Use the IDs as an initial filter on the query
var composablequery = context.SomeEntities.Where(d => ids.Contains(d.Id));
//add on whatever other parameters were captured to the 'composablequery' variable
composablequery = composablequery.Where(.....)
I had this same issue recently:
EF 5 Code First FTS Queriable
Let me extend that post.
Your first option was mine first as well - using SqlQuery
I also needed to do more filtering, so instead of always writing full sql I used QueryBuilder, to which I made some changes and added more functions to fit my needs(I could upload it somewhere if needed):
QueryBuilder
After I have found another idea which I implemented.
Someone already mention it here, and that is to use SqlQuery that will return HashSet of Ids and that you can use it in EF queries with Contains.
This is better but not most optimal since you need 2 queries and Id list in memory.
Example:
public IQueryable<Company> FullTextSearchCompaniesByName(int limit, int offset, string input, Guid accountingBureauId, string orderByColumn)
{
FtsQueryBuilder ftsQueryBuilder = new FtsQueryBuilder();
ftsQueryBuilder.Input = FtsQueryBuilder.FormatQuery(input);
ftsQueryBuilder.TableName = FtsQueryBuilder.GetTableName<Company>();
ftsQueryBuilder.OrderByTable = ftsQueryBuilder.TableName;
ftsQueryBuilder.OrderByColumn = orderByColumn;
ftsQueryBuilder.Columns.Add("CompanyId");
if (accountingBureauId != null && accountingBureauId != Guid.Empty)
ftsQueryBuilder.AddConditionQuery<Guid>(Condition.And, "" , #"dbo.""Company"".""AccountingBureauId""", Operator.Equals, accountingBureauId, "AccountingBureauId", "");
ftsQueryBuilder.AddConditionQuery<bool>(Condition.And, "", #"dbo.""Company"".""Deleted""", Operator.Equals, false, "Deleted", "");
var companiesQuery = ftsQueryBuilder.BuildAndExecuteFtsQuery<Guid>(Context, limit, offset, "Name");
TotalCountQuery = ftsQueryBuilder.Total;
HashSet<Guid> companiesIdSet = new HashSet<Guid>(companiesQuery);
var q = Query().Where(a => companiesIdSet.Contains(a.CompanyId));
return q;
}
However EF 6 now has something called Interceptors that can be used to implement queriable FTS, and it is pretty simple and generic(last post):
EF 6 Interceptors for FTS.
I have tested this and it works fine.
!! REMARK: EF Code First, even with version 6, does not support Custom Stored Procedures.
There are only some for predefined CUD operations if I understood it well:
Code First Insert/Update/Delete Stored Procedure Mapping, so it can't be done with it.
Conclusion: if you can use EF 6 go for third options, is gives all you need.
If you are stucked with EF 5 or less, second option is better then first but not most optimal.