If I want to increase the value in a table column with SQL I can do it as follows:
UPDATE mytable m SET m.mycolumn = m.mycolumn + 1;
This is great because it doesn't rely on being executed in any order and uses an absolute minimum of locking.
How can this be done using C# and Entity Framework, with the same (or as close as possible) minimal overhead?
In short, you can't. You can execute a sql statement through the ado.net properties available on the DbContext.Database. You could also created a stored procedure if this is a specific action that occurs where the stored proc does this. That is the closest you will get to this.
This answer assumes you already know how to retrieve all the records, iterate over them to change the value of the property, and then persist the changes using SaveChanges on the DbContext and that this is not what you are looking for as it generates 1 update statement per record.
As already said, it's not possible in Pure EF.
As always you're free to create your pure SQL statements and run them through your context.
The closest thing to your question is to use Entity Framework Extensions. The have an UpdateFromQuery method: https://entityframework-extensions.net/update-from-query
This would look like this:
context.mytable.UpdateFromQuery(x => new mytable() {mycolumn = x.mycolumn + 1});
You can use EF Core extension linq2db.EntityFrameworkCore (disclaimer: I'm one of the creators)
context.mytable
.Set(x => x.mycolumn, x => x.mycolumn + 1)
.Update();
Library has it's own LINQ Translator which eliminates a lot of EF Core limitations.
dbContext.Entity.ForEach(x -> x.property = value);
or
dbContext.Entity.Select(x -> {x.property =100; return c;}).ToList();
See Using LINQ to Update A Property in a List of Entities
Related
I am currently using EF Extensions. One thing I don't understand, "its supposed to help with performance"
however placing a million+ records into List variable, is a Memory Issue itself.
So If wanting to update million records, without holding everything in memory, how can this be done efficiently?
Should we use a for loop, and update in batches say 10,000? Does EFExtensions BulkUpdate have any native functionality to support this?
Example:
var productUpdate = _dbContext.Set<Product>()
.Where(x => x.ProductType == 'Electronics'); // this creates IQueryable
await productUpdate.ForEachAsync(c => c.ProductBrand = 'ABC Company');
_dbContext.BulkUpdateAsync(productUpdate.ToList());
Resource:
https://entityframework-extensions.net/bulk-update
This is actually something that EF is not made for. EF's database interactions start from the record object, and flow from there. EF cannot generate a partial UPDATE (i.e. not overwriting everything) if the entity wasn't change tracked (and therefore loaded), and similarly it cannot DELETE records based on a condition instead of a key.
There is no EF equivalent (without loading all of those records) for conditional update/delete logic such as
UPDATE People
SET FirstName = 'Bob'
WHERE FirstName = 'Robert'
or
DELETE FROM People
WHERE FirstName = 'Robert'
Doing this using the EF approach will require you to load all of these entities just to send them back (with an update or delete) to the database, and that's a waste of bandwidth and performance as you've already found.
The best solution I've found here is to bypass EF's LINQ-friendly methods and instead execute the raw SQL yourself. This can still be done using an EF context.
using (var ctx = new MyContext())
{
string updateCommand = "UPDATE People SET FirstName = 'Bob' WHERE FirstName = 'Robert'";
int noOfRowsUpdated = ctx.Database.ExecuteSqlCommand(updateCommand);
string deleteCommand = "DELETE FROM People WHERE FirstName = 'Robert'";
int noOfRowsDeleted = ctx.Database.ExecuteSqlCommand(deleteCommand);
}
More information here. Of course don't forget to protect against SQL injection where relevant.
The specific syntax to run raw SQL may vary per version of EF/EF Core but as far as I'm aware all versions allow you to execute raw SQL.
I can't comment on the performance of EF Extensions or BulkUpdate specifically, and I'm not going to buy it from them.
Based on their documentation, they don't seem to have the methods with the right signatures to allow for conditional update/delete logic.
BulkUpdate doesn't seem to allow you to input the logical condition (the WHERE in your UPDATE command) that would allow you to optimize this.
BulkDelete still has a BatchSize setting, which suggests that they are still handling the records one at a time (well, per batch I guess), and not using a single DELETE query with a condition (WHERE clause).
Based on your intended code in the question, EF Extensions isn't really giving you what you need. It's more performant and cheaper to simply execute raw SQL on the database, as this bypasses EF's need to load its entities.
Update
I might stand corrected, there is some support for conditional update logic, as seen here. However, it is unclear to me while the example still loads everything in memory and what the purpose of that conditional WHERE logic then is if you've already loaded it all in memory (why not use in-memory LINQ then?)
However, even if this works without loading the entities, it's still:
more limited (only equality checks are allowed, compared to SQL allowing any boolean condition that is valid SQL),
relatively complex (I don't like their syntax, maybe that's subjective)
and more costly (still a paid library)
compared to rolling your own raw SQL query. I would still suggest rolling your own raw SQL here, but that's just my opinion.
I found the "proper" EF Extensions way to do a bulk update with a query-like condition:
var productUpdate = _dbContext.Set<Product>()
.Where(x => x.ProductType == 'Electronics')
.UpdateFromQuery( x => new Product { ProductBrand = "ABC Company" });
This should result in a proper SQL UPDATE ... SET ... WHERE, without the need to load entities first, as per the documentation:
Why UpdateFromQuery is faster than SaveChanges, BulkSaveChanges, and BulkUpdate?
UpdateFromQuery executes a statement directly in SQL such as UPDATE [TableName] SET [SetColumnsAndValues] WHERE [Key].
Other operations normally require one or multiple database round-trips which makes the performance slower.
You can check the working syntax on this dotnet fiddle example, adapted from their example of BulkUpdate.
Other considerations
No mention of batch operations for this, unfortunately.
Before doing a big update like this, it might be worth considering deactivating indexes you may have on this column, and rebuild them afterward. This is especially useful if you have many of them.
Careful about the condition in the Where, if it can't be translated as SQL by EF, then it will be done client side, meaning the "usual" terrible roundtrip "Load - change in memory - update"
I created my temporal table following instructions on this link "Cutting Edge - Soft Updates with Temporal Tables"
The procedure generally takes two steps:
create the regular table; this is just the kind of work that Code First usually does
alter the table by adding SysStartTime, SysEndTime columns and turning on the SYSTEM_VERSIONING setting
Everything looks good so far.
However, please note that the entity class doesn't have SysStartTime property and SysEndTime property as they are added later. This gives me trouble as I need to get information of SysStartTime from table.
My question is: How can I get it with Entity Framework?
The link also says:
In EF 6, you can only leverage the SqlQuery method of the DbSet class:
using (var db = new EF6Context())
{
var current = db.Bookings.Single(b => b.Id == 1);
var time = DateTime.Now.AddMinutes(-5);
var old = db.Bookings
.SqlQuery("SELECT * FROM dbo.Bookings
FOR SYSTEM_TIME AS OF {0} WHERE Id = 1", time)
.SingleOrDefault();
}
Note that for EF 6, the column names returned in the query need to match the property names on the class. This is because SqlQuery doesn’t use mappings. If column and property names don’t match, then you’d need to alias the columns in the SELECT list, rather than just SELECT *.
I have no clue how to do this or if it solves my problem. Does anyone have any experience on this?
I can think of one solution is to added an extra column AppliedTime on my table by adding AppliedTime to my entity class. It has (almost, but good enough for me) same value as SysStartTime.
Another solution could be use plain SQL to query the table directly.
Thanks.
I have been worked on this one and found a solution. It is actually quite simple. Just use Database.SqlQuery Method
DateTime time = context.Database.SqlQuery<DateTime>("SELECT SysStartTime
FROM dbo.Calibration
WHERE CalibrationID = 1")
.SingleOrDefault();
See Also: Raw SQL Queries
I have few related objects and relation is like
public class Project
{
public List<ProjectEdition> editions;
}
public class ProjectEdition
{
public List<EditionItem> items;
}
public class EditionItem
{
}
I wanted to fetch the EditionItems from Last entries of ProjectEditions only for each Project
Example
Project#1 -> Edition#1 [contains few edition items ] , Edition#2 [contains few edition items]
Project#2 -> Edition#1 ,Edition#2 and Edition#3
My required output contains EditionItems from Edition#2 of Project#1 and Edition#3 of Project#2 only . I mean EditionItems from latest edition of a Project or last edition of a Project only
To get this i tried this query
List<EditionItem> master_list = context.Projects.Select(x => x.ProjectEditions.LastOrDefault())
.SelectMany(x => x.EditionItems).ToList();
But its returns error at LatsOrDefault() section
An exception of type 'System.NotSupportedException' occurred in EntityFramework.SqlServer.dll but was not handled in user code
Additional information: LINQ to Entities does not recognize the method '---------.Models.ProjectEdition LastOrDefault[ProjectEdition](System.Collections.Generic.IEnumerable`1
so how can i filter for last edition of a project and then get the list of EditionItems from it in a single LINQ call
Granit got the answer right, so I won't repeat his code. I would like to add the reasons for this behaviour.
Entity Framework is magic (sometimes too much magic) but it yet translates your LINQ queries into SQL and there are limitations to that of what your underlying database can do (SQL Server in this case).
When you call context.Projects.FirstOrDefault() it is translated into something like Select TOP 1 * from Projects. Note the TOP 1 part - this is SQL Server operator that limits number of rows returned. This is part of query optimisation in SQL Server. SQL Server does not have any operators that will give you LAST 1 - because it needs to run the query, return all the results, take the last one and dump the rest - this is not very efficient, think of a table with a couple (bi)million records.
So you need to apply whatever required sort order to your query and limit number of rows you return. If you need last record from the query - apply reverse sort order. You do need to sort because SQL Server does not guarantee order of records returned if no Order By is applied to the query - this is due to the way the data is stored internally.
When you write LINQ queries with EF I do recommend keep an eye on what SQL is generated by your queries - sometimes you'll see how complex they come out and you can easily simplify the query. And sometimes with lazy-loading enabled you introduce N+1 problem with a stroke of a key (literally). I use ExpressProfiler to watch generated SQL, LinqPad can also show you the SQL queries and there are other tools.
You cannot use method LastOrDefault() or Last() as discussed here.
Insetad, you can use OrderByDescending() in conjunction with FirstOrDefault() but first you need to have a property in you ProjectEdition with which you want to order the entities. E.g. if ProjectEdition has a property Id (which there is a good chance it does), you can use the following LINQ query:
List<EditionItem> master_list = context.Projects.Select(
x => x.ProjectEditions
.OrderByDescending(pe => pe.Id)
.FirstOrDefault())
.SelectMany(x => x.EditionItems).ToList();
List<EditionItem> master_list = context.Projects
.Select(p => p.editions.LastOrDefault())
.SelectMany(pe => pe.items).ToList();
IF LastOrDefault not supported you can try using OrderByDescending
List<EditionItem> master_list = context.Projects
.Select(p => p.editions.OrderByDescending(e => e.somefield).FirstOrDefault())
.SelectMany(pe => pe.items).ToList();
from p in context.project
from e in p.projectEdition.LastOrDefault()
select new EditionItem
{
item1 = e.item1
}
Please try this
I have a sql server with the table "contacts", in this table i have column "clicks" of type int, i want to increment its value without making a query.
This is a copy of: "Entity framework update one column by increasing the current value by one without select"
the solution that suggested there is using EntityFramework.Utilities
the problem is that its not a IQueryable extension like EntityFramework.Extended and that makes it hard to implement for me.
the question is there any other way/solution to incremenet int value on sql with using IQueryable(without actually writing sql query)?
the result of the IQueryable query should look like:
UPDATE Contacts SET clicks=clicks+1 WHERE id=8432
I just found the solution after some research and testing, actually EntityFramework.Extended do exactly what i want it to do, if you want to increment a column just call the Update method with adding assignment:
db.Contacts.Where(c => c.Id == id).Update(c => new Contact { Clicks = c.Clicks + 1 });
After checking the source code of EntityFramework.Extended and debugging it i saw that the query that executed in the end is exactly what i wanted,
so thanks everybody for trying to help, the solution were right under my nose ;-)
I had the following:
List<Message> unreadMessages = this.context.Messages
.Where( x =>
x.AncestorMessage.MessageID == ancestorMessageID &&
x.Read == false &&
x.SentTo.Id == userID ).ToList();
foreach(var unreadMessage in unreadMessages)
{
unreadMessage.Read = true;
}
this.context.SaveChanges();
But there must be a way of doing this without having to do 2 SQL queries, one for selecting the items, and one for updating the list.
How do i do this?
Current idiomatic support in EF
As far as I know, there is no direct support for "bulk updates" yet in Entity Framework (there has been an ongoing discussion for bulk operation support for a while though, and it is likely it will be included at some point).
(Why) Do you want to do this?
It is clear that this is an operation that, in native SQL, can be achieved in a single statement, and provides some significant advantages over the approach followed in your question. Using the single SQL statement, only a very small amount of I/O is required between client and DB server, and the statement itself can be completely executed and optimized by the DB server. No need to transfer to and iterate through a potentially large result set client side, just to update one or two fields and send this back the other way.
How
So although not directly supported by EF, it is still possible to do this, using one of two approaches.
Option A. Handcode your SQL update statement
This is a very simple approach, that does not require any other tools/packages and can be performed Async as well:
var sql = "UPDATE TABLE x SET FIELDA = #fieldA WHERE FIELDB = #fieldb";
var parameters = new SqlParameter[] { ..., ... };
int result = db.Database.ExecuteSqlCommand(sql, parameters);
or
int result = await db.Database.ExecuteSqlCommandAsync(sql, parameters);
The obvious downside is, well breaking the nice linqy paradigm and having to handcode your SQL (possibly for more than one target SQL dialect).
Option B. Use one of the EF extension/utility packages
Since a while, a number of open source nuget packages are available that offer specific extensions to EF. A number of them do provide a nice "linqy" way to issue a single update SQL statement to the server. Two examples are:
Entity Framework Extended Library that allows performing a bulk update using a statement like:
context.Messages.Update(
x => x.Read == false && x.SentTo.Id == userID,
x => new Message { Read = true });
It is also available on github
EntityFramework.Utilities that allows performing a bulk update using a statement like:
EFBatchOperation
.For(context, context.Messages)
.Where(x => x.Read == false && x.SentTo.Id == userID)
.Update(x => x.Read, x => x.Read = true);
It is also available on github
And there are definitely other packages and libraries out there that provide similar support.
Even SQL has to do this in two steps in a sense, in that an UPDATE query with a WHERE clause first runs the equivalent of a SELECT behind the scenes, filtering via the WHERE clause, then applying the update. So really, I don't think you need to be worried about improving this.
Further, the reason why it's broken into two steps like this in LINQ is precisely for performance reasons. You want that "select" to be as minimal as possible, i.e. you don't want to load any more objects from the database into in memory objects than you have to. Only then do you alter objects (in the foreach).
If you really want to run a native UPDATE on the SQL side, you could use a System.Data.SqlClient.SqlCommand to issue the update, instead of having LINQ give you back objects that you then update. That will be faster, but then you conceptually move some of your logic out of your C# code object model space into the database model space (you are doing things in the database, not in your object space), even if the SqlCommand is being issued from your code.