I have updated my ASP.NET app from NpgSQL 2.2.5 to 3.0.1. In the breaking changes it's specified that they have removed the Preload Reader support. So I remove it from the string connection.
Testing my web app, I got the error "An operation is already in progress." specially in the linq query like this:
var plugins =
from p in _pluginRepository.GetPlugins() // this method return this: GetAll().OrderBy(p => p.Created)
join e in _userPluginRepository.GetByUserId(user.Id).ToList() on p.Id equals e.Plugin.Id into pe
from e in pe.DefaultIfEmpty()
select new PluginViewModel
{
Active = e != null,
Name = p.Translations.ToUserLanguage(loggedInUser),
Key = p.Key,
PluginId = p.Id,
SettingId = e == null ? 0 : e.Id,
ExpireDate = e != null && e.ExpireDate.HasValue ? e.ExpireDate.Value : (DateTime?) null,
Grants = e == null ? UserPluginGrants.None.GetHashCode().ToString() : e.Grants.GetHashCode().ToString()
};
To solve this error, I have to append a ToList after the GetPlugins method.
Is this the correct behavior to use without Preload Reader? Why?
In Npgsql 2.x, using the Preload Reader made Npgsql pull the entire result set of the query from the database into the memory of your application. This freed the connection and allowed another command to be executed while still traversing the resultset of the first query. In other words, it allowed you to program as if you could execute multiple queries concurrently (known sometimes as MARS), although behind the scenes this was implemented inefficiently.
Adding a ToList() does exactly the same thing - pull everything into client memory, only it happens in your application code instead of in the database driver. So it's definitely an acceptable way to port your application from Npgsql 2.x to 3.x.
Now, if the result set being pulled (in this case GetPlugins) is small, this is a perfectly valid approach. If it's big, however you should look into alternatives. In your example, the join could be sent to the database, making your Linq expression translate into a single SQL query and eliminating the need for multiple queries (ORMs such as Entity Framework can usually do this for you). A more extreme solution would be to use multiple database connections, but that is heavier and also problematic if you're using transactions.
Note that there's an issue open for implementing true MARS in Npgsql (although it isn't likely to be implemented very soon): https://github.com/npgsql/npgsql/issues/462
Related
We have a number of customer databases (all single tenant) and a number of APIs that access these databases.
The APIs are aware of the customer accessing the various endpoints by virtue of access tokens, and a DbContext is created for each request that comes in, to ensure we hit the correct database. The API is pretty busy so we've got DbContext's being created and disposed of all the time.
We use a PredicateBuilder to deal with query string parameters that are converted into SQL via LINQ and Entity Framework. Here is an example that is applied to our queryable as part of a query string parameter:
var predicate = PredicateBuilder.New<Property>(true);
if (query.OwnerIds != null && query.OwnerIds.Any())
{
predicate = predicate.And(p => query.OwnerIds.Contains(p.PrimaryOwner) || query.OwnerIds.Contains(p.SecondaryOwner));
}
return predicate;
The predicate is then used in the Where condition when calling the DbContext eg
this.dbContext.Where(predicate).ToListAsync();
I am seeing a problem where the compiled query uses values from a previous web request, which results in the wrong data being returned. Take the following scenario:
Customer A makes a called to /endpoint?ownerId=ABC
This gets converted into SQL that looks something similar to
SELECT EXISTS (
SELECT 1
FROM `data` AS `p`
WHERE (((`p`.`primaryOwner` = 'ABC') OR (`p`.`secondaryOwner` = 'ABC')))
Customer B makes a similar request, but with different parameters to /endpoint?ownerIds=XYZ
What I would expect here is that a query similar to the above would be built:
SELECT EXISTS (
SELECT 1
FROM `data` AS `p`
WHERE (((`p`.`primaryOwner` = 'XYZ') OR (`p`.`secondaryOwner` = 'XYZ')))
However the first query runs, as though the query string parameters have never changed.
I cannot find anything online that relates to this problem and am struggling to resolve it. Can anybody provide any pointers?
I am currently trying to convert an existing SQL server query to EF Core. The goal is to get all users and get their latest order date-time and latest support request date-time. I want to ensure users are returned even if they don't have an order yet or a support request yet. If they have not placed an order, the column for "latest order date-time" should be NULL. If they have not filed a support request, the column for "latest support request date-time" should be NULL.
The outputted columns should be: Id, Name, Email, LatestOrderDateTime, LatestSupportRequestDateTime
Here is my working SQL server query:
SELECT [User].[Id], [User].[Name], [User].[Email], MAX([Order].[DateTime]) as LatestOrderDateTime, MAX([SupportRequest].[DateTime]) as LatestSupportRequestDateTime FROM [User]
LEFT JOIN [Order] on [User].[Id] = [Order].[UserId]
LEFT JOIN [SupportRequest] on [User].[Id] = [SupportRequest].[ConsumerId]
GROUP BY [User].[Id], [User].[Name], [User].[Email]
ORDER BY [User].[Id]
This is what I've tried, however it does not evaluate on the server:
await this.context.User
.GroupBy(u => new { u.Id, u.Name, u.Email })
.Select(g => new
{
id = g.Key.Id,
name = g.Key.Name,
email = g.Key.Email,
lastOrderDateTime = g.Max(o => o.Orders.Select(o => o.DateTime)),
lastSupportRequestDateTime = g.Max(o => o.SupportRequests.Select(s => s.DateTime)),
})
.OrderBy(c => c.id)
.ToListAsync();
I just want to convert this query to EF core (where the query DOES NOT get evaluated locally).
If you could do it in method syntax, that'd be great, but no worries if not since I can convert it with JetBrains Rider.
Thank you so much for your help!
I just want to convert this query to EF core (where the query DOES NOT get evaluated
locally).
Can not be done, use EntityFramework 6.4, not core, if you want this.
The SQL generation in current EntityFramework (and I mean current up to the nightly builds of veryion 5) is EXTREMELY Limited in the SQL it can generate, combined with what looks like utter ignorance to even accept that fact from the team (which reminds me of the times of EntityFramework 2 and 3 until that team started being serious about LINQ in their version 4).
If it tells you it can not generate this as SQL then your only 2 choises are:
Use EntityFramework 6.4 (which works in dotnetcore 3.1) and get server side execution
Open a bug report, HOPE someone picks it up and then either wait until November for the release of version 5 OR - once it possibly is fixed - work with nightly builds until then.
This is not a syntax issue. They deactivated client evaluation of SQL and their SQL generator is not able to handle a LOT of standard cases. Given you do not want the first (which is what we do at the moment), their feature set just means it can not be done.
You could try to explicitly spell out the left joins in Linq (left join syntax is a bit un-intuitive iirc so it may take some doing to sort it out).
You can find more information at:
https://learn.microsoft.com/en-us/dotnet/csharp/linq/perform-left-outer-joins
The way your Linq is set up specifically asks for object linking, which is why it happens client side. I believe what you're trying to do has a solution in EF Core.
I have the following code:
ILookup<string, costcenterdata> areas = p_data.CostcenterDatas.ToLookup(p => OracleDataProvider.GetAreaForClient(p_data.ClientID), p => p);
it takes acollection of pocos and generates a lookup. For determining the key it goes to an ORACLE table and performs a select foo from data where bar.
Nothing fancy. This is done in GetAreaFromClient which does open conncection, read, close connection with ODP.net. No Entity Framework or other newfangled stuff.
But sometimes it seems to produce a lock. the code above waits endlessly and in the end, we have to cancel the connection in the database.
Can it be tha the toLockup somehow produces a parallel access that may or may not produce this lock?
Oracle's locking/blocking is more akin to sql server's snapshot isolation, so there shouldn't be any blocking of a simple select.
Are you sure that method is only doing a select or that it's connection wasn't used for something else?
In any case a dba should be able to help you determine why the connection is blocked. You could also try looking yourself if you have permissions with something like:
select
ao.object_name,
ao.object_type,
s.sid,
s.username,
s.status,
s.osuser,
s.machine,
s.program,
s.module,
s.wait_time,
s.seconds_in_wait,
lo.inst_id,
s.*, lo.*
from
gv$locked_object lo ,
gv$session s,
all_objects ao
where
s.sid = lo.session_id
and
lo.object_id = ao.object_id
and s.username = 'connectionUserName';
I had the following:
List<Message> unreadMessages = this.context.Messages
.Where( x =>
x.AncestorMessage.MessageID == ancestorMessageID &&
x.Read == false &&
x.SentTo.Id == userID ).ToList();
foreach(var unreadMessage in unreadMessages)
{
unreadMessage.Read = true;
}
this.context.SaveChanges();
But there must be a way of doing this without having to do 2 SQL queries, one for selecting the items, and one for updating the list.
How do i do this?
Current idiomatic support in EF
As far as I know, there is no direct support for "bulk updates" yet in Entity Framework (there has been an ongoing discussion for bulk operation support for a while though, and it is likely it will be included at some point).
(Why) Do you want to do this?
It is clear that this is an operation that, in native SQL, can be achieved in a single statement, and provides some significant advantages over the approach followed in your question. Using the single SQL statement, only a very small amount of I/O is required between client and DB server, and the statement itself can be completely executed and optimized by the DB server. No need to transfer to and iterate through a potentially large result set client side, just to update one or two fields and send this back the other way.
How
So although not directly supported by EF, it is still possible to do this, using one of two approaches.
Option A. Handcode your SQL update statement
This is a very simple approach, that does not require any other tools/packages and can be performed Async as well:
var sql = "UPDATE TABLE x SET FIELDA = #fieldA WHERE FIELDB = #fieldb";
var parameters = new SqlParameter[] { ..., ... };
int result = db.Database.ExecuteSqlCommand(sql, parameters);
or
int result = await db.Database.ExecuteSqlCommandAsync(sql, parameters);
The obvious downside is, well breaking the nice linqy paradigm and having to handcode your SQL (possibly for more than one target SQL dialect).
Option B. Use one of the EF extension/utility packages
Since a while, a number of open source nuget packages are available that offer specific extensions to EF. A number of them do provide a nice "linqy" way to issue a single update SQL statement to the server. Two examples are:
Entity Framework Extended Library that allows performing a bulk update using a statement like:
context.Messages.Update(
x => x.Read == false && x.SentTo.Id == userID,
x => new Message { Read = true });
It is also available on github
EntityFramework.Utilities that allows performing a bulk update using a statement like:
EFBatchOperation
.For(context, context.Messages)
.Where(x => x.Read == false && x.SentTo.Id == userID)
.Update(x => x.Read, x => x.Read = true);
It is also available on github
And there are definitely other packages and libraries out there that provide similar support.
Even SQL has to do this in two steps in a sense, in that an UPDATE query with a WHERE clause first runs the equivalent of a SELECT behind the scenes, filtering via the WHERE clause, then applying the update. So really, I don't think you need to be worried about improving this.
Further, the reason why it's broken into two steps like this in LINQ is precisely for performance reasons. You want that "select" to be as minimal as possible, i.e. you don't want to load any more objects from the database into in memory objects than you have to. Only then do you alter objects (in the foreach).
If you really want to run a native UPDATE on the SQL side, you could use a System.Data.SqlClient.SqlCommand to issue the update, instead of having LINQ give you back objects that you then update. That will be faster, but then you conceptually move some of your logic out of your C# code object model space into the database model space (you are doing things in the database, not in your object space), even if the SqlCommand is being issued from your code.
I'am using wrapper to get some data from table User
IQueryable<StarGuestWrapper> WhereQuery =
session.Linq<User>().Where(u => u.HomeClub.Id == clubId && u.IsActive).Select(
u =>
new StarGuestWrapper()
{
FullName = u.Name + " " + u.LastName,
LoginTime = u.SomeDateTime,
MonthsAsMember = u.SomeIntergerValue,
StarRating = u.SomeOtherInteregValue,
UserPicture = u.Photo.PhotoData,
InstructorFullName = u.SomeInstructorName,
TalkInteractionDuringSession = u.SomeBoolValue,
GoalInteractionDuringSession = u.SomeOtherBoolValue
});
I use this without a problem as a IQueryable so I can do useful things before actually running the query. Like :
WhereQuery.Skip(startRowIndex).Take(maximumRows).ToList();
and so on.
The problem occurs using 'where' statement on query.
For example:
WhereQuery.Where(s => s.StarRating == 1)
will throw an exception in runtime that 'StarRating' doesn't exist in User table - of course it doesn't it's a wrappers property. It will work if I materialize query by
WhereQuery.AsEnumerable().Where(s => s.StarRating == 1)
but then it loses all the sens of using IQueryable and I don't want to do this.
What is strange and interesting that not all properties from wrapper throw error, all the bool values can be used in where statement. Example :
WhereQuery.Where(s => s.TalkInteractionDuringSession)
It works in EntityFramework , why do I get this error in NHibernate and how to get it working the way I want it to ?
Keep in mind the older nHibernate Linq provider is only a partial implementation and is no longer being actively worked on. A new and more complete linq provider is being developed now and will be part of NH3.0 (you can checkout the trunk and build it to see if it addresses this problem).
My recommendation is to change your code to call ToList() at the point when you explicitly wish to hit the database. You are passing a future valued query back from your repository at which point anything could technically happen to the query. Even EF and LINQ2SQL cannot translate any possible linq query into SQL.
I do realize this isn't what you want to be able to do, but I think you are trying to bend the framework to do something in a way this isn't very natural at all.