We Keep Coding

How to execute manual SQL as part of SaveChanges call in Entity Framework Core?

I am looking for a way to add manual raw SQL call to standard AppDbContext interaction. Say I have following code:
Message m = new Message
{
Title = message.Title,
Body = message.Body,
Date = DateTime.Now,
Foo = "foo"
};
_context.Messages.Add(m);
// Add some manual UPDATE SQL here, like:
// UPDATE SomeOtherTable SET Foo = 'Something' WHERE SomeComplex = 'Condition'
_context.SaveChanges();
The most important thing is to make that manual SQL part of command / transaction that gets created and executed as part of SaveChanges. If there is a way to inject my modification into SQL generated by SaveChanges - that's what I'm looking for.

I use an overload for SaveChangesAsync on my DbContext base that accepts any arbitrary raw sql, the key is to wrap all the processing within a manually created transaction:
public async Task<int> SaveChangesAsync(string customCommand,
CancellationToken cancellationToken = default(CancellationToken),
params object[] parameters)
{
using (var transaction = Database.BeginTransaction())
{
try
{
var result = await base.SaveChangesAsync(true, cancellationToken);
await Database.ExecuteSqlCommandAsync(customCommand, parameters);
transaction.Commit();
return result;
}
catch (Exception ex)
{
transaction.Rollback();
throw ex.Handle();
}
}
}
Note:
Handle() is an extension method where I transform the t-sql exceptions into my business exceptions, so that method is not relevant here.

Unable to delete specific document from cosmos db via sdk?

I am using DeleteDocumentAsync as per below code.
public static void DeleteErrorLog(List<string> LogID, string collectionName)
{
FeedOptions queryOptions = new FeedOptions { EnableCrossPartitionQuery = true };
try
{
//getting resource id for a document with given logid
var db = client.CreateDatabaseQuery().Where(x => x.Id == databaseName).ToList().First();
var coll = client.CreateDocumentCollectionQuery(db.SelfLink).Where(x => x.Id == collectionName).ToList().First();
var docs = client.CreateDocumentQuery(coll.SelfLink, queryOptions).Where(x => x.Id == LogID[0]).AsEnumerable().Single();
var collectionUri = UriFactory.CreateDocumentUri(databaseName, collectionName, docs.ResourceId);
client.DeleteDocumentAsync(collectionUri);
}
catch (Exception) { throw; }
}
Every value is populating correctly and I am getting the document that I need to delete still unable to delete it? Any help is greatly appreciated

You are probably missing the PartitionKey for the delete operation:
var documentUri = UriFactory.CreateDocumentUri(databaseName, collectionName, LogID[0]);
client.DeleteDocumentAsync(documentUri, new RequestOptions(){PartitionKey=new PartitionKey(LogID[0])}).GetAwaiter().GetResult();
Also, if you already know the Id, which in your case is LogID[0], you don't need the Query.

You're not awaiting client.DeleteDocumentAsync, which means your catch will not catch exceptions that occur in the Task that's created - it will silently fail.
My guess is that due to this, there is an exception being thrown in DeleteDocumentAsync that's not subsequently caught where expected.
Ideally, this method would be re-written to use async/await, then your try catch will pick up any exceptions being thrown:
public static async Task DeleteErrorLog(List<string> LogID, string collectionName)
{
...
await client.DeleteDocumentAsync(collectionUri);
...
}
You'll have to make sure that the code calling this method also uses await.

Constructing an IEnumerable based on a collection of collections

Good afternoon,
I'm having a highly entertaining time trying to convince the 'yield' keyword to function in a way that I can understand, but unfortunately I'm not having much luck. Here's the scenario:
Based on the property of a user, I want to look up a set of RSS feed addresses and display the seven most recent articles from all of those feeds, not each of those feeds. To do so, I'm trying to build a collection of the five most recent articles from each feed, then take the 7 most recent from those. The (very dull) pseudo-code-type-process goes something like:
Look up the the member
Get the relevant property ( a group name) for the member
Look up the addresses of the RSS feeds for that group
For each address in the collection, get the five most recent articles and place them in another collection
Take the seven most recent articles from that
subsequent collection and display them.
I've done some research and been able to produce the following:
public static class RSSHelper
{
public static IEnumerable<SyndicationItem> GetLatestArticlesFromFeeds(List<string> feedList, short articlesToTake)
{
foreach (string Feed in feedList)
{
yield return GetLatestArticlesFromFeed(Feed).OrderByDescending(o => o.PublishDate).Take(articlesToTake).First();
}
yield return null;
}
private static IEnumerable<SyndicationItem> GetLatestArticlesFromFeed(string feedURL)
{
// We're only accepting XML based feeds, so create an XML reader:
SyndicationItem Result = new SyndicationItem();
int SkipCount = 0;
for (int Curr = 1; Curr <= 5; Curr++)
{
try
{
XmlReader Reader = XmlReader.Create(feedURL);
SyndicationFeed Feed = SyndicationFeed.Load(Reader);
Reader.Close();
Result = Feed.Items.OrderByDescending(o => o.PublishDate).Skip(SkipCount).Take(1).Single();
SkipCount++;
}
catch (Exception ex)
{
// Do nothing, else the Yield will fail.
}
yield return Result;
}
}
}
What seems to be happening is that I get five results (articlesToTake is 7, not 5), and occasionally either the whole SyndicationItem is null, or properties of it are null. I'm also convinced that this is a really, really poorly performing approach to tackling this problem, but I can't find much direction on using the yield keyword in this context.
I did find this question but it's not quite helping me understand anything.
Is what I'm trying to do achievable in this way, or do I just need to bite the bullet and use a couple of foreach loops?

Load your RSS feed in memory using async and await, then order them by date, and just Take the first 7

Considering all you want to do in GetLatestArticlesFromFeed is to get the 5 latest items, wouldn't it be easier to only order the list once and then take the first 5 items? It would look like this (together with a SelectMany based approach to the first method)
public static class RSSHelper
{
public static IEnumerable<SyndicationItem> GetLatestArticlesFromFeeds(List<string> feedList, short articlesToTake)
{
return feedList.SelectMany(f => GetLatestArticlesFromFeed(f)).OrderByDescending(a => a.PublishDate).Take(articlesToTake);
}
private static IEnumerable<SyndicationItem> GetLatestArticlesFromFeed(string feedURL)
{
// We're only accepting XML based feeds, so create an XML reader:
SyndicationFeed feed = null;
try
{
using (XmlReader reader = XmlReader.Create(feedURL))
{
feed = SyndicationFeed.Load(reader);
}
return feed.Items.OrderByDescending(o => o.PublishDate).Take(5);
}
catch
{
return Enumerable.Empty<SyndicationItem>();
}
}
}
Let me know if this doesn't work!

Morning,
Now that I don't feel like death warmed up, I've got it working! Thanks to #Rodrigo and #McKabue for their help in finding the eventual answer, and #NPSF3000 for pointing out my original stupidity!
I've settled on this as a result:
public static class RSSHelper
{
public static IEnumerable<SyndicationItem> GetLatestArticlesFromFeeds(List<string> feedList, short articlesToTake)
{
return GetLatestArticlesFromFeedsAsync(feedList, articlesToTake).Result;
}
private async static Task<IEnumerable<SyndicationItem>> GetLatestArticlesFromFeedsAsync(List<string> feedList, short articlesToTake)
{
List<Task<IEnumerable<SyndicationItem>>> TaskList = new List<Task<IEnumerable<SyndicationItem>>>();
foreach (string Feed in feedList)
{
// Call and start a task to evaluate the RSS feeds
Task<IEnumerable<SyndicationItem>> T = Task.FromResult(GetLatestArticlesFromFeed(Feed).Result);
TaskList.Add(T);
}
var Results = await Task.WhenAll(TaskList);
// Filter the not null results - on the balance of probabilities, we'll still get more than 7 results.
var ReturnList = Results.SelectMany(s => TaskList.Where(w => w.Result != null).SelectMany(z => z.Result).OrderByDescending(o => o.PublishDate)).Take(articlesToTake);
return ReturnList;
}
private async static Task<IEnumerable<SyndicationItem>> GetLatestArticlesFromFeed(string feedURL)
{
// We're only accepting XML based feeds, so create an XML reader:
try
{
XmlReader Reader = XmlReader.Create(feedURL);
SyndicationFeed Feed = SyndicationFeed.Load(Reader);
Reader.Close();
return Feed.Items.OrderByDescending(o => o.PublishDate).Take(5);
}
catch (Exception ex)
{
return null;
}
}
}
It took me a while to wrap my head around, as I was forgetting to define the result type of the task I was kicking off, but thankfully I stumbled across this question this morning, which helped everything fall nicely into place.
I feel a bit cheeky answering my own question, but on balance I think this is a nice, tidy balance between the proposed answers and it certainly seems functional. I'll add some hardening and comments, and of course if anyone has feedback I'll receive it gratefully.
Thanks!
Ash

There is already an open DataReader ... even though it is not

Note: I've went through millions of questions when the issue is not disposing the reader/connection properly, or when the error is because of badly handled lazy loading. I believe that this issue is a different one, and probably related to MySQL's .NET connector.
I'm using MySQL server (5.6) database extensively through its .NET connector (6.8.3). All tables are created with MyISAM engine for performance reasons. I have only one process with one thread (update: in fact, it's not true, see below) accessing the DB sequentially, so there is no need for transactions and concurrency.
Today, after many hours of processing the following piece of code:
public IEnumerable<VectorTransition> FindWithSourceVector(double[] sourceVector)
{
var sqlConnection = this.connectionPool.Take();
this.selectWithSourceVectorCommand.Connection = sqlConnection;
this.selectWithSourceVectorCommand.Parameters["#epsilon"].Value
= this.epsilonEstimator.Epsilon.Min() / 10;
for (int d = 0; d < this.dimensionality; ++d)
{
this.selectWithSourceVectorCommand.Parameters["#source_" + d.ToString()]
.Value = sourceVector[d];
}
// *** the following line (201) throws the exception presented below
using (var reader = this.selectWithSourceVectorCommand.ExecuteReader())
{
while (reader.Read())
{
yield return ReaderToVectorTransition(reader);
}
}
this.connectionPool.Putback(sqlConnection);
}
threw the following exception:
MySqlException: There is already an open DataReader associated with this Connection which must be closed first.
Here is the relevant part of the stack trace:
at MySql.Data.MySqlClient.ExceptionInterceptor.Throw(Exception exception)
at MySql.Data.MySqlClient.MySqlConnection.Throw(Exception ex)
at MySql.Data.MySqlClient.MySqlCommand.CheckState()
at MySql.Data.MySqlClient.MySqlCommand.ExecuteReader(CommandBehavior behavior)
at MySql.Data.MySqlClient.MySqlCommand.ExecuteReader()
at implementation.VectorTransitionsMySqlTable.d__27.MoveNext() in C:\Users\bartoszp...\implementation\VectorTransitionsMySqlTable.cs:line 201
at System.Linq.Enumerable.d__3a1.MoveNext()
at System.Linq.Buffer1..ctor(IEnumerable1 source)
at System.Linq.Enumerable.ToArray[TSource](IEnumerable1 source)
at implementation.VectorTransitionService.Add(VectorTransition vectorTransition) in C:\Users\bartoszp...\implementation\VectorTransitionService.cs:line 38
at Program.Go[T](Environment`2 p, Space parentSpace, EpsilonEstimator epsilonEstimator, ThresholdEstimator thresholdEstimator, TransitionTransformer transitionTransformer, AmbiguityCalculator ac, VectorTransitionsTableFactory vttf, AxesTableFactory atf, NeighbourhoodsTableFactory ntf, AmbiguitySamplesTableFactory astf, AmbiguitySampleMatchesTableFactory asmtf, MySqlConnectionPool connectionPool, Boolean rejectDuplicates, Boolean addNew) in C:\Users\bartoszp...\Program.cs:line 323
The connectionPool.Take returns the first connection that satisfies the following predicate:
private bool IsAvailable(MySqlConnection connection)
{
var result = false;
try
{
if (connection != null
&& connection.State == System.Data.ConnectionState.Open)
{
result = connection.Ping();
}
}
catch (Exception e)
{
Console.WriteLine("Ping exception: " + e.Message);
}
return result && connection.State == System.Data.ConnectionState.Open;
}
(This is related to my previous question, when I resolved a different, but similar issue: MySQL fatal error during information_schema query (software caused connection abort))
The FindWithSourceVector method is called by the following piece of code:
var existing
= this.vectorTransitionsTable
.FindWithSourceVector(vectorTransition.SourceVector)
.Take(2)
.ToArray();
(I need to find at most two duplicate vectors) - this is the VectorTransitionService.cs:line 38 part of the stack trace.
Now the most interesting part: when the debugger stopped execution after the exception occured, I've investigated the sqlConnection object to find, that it doesn't have a reader associated with it (picture below)!
Why is this happening (apparently at "random" - this method was being called almost every minute for the last ~20h)? Can I avoid that (in ways other then guess-adding some sleeps when Ping throws an exception and praying it'll help)?
Additional information regarding the implementation of the connection pool:
Get is intended for methods that call only simple queries and are not using readers, so the returned connection can be used in a re-entrant way. It is not used directly in this example (because of the reader involved):
public MySqlConnection Get()
{
var result = this.connections.FirstOrDefault(IsAvailable);
if (result == null)
{
Reconnect();
result = this.connections.FirstOrDefault(IsAvailable);
}
return result;
}
The Reconnect method just iterates though the whole array and recreates and opens the connections.
Take uses Get but also removes the returned connection from the list of available connections so in case of some methods that during their usage of a reader call other methods that also need a connection, it will not be shared. This is also not the case here, as the FindSourceVector method is simple (doesn't call other methods that use the DB). However, the Take is used for the sake of convention - if there is a reader, use Take:
public MySqlConnection Take()
{
var result = this.Get();
var index = Array.IndexOf(this.connections, result);
this.connections[index] = null;
return result;
}
Putback just puts a connection to the first empty spot, or just forgets about it if the connection pool is full:
public void Putback(MySqlConnection mySqlConnection)
{
int index = Array.IndexOf(this.connections, null);
if (index >= 0)
{
this.connections[index] = mySqlConnection;
}
else if (mySqlConnection != null)
{
mySqlConnection.Close();
mySqlConnection.Dispose();
}
}

I suspect this is the problem, at the end of the method:
this.connectionPool.Putback(sqlConnection);
You're only taking two elements from the iterator - so you never complete the while loop unless there's actually only one value returned from the reader. Now you're using LINQ, which will automatically be calling Dispose() on the iterator, so your using statement will still be disposing of the reader - but you're not putting the connection back in the pool. If you do that in a finally block, I think you'll be okay:
var sqlConnection = this.connectionPool.Take();
try
{
// Other stuff here...
using (var reader = this.selectWithSourceVectorCommand.ExecuteReader())
{
while (reader.Read())
{
yield return ReaderToVectorTransition(reader);
}
}
}
finally
{
this.connectionPool.Putback(sqlConnection);
}
Or ideally, if your connection pool is your own implementation, make Take return something which implements IDisposable and returns the connection back to the pool when it's done.
Here's a short but complete program to demonstrate what's going on, without any actual databases involved:
using System;
using System.Collections.Generic;
using System.Linq;
class DummyReader : IDisposable
{
private readonly int limit;
private int count = -1;
public int Count { get { return count; } }
public DummyReader(int limit)
{
this.limit = limit;
}
public bool Read()
{
count++;
return count < limit;
}
public void Dispose()
{
Console.WriteLine("DummyReader.Dispose()");
}
}
class Test
{
static IEnumerable<int> FindValues(int valuesInReader)
{
Console.WriteLine("Take from the pool");
using (var reader = new DummyReader(valuesInReader))
{
while (reader.Read())
{
yield return reader.Count;
}
}
Console.WriteLine("Put back in the pool");
}
static void Main()
{
var data = FindValues(2).Take(2).ToArray();
Console.WriteLine(string.Join(",", data));
}
}
As written - modelling the situation with the reader only finding two values - the output is:
Take from the pool
DummyReader.Dispose()
0,1
Note that the reader is disposed, but we never get as far as returning anything from the pool. If you change Main to model the situation where the reader only has one value, like this:
var data = FindValues(1).Take(2).ToArray();
Then we get all the way through the while loop, so the output changes:
Take from the pool
DummyReader.Dispose()
Put back in the pool
0
I suggest you copy my program and experiment with it. Make sure you understand everything about what's going on... then you can apply it to your own code. You might want to read my article on iterator block implementation details too.

TyCobb and Jon Skeet have correctly guessed, that the problem was the pool implementation and multi-threading. I forgot that actually I did start some tiny Tasks in the Reconnect method. The first connection was created and opened synchronously but all other where opened asynchronously.
The idea was that because I only need one connection at time, there others can reconnect in different threads. However, because I didn't always put the connection back (as explained in Jon's answer) reconnecting was happening quite frequently, and because the system was quite loaded these reconnection threads weren't fast enough, which eventually led to race conditions. The fix is to reconnect in a more simple and straightforward manner:
private void Reconnect()
{
for (int i = 0; i < connections.Length; ++i)
{
if (!IsAvailable(this.connections[i]))
{
this.ReconnectAt(i);
}
}
}
private void ReconnectAt(int index)
{
try
{
this.connections[index] = new MySqlConnection(this.connectionString);
this.connections[index].Open();
}
catch (MySqlException mse)
{
Console.WriteLine("Reconnect error: " + mse.Message);
this.connections[index] = null;
}
}

Cannot return type IQueryable or IEnumerable

Since I cannot return either of these types from a Linq to Entities query, I am sticking with returning a List.
On return from the DB call (which is in a separate WCF service or DLL), the code in the Controller fails because the dbcontext connection is closed.
Note the code in the following. For both IEnumerable and IQueryable, the data does not come back because of the above description.
Method in MVC Controller
// Controller
IEnumerable<ProjectDescription> projectDdl = db.GetProjectDropDownList().AsEnumerable();
// Error coming back because dbcontext connection was closed.
ViewBag.ProjectsCust = new SelectList(projectDdl, "ProjectId", "Name");
Method in WCF service or DLL
// WCF Service or DLL
public IEnumerable<ProjectDescription> GetProjectDropDownList()
{
try
{
//IQueryable<ProjectDescription> project = null;
using (YeagerTechEntities DbContext = new YeagerTechEntities())
{
DbContext.Configuration.ProxyCreationEnabled = false;
DbContext.Database.Connection.Open();
IEnumerable<ProjectDescription> project = DbContext.Projects.Select(s =>
new ProjectDescription()
{
ProjectID = s.ProjectID,
Description = s.Description
}
);
return project;
}
}
catch (Exception ex)
{
throw ex;
}
}
I even tried the following establishing a DbContext instance before the DB call which still didn't work actually trying to pass down the DbContext to the DB method.
Controller:
DbContext = new YeagerTechEntities();
IEnumerable<ProjectDescription> projectDdl = db.GetProjectDropDownList(DbContext).AsEnumerable();
ViewBag.ProjectsCust = new SelectList(projectDdl, "ProjectId", "Name");
DbContext.Dispose();
DB method:
public IEnumerable<ProjectDescription> GetProjectDropDownList(YeagerTechEntities DbContext)
{
try
{
//IQueryable<ProjectDescription> project = null;
DbContext.Configuration.ProxyCreationEnabled = false;
DbContext.Database.Connection.Open();
IEnumerable<ProjectDescription> project = DbContext.Projects.Select(s =>
new ProjectDescription()
{
ProjectID = s.ProjectID,
Description = s.Description
}
);
return project;
}
catch (Exception ex)
{
throw ex;
}
}
The only thing that worked besides using List was to actually place the DB method inside the Controller, which obviously is not good practice whatsoever.
Here is the List convention that works fine:
Controller:
IEnumerable<ProjectDescription> projectDdl = db.GetProjectDropDownList();
ViewBag.ProjectsCust = new SelectList(projectDdl, "ProjectId", "Name");
DB method:
public List<ProjectDescription> GetProjectDropDownList()
{
try
{
using (YeagerTechEntities DbContext = new YeagerTechEntities())
{
DbContext.Configuration.ProxyCreationEnabled = false;
DbContext.Database.Connection.Open();
IEnumerable<ProjectDescription> project = DbContext.Projects.Select(s =>
new ProjectDescription()
{
ProjectID = s.ProjectID,
Description = s.Description
}
);
List<ProjectDescription> myProjects = new List<ProjectDescription>();
myProjects = project.ToList();
return myProjects;
}
}
catch (Exception ex)
{
throw ex;
}
}
If anyone can explain to me what the correct model should be when using IQueryable or IEnumerable, that would be fine. However, after reading this link, it definitely seems like List is the way to go:
IEnumerable vs IQueryable for Business Logic or DAL return Types

The reason why your code failed when using IEnumerable/IQueryable is because the EF query is deferred; that is to say it doesn't actually get executed against the database until you start to enumerate (foreach, etc) or "materialise" (ToList(), etc) the results.
Therefore because you create an effectively temp DbContext with your using{}, the IEnumerable that you pass out of the function is a landmine - the underlying DbContext ceases to exist after the using block, and any attempt to enumerate or materialise the collection will fail.
Even when you created the DBContext without a using block, you then went ahead and Dispose'd of it before you executed the query - hence getting the same error.
This is one of the main reasons for not returning IQueryable from a DAL/Repository - because the recipient cannot know if the underlying context that the query relies on has been closed before they try to consume the results. IEnumerable is much better (List<T> implements IEnumerable<T> of course) and IMHO List is even better because it forces you to return a fully materialised collection and makes it clear to the caller that the result is not connected to the database any more.
So the working version that you have is probably fine - create your context, execute the query and then immediately materialise it which executes the actual SQL and then disconnects the result set from the database. So that pattern will work fine for you.
However I would recommend looking at having things like DbContext injected for you using an IoC container such as Ninject as it frees you from having to worry about the lifecycle of your DbContexts.
Hope that helps.

As you already knows, your DbContext enables you to query your database and so it has to be alive when SQL requests are sent to your SGBD.
The key point here is to know exactly how IQueryable works :
The IQueryable interface inherits the IEnumerable interface so that
if it represents a query, the results of that query can be enumerated.
Enumeration causes the expression tree associated with an IQueryable
object to be executed. The definition of "executing an expression
tree" is specific to a query provider. For example, it may involve
translating the expression tree to an appropriate query language for
the underlying data source.
Which means that as long as your Linq query has not been enumerated (with .ToList() or foreach), no query is sent to the database. The execution is deferred!
In your first attempt, you're :
Calling the GetProjectDropDownList method
Returning an IEnumerable (which is a Ling query) without "enumerating" it
Disposing the DbContext
Enumerating your Linq query through new SelectList when your view is generated but... Your DbContext has already been disposed at that time.
The same applies for your second attempt, the DbContext is disposed a little bit later but also already disposed when the view is generated.
In your last attempt, everything works fine because your DbContextis still alive when you're enumerating your Linq query (with project.ToList();)
In my opinion, isolating your DbContext calls and instances into a Data Access Layer and returning Lists or simple disconnected objects is not a bad practice at all.