I cannot figure out why the HasChanged value of my SqlCacheDependency object is coming back originally from the command execution as false, but somewhere almost immediately after it comes back from the database, the value changes to true.
Sometimes this happens before the item is even inserted into the cache, causing the cache to discard it immediately, sometimes it's after the insert, and I can grab an enumerator which sees the key in the cache but before I even loop to that item in the cache it's been deleted.
SPROC:
ALTER PROCEDURE [dbo].[ntz_dal_ER_X_Note_SelectAllWER_ID]
#ER_ID int
AS
BEGIN
SELECT
ER_X_Note_ID,
ER_ID,
Note_ID
FROM dbo.ER_X_Note e
WHERE
ER_ID = #ER_ID
END
The database is MS SQL Server 2008, broker service is enabled, and SOME output does cache and remain cached. For instance, this one works just fine:
ALTER PROC [dbo].[ntz_dal_GetCacheControllerByEntityName] (
#Name varchar(50)
) AS
BEGIN
SELECT
CacheController_ID,
EntityName,
CacheEnabled,
Expiration
From dbo.CacheController cc
WHERE EntityName = #Name
END
The code which calls the SPROC in question that fails:
DataSet toReturn;
Hashtable paramHash = new Hashtable();
paramHash.Add("ER_ID", _eR_ID.IsNull ? null : _eR_ID.Value.ToString());
string cacheName = BuildCacheString("ntz_dal_ER_X_Note_SelectAllWER_ID", paramHash);
toReturn = (DataSet)GetFromCache(cacheName);
if (toReturn == null)
{
// Set up parameters (1 input and 0 output)
SqlParameter[] arParms = {
new SqlParameter("#ER_ID", _eR_ID),
};
SqlCacheDependency scd;
// Execute query.
toReturn = _dbTransaction != null
? _dbConnection.ExecuteDataset(_dbTransaction, "dbo.[ntz_dal_ER_X_Note_SelectAllWER_ID]", out scd, arParms)
: _dbConnection.ExecuteDataset("dbo.[ntz_dal_ER_X_Note_SelectAllWER_ID]", out scd, arParms);
AddToCache(cacheName, toReturn, scd);
}
return toReturn;
Code that works
const string sprocName = "ntz_dal_GetCacheControllerByEntityName";
string cacheControlPrefix = "CacheController_" + CachePrefix;
CacheControl controller = (CacheControl)_cache[cacheControlPrefix];
if (controller == null)
{
try
{
SqlParameter[] arParms = {
new SqlParameter("#Name", CachePrefix),
};
SqlCacheDependency sqlCacheDependency;
// Execute query.
DataSet result = _dbTransaction != null
? _dbConnection.ExecuteDataset(_dbTransaction, sprocName, out sqlCacheDependency, arParms)
: _dbConnection.ExecuteDataset(sprocName, out sqlCacheDependency, arParms);
controller = result.Tables[0].Rows.Count == 0
? new CacheControl(false)
: new CacheControl(result.Tables[0].Rows[0]);
_cache.Insert(cacheControlPrefix, controller, sqlCacheDependency);
}
catch (Exception ex)
{
// if sproc retreival fails cache the result of false so we don't keep trying
// this is the only case where it can be added with no expiration date
controller = new CacheControl(false);
// direct cache insert, no dependency, no expiration, never try again for this entity
if (HttpContext.Current != null && UseCaching && _cache != null) _cache.Insert(cacheControlPrefix, controller);
}
}
return controller;
The AddToCache method is overloaded and has more tests in it; The direct _cache.Insert in the working method is to bypass those other tests. The working code helps determine if db caching should happen at all.
You can see that when the "non working" data is retrieved initially, all is OK:
But somewhere random beyond that point, in this instance, just stepping into the next method
And yet the data is NOT changing at all; I'm the only one touching this instance of the database.
It was really, really simple, so simple I completely overlooked it.
In this article Creating a Query for Notification, which I DID scour multiple times, it clearly states:
SET Option Settings
When a SELECT statement is executed under a notification request, the
connection that submits the request must have the options for the
connection set as follows:
ANSI_NULLS ON
ANSI_PADDING ON
ANSI_WARNINGS ON
CONCAT_NULL_YIELDS_NULL ON
QUOTED_IDENTIFIER ON
NUMERIC_ROUNDABORT OFF
ARITHABORT ON
Well, I read and re-read and RE-re-read the sproc, and I still didn't see that both ANSI_NULLS and QUOTED_IDENTIFIER were "OFF", not ON.
My dataset is now caching and retaining the data properly without false indicators of change.
I have a hunch that the issue is with your _eR_ID. I think that you should try adding a local variable to the failing procedure that uses an impossible value for _eR_ID, such as -1. I never trust what is going to happen when nulls are involved and I think this could be the source of your problem.
Here is the modified version that I recommend trying:
DataSet toReturn;
Hashtable paramHash = new Hashtable();
int local_er_ID = eR_ID.IsNull ? -1 : _eR_ID.Value;
paramHash.Add("ER_ID", local_eR_ID.ToString());
string cacheName = BuildCacheString("ntz_dal_ER_X_Note_SelectAllWER_ID", paramHash);
toReturn = (DataSet)GetFromCache(cacheName);
if (toReturn == null)
{
// Set up parameters (1 input and 0 output)
SqlParameter[] arParms = {
new SqlParameter("#ER_ID", local_eR_ID),
};
SqlCacheDependency scd;
// Execute query.
toReturn = _dbTransaction != null
? _dbConnection.ExecuteDataset(_dbTransaction, "dbo.[ntz_dal_ER_X_Note_SelectAllWER_ID]", out scd, arParms)
: _dbConnection.ExecuteDataset("dbo.[ntz_dal_ER_X_Note_SelectAllWER_ID]", out scd, arParms);
AddToCache(cacheName, toReturn, scd);
}
return toReturn;
Important
While creating the above code, I think I discovered the source of your problem: when setting the stored proc parameter, you are using _eR_ID but when you set the paramHash you are using _eR_ID.Value.
The code rewrite will solve this problem, but I suspect that this is the root of the problem.
Running into the same issue and finding the same answers online without any help, I was reasearching the xml invalid subscription response from profiler.
I found an example on msdn support site that had a slightly different order of code. When I tried it I realized the problem - Don't open your connection object until after you've created the command object and the cache dependency object. Here is the order you must follow and all will be good:
Be sure to enable notifications (SqlCahceDependencyAdmin) and run SqlDependency.Start first
Create the connection object
Create the command object and assign command text, type, and connection object (any combination of constructors, setting properties, or using CreateCommand).
Create the sql cache dependency object
Open the connection object
Execute the query
Add item to cache using dependency.
If you follow this order, and follow all other requirements on your select statement, don't have any permissions issues, this will work!
I believe the issue has to do with how the .NET framework manages the connection, specifically what settings are set. I tried overriding this in my sql command test but it never worked. This is only a guess - what I do know is changing the order immediately solved the issue.
I was able to piece it together from the following to msdn posts.
This post was one of the more common causes of the invalid subscription, and shows how the .Net client sets the properties that are in contrast to what notification requires.
https://social.msdn.microsoft.com/Forums/en-US/cf3853f3-0ea1-41b9-987e-9922e5766066/changing-default-set-options-forced-by-net?forum=adodotnetdataproviders
Then this post was from a user who, like me, had reduced his code to the simplest format. My original code pattern was similar to his.
https://social.technet.microsoft.com/Forums/windows/en-US/5a29d49b-8c2c-4fe8-b8de-d632a3f60f68/subscriptions-always-invalid-usual-suspects-checked-no-joy?forum=sqlservicebroker
Then I found this post, also a very simple reduction of the problem, only his was a simple issue - needing 2 part name for tables. In his case the suggestion resolved the issue. After looking at his code I noticed the main difference was waiting to open the connection object until AFTER the command object AND the dependency object were created. My only assumption is under the hood (I have not yet started reflector to check so only an assumption) the Connection object is opened differently, or order of events and command happen differently, because of this association.
https://social.msdn.microsoft.com/Forums/sqlserver/en-US/bc9ca094-a989-4403-82c6-7f608ed462ce/sql-server-not-creating-subscription-for-simple-select-query-when-using-sqlcachedependency?forum=sqlservicebroker
I hope this helps someone else in a similar issue.
Related
Is there a way to dump the generated sql to the Debug log or something? I'm using it in a winforms solution so the mini-profiler idea won't work for me.
I got the same issue and implemented some code after doing some search but having no ready-to-use stuff. There is a package on nuget MiniProfiler.Integrations I would like to share.
Update V2: it supports to work with other database servers, for MySQL it requires to have MiniProfiler.Integrations.MySql
Below are steps to work with SQL Server:
1.Instantiate the connection
var factory = new SqlServerDbConnectionFactory(_connectionString);
using (var connection = ProfiledDbConnectionFactory.New(factory, CustomDbProfiler.Current))
{
// your code
}
2.After all works done, write all commands to a file if you want
File.WriteAllText("SqlScripts.txt", CustomDbProfiler.Current.ProfilerContext.BuildCommands());
Dapper does not currently have an instrumentation point here. This is perhaps due, as you note, to the fact that we (as the authors) use mini-profiler to handle this. However, if it helps, the core parts of mini-profiler are actually designed to be architecture neutral, and I know of other people using it with winforms, wpf, wcf, etc - which would give you access to the profiling / tracing connection wrapper.
In theory, it would be perfectly possible to add some blanket capture-point, but I'm concerned about two things:
(primarily) security: since dapper doesn't have a concept of a context, it would be really really easy for malign code to attach quietly to sniff all sql traffic that goes via dapper; I really don't like the sound of that (this isn't an issue with the "decorator" approach, as the caller owns the connection, hence the logging context)
(secondary) performance: but... in truth, it is hard to say that a simple delegate-check (which would presumably be null in most cases) would have much impact
Of course, the other thing you could do is: steal the connection wrapper code from mini-profiler, and replace the profiler-context stuff with just: Debug.WriteLine etc.
You should consider using SQL profiler located in the menu of SQL Management Studio → Extras → SQL Server Profiler (no Dapper extensions needed - may work with other RDBMS when they got a SQL profiler tool too).
Then, start a new session.
You'll get something like this for example (you see all parameters and the complete SQL string):
exec sp_executesql N'SELECT * FROM Updates WHERE CAST(Product_ID as VARCHAR(50)) = #appId AND (Blocked IS NULL OR Blocked = 0)
AND (Beta IS NULL OR Beta = 0 OR #includeBeta = 1) AND (LangCode IS NULL OR LangCode IN (SELECT * FROM STRING_SPLIT(#langCode, '','')))',N'#appId nvarchar(4000),#includeBeta bit,#langCode nvarchar(4000)',#appId=N'fea5b0a7-1da6-4394-b8c8-05e7cb979161',#includeBeta=0,#langCode=N'de'
Try Dapper.Logging.
You can get it from NuGet. The way it works is you pass your code that creates your actual database connection into a factory that creates wrapped connections. Whenever a wrapped connection is opened or closed or you run a query against it, it will be logged. You can configure the logging message templates and other settings like whether SQL parameters are saved. Elapsed time is also saved.
In my opinion, the only downside is that the documentation is sparse, but I think that's just because it's a new project (as of this writing). I had to dig through the repo for a bit to understand it and to get it configured to my liking, but now it's working great.
From the documentation:
The tool consists of simple decorators for the DbConnection and
DbCommand which track the execution time and write messages to the
ILogger<T>. The ILogger<T> can be handled by any logging framework
(e.g. Serilog). The result is similar to the default EF Core logging
behavior.
The lib declares a helper method for registering the
IDbConnectionFactory in the IoC container. The connection factory is
SQL Provider agnostic. That's why you have to specify the real factory
method:
services.AddDbConnectionFactory(prv => new SqlConnection(conStr));
After registration, the IDbConnectionFactory can be injected into
classes that need a SQL connection.
private readonly IDbConnectionFactory _connectionFactory;
public GetProductsHandler(IDbConnectionFactory connectionFactory)
{
_connectionFactory = connectionFactory;
}
The IDbConnectionFactory.CreateConnection will return a decorated
version that logs the activity.
using (DbConnection db = _connectionFactory.CreateConnection())
{
//...
}
This is not exhaustive and is essentially a bit of hack, but if you have your SQL and you want to initialize your parameters, it's useful for basic debugging. Set up this extension method, then call it anywhere as desired.
public static class DapperExtensions
{
public static string ArgsAsSql(this DynamicParameters args)
{
if (args is null) throw new ArgumentNullException(nameof(args));
var sb = new StringBuilder();
foreach (var name in args.ParameterNames)
{
var pValue = args.Get<dynamic>(name);
var type = pValue.GetType();
if (type == typeof(DateTime))
sb.AppendFormat("DECLARE #{0} DATETIME ='{1}'\n", name, pValue.ToString("yyyy-MM-dd HH:mm:ss.fff"));
else if (type == typeof(bool))
sb.AppendFormat("DECLARE #{0} BIT = {1}\n", name, (bool)pValue ? 1 : 0);
else if (type == typeof(int))
sb.AppendFormat("DECLARE #{0} INT = {1}\n", name, pValue);
else if (type == typeof(List<int>))
sb.AppendFormat("-- REPLACE #{0} IN SQL: ({1})\n", name, string.Join(",", (List<int>)pValue));
else
sb.AppendFormat("DECLARE #{0} NVARCHAR(MAX) = '{1}'\n", name, pValue.ToString());
}
return sb.ToString();
}
}
You can then just use this in the immediate or watch windows to grab the SQL.
Just to add an update here since I see this question still get's quite a few hits - these days I use either Glimpse (seems it's dead now) or Stackify Prefix which both have sql command trace capabilities.
It's not exactly what I was looking for when I asked the original question but solve the same problem.
Almost every tutorial I have read seems to incorrectly setup SqlCacheDependency. I believe they normally mix up the outdated polling method with the query notification method.
Here are two of many examples:
Web Caching with SqlCacheDependency Simplified (non-microsoft)
SqlCacheDependency Class (Microsoft)
Based on my testing, if you are using the broker (MSSQL 2015+) you don't need to make any .config changes nor do you need to make any SqlCacheDependencyAdmin calls (Don't need to define tables, etc).
I simplify just do this...
SqlDependency.Start(connString)
...
queryString = "SELECT ...";
cacheName = "SqlCache" + queryString.GetHashCode();
...
using (var connection = new SqlConnection(connString))
{
connection.Open();
var cmd = new SqlCommand(queryString, connection)
{
Notification = null,
NotificationAutoEnlist = true
};
var dependency = new SqlCacheDependency(cmd);
SqlDataReader reader = cmd.ExecuteReader();
try
{
while (reader.Read())
{
// Set the result you want to cache
data = ...
}
}
finally
{
reader.Close();
}
HostingEnvironment.Cache.Insert(cacheName, data, dependency);
}
(The code that checks if the cache is null or not is not included, as that's all just setup. I just want to show the setting of the cache)
This seems to work without the need to define which tables are involved in the query and make complicated triggers on each table. It just works.
More surprising to me is that the rules for making a query have notification :
Creating a Query for Notification (Can't find documentation newer than 2008) don't seem to apply. I purpose to do a TOP in my SQL and it still works.
For a test, I have it run a query 1000 times involving a table named "Settings". Then I update a value in the table and repeat the query.
I watch the Profiler for any queries involving the word "Settings" and I see the query is executed just 1 time (to set the cache) and then the update statement occurs, and then the query is re-executed one more time (the cache was invalidated and the query ran again)
I am worried that in my 2-3 hours of struggling with the proper way to do this I am missing something and it really is this simple?
Can I really just put any query I want and it'll just work? I am looking for any pointers where I am doing something dangerous/non-standard or any small print that I am missing
var dependency = new SqlCacheDependency(cmd);
when you write query like this you autiomatically define table name in it.Your connection already has db name.
It is non explicit way to do same.
Explicit way to catch exception and to know what went wrong is this.
// Declare the SqlCacheDependency instance, SqlDep.
SqlCacheDependency SqlDep = null;
// Check the Cache for the SqlSource key.
// If it isn't there, create it with a dependency
// on a SQL Server table using the SqlCacheDependency class.
if (Cache["SqlSource"] == null) {
// Because of possible exceptions thrown when this
// code runs, use Try...Catch...Finally syntax.
try {
// Instantiate SqlDep using the SqlCacheDependency constructor.
SqlDep = new SqlCacheDependency("Northwind", "Categories");
}
// Handle the DatabaseNotEnabledForNotificationException with
// a call to the SqlCacheDependencyAdmin.EnableNotifications method.
catch (DatabaseNotEnabledForNotificationException exDBDis) {
try {
SqlCacheDependencyAdmin.EnableNotifications("Northwind");
}
// If the database does not have permissions set for creating tables,
// the UnauthorizedAccessException is thrown. Handle it by redirecting
// to an error page.
catch (UnauthorizedAccessException exPerm) {
Response.Redirect(".\\ErrorPage.htm");
}
}
// Handle the TableNotEnabledForNotificationException with
// a call to the SqlCacheDependencyAdmin.EnableTableForNotifications method.
catch (TableNotEnabledForNotificationException exTabDis) {
try {
SqlCacheDependencyAdmin.EnableTableForNotifications("Northwind", "Categories");
}
// If a SqlException is thrown, redirect to an error page.
catch (SqlException exc) {
Response.Redirect(".\\ErrorPage.htm");
}
}
// If all the other code is successful, add MySource to the Cache
// with a dependency on SqlDep. If the Categories table changes,
// MySource will be removed from the Cache. Then generate a message
// that the data is newly created and added to the cache.
finally {
Cache.Insert("SqlSource", Source1, SqlDep);
CacheMsg.Text = "The data object was created explicitly.";
}
}
else {
CacheMsg.Text = "The data was retrieved from the Cache.";
}
As documented in https://learn.microsoft.com/en-us/dotnet/api/system.web.caching.sqlcachedependency?view=netframework-4.8 "Using a SqlCacheDependency object with SQL Server 2005 query notification does not require any explicit configuration."
So, the CMD has explicit table names in it, and ADO.net is issuing the correct Service Broker configuration commands for you. When the table is updated, SQL Server posts a Service Broker message saying the table has been updated. When ADO.net validates the CMD it checks the explicit tables in the broker for updates.
This is why the SQlCacheDependency associated CMD must use explicit tables.
I have an intermittent Devart.Data.Linq.ChangeConflictException: Row not found or changed raising it's ugly head. The funny thing is, the change is still written to the database!
The stack trace says:
Devart.Data.Linq.ChangeConflictException: Row not found or changed.
at Devart.Data.Linq.Engine.b4.a(IObjectEntry[] A_0, ConflictMode A_1, a A_2)
at Devart.Data.Linq.Engine.b4.a(ConflictMode A_0)
at Devart.Data.Linq.DataContext.SubmitChanges(ConflictMode failureMode)
at Devart.Data.Linq.DataContext.SubmitChanges()
at Billing.Eway.EwayInternal.SuccessCustomerRenewal(String username, Bill bill, EwayTransaction transaction) in c:\Users\Ian\Source\Repos\billing-class-library\Billing\Billing\Eway\EwayInternal.cs:line 552
at Billing.Eway.Eway.BillAllUsers() in c:\Users\Ian\Source\Repos\billing-class-library\Billing\Billing\Eway\Eway.cs:line 138
And my code for Billing.Eway.EwayInternal.SuccessCustomerRenewal:
internal static void SuccessCustomerRenewal(string username, Bill bill, EwayTransaction transaction)
{
// Give them their points!
ApplyBillToCustomerAccount(username, bill, true);
BillingEmail.SendRenewalSuccessEmail(username, bill, transaction);
using (MsSqlDataClassesDataContext msSqlDb = new MsSqlDataClassesDataContext())
{
// TODO: Remove this logging
msSqlDb.Log = new StreamWriter(#"logs\db\" + Common.GetCurrentTimeStamp() + "-MsSQL.txt", true) { AutoFlush = true };
EwayCustomer ewayCustomer = msSqlDb.EwayCustomers.First(c => c.Username == username);
ewayCustomer.NextBillingDate = Common.GetPlanExpiry(bill.BillPlan);
using (MySqlDataContext mySqlDb = new MySqlDataContext())
{
// TODO: Remove this logging
mySqlDb.Log = new StreamWriter(#"logs\db\" + Common.GetCurrentTimeStamp() + "-MySQL.txt", true) { AutoFlush = true };
BillingMySqlContext.Customer grasCustomer = mySqlDb.Customers.First(c => c.Username == username);
// Extend their membership date out so that the plan doesn't expire because of a failed credit card charge.
grasCustomer.MembershipDate =
ewayCustomer.NextBillingDate.AddDays(1);
mySqlDb.SubmitChanges(); // <-- This is line 552
}
msSqlDb.SubmitChanges();
}
}
I know that the issue occurs on the mySqlDb.SubmitChanges() line, since that DB context is the one using Devart (Linq solution for MySQL databases): the other context uses pure MS Linq.
Not only is the change written to the MySql DB (inner using block), but it is also written to the MsSql DB (outer using block). But that's where the magical success ends.
If I could I would write a Minimal, Complete and Verifiable example, but strangely I'm unable to generate a Devart ChangeConflictException.
So, why does the change get saved to the database after a Devart.Data.Linq.ChangeConflictException? When I previously encountered System.Data.Linq.ChangeConflictException changes weren't saved.
Edit 1:
I've also now included the .PDB file and gotten line number confirmation of the exact source of the exception.
Edit 2:
I now understand why I can't generate a ChangeConflictException, so how is it happening here?
These are the attributes for MembershipDate:_
[Column(Name = #"Membership_Date", Storage = "_MembershipDate", CanBeNull = false, DbType = "DATETIME NOT NULL", UpdateCheck = UpdateCheck.Never)]
I know I can explicitly force my changes through to override any potential conflict, but that seems undesirable (I don't know what I would be overriding!). Similarly I could wrap the submit in a try block, and retry (re-reading each time) until success, but that seems clunky. How should I deal with this intermittent issue?
Edit 3:
It's not caused by multiple calls. This function is called in one place, by a single-instance app. It creates log entries every time it is run, and they are only getting created once. I have since moved the email call to the top of the method: the email only gets sent once, the exception occurs, and database changes are still made.
I believe it has something to do with the using blocks. Whilst stepping through the debugger on an unrelated issue, I entered the using block, but stopped execution before the SubmitChanges() call. And the changes were still written to the database. My understanding was that using blocks were to ensure resources were cleaned up (connections closed, etc), but it seems that the entire block is being executed. A new avenue to research...
But it still doesn't answer how a ChangeConflictException is even possible given Devart explicitly ignores them.
Edit 4:
So I wasn't going crazy, the database change did get submitted even after I ended execution in the middle of the using block, but it only works for websites.
Edit 5:
As per #Evk's suggestion I've included some DB logging (and updated the stacktrace and code snippet above). The incidence rate of this exception seems to have dropped, as it has only just happened since I implemented the logging. Here are the additional details:
Outer (MS SQL) logfile:
SELECT TOP (1) [t0].[id], [t0].[Username], [t0].[TokenId], [t0].[PlanId], [t0].[SignupDate], [t0].[NextBillingDate], [t0].[PaymentType], [t0].[RetryCount], [t0].[AccountStatus], [t0].[CancelDate]
FROM [dbo].[EwayCustomer] AS [t0]
WHERE [t0].[Username] = #p0
-- #p0: Input NVarChar (Size = 4000; Prec = 0; Scale = 0) [dyonis]
-- Context: SqlProvider(Sql2008) Model: AttributedMetaModel Build: 4.0.30319.18408a
(It just shows the SELECT call (.First()), none of the updates show).
Inner (MySQL) logfile:
SELECT t1.Customer_ID, t1.Username, t1.Account_Group, t1.Account_Password, t1.First_Name, t1.Last_Name, t1.Account_Type, t1.Points, t1.PromoPoints, t1.Phone, t1.Cell, t1.Email, t1.Address1, t1.Address2, t1.City, t1.State, t1.Country, t1.Postcode, t1.Membership_Group, t1.Suspend_On_Zero_Points, t1.Yahoo_ID, t1.MSN_ID, t1.Skype_ID, t1.Repurchase_Thresh, t1.Active, t1.Delete_Account, t1.Last_Activity, t1.Membership_Expires_After_x_Days, t1.Membership_Date, t1.auth_name, t1.created_by, t1.created_on, t1.AccountGroup_Points_Used, t1.AccountGroup_Points_Threashold, t1.LegacyPoints, t1.Can_Make_Reservation, t1.Gallery_Access, t1.Blog_Access, t1.Private_FTP, t1.Photometrica, t1.Promo_Code, t1.Promo_Expire_DTime, t1.Gift_FirstName, t1.Gift_LastName, t1.Gift_Email, t1.Gift_Phone, t1.Gift_Active, t1.NoMarketingEmail, t1.Can_Schedule, t1.Refered_By, t1.Q1_Hear_About_Us, t1.Q2_Exp_Level, t1.Q3_Intrests, t1.GIS_DTime_UTC, t1.Membership_Expire_Notice_Sent, t1.Promo_Expire_Notice_Sent, t1.isEncrypted, t1.PlanId
FROM grasbill.customers t1
WHERE t1.Username = :p0 LIMIT 1
-- p0: Input VarChar (Size = 6; DbType = AnsiString) [dyonis]
-- Context: Devart.Data.MySql.Linq.Provider.MySqlDataProvider Mapping: AttributeMappingSource Build: 4.4.519.0
UPDATE grasbill.customers SET Membership_Date = :p1 WHERE Customer_ID = :key1
-- p1: Input DateTime (Size = 0; DbType = DateTime) [8/3/2016 4:42:53 AM]
-- key1: Input Int (Size = 0; DbType = Int32) [7731]
-- Context: Devart.Data.MySql.Linq.Provider.MySqlDataProvider Mapping: AttributeMappingSource Build: 4.4.519.0
(Shows the SELECT and UPDATE calls)
So the log files don't really give any clue as to what's happening, but again the MS SQL database has been updated! The NextBillingDate field has been set correctly, as per this line:
ewayCustomer.NextBillingDate = Common.GetPlanExpiry(bill.BillPlan);
If it hadn't been updated, the user would have been billed again on the next timer tick (5 mins later), and I can see from logging that didn't happen.
One other interesting thing to note is the log file timestamps. As you can see from the code above I grab the current (UTC) time for the log filename. Here is the information shown by Windows File Explorer:
The MS SQL logfile was created at 04:42 (UTC) and last modified at 14:42 (UTC+10, Windows local-time), but the MySQL logfile was last modified at 15:23 (UTC+10), 41 minutes after it was created. Now I assume the logfile StreamWriter is closed as soon as it leaves scope. Is this delay an expected side effect of the exception? Did it take 41 minutes for the garbage collector to realise I no longer needed a reference to the StreamWriter? Or is something else going on?
Well 6 months later I finally got to the bottom of this problem. Not sure if it will ever help anyone else, but I'll detail it anyway.
There were 2 problems in play here, and 1 of them was idiocy (as they usually are), but one was legitimately something I did not know or expect.
Problem 1
The reason the changes were magically made to the database even though there was an exception was because the very first line of code in that function ApplyBillToCustomerAccount(username, bill, true); updates the database! <facepalm>
Problem 2
The (Devart) ChangeConflictException isn't only thrown if the data has changed, but also if you're not making any changes. MS SQL stores DateTimes with great precision, but MySQL (or the one I'm running at least) only stores down to seconds. And here's where the intermittency came in. If my database calls were quick enough, or just near the second boundary, they both got rounded to the same time. Devart saw no changes to be written, and threw a ChangeConflictException.
I recently made some optimisations to the database which resulted in far greater responsiveness, and massively increased incidence of this exception. That was one of the clues.
Also I tried changing the Found Rows parameter to true as instructed in the linked Devart post but found it did not help in my case. Or perhaps I did it wrong. Either way now that I've found the source of the issue I can eliminate the duplicate database updates.
I have a distributed client server system which has multiple 'worker' engines and a WCF service, lets call it, FileManagerService, which adds 'jobs' to a database for the 'worker' engines to pick up and process.
The database consists of two tables: Jobs and File. Every job operates on a File ( many to one relationship ). Multiple jobs can be added to the Jobs table at once, they are processed in the fashion of a queue.
Example of Job's are actions such as : MoveFile or DeleteFile, etc.
I am trying to achieve mutual exclusion on the File record by using a 'locked' ( bool ) column in the File table itself.
The mutual exclusion zones that I think are needed are within the WCF service FileManagerService's method: addJob() and within the 'worker' engines' TakeAndProcessAJob().
At the moment I am having trouble with working out how linq-to-sql deferred execution and how the linq-to-sql transaction system works.
I currently have the following method which is supposed to take a Job from the Jobs table, check if its related File is locked, if it is locked we will return nothing, if it is not locked we will lock the File and return a Job, and delete that Job from the Jobs queue:
public static Job GetAnAvailableJob()
{
using (DBDataContext db = new DBDataContext())
{
DataLoadOptions loadOptions = new DataLoadOptions();
loadOptions.LoadWith<Job>(f => f.File);
db.LoadOptions = loadOptions;
var jobAvailable = from ja in db.Jobs
where ja.File.locked == false
select ja;
var jobToTake = jobAvailable.FirstOrDefault();
// This file temp is here so that we can return
// an associated File with the Job.
// We delete
File fileTemp = null;
if (jobToTake != null)
{
fileTemp = jobToTake.File;
jobAvailable.File.locked = true;
Console.WriteLine("Locked file:" + fileTemp.FileID);
db.Jobs.DeleteOnSubmit(jobToTake);
db.SubmitChanges();
jobToTake.Asset = fileTemp;
}
return jobToTake;
}
}
The 'worker' engines TakeAndProcessAJob() basically does the following: Call GetAnAvailableJob(), if return object is not null -> proccess the job, if it is null -> sleep.
The WCF Service FileManagerService uses this locking method:
public static bool LockAsset(long FileID)
{
using (DBDataContext db = new DBDataContext())
{
var fileToLock = db.Files.Where(f => f.FileID == FileID && f.locked == false).Single();
if (fileToLock == null)
{
return false;
}
else
{
Console.WriteLine("Locked File:" + fileToLock.AssetID);
fileToLock.locked = true;
db.SubmitChanges();
return true;
}
}
}
What do I need to change in these two methods to make then behave in a transactional manner and have mutual exclusion on the File record. Currently I am getting change conflict exceptions - row not changed or found - on the SubmitChanges() within the TakeAndProcessAJob() method.
Thanks
That "row not changed or found" error can be something as simple as your database model file not matching the actual physical database schema.
Another reason might be that LINQ didn't detect any changes to submit. Say if the record you were selecting was already true and you mark it as true. You could avoid this by not updating if its already true or possibly the cause is there's a setting on the modal file under the properties for that field on the table that you can actually turn off UpdateCheck, but it doesn't sound like thats what you want.
In an unrelated story:
doesn't ".Single()" throw an error when nothing is found as aposed to firstOrDefault() which just returns null ?
I could not figure out the answer to my question and so I did some logic restructuring and devised a workaround using plain old SQL. To lock the record I now use a single line ( atomic ) command on the record:
UPDATE dataTable SET locked = 'True' WHERE (locked = 'False') AND (recordID= 15)
I then use SqlDataReader and check the RecordsAffected field. If records have changed, it means that I have successfully locked the record. If records have not changed, it means the record is already locked, and so I return false on my locking method.
The process that is trying to lock the record then sits in a while loop calling lockRecord and waiting for a successful callback.
I'm not sure if it was completely necessary to make the switch from LINQ to SQL. I made the logic changes in my application, and the switch to plan SQL at the same time and so it is difficult to identify if using LINQ was THE problem.
So, we've got this set of code that, for some reason, keeps timing out. It's not the stored procedure that it's running, because that runs fine. Also, if we remove the parameter from the c# code, the code runs. The parameter keeps breaking (causing it to time out) and we can't figure out why.
c#:
public static PTWViewList GetList(int studynumber)
{
PTWViewList tempList = new PTWViewList();
using (SqlConnection myConnection = new SqlConnection(AppConfiguration.cnARDB))
{
string spName = "ardb.PTWViewSelect";
SqlCommand myCommand = new SqlCommand(spName, myConnection);
myCommand.CommandType = CommandType.StoredProcedure;
myCommand.Parameters.AddWithValue("#study", studynumber);
myConnection.Open();
using (NullableDataReader myReader = new NullableDataReader(myCommand.ExecuteReader())) /*this is where the code times out*/
{
tempList = new PTWViewList();
while (myReader.Read())
{
tempList.Add(FillDataRecord(myReader));
}
myReader.Close();
}
}
tempList.ListCount = tempList.Count;
return tempList;
}
stored procedure:
CREATE PROCEDURE [ardb].[PTWViewSelect]
#studynumber int = NULL,
#quoteid uniqueidentifier = NULL,
#lineitemid uniqueidentifier = NULL
AS
BEGIN
SET NOCOUNT ON;
SELECT
[Study]
,[LineItemID]
,[QuoteID]
,[Total]
,[COOP]
,[VendorCost]
,[CustCost]
,[LineItemNumber]
,[StudyTypeCode]
,[GroupLeader]
,[PTWDate]
,[PONumber]
,[POStatus]
,[StudyDirector]
,[SL_DESC_L]
,[SL_Code]
,ProjectDescription
,CreatedBy
,chARProcess
,CODate
FROM
[ARDB].[dbo].[PTWView]
WHERE
(#studynumber is null or StudyNumber=#studynumber)
AND (#quoteid is null or QuoteID=#quoteid)
AND (#lineitemid is null or LineItemID = #lineitemid)
END
have you tried
myCommand.Parameters.AddWithValue("#studynumber", studynumber);
instead of:
myCommand.Parameters.AddWithValue("#study", studynumber);
EDIT
If passing parameters is the problem, then it comes down to how much time the stored procedure takes to execute. Default timeout for SQL server is usually 120 secs. You can add "Connect Timeout" to increase timeout in your DB connection string and check out.
** Old Answer -- Ignore **
Without stack trace, and taking your word that the stored procedure is fine, I am guessing that it is timing out due to the connection failure. The code is unable to connect to your DB server and hence timing out.
setting arithabort off made the sp take 45 seconds as opposed to 1. setting it back on changed it back to 1. I updated the stored procedure to set it on, no change in the app. Changed it to off, no change. I then removed the update and then the app worked fine.
I believe what happened is that updating the stored procedure caused it to recompile, fixing the issue. I'm not 100% sure on this though.
One thing could be the ARITHABORT setting, set it to ON...NET defaults to OFF
run the proc in SSMS with ARITHABORT set to OFF and see if it runs slower now like from .NET
example
MyConnection.Execute "SET ARITHABORT ON"
Another thing is that your WHERE clause is not optimal, take a look at Do you use Column=#Param OR #Param IS NULL in your WHERE clause? Don't, it doesn't perform
does the proc run slow with parameters in SSMS? Can you show the execution plan?