Is there a way to send a correlation ID from C# code to SQL Server at the command level?
For instance, using x-correlation-id is an accepted way to track a request down to all parts of the system. we are looking for a way to pass this string value to stored procedure calls in SQL Server.
I spent sometime reading thru documents and posts but I was not able to find anything useful.
Can someone please let me know if there is a way to do this? The goal is to be able to track a specific call thru all services (which we can now) and DB calls (which we cannot and looking for a solution.)
I know the answer here is one year later. But in case, somebody has the same question.
Since EF core 2.2, MS provides a new method called "TagWith()" which you could pass your own annotation with the EF query into SQL server. In this way, you could easily track the SQL query with the same correlation id generated in your C# code.
https://learn.microsoft.com/en-us/ef/core/querying/tags
Unfortunately, this new feature is not available in EF 6. But it is not only us in this situation. If you just need a simple solution, you could check the thread here and MS documents.
If you need a more stable solution, you could check this NuGet plugin for EF 6 as well.
To pass your correlation id to SQL Server you have two options:
explicitly pass it as a parameter to your queries & stored procedures.
This is annoying as it requires work to change all your db calls to have a parameter like #correlationId, and often doesn't make sense having that parameter for simple data-retrieval queries. Perhaps you decide to only pass it for data-modification operations.
But on the positive side it's really obvious where the correlation info comes from (i.e. nobody reading the code will be confused) and doesn't require any additional db calls.
If all your data-modification is done using stored procs I think this is a good way to go.
use SQL Server's SESSION_CONTEXT(), which is a way you can set session state on a connection that can be retrieved from within stored procs etc.
You can find a way to inject it into your db layer (e.g. this) so the session context is always set on a connection before executing your real db calls. Then within your procs/queries you get the correlation id from the SESSION_CONTEXT and write to wherever you want to store it (e.g. some log table or as a column on tables being modified)
This can be good as you don't need to change each of your queries or procs to have the #correlationId parameter.
But it's often not so transparent how the session context is magically set. Also you need to be sure it's always set correctly which can be difficult with ORMs and connection pooling and other architectural complexities.
If you're not already using stored procs for all data modification, and you can get this working with your db access layer, and you don't mind the cost of the extra db calls this is a good option.
I wish this was easier.
Another option is to not pass it to SQL Server, but instead log all your SQL calls from the tier that makes the call and include the correlation id in those logs. That's how Application Insights & .NET seems to do it by default: logging SQL calls as a dependency along with the SQL statement and the correlation id.
Related
I'm creating a website content management system which stores a whole bunch of website articles and let user be able to modify these articles through the system. I'm a typical SQL Server developer however I'm thinking maybe this system can be done in DocumentDB.We are using C# plus WebAPI to do the read and write. I'm testing different data access technology to see which one performs better. I have been trying Ling, Linq Lambda, SQL and Stored Procedure. The thing is all these query methods seems all running around 600ms to 700ms when I test via Postman. For example, one of my test is a simple Get http://localhost:xxxxxx/multilanguage/resources/1, which would take 600ms+. That was only a 1 kb document and there are only have 5 documents stored in my collection so far. So I guess what I want to ask is: is there a quicker way to query DocumentDB than this. The reason I ask is because I did something similar in SQL Server before(not to query document, it was for relational tables). A much more complex query in a stored procedure on multiple joined tables only takes around 300ms. So I guess there should be a quicker way to do this. Thanks for any suggestions!
Most probably if you will change implementation to stab you will get same performance since actually you are testing connection time between yours server and client (postman).
There's a couple things you can do, but do keep in mind that DocumentDB, and other NoSQL solutions behave very differently than standard SQL Server. For example, the more nodes and RAM available to DocumentDB the better it will perform overall. The development instance of DocumentDB on Azure is understandably going to use fewer resources than a production instance. Since Azure takes care of scaling, one way to think about it is that the more data you have the better it will perform.
That said, something you are probably not used to is sharing your connection object for your whole application. That avoids the start up penalties every time you want to get your data. Summarizing Performance Tips:
Use TCP connection instead of HTTPS when you can
Use await client.OpenAsync() to avoid pausing on start up latency for the first request
Connect to the DocumentDB in the same region (keep in mind if you host across regions)
Use a singleton to access DocumentDB (it's threadsafe)
Cache your SelfLinks for quick access
Tune your page sizes so that you get only the data you intend to use
The more advanced performance tips cover index policies, etc. DocumentDB and other NoSQL databases behave differently than SQL databases. That also means your assumptions about how the APIs work are probably wrong. Make sure you are testing similar concepts. The SQL Server database connection object needs you to create/dispose of objects for each transaction so it can return those connections back to a connection pool. Treating DocumentDB the same way is going to cause the same kind of performance problems as if you didn't use a connection pool.
I have a SQL Server 2012 stored procedure that returns a table. I have to modify that SP to add an additional value to the returned table. Unfortunately, that added value comes from a call to a web-service. From my research, I gather the main ways to do this are using the OLE Automation procedures (sp_OA...) in SQL, or a SQLCLR stored procedure. Given the security context in which the sp_OA... procedures run, the single return value is a VARCHAR(10) registration key, and calls to the service are few (ten to twenty per hour), I'm guessing the SQLCLR method is the way to go. Also, the web-service is hosted on our intranet, and not accessible to the outside world.
Is there a better way to accomplish what I need? Better meaning more performant, better security, easier to code and maintain
Please do not use the sp_OA* OLE Automation procedures. They do not appear to be officially deprecated, but SQLCLR replaces both the OLE Automation procedures as well as Extended Stored Procedures.
Yes, this can be done easily enough in SQLCLR. You can find examples on using WCF (as shown in #CodeCaster's answer) or using HttpWebRequest / HttpWebResponse (I have more info in this answer: How to invoke webservice from SQL Server stored procedure ). Also, please be aware that sometimes you will need to also add the Serialization Assembly: Using Webservices and Xml Serialization in CLR Integration
Coding and Maintenance
Web Services provide a nice API, but if you change the structure you will have to recompile and redeploy at least some part of this. Assuming the information being exchanged is simple enough, I tend to think that treating this as a standard web request adds a lot of flexibility. You can create a generic web request function (scalar or TVF) that takes in the parameters and URI and constructs the properly formatted XML request and sends it to the URI. It then gets the response and merely returns the XML. So you shift a little bit of the responsibility since you now need to parse the XML response rather than getting a nice object. But, XML is easy to parse in SQL Server, and you can re-use this function in any number of places. And, if the remote service is ever updated, updating a Stored Procedure to change the query string that is passed to the Web Service and/or change the parsing of the XML response is a simple ALTER PROCEDURE and should be easy to test. No need to recompile / redeploy the SQLCLR Assembly.
Security
Regardless of how "pure" of a web service call you want, the main thing, security wise, is to NOT be lazy and turn TRUSTWORTHY ON (as also shown in the linked page from #CodeCaster's answer, and unfortunately most other examples here on the interwebs). The proper way to make this secure is to do the following:
Sign your Assembly
In the [master] database, create an Asymmetric Key from the DLL of your Assembly.
Also, in [master], create a Login from that Asymmetric Key
Grant your new Login the EXTERNAL ACCESS ASSEMBLY permission
Create your Assembly with a PERMISSION_SET of EXTERNAL_ACCESS, not UNSAFE
For more details on:
using SQLCLR in general, please visit: SQLCLR Info
using Module Signing, please visit: Module Signing Info
not using TRUSTWORTHY ON, please read: PLEASE, Please, please Stop Using Impersonation, TRUSTWORTHY, and Cross-DB Ownership Chaining
You can definitely call a WCF service using SQL CLR.
If you don't want that, you could write a Windows Service in C# that watches or polls the table for changes. Depending on how you implement this service, the reaction to a new record would be near immediate. Read also How to notify a windows service(c#) of a DB Table Change(sql 2005)?.
Then you can perform the service call from C#, perform the required work and store the result in the column.
When you require more information, for example extra variables obtained during the exchange, you could introduce a new table for storing that, and the actual result you're interested in. Then join that table from the table in your question.
In the beginning of our app's development, we were using SqlDependency quite heavily to cache DB results until the notifications told our app to grab a fresh copy.
During testing, we've noticed that the SQL DB's performance was getting hammered by the SqlDependency notification service. We scaled back the number of tables that we were using SqlDependency and noticed a large gain in performance. So, we thought we were just over using it and we moved on. We are down to only a few tables now.
Later, we discovered that we couldn't scale back the security access level for the username that will establish the dependency. We could have more than one connection string for each DB (one for dependency and one for the rest of the app), but with multiple DBs and DB mirroring, this is a pain (from SQL DB admin point of view and app development).
At this point, we are just thinking about moving away from SqlDependency altogether based on the following logic:
We don't need "instant" notification that the data has changed. If we knew within 1 second, that would be fast enough.
With some slight refactoring, we could get it down to just 1 table and poll that table once a second.
Does anyone see a flaw in this logic?
Would polling one table once a second cause more or less load on the DB than SqlDependency?
Has anyone had similar performance issue with SqlDependency?
I do dare try answer your question. But I am not sure you'll get the answer you was hoping for...
I remember back in the early 90ies when Borland promoted this grand new feature of 'callbacks' in their database Interbase that would give the caller (Delphi) 'notifications' via some very nifty new tech where promises was made that the database could be 'active'.
This was later known as the 'waste of time theory'.
And I guess why this never took of is perhaps that while the concept of DBMS was looking very promising, the database is one of your tiers that you can only scale up and not horizontally.
So programming languages to the rescue. Or rather the idea of Service Oriented Architecture (SOA). Many confuse SOA for 'Webservices' that was indeed an included hype in this new concept.
But if you check out the Fiefdom/Emissary design pattern (or Master/Agent pattern renamed to make it sound more cool and professional), you will find that the major idea is having exclusive control of its resources (read databases) and that all calls are being funneled via one single data adapter.
Obviously such a design does not work at all with triggers nor any callback frameworks.
But I think you should reconsider your entire design. If you funnel all actions and all calls via a single 'DataLayer', perhaps using Entity Framework, and perhaps on top on that a Caching mechanism you would not have to rely on your database to forward messages back up the food chain.
To show how weird things can get when being to 'database-centric', here is an extreme actual live example of how not to send an email, written a long long time ago, by a coder I was not so much impressed with:
Fact 1: Sql Server can send emails.
Fact 2: Asp3 coder does not know if or how this can be done in VbScript.
Asp3: read textbox email-address, send to com+ layer
Com+: take email-address and forward to datalayer
Datalayer: take email-address and forward to a stored procedure
Sproc: take email-address and forward to sql function
function: do weird sub-string things to check that email-adress has # . in it. return true or false.
Sproc: return a recordset with one column and one row containing 1 or 0
Datalayer: return the table as is.
Com+: convert the first column and row with value 1 or 0 to true or false
Asp3: if true, send email-adress with email subject and email text to com+
Com+: sends the exact information to datalayer
Datalayer: calls an stored procedure..
Sproc: calls a sql-function...
function: uses sql server email agent to send the email
If you read this far, my advice is to let sql server manage tables, relations, indexes and transactions. It is very good at that. Anything beyond those tasks, and with that I do include cursors in stored procedures, is better handled via proper code.
I am migrating an existing .NET 2.0, SQL Server codebase to a .NET 4.0, SQL Server 2008 environment.
The design pattern is that all app calls to the database go through a stored procedure. So there's a get[object name] stored procedure that needs to be created or altered for most select statements.
So the disadvantages of this architecture to me are already evident: inconvenience.
What are the advantages of this highly enforced stored procedure design? (now in .NET 4.0 as opposed to using an ORM).
Actually - contrary to popular belief - performance isn't one of the advantages of stored procedures - not anymore. Properly written "inline" SQL queries with parameters are just as fast, get "compiled" once by SQL Server (before first use) and remain in the procedure cache of SQL Server just as long as any stored procedure.
But stored procedures do have advantages - two main ones I'd like to mention:
the shield the user from the underlying tables. Which also means: it's another layer in your security system. Your database users do not need access to the tables - and thus they won't be able to cause any grief on those tables, either - by accessing them via Excel or Access or some other tool. This alone can be a huge benefit.
the second point is having a layer of stored procedure can give your DBA a place to optimize. You as a developer only call stored procedures - and the DBA can tweak them, fine tune them, make them run faster. As long as the parameter list and the return result set remain the same - you as a frontend developer won't even notice (at least not in a negative way!)
I take the approach of stored procs for INSERT/ UPDATE / DELETE for objects and do SELECTs in application code. Advantages:
Clear separation of business logic and data
Data security is better because it is controlled at the database layer.
Doing SELECTs in business logic is a compromise that anyone can read table data if they get the database login credentials, but they cant modify it (assuming you setup object level permissions correctly (tables read-only)), but i don't have to write a stored proc for every variant of where criteria.
its easier to customize data operations when you write your own data adapters vs ORMs
ORMs are fine, but there's typically alot of overhead in ORMs and i like the approach of my applications creating the least amount of work possible for the machines they run on. Plus I know exactly what is happening and there's less 'magic' happening behind the scenes
Disadvantages:
You can generate alot of code if you don't use ORMs, which means more to maintain.
It's fair to say that writing your own data adapters is re-inventing the wheel. more control always comes with a cost
Stored procedures greatest benefit is - execution time. If you have "heavy" SQL queries you should use SP.
I'm working on an application that uses SQL Server and NHibernate. We have the concept of default data (complex entities) that needs to be created for each new entity. This data can be changed on a per-user basis. However, we're struggling with the best way to create this data.
For example, lets say my application has a Store entity which has several default Products that I want to create when a new Store gets created. Anything about aProduct can be modified by managers of each Store.
As I see it, there are two main options:
Keep the default data in code and write it to the database once the new entity is created.
Keep the default data in the database and move it over with a stored procedure/raw SQL when the entity is created.
Instinctively, I lean toward option two, since databases are great at moving and manipulating sets of data, and option one would require a ton of messy code that could get out of hand.
However, writing a stored procedure or raw SQL presents its own issues:
We would have to re-write the stored procedure or SQL depending on the database we're using
We would be subverting the ORM in a way (not sure if this is actually wrong). That is, we'd be moving data around without using NHibernate
I found this article by Ayende Rahien which outlines how to perform a bulk delete. I am thinking that doing something similar for inserting default data would be fine. I also found an nhibernate users groups post (called "Schema export and default data"--SO won't let me post two links) that describes a similar situation, but it doesn't seem like there's a consensus on what the right solution is (although Ayende does offer some feedback and suggests that the data live in the database).
After writing this, I'm leaning even more toward using a stored procedure, I'm just worried about possible pitfalls of mixing two database access strategies (directly calling SProcs and using an ORM).
Any feedback is appreciated!
Edit: Removed "immutable" language. I'm specifically talking about default data that can change so I think this term was incorrect/confusing here.
I would create a default data service that creates those data in code, and use a factory to create your store and use the default data service to generate the default entities.
Using a Stored Procedure definitely defeats the point of having an ORM.