We have a process that needs to fire when a change occurs to a specific database table in Oracle. At the moment a similar process has been developed using triggers and a bunch of subsequent database actions that occur when that trigger is fired.
However, in this solution we want to call a .NET component (most likely a service) when the change occurs to a row or bunch of rows in a database table. Now, you could implement a polling mechanism that will check the table at regular intervals for those modifications and then instantiate the service when it finds any. However, I would prefer a more event driven approach.
I assume this is something that has been done elsewhere so I was wondering what approaches other people have used for dealing with such requirements?
Thanks in advance
Edit: The process that fires when a change occurs to the underlying data is essentially a call to an external web service using some of the related data. I am beginning to think whether this call should occur as part of the same process that is submitting the data into the database, rather than being triggered by the data change itself.
You should look at Oracle Database Extensions for .NET.
From the linked article:
Oracle Database Extensions for .NET provides the following:
A Common Language Runtime (CLR) host for Oracle Database
Data access through Oracle Data Provider for .NET classes
Oracle Deployment Wizard for Visual Studio .NET
You would still use triggers to detect the db changes but instead of firing all the db-side logic you describe you would now be able to execute that logic from a .NET module.
If you are using Oracle's .NET driver, you can use Oracle Continuous Query Notification (CQN) to do that.
You just give it a normal SELECT query, and it will fire a callback in your app whenever the resultset for that query changes.
The one caveot I know of is that when it initially runs the query to subscribe for continuous notification, it momentarily requires an exclusive lock. Usually its not a big deal since you just evecute it once at startup, so any other DB queries on the same table will be blocked for a fraction of a second.
It sounds possible but will likely take some leg work. I think you want to look into the Oracle Access Manager
http://download.oracle.com/docs/cd/E12530_01/oam.1014/e10355/toc.htm
This is similar to Paul's; but does not assume that you have Oracle installed on a Windows machine.
You may use dbms_scheduler to create a job that will call your external process.
You may directly call an remote external job from Oracle (this requires Oracle Scheduler Agent to be installed but nothing else)
it requires a bit of leg work to get the authentication set up and such, but this works.
then you utilize an event to start your job(called from your trigger).
This way, you may actually be able to utilize a lot of what you already have coded and just have the oracle scheduler to handle the rest.
Oracle provides two mechanisms to deal with what describe in a rather nice way... on the DB side you implement triggers for detecting the changes or whatever should result in an action on the .NET side...
For the communication you use a publish/subsribe mechanism based on the Oracle built-in queueing technology (called Advanced Queueing Technology, AQ) - for a nice example see http://www.devart.com/dotconnect/oracle/docs/AQ.html (not affiliated, just a happy customer).
Another option is to use built-in DBMS_ALERTER package for communication which is transactional and asynchronous - see for an example http://www.devart.com/dotconnect/oracle/docs/Devart.Data.Oracle~Devart.Data.Oracle.OracleAlerter.html
Just to be clear:
the above technologies (DBMS_ALERTER and AQ) are Oracle built-in ones, not specific to any 3rd-party libraries... you just need an ADO.NET provider supporting them...
EDIT - after the EDIT from the OP:
If you have control over the code or the call to the code that triggers the data change (WebService?) then it is indeed the best way to deal with it is purely on the .NET side of things... this also helps to deal with situations where such a change runs into an error etc.
Related
I have been working on an application for a couple of years that I updated using a back-end database. The whole key is that everything is cached on the client, so that it never requires an network connection to operate, but when it does have a connection it will always pickup the latest updates. Every application updated is shipped with the latest version of the database and I wanted it to download only the minimum amount of data when the database has been updated.
I currently use a table with a timestamp to check for updates. It looks something like this.
ID - Name - Description- Severity - LastUpdated
0 - test.exe - KnownVirus - Critical - 2009-09-11 13:38
1 - test2.exe - Firewall - None - 2009-09-12 14:38
This approach was fine for what I previously needed, but I am looking to expand more function of the application to use this type of dynamic approach. All the data is currently stored as XML, but I do not want to store complete XML files in the database and only transmit changed data.
So how would you go about allowing a fairly simple approach to storing dynamic content (text/xml/json/xaml) in a database, and have the client only download new updates? I was thinking of having logic that can handle XML inserted directly
ID - Data - Revision
15 - XXX - 15
XXX would be something like <Content><File>Test.dll<File/><Description>New DLL to load.</Description></Content> and would be inserted into the cache, but this would obviously be complicated as I would need to load them in sequence.
Another approach that has been mentioned was to base it on something similar to Source Control, storing the version in the root of the file and calculating the delta to figure out the minimal amount of data that need to be sent to the client.
Anyone got any suggestions on how to approach this with no risk for data corruption? I would also to expand with features that allows me to revert possibly bad revisions, and replace them with new working ones.
It really depends on the tools you are using and the architecture you already have. Is there already a server with some logic and a data access layer?
Dynamic approaches might get complicated, slow and limit the number of solutions. Why do you need a dynamic structure? Would it be feasible to just add data by using a name-value pair approach in a relational database? Static and uniform data structures are much easier to handle.
Before going into detail, you should consider the different scenarios.
Items can be added
Items can be changed
Items can be removed (I assume)
Adding is not a big problem. The client needs to remember the last revision number it got from the server and you write a query which get everything since there.
Changing is basically the same. You should care about identification of the items. You need an unchangeable surrogate key, as it seems to be the ID you already have. (Guids may be useful here.)
Removing is tricky. You need to either flag items as deleted instead of actually removing them, or have a list of removed IDs with the revision number when they had been removed.
Storing the data in the client: Consider using a relational database like SQLite in the client. (It doesn't need installation, it is just storing in a file. Firefox for instance stores quite a lot in SQLite databases.) When using the same in the server, you can probably reuse some code. It is also transaction based, which helps to keep it consistent (rollback in case of error during synchronization).
XML - if you really need it - can be stored just as a string in the database.
When using an abstraction layer or ORM that supports SQLite (eg. NHibernate), you may also reuse some code even when there is another database used by the server. Note that the learning curve for such an ORM might be rather steep. If you don't know anything like this, it could be too much.
You don't need to force reuse of code in the client and server.
Synchronization itself shouldn't be very complicated. You have a revision number in the client and a last revision in the server. You get all new / changed and deleted items since then in the client and apply it to the local store. Update the local revision number. Commit. Done.
I would never update only a part of a revision, because then you can't really know what changed since the last synchronization. Because you do differential updates, it is essential to have a well defined state of the client.
I would go with a solution using Sync Framework.
Quote from Microsoft:
Microsoft Sync Framework is a comprehensive synchronization platform enabling collaboration and offline for applications, services and devices. Developers can build synchronization ecosystems that integrate any application, any data from any store using any protocol over any network. Sync Framework features technologies and tools that enable roaming, sharing, and taking data offline.
A key aspect of Sync Framework is the ability to create custom providers. Providers enable any data sources to participate in the Sync Framework synchronization process, allowing peer-to-peer synchronization to occur.
I have just built an application pretty much exactly as you described. I built it on top of the Microsoft Sync Framework that DjSol mentioned.
I use a C# front end application with a SqlCe database, and a SQL 2005 Server at the other end.
The following articles were extremely useful for me:
Tutorial: Synchronizing SQL Server and SQL Server Compact
Walkthrough: Creating a Sync service
Step by step N-tier configuration of Sync services for ADO.NET 2.0
How to Sync schema changed database using sync framework?
You don't say what your back-end database is, but if it's SQL Server you can use SqlCE (SQL Server Compact Edition) as the client DB and then use RDA merge replication to update the client DB as desired. This will handle all your requirements for sure; there is no need to reinvent the wheel for such a common requirement.
Hy guys,
to monitor a specified table on db I'm polling this one every few milliseconds (10-20 ms).
Is there any chance to have a notification (avoiding SqlDependency, that in my scenario is too slow) instead polling?
Do you have any idea?
My scenario is .net + Sql Server 2008
Thanks!
It can be done, using a CLR stored procedure that will call a WCF/or a webservice. It is not something very difficult to do.
This needs practically 2 steps.
The modification of data.
After you modified the data you have to send the data to the clr stored procedure. The easiest way is to write it into one ore more temporary tables.
The clr stored procedure.
The clr store procedure will connect to the db with
"context connection=true"
so that you will have access to the stored procedures that you need. After loading the data you send it to a server (WCF/webservice). In the CLR you just need to add the service references that you need. Also on the server you will have to register some dlls for the server to use:
system.web
smdiagnostics
system.runtime.serialization
system.identitymodel
system.identitymodel.selectors
system.messagng
system.transactions.bridge
system.servicemodel
Everything else is plain .NET code to call a WCF/Web service.
This approach is very fast and very reliable.
To get onChange event notification in SQL, you can use the Query Notification feature which is built on Server Broker.
Alternatives are:
DB trigger
SignalR
Change Data Capture
Change Tracking
You could use the SqlChangeMonitor class, but that wraps SqlDependency in cached data scenarios. Your question is a little vague on why you want to do this, though.
I see you say you need notifications (and SqlDependency is available for that), but maybe you don't need instant notifications, and efficiently reading the changes periodically will do the task. If so, go google Change Data Capture, and Change Tracking.
Can the calling application be changed to say write to a queue instead?
If not I guess a trigger on the database which calls a CLR Stored Procedure? That could fire off any kind of event required.
I am working on an assignment to monitor and log various "system-level messages” within our multi-tier .NET 4 WPF, WCF, SQL 2008 application. By messages, I mean details about an event that has occurred in our application, e.g. user logs in and out or data is saved. By logging, I mean that we will be inserting messages into a light-weight SQL table from which clients can poll and display the latest messages.
The source of these messages could come from several different components in our application e.g. Windows Services, WCF Host in IIS, and even stored procedures in the Database. Ultimately these components modify SQL tables each in their own way. So rather than attacking each of these components independently, I’m thinking I might be able to simplify things by saying that the messages need to be “triggered” when certain tables in SQL are modified (updated, inserted).
The first thing that comes to mind would be a trigger on each of the tables which monitor changes and insert records into the light-weight message table. I’ve always (99% of the time) been of the mindset that database triggers are bad news (Are database triggers evil?). And I would personally much rather develop and debug C# than a SQL trigger.
So, in the pursuit of alternatives, I’ve come across Using SqlDependency to Monitor Database Changes which discusses using the SqlDependency class to detect changes. A quick proof of concept seems to work; however, after looking at several code samples, it looks like with each change that’s detected, new SqlConnection, SqlCommand, and SqlDependency objects will have to be reinitialized and I will probably need 3 or 4 queries that will need to be monitored.
Are there better options to detecting changes to SQL tables in C#?
Also, trigger bias aside, this seems like round-about way to simply monitor changes in one set of tables just to re-insert data into another table. There will be some other logic that will need to take place that I would rather implement in C#, but should I just write triggers to do this and be done with it?
Thoughts?
I think your bias against triggers in general is unfounded. People look down upon triggers for the same reason they look down on cursors: in many cases, they are misused and abused. But like a lot of things they certainly have their place and, when used correctly, are the right answer.
There are some other native technologies you should look into:
Change Tracking
http://msdn.microsoft.com/en-us/library/cc280462(SQL.100).aspx
Change Data Capture
http://msdn.microsoft.com/en-us/library/bb522489(SQL.100).aspx
SQL Server Audit
http://msdn.microsoft.com/en-us/library/cc280386(SQL.100).aspx
Though I don't know your edition (some of these are Enterprise features). Some 3rd party solutions exist as well (I haven't used any so I'll leave it up to you to search / research).
I'm wondering what the best way to implement this would be.
Basically our project has a requirement that any change made to records in the database should be logged. I already have it completed in C# using Reflection and Generics but I'm 100% sure that I used the best method.
Is there a way to do this from inside the SQL database?
The big key is that the way our project works, the ObjectContext is disconnected, so we couldn't use the built in Change Tracking and had to do our own compares against previous Log items.
If you're using SQL Server 2008 or higher, you can implement either change tracking or change data capture directly on the database. Note that the latter is only available in the Enterprise edition engine. There are pros and cons to each method. You'll have to review each solution for yourself as there isn't enough requirement information to go on in the question.
If you're using SQL Server 2005 or below, you'll have to resort to a trigger-based solution, as suggested by the other answers.
You want to look at database triggers.
depending on the complexity of your datamodel you could setup on update/insert/delete triggers on the relevant tables - these triggers could log whatever is needed (old/new values, User, timestamp etc.)... see http://msdn.microsoft.com/de-de/library/ms189799.aspx
Look at my blog to see how you can track data changes without database scheme modification:
part1,part2
For your project requirement, SQL trigger is the better solution than the current C# reflection. Becaz triggers provides a way for the database management system to actively control, monitor, and manage a group of tables whenever an insert, update, or delete operation is performed. More over, the requirement is full filled at DataBase layer itself and so hosted as the single solution for various front end applications.
I was wondering how to monitor a database for changes programmatically.
Suppose I want to have a .net application which would run after every 100th or(nth) row insertion or row deletion or updation . how can it be achieved?
I know little about triggers.they can be used to fire executable.
But I heard that it isn't a good practice.
Is there any other way?
2]Do database fire events on table updations? and can they be caught in a program?
3]Can SQL reporting services be used here?
(Also assuming that this application is independent from the actual program which does
database manipulation.)
SQL Server 2005 introduced query
notifications, new functionality that
allows an application to request a
notification from SQL Server when the
results of a query change. Query
notifications allow programmers to
design applications that query the
database only when there is a change
to information that the application
has previously retrieved.
Check out the MSDN link for more clarity
and sample immplementation
A trigger is really going to be your only way unless you aren't concerned about the accuracy of "100th" or "nth".
The answer to 2 and 3 are no.
You can write managed stored procedures (MSDN example) but that doesn't help you here really. In general triggers can be bad practice since they can block the initial caller but sometimes they are the only solution.
I think you need to question your requirement to place this low-level data monitoring in a separate application. Think about where your data changes could originate -
Do you have full understanding of every:
stored proc within your db (now and future) and which ones update this table?
application that may hit your database (now and future)
If not, then watching the changes right down at the data level (ie within the db) is probably the best option, and that probably means triggers...
Read about "Service Broker" at http://msdn.microsoft.com/en-us/library/ms166104(v=SQL.90).aspx