Using only microsoft based technologies (MS SQL Server, C#, EAB, etc) if you needed keep the track of changes done on a record in a database which strategy would you will use? Triggers, AOP on the DAL, Other? And how you will display the collected data? Is there a pattern about it? Is there a tool or a framework that help to implement this kind of solution?
The problem with Change Data capture is that it isn't flexible enough for real auditing. You can't add the columns you need. Also it dumps the records every three days by default (you can change this, but I don't think you can store forever) so you have to have a job dunping the records to a real audit table if you need to keep the data for a long time which is typical of the need to audit records (we never dump our audit records).
I prefer the trigger approach. You have to be careful when you write the triggers to ensure that they will capture the data if multiple records are changed. We have two tables for each table audited, one to store the datetime and id of the user or process that took the action and one to store the old and new data. Since we do a lot of multiple record processes this is critical for us. If someone reports one bad record, we want to be able to see if it was a process that made the change and if so, what other records might have been affected as well.
At the time you create the audit process, create the scripts to restore a set of audited data to the old values. It's a lot easier to do this when under the gun to fix things, if you already have this set up.
Sql Server 2008 R2 has this built-in - lookup Change Data Capture in books online
This is probably not a popular opinion, but I'm going to throw it out there anyhow.
I prefer stored procedures for all database writes. If auditing is required, it's right there in the stored procedure. There's no magic happening outside the code, everything that happens is documented right at the point where writes occur.
If, in the future, a table needs to change, one has to go to the stored procedure to make the change. The need to update the audit is documented right there. And because we used a stored procedure, it's simpler to "version" both the table and its audit table.
Related
My team works with a large Azure SQL database where several other teams insert and read data from our database. They sometimes need to create or alter tables but those actions should be coordinated with our team and unfortunately has not been the case. We've had a couple scenario's where one of those teams updated a stored procedure. As a result their changes are not under our source control and if we create a local database for development or do a backup/restore we get errors because of missing references.
We are looking for a way to programmatically determine if a table was altered or modified. It doesn't need to be real-time. I considered reading logs and looking for alter or create commands. I've not has much success as the logs are binary and I don't currently know how to parse them. My other thought is to keep a copy of the master database sys tables and routinely compare them to see if something changed. I'm not sure how well that would work or if I could determine who made the change. Thoughts, Ideas?
Please keep in mind that this is using an Azure SQL Database which is a bit more limited than a standard SQL database.
You can use DDL Triggers as explained here.
CREATE TRIGGER safety
ON DATABASE
FOR DROP_TABLE, ALTER_TABLE
AS
PRINT 'Save change on a log'
SELECT EVENTDATA().value('(/EVENT_INSTANCE/TSQLCommand/CommandText)[1]','nvarchar(max)');
Additionally you can use Extended Events to track schema changes. Look at samples here.
Finally you can also see how Azure SQL Auditing may fit your needs.
I'm writing a process that will run on a schedule and periodically scan records in a SQL Server database and check for changes to those records. Unfortunately the records in this database are not versioned/timestamped in any way and I can't change the schema because it's not my database. Short of comparing every single field I don't have any idea how to check to see if a change has actually occurred.
Right now I'm using entity framework for all data access, but I can switch to direct queries if needed. Is there an easy way to do this?
EDIT: I need a way to check each individual record for changes, not a database/table as a whole.
I don't think checking each record can work for big databases (it can take forever)
But you could calculate hash for each row and store it in your apps database. Then on every run recalculate hash and compare with what's in your apps db.
Also I know you can't change schema, but maybe you could add triggers to monitored dataabse that will log changes?
Another option could be setting up replication and detecting changes (with triggers as above) in database that receive's replicated data (but I have to admit I never configured replication and I may be wrong about how it works).
If you're using SQL Server 2008 or greater, you may be able to make use of either Change Data Capture or Change Tracking. Here's a link to the MSDN documentation.
Not sure if this question is suitable for StackOverflow as it's much more 'general'. Basically, I have a database driven business application made in ASP.NET and C#, which'll be used by around 20 members of a company. A crucial aspect of this is auditing - I need to log on any changes to any of the tables, and have them viewable by senior members of the staff.
My current solution uses SQL triggers, but I need to create something much more robust and user friendly. The database is gigantic, with a lot of tables with relations etc, and the audits currently are very uninformative to the users - telling the staff that x user modified an order to have a customer of ID of 837 is near enough useless - I need to be able to dictate which field is displayed in the audit log.
My idea is to create a class in my code that'll handle all these, and somehow map out what fields to display to the user, and also somehow tell the code which table was modified and which record.
Can anyone offer any general advice on how to do what I want, and whether it's actually possibile? I'm a heavy user of LINQ-to-SQL in my code, so I'm hoping that'll help...
You could also try using DoddleAudit for your needs. It provides automatic auditing of all inserts/updates/deletes for any table in your database with a single line of code, including:
What table was modified?
What fields changed?
Who made the change?
When did it occur?
You can find it here: http://doddleaudit.codeplex.com/
I've had similar audit requirements for a healthcare application, which used linq-to-sql for data access.
One way to do it centrally in Linq-to-sql is to override SubmitChanges in the data context class. Before submitting the changes, call GetChangeSet() to get data about the pending changes. Then add change tracking information as appropriate to a relevant log table before calling base.SubmitChanges(). In my application I used an xml column to be able to store change data for different tables in a structured manner, without having to create special history tables for each table in the system.
You could also try using SQL Server 2008's Change Data Capture feature. It basically captures inserts, updates and deletes on the desired tables, and stores changes made into a separate set of relational tables.
http://www.mssqltips.com/sqlservertip/1474/using-change-data-capture-cdc-in-sql-server-2008/
I'm using an sql server and i have a specific table that can contain ~1million-~10 million recrdords max.
In each record i retrieve I do some checkings (i run a few simple lines of code), and then I want to mark that the records was checked in DateTime.Now;
so what i do is retrieve a record, check some stuff, run an 'update' query to set the 'last_checked_time' field to DateTime.Now, and then move to the next record.
I can then get all the records ordered by their 'last_checked_time' field (ascending), and then i can iterate over them ordered by the their check time..
Is this a good practice ? Can it still remain speedy as long as i have no more than 10 million records on that table ?
I've read somewhere that every 'update' query is actually a deletion and a creation of a new record.
I'd also like to mention that these records will be frequently retrieved by my ASP.net website ..
I was thinking of writing down the 'last_checked_time' on a local txt file/binary file,but i'm guessing it would mean implementing something that the database can already do for you.
If you need that "last checked time" value then the best, most efficient, place to hold it is on the row in the table. It doesn't matter how many rows there are in the table, each update will affect just the row(s) you updated.
How an update is implemented is up to the DBMS, but it is not generally done by deleting and re-inserting the row.
I would recommend retrieving your data or a portion of the data, doing your checks on all of them and sending the updates back in transactions to let the database operate more effectively. This would provide for fewer round trips.
As to if this is a good practice, I would say yes especially since you are using in in your queries. Definitely, do not store the last checked time in a file and try to match up after you load your database data. The database RDBMS is designed to effeciently handle this for you. Don't reinvent the wheel using cubes.
Personally, I see no issues with it. It seems perfectly reasonable to store the last checked time in the database, especially since it might be used in queries (for example, to find records that haven't been checked in over a week).
Maybe (just maybe) you could create a new table containing two rows: the id of the row in the first table and the checked date.
That way you wouldn't alter the original table, but depending on the usage of the data and the check date you would be forced to make a joined query which is maybe something you also don't want to do.
It makes sense to store the 'checked time' as part of the row you're updating, rather than in a separate file or even a separate table in the database. This approach should provide optimal performance and help to maintain consistency. Solutions involving more than one table or external data stores may introduce a requirement for distributed or multi-table transactional updates that involve significant locking, which can negatively impact performance and make it much more difficult to guarantee consistency.
In general, solutions that minimize the scope of transactions and, by extension, locking, are worth striving for. Also, simplicity itself is a useful goal.
So I need to track changes that happen on a Mysql table. I was thinking of using triggers to log all the changes made to it and then save these changes in another table. Then I will have a cron script get all these changes and propagate the changes into the Mssql database.
I really dont expect a lot of information to be proporgated, but the data is very time sensitive. Ideally the MSSQL will see these changes within a minute, but I know that this requirement may be too high.
I was wondering if anyone had a better solution.
I have the bulk of the site written in .net but use vbulletin as the forums (sorry but there are no .net forums as powerful or feature rich like vbulletin)
The majority of the replicator tools use this technique. Fill another table on insert/update/delete triggers that containt the tablename and the PK or a unique key.
Then a reader reads this table, do the proper "select" if insert/update to get the data, then updates the other database.
HTH