We Keep Coding

C# script to parse stored procedures and extract meta data

I have around 500 stored procedures that are used for our ETL process. I have been asked to identify all the source and target tables used by each stored procedure. So, a stored procedure could have a connection to an Oracle linked server, or another SQL Server. It could also be using an OPENQUERY to extract data from our transactional systems.
Since I have some basic .NET/C# programming chops, I was hoping to leverage the .NET RegEx class to get started. However, I am looking for suggestions on how I should approach this. I really don't have to reinvent the wheel if someone already has a solution for this.
As a context, we are working on implementing PowerDesigner to store metadata repository. So, we are looking to extract metadata from our BI reports (map reports to it's source tables/views) and our Informatica and T-SQL ETL scripts.
Thanks

I'd suggest a dual-approach. Firstly, I'd avoid using regex for something as complex as SQL Query parsing, especially since there are tools in place for this kind of thing.
https://msdn.microsoft.com/en-us/library/microsoft.sqlserver.management.smo.dependencywalker.aspx
The SMO library exposes a class that will let you connect to a server and retrieve a dependency tree for a given stored procedure. How to do this exactly is left as an exercise for the reader :)
However, this class won't pick up dependencies that are introduced via dynamic SQL or through OPENQUERY. If the number of procedures that do this are small, I'd recommend doing this manually, and then merging the results. You could use the SMO scripting capabilities to pick up all instances of either OPENQUERY or exec/sp_executesql; at least then you would have an idea of 'suspect' pieces of code.
Merging the results will be tricky. Not only do you have to manually update dependencies for procedures containing dynamic dependencies, but you have to update procedures that depend on procedures containing dynamic dependencies.

You can use a dynamic management view dm_sql_referenced_entities to get some dependency information from SQL Server itself but there are some limitations. Not sure if the Dependency Walker leverages this view, but the pros and cons are very similar.
The same main limitation that I know of and have experienced is that you won't get any dependency information for an object that is leveraged through dynamic sql. We have very contained usages of dynamic sql so I can feel pretty confident leveraging this DMV and manually accounting for the objects hit by those specific procs.
We don't do linked servers, but in my understanding is that those would show in this DMV. I don't know about the OPENQUERY ... I did a little bit of research but I did not test it out but I am guessing those would not be surfaced by the view. Like the previous poster said, you may need a two-pronged approach to get everything you're looking for.
And just for reference, a simple example of using that DMV:
SELECT DISTINCT
[database] = COALESCE(r.referenced_database_name, DB_NAME())
, [schema] = r.referenced_schema_name
, name = r.referenced_entity_name
, r.referenced_id
FROM sys.dm_sql_referenced_entities('dbo.procName_sp', 'OBJECT') AS r
WHERE r.referenced_id IS NOT NULL;

I wouldn't use C# for this. However, maybe something like this will do the job.
select *
from DatabaseName.information_schema.routines
where routine_type = 'PROCEDURE'
SELECT name, type
FROM dbo.sysobjects
WHERE type IN (
'P', -- stored procedures
'FN', -- scalar functions
'IF', -- inline table-valued functions
'TF' -- table-valued functions
)
ORDER BY type, name
Or, if you want SProcs and parameters:
select * from information_schema.parameters
Finally, this link looks pretty helpful for your situation.
http://blog.sqlauthority.com/2010/02/04/sql-server-get-the-list-of-object-dependencies-sp_depends-and-information_schema-routines-and-sys-dm_sql_referencing_entities/

Accessing random SQL server

The application I'm designing will have to be able to access any MS or Oracle database through the network, will have to read all the databases inside the database, list all the tables.
In the end, the user must be able to download a whole table, and load it into the application.
The user will have to insert some parameters :
Ip address and port
Login and password.
Then he will have to choose the database, and then he will have to choose the table.
I've looked up through the internet to find something almost out of the box, and the best result I've found is actually Dapper (http://code.google.com/p/dapper-dot-net/)
Ain't there any better library that could meet my needs?

There is no library that will meet your needs. You're building a database browsing tool, like SSMS, so Dapper is only a means to an end to get data in a dynamic fashion. However, you could just as easily do it with basic ADO.NET. The DataTable is a very dynamic structure and quite capable of what you need.
In fact, though I love Dapper, I would argue that if I were in your shoes I'd just use ADO.NET because you could abstract your interaction to using just the interfaces: IDbConnection and IDbCommand for example.

Before you go rolling your own, I would suggest you take a look at the Microsoft Enterprise Library Data Access Application Block; it builds on ADO.Net but provides a database agnostic API for applications to code against. I've used it against both Sql Server and Oracle.
http://msdn.microsoft.com/en-us/library/dn169621.aspx

A few notes:
I agree with the idea of using ADO.NET. And getting back to the interfaces that SqlConnection and the ODP.NET share.
A few ideas:
IoC Link
and
INFORMATION_SCHEMA
Google that. It is what the standard is for different RDBMS.
My guess is that you could write 2 "Concretes" that implement the same abstract class (see IoC link above)... where the abstract class has "common logic"...but then you have a "safety value" if something does work exactly the same between the 2 RDBMS.
But since your results will be dynamic, in the sense you won't know ahead of time what the tables/columns, a DataTable isn't a bad idea.
A different idea is to look at this, the "Dynamic Data" project.
http://msdn.microsoft.com/en-us/library/cc488545%28v=vs.100%29.aspx
But I don't know (maybe not) if that ever got an implementation for Oracle, probably just Sql Server.

I finally found happyness using DbProviderFactory.
I've created a class that creates connexion strings, and that has a property that return the invariant name of the connexion (eg : "System.Data.SqlClient").
I use this invariant name and this connexion string that way :
this.Factory = DbProviderFactories.GetFactory(connexionStringBuilder.InvariantName);
this.Connection = Factory.CreateConnection();
Connection.ConnectionString = connexionStringBuilder.ConnexionString;
And then, I have all the ADO.Net queries that I had in the past.

What is best way access different database on different engine but with similar structure

I usually work with MySql, but also with SQL Server, Oracle and Access, the database structure is almost the same. My database stores configuration and recorded data of a SCADA application ("Supervisory Control And Data Acquisition").
Most of the tables are usually the same but sometime my teammates adds fields, tables or changes some fields type.
I'm writing an application that need to load some config parameters from db, then load data, process it and store the new values on db. It also need to add new records.
I have a class that, independently from db type, given the correct connection params, gets a IDbConnection object. With some methods I can specified a SQL query and it give me and IDataReader or a also Dataset.
Now, how should i query data from the db, analyze, recalculate, and finally store them again?
I'm a bit scared of building a detailed object mapping because of the possibility of changed fields. A simple dataset/datatable/datarow should be ok but i'd like to use linq to query in a simpler way the extracted data from the database.
Finally, my db has about 60 tables but in this application I work only with a dozen of them. I have only a few time to build that application, so I need a fast way, also if it's not "very beautiful".
Thanks.

you should try an ORM that configures itself automatically according to schema
i have found this one. I didn't use similar things in c# but it works nicely in other (dynamic) languages.
http://www.codeproject.com/Articles/117666/Kerosene-ORM

Using an ORM would most probably be the fastest. You could use NHibernate which has multiple DB support. NHibernate does have a learning curve, so something like a micro ORM could be easier to use perhaps. Petapoco is a great micro ORM and supports SQL Server, SQL Server CE, MySQL, PostgreSQL and Oracle.
These ORMs would create a mapping file for each DB you use which needs to be updated or recreated when changes are made in the DB.

How to make data access layer for SQL Server and MySql

I'm building a data access layer and need to be able switch between two providers in different environments.
How do I structure this? I'm using a repository pattern and have e.g. a CarRepository class and a Car class. The CarRepository class is responsible for saving, deleting and loading from the database.
I have a Database class, responsible for connecting to the database, and executing the query (sending a SqlCommand for SQL Server). The SQL syntax for the underlying databases is different and the parameter syntax is also different (SQL Server uses # and MySql uses ?).
I would like an approach where I can make the least effort in making my application run on both platforms.
The obvious method is making a MySqlCarRepository and a SqlServerCarRepository, but that introduces a heavy amount of maintenance. Are there any good approaches to this scenario? Maybe keeping a static class with static strings containing the SQL statements for the different SQL flavours? (how about parameter syntax then?)
Any advice is welcome
(Please note that ORM (Nhibernate, Linq2Sql etc) is not an option)

The approach I follow is to first-of-all use the ADO Provider Factories to abstract the data access implementation. So I will use IDbConnection and so forth in the code.
Then I have an abstraction for a query. I can then use Query objects that contain the actual sql statements. These Query objects are created from RawQuery or various query builders (insert/update/delete/etc.) that have implementations for each provider type. The specific raw queries will need to be coded and obtained specific to the DB you need since there is no gettin passed that.
There is quite a bit of leg work involved in coding this 'plumbing' and I have not had a situation where I actually require different platforms so I have not bothered coding some small bits that I know need some ironing out but you are welcome to contact if you are interested in seeing some code.

Can you use any code generation tools?
I used to use Code Smith in another life and had templates that would generate POCO objects from DB tables, repository classes for each object and stored procedures. Worked alright after fine tuning the templates, and there were plenty examples on the net.
But this was way before I saw the light with NHibernate!

A pattern for accessing multiple database types is the DAO (Data Access Object) pattern. This could suit your particular need if you can't/don't use an ORM. The following article explains the pattern for Java but it is still very relevant for C#:
http://java.sun.com/blueprints/corej2eepatterns/Patterns/DataAccessObject.html

Where do you put SQL Statements in your c# projects?

I will likely be responsible for porting a vb6 application to c#. This application is a windows app that interacts with an access db. The data access is encapsulated in basic business objects. One class for one table basically. The existing vb6 business objects read and write to the DB via DAO. I have written DALs and ORMs a few times before but they all targeted SQL Server only. This one will need to target access and sql server. In previous projects, I would place the SQL strings in the private parts of the business object and maybe move the redundant sql code like connecting, creating command, in into a common base class to reduce the code.
This time, i'm thinking about writing the SQL strings into a .settings file or some other key/value type text file. I would then write a sql utility to edit this file and allow me to run and test the parameterized queries. These queries would be referenced by name in the business object instead of embedding the sql into code.
I know a standard approach is to create a DAL for each targeted database and have the configuration state which DAL to use. I really don't want to create the two DAL classes for each database. It seems like it would be less code if I just referenced the correct query by keyname and have the proper type of connection.
So, are you guys doing things like this? How would or have you approached this problem?
What works best for you?
Thanks!

Well, there's a lot of options - so it really depends on what your most pressing needs are :-)
One approach might be to create SQL statements as text files inside your VS solution, and mark them as "embedded resource" in the "build action". That way, the SQL is included in your resulting assembly, and can be retrieved from it at runtime using the ResourceManifestStream of the .NET framework:
private string LoadSQLStatement(string statementName)
{
string sqlStatement = string.Empty;
string namespacePart = "ConsoleApplication1";
string resourceName = namespacePart + "." + statementName;
using(Stream stm = Assembly.GetExecutingAssembly().GetManifestResourceStream(resourceName))
{
if (stm != null)
{
sqlStatement = new StreamReader(stm).ReadToEnd();
}
}
return sqlStatement;
}
You need to replace "ConsoleApplication1" with your actual namespace, in which the sql statement files reside. You need to reference them by means of the fully qualified name. Then you can load your SQL statement with this line:
string mySQLStatement = LoadSQLStatement("MySQLStatement.sql");
This however makes the queries rather "static", e.g. you cannot configure and change them at runtime - they're baked right into the compiled binary bits. But on the other hand, in VS, you have a nice clean separation between your C# program code, and the SQL statements.
If you need to be able to possibly tweak and change them at runtime, I'd put them into a single SQL table which contains e.g. a keyword and the actual SQL query as fields. You can then retrieve them as needed, and execute them. Since they're in the database table, you can also change, fix, amend them at will - even at runtime - without having to re-deploy your whole app.
Marc

When I really need it, I put the queries into individual *.sql files, then include them into Resources.resx. There is a 'Files' section in it, which allows you to include Embedded Resource files.
After that, I can use generated Resources.MyQuery property which both guarantees that resource exists and saves me from writing a custom resource load method.

LINQ to DataSet sounds like the way to go for you.
If you havent used the .NET 3.5 before / LINQ then you're in for a treat. LINQ will save you writing your raw sql in string literals and provide you with a more logical way to creating querys.
Anyway, check this link out for using LINQ on Access databases - http://msdn.microsoft.com/en-us/library/bb386977.aspx

If i'd had to create application for both SQL and Access, I'd use some IDAL interface, DALCommon with common functionality implementation and separate DALSql and DALAccess, inherited from DALCommon, with some specific stuff, like exceptions, transactions handling, security etc.
I used to keep stored procedure names or queries in resource files.

I'll tell where I won't put it ever, something I saw done in some code I inherited. It was in Java, but applies to any language
A base class that declared protected static member variables for for SQL statements, inited to null, with a get method that returns individual SQL statements
A sub class for each supported database server, with an init method that assigns to the base class member variables
Several DA classes that use the base class method to retrieve SQL statements
The application start-up class with the responsibility to create the correct sub-class object and call its init method
I will also not go into explaining why I will not do this ever :-)

One method we used is to have a class that would connect to the DB and methods to call procedures and in the method parameter you would provide the procedure name. so all the SQL code is in the procedure. we would use overloads for the different return types
class ConnectToSQL()
{
//connectSql code (read from setting file i assume)
XMLDataDocument runProcedure(string procedureName);
int runProcedure(string procedureName);
//etc....
}

Sometimes, like with custom reporting apps, you really need to embrace the impedance mismatch, and give special importance to the SQL. In these cases I recommend the following: For each module that contains SQL strings, create a single static "SQL" class to hold them all. Some of the SQL strings will likely require parameters, so be consistent and put each string behind it's own static method.
I only do this for the occasional custom reporting app, but it always works out great and feels refreshing and liberating. And it's quite nice to come back months later to make an enhancement, and find all of the SQL waiting for you in a single SQL.cs file. Just by reading that one file, it all comes back, and often this is the only file that needs to be changed.
I don't see a need in these cases for hiding the SQL in resources or elsewhere. When SQL is important, then it's important. Interestingly, more and more developers are now freely mixing SQL with C#, including I believe this site, because essentially, that's what LINQ is.
Finally, as always, make sure you are not susceptible to SQL injection attacks. Especially if user input is involved, make sure you are using some kind of parameterization and that you are not using string concatenation.

Embedding solutions shown above may not work if SQL Query has a "where" cause like , but for the same Query the next run needs PropertyID='113' as the PropertyID is read-in.

Glad you asked! Put your sql in a QueryFirst .sql template.
It's automatically compiled into your app as an embedded resource, but you don't care. You just write it, in a real sql window, connected to your DB, with syntax validation and intellisense for tables and columns, then use it, via the generated Execute() methods, with intellisense for your inputs and results.
disclaimer : I wrote QueryFirst.