We Keep Coding

C# and SQLServer normalizing large sets of Urls

I have many tables in the database that have at least one column that contains a Url. And these are repeated a lot through-out the database. So I normalize them to a dedicated table and I just use numeric IDs everywhere I need them. I often need to join them so numeric ids are much better than full strings.
In MySql + C++, to insert a lot of Urls in one strike, I used to use multi-row INSERT IGNOREs or mysql_set_local_infile_handler(). Then batch SELECT with IN () to pull the IDs back from the database.
In C# + SQLServer I noticed there's a SqlBulkCopy class that's very useful and fast in mass-insertion. But I also need mass-selection to resolve the Url IDs after I insert them. Is there any such helper class that would work the same as SELECT WHERE IN (many, urls, here)?
Or do you have a better idea for turning Urls into numbers in a consistent manner in C#? I thought about crc32'ing the urls or crc64'ing them but I worry about collisions. I wouldn't care if collisions are few, but if not... it would be an issue.
PS: We're talking about tens of millions of Urls to get an idea of scale.
PS: For basic large insert, SQLBulkCopy is faster than SqlDbType.Structured. Plus it has the SqlRowsCopied event for a status tracking callback.

There is even a better way than SQLBulkCopy.
It's called Structured Parameters and it allows you to pass a table-valued parameter to stored procedure or query through ADO.NET.
There are code examples in the article, so I will only highlight what you need to do to get it up and working:
Create a user defined table type in the database. You can call it UrlTable
Setup a SP or query which does the SELECT by joining with a table variable or type UrlTable
In your backing code (C#), create a DataTable with the same structure as UrlTable, populate it with URLs and pass it to an SqlCommand through as a structured parameter. Note that column order correspondence is critical between the data table and the table type.
What ADO.NET does behind the scenes (if you profile the query you can see this) is that before the query it declares a variable of type UrlTable and populates it (INSERT statements) with what you pass in the structured parameter.
Other than that, query-wise, you can do pretty much everything with table-valued parameters in SQL (join, select, etc).

I think you could use the IGNORE_DUP_KEY option on your index. If you set IGNORE_DUP_KEY = ON on the index of the URL column, the duplicate values are simply ignored and the rest are inserted appropriately.

Is it possible to insert data in to table through coding with out using table name

My question is generally we write the following through code while we are inserting data to a table
insert into tblname values('"+txt.text+"','"+txt1.text+"');
As we pass the data form the text boxes like that is it possible to insert in to table with out using table name directlty

Well you obviously need to know what table to insert into, so there has to be a table name identified to the INSERT statement. The options include:
an INSERT statement with actual table name as per your existing example
an INSERT statement with a synonym as the target (alias for an actual table - see: http://blog.sqlauthority.com/2008/01/07/sql-server-2005-introduction-and-explanation-to-synonym-helpful-t-sql-feature-for-developer/)
an INSERT statement with an updateable view as the target
a sproc call whereby the sproc knows the table to INSERT into (but the calling code does not need to know)
You should also be aware of SQL injection risks with your example - avoid concatenating values directly into a SQL string to execute. Instead, parameterise the SQL.
If you need to dynamically specify the table to insert into at run time, you have to concatenate the table name into the SQL statement you then execute. However, be very wary of SQL injection - make sure you fully validate the tablename to make sure there are no nasties in it. You could even check it is a real table by checking for it in sys.tables.

Not possible without name of table.
But you can make use of Linq To SQL (i.e any ORM) or DataAdapter.Update if you have filled it with the proper table....

You cannot do that without the table name, no. However, the bigger problem is that your code is horribly dangerous and at rick from SQL injection. You should fix this right now, today, immediately. Injection, even for internal apps, is the single biggest risk. Better code would be:
insert into tblname (Foo, Bar) values(#foo, #bar)
adding the parameters #foo and #bar to your command (obviously, replace with sensible names).
Before you ask: no, the table name cannot be parameterised; you cannot use
insert into #tblname -- blah
The table name(s) is(/are) fundamental in any query or operation.

I suppose that if it's possible you have to use parameters.
Here you have a little example.

Is there any way to get the table hierarchy from a connection string in c#?

I have a current requirement to determine the table hierarchy from a sql statement within c#. For example, consider the following sql statement:
Select Table1.*, Table2.* from Table1
left join table2 on Table1.parentCol = Table2.childCol
That might return 7 columns, 3 for Table1 and 4 for table2. I need to know the column names, and ideally (though not mandatory) their types.
I have no control over what SQL Statement will be used, as this is a user entered field. In C# it's a very basic task to open a connection and create an SqlCommand using that statement. I have freedom to run the SQL into a SqlDataReader, or any other System.Data.SqlClient class if necessary, however I cannot find any combination that will return the columns, rather than the actual column values.
Is anyone able to help?
Many thanks and best regards

You cannot do what you are asking (easily).
More to the point, do not let users enter arbitrary TSQL (You will regret it at some point...).
Instead, create a 'Search' form that allows entering various params and use a parameterised query onto a view that joins all the tables/columns required.

There's no direct way. You'll need to parse names of all the tables from the sql query.
Once you have done that you'll need to write few queries on Information_Schema to get raw data for what you are looking for.
If you are on SQL Server, you may want to use Catalog View
ex-
Select * from sys.tables where [Name] = 'MyTable'

How do I securely create queries in ADO.NET where the tables being selected from change?

In ADO.NET you can add parameters to a command object to securely add user input to a SQL query. What is the equivalent for the other predicates common to a SQL query?
I am writing a program that is essentially a very limited O-R mapper and SQL generator (it's focused heavily around a database with meta-information and other databases that conform to that meta-data). As a result I need to be able to call stuff like:
string sql = "select " + USER_SELECTED_COLUMNS +
" from " + USER_SELECTED_TABLE +
" where " + USER_CRITERIA;
Some of it (like the criteria) is literally entered into my program by trusted users (other developers in my company), while other data is entered into my program by untrusted users (clients) through their searches, etc.
I'd like to make this program secure, and I'm aware that the above is not. Currently I have the USER_SELECTED_COLUMNS replaced with command parameters, but I've not been able to find the equivalent for the CRITERIA and TABLEs. (Or the order-by columns). Are there any ADO.NET features similar to SqlParameter that I can use for non-selection predicates?

I dont think I could tell you how to avoid SQL Injection in 1 response, however, the main pointer I can give you is this:
USE WHITELISTS, NOT BLACKLISTS.
That is to say, when sanitizing the users input for USER_SELECTED_TABLE, the possible input should only be the possible tables. Equaly, the input for USER_SELECTED_COLUMNS should be limited to the possible columns for USER_SELECTED_TABLE.

when you build the screens that allow the user to select the table and columns don't use the actual names. Kind of how you would have a UserID but show the UserName. Use the object_id and column_id (like form sys.tables.object_id, and sys.columns.object_id+column_id). Pas those into your procedure and build your SQL using only the numeric IDs that join to the system views:
sys.tables (Transact-SQL)
sys.columns (Transact-SQL)
you can concatenate the string table and column names, but they will come from the system views and not from the user input.

Run all of the variables that you indicated above through the Microsoft Web Protection Library. It will provide safety from both SQL Injection and XSS attacks. There are examples in the download to show you how to use it in codebehind.

How do I protect this function from SQL injection?

public static bool TruncateTable(string dbAlias, string tableName)
{
string sqlStatement = string.Format("TRUNCATE TABLE {0}", tableName);
return ExecuteNonQuery(dbAlias, sqlStatement) > 0;
}

The most common recommendation to fight SQL injection is to use an SQL query parameter (several people on this thread have suggested it).
This is the wrong answer in this case. You can't use an SQL query parameter for a table name in a DDL statement.
SQL query parameters can be used only in place of a literal value in an SQL expression. This is standard in every implementation of SQL.
My recommendation for protecting against SQL injection when you have a table name is to validate the input string against a list of known table names.
You can get a list of valid table names from the INFORMATION_SCHEMA:
SELECT table_name
FROM INFORMATION_SCHEMA.Tables
WHERE table_type = 'BASE TABLE'
AND table_name = #tableName
Now you can pass your input variable to this query as an SQL parameter. If the query returns no rows, you know that the input is not valid to use as a table. If the query returns a row, it matched, so you have more assurance you can use it safely.
You could also validate the table name against a list of specific tables you define as okay for your app to truncate, as #John Buchanan suggests.
Even after validating that tableName exists as a table name in your RDBMS, I would also suggest delimiting the table name, just in case you use table names with spaces or special characters. In Microsoft SQL Server, the default identifier delimiters are square brackets:
string sqlStatement = string.Format("TRUNCATE TABLE [{0}]", tableName);
Now you're only at risk for SQL injection if tableName matches a real table, and you actually use square brackets in the names of your tables!

As far as I know, you can't use parameterized queries to perform DDL statements/ specify table names, at least not in Oracle or Sql Server. What I would do, if I had to have a crazy TruncateTable function, that had to be safe from sql injection would be to make a stored procedure that checks that the input is a table that is safe to truncate.
-- Sql Server specific!
CREATE TABLE TruncableTables (TableName varchar(50))
Insert into TruncableTables values ('MyTable')
go
CREATE PROCEDURE MyTrunc #tableName varchar(50)
AS
BEGIN
declare #IsValidTable int
declare #SqlString nvarchar(50)
select #IsValidTable = Count(*) from TruncableTables where TableName = #tableName
if #IsValidTable > 0
begin
select #SqlString = 'truncate table ' + #tableName
EXECUTE sp_executesql #SqlString
end
END

If you're allowing user-defined input to creep into this function via the tablename variable, I don't think SQL Injection is your only problem.
A better option would be to run this command via its own secure connection and give it no SELECT rights at all. All TRUNCATE needs to run is the ALTER TABLE permission. If you're on SQL 2005 upwards, you could also try using a stored procedure with EXECUTE AS inside.

CREATE OR REPLACE PROCEDURE truncate(ptbl_name IN VARCHAR2) IS
stmt VARCHAR2(100);
BEGIN
stmt := 'TRUNCATE TABLE '||DBMS_ASSERT.SIMPLE_SQL_NAME(ptbl_name);
dbms_output.put_line('<'||stmt||'>');
EXECUTE IMMEDIATE stmt;
END;

Use a stored procedure. Any decent db library (MS Enterprise Library is what I use) will handle escaping string parameters correctly.
Also, re:parameterized queries: I prefer to NOT have to redeploy my app to fix a db issue. Storing queries as literal strings in your source increases maintenance complexity.

Have a look at this link
Does this code prevent SQL injection?
Remove the unwanted from the tableName string.
I do not think you can use param query for a table name.

There are some other posts which will help with the SQL injection, so I'll upvote those, but another thing to consider is how you will be handling permissions for this. If you're granting users db+owner or db_ddladmin roles so that they can truncate tables then simply avoiding standard SQL injection attacks isn't sufficient. A hacker can send in other table names which might be valid, but which you wouldn't want truncated.
If you're giving ALTER TABLE permissions to the users on the specific tables that you will allow to be truncated then you're in a bit better shape, but it's still more than I like to allow in a normal environment.
Usually TRUNCATE TABLE isn't used in normal day-to-day application use. It's used for ETL scenarios or during database maintenance. The only situation where I might imagine it would be used in a front-facing application would be if you allowed users to load a table which is specific for that user for loading purposes, but even then I would probably use a different solution.
Of course, without knowing the specifics around why you're using it, I can't categorically say that you should redesign, but if I got a request for this as a DBA I'd be asking the developer a lot of questions.

Use parameterized queries.

In this concrete example you need protection from SQL injection only if table name comes from external source.
Why would you ever allow this to happen?
If you are allowing some external entity (end user, other system, what?)
to name a table to be dropped, why won't you just give them admin rights.
If you are creating and removing tables to provide some functionality for end user,
don't let them provide names for database objects directly.
Apart from SQL injection, you'll have problems with name clashes etc.
Instead generate real table names yourself (e.g DYNTABLE_00001, DYNTABLE_00002, ...) and keep a table that connects them to the names provided by user.
Some notes on generating dynamic SQL for DDL operations:
In most RDBMS-s you'll have to use dynamic SQL and insert table names as text.
Be extra careful.
Use quoted identifiers ([] in MS SQL Server, "" in all ANSI compliant RDBMS).
This will make avoiding errors caused by invalid names easier.
Do it in stored procedures and check if all referenced objects are valid.
Do not do anything irreversible. E.g. don't drop tables automatically.
You can flag them to be dropped and e-mail your DBA.
She'll drop them after the backup.
Avoid it if you can. If you can't, do what you can to minimize rights to other
(non-dynamic) tables that normal users will have.

You could use SQLParameter to pass in tableName value. As far as I know and tested, SQLParameter takes care of all parameter checking and thus disables possibility of injection.

If you can't use parameterized queries (and you should) ... a simple replace of all instances of ' with '' should work.
string sqlStatement = string.Format("TRUNCATE TABLE {0}", tableName.Replace("'", "''"));