Related
I am really clueless here.
I import ~2 million rows into my Azure SQL database. I create a temp table where I put my values, when I use merge technique to insert only rows that do not have duplicates. My code and scripts are shown below.
public async Task BulkImportWithoutDuplicates(DataTable reader)
{
var tableName = "##tempImport";
using (var connection = new SqlConnection(sqlCOnn.ConnectionString))
{
using (SqlCommand command = new SqlCommand("", sqlCOnn))
{
try
{
sqlCOnn.Open();
// Creating temp table on database
command.CommandText = Scripts.GetTempTableScript();
command.ExecuteNonQuery();
// Bulk insert into temp table
using (SqlBulkCopy b = new SqlBulkCopy(conString, SqlBulkCopyOptions.TableLock))
{
b.BulkCopyTimeout = 0;
b.BatchSize = reader.Rows.Count;
b.DestinationTableName = tableName;
//dataTable
await b.WriteToServerAsync(reader);
b.Close();
}
// Updating destination table, and dropping temp table
command.CommandText = Scripts.GetMergeScript();
var rows = command.ExecuteNonQuery();
}
catch (Exception ex)
{
// Handle exception properly
}
finally
{
connection.Close();
}
}
}
}
public static string GetTempTableScript()
{
return $#"
IF OBJECT_ID('tempdb.dbo.##tempImport', 'U') IS NOT NULL
BEGIN
DROP TABLE ##tempImport;
END
CREATE TABLE ##tempImport ( ... all the columns);";
}
public static string GetMergeScript()
{
return $#"MERGE INTO dbo.Data AS target
USING ##tempImport AS source
ON (source.TransactionId = target.TransactionId AND source.UserId = target.UserId)
WHEN NOT MATCHED THEN
INSERT (Start, Spend, UserId, Product, Shop, ClientId, UploadDataId, UniqueId, TransactionId, q, cq, c2)
VALUES (source.Start, source.Spend, source.UserId, source.Product, source.Shop,
source.ClientId, source.UploadDataId, source.UniqueId, source.TransactionId, source.q, source.c1, source.c2);
";
}
I really do not get why it takes ages until it finishes. I waited for 24 minutes until it was added to temporary table alone.
I was reading this article and it seems that it shouldn't take long. https://www.adathedev.co.uk/2011/01/sqlbulkcopy-to-sql-server-in-parallel.html?m=1
What I am doing wrong here? How can I improve the speed?
I tried using both IDataReader and DataTable but both of them do not work well for me...
We have a big list around 100000 records and want to insert it into a sql table.
What are we doing is; converting that list into data table and passing datatable to SqlBulkcopy method.
This conversion from list to Datatable taking more time. Tried using Parallel but as Datatable is not thread safe so avoided that.
Adding sample poc code which generates integer list and insert it into temp table
static void Main(string[] args)
{
List<int> valueList = GenerateList(100000);
Console.WriteLine("Starting with Bulk Insert ");
DateTime startTime = DateTime.Now;
int recordCount = BulkInsert(valueList);
TimeSpan ts = DateTime.Now.Subtract(startTime);
Console.WriteLine("Bulk insert for {0} records in {1} miliseconds.-> ", recordCount, ts.Milliseconds);
Console.WriteLine("Done.");
Console.ReadLine();
}
private static int BulkInsert(List<int> valueList)
{
SqlBulkHelper sqlBulkHelper = new SqlBulkHelper();
var eventIdDataTable = CreateIdentityDataTable(valueList, "SqlTable", "Id");
return FillBulkPoundTable(eventIdDataTable, "#SqlTable");
}
private static List<int> GenerateList(int size)
{
return Enumerable.Range(0, size).ToList();
}
private static DataTable CreateIdentityDataTable(List<int> ids, string dataTableName, string propertyName)
{
if (ids == null) return null;
using (var dataTable = new DataTable(dataTableName))
{
dataTable.Locale = CultureInfo.CurrentCulture;
var dtColumn = new DataColumn(propertyName, Type.GetType("System.Int32"));
dataTable.Columns.Add(dtColumn);
foreach (int id in ids)
{
DataRow row = dataTable.NewRow();
row[propertyName] = id;
dataTable.Rows.Add(row);
}
return dataTable;
}
}
private static int FillBulkPoundTable(DataTable dataTable, string destinationTableName)
{
int totalInsertedRecordCount = 0;
using (SqlConnection _connection = new SqlConnection(CongifUtil.sqlConnString))
{
string sql =
#"If object_Id('tempdb..#EventIds') is not null drop table #EventIds
CREATE TABLE #EventIds(EvId int) ";
_connection.Open();
using (var command = new SqlCommand(sql, _connection))
{
command.ExecuteNonQuery();
}
using (var sqlBulkCopy = new SqlBulkCopy(_connection))
{
sqlBulkCopy.BulkCopyTimeout = 0;
sqlBulkCopy.DestinationTableName = destinationTableName;
sqlBulkCopy.WriteToServer(dataTable);
}
using (var command = new SqlCommand(sql, _connection))
{
command.CommandText = "Select Count(1) as RecordCount from #EventIds";
SqlDataReader reader = command.ExecuteReader();
if (reader.HasRows)
{
while (reader.Read())
{
totalInsertedRecordCount = Convert.ToInt32(reader["RecordCount"]);
}
}
}
}
return totalInsertedRecordCount;
}
Currently it is taking around 8 seconds but we need to make it more faster. Reason is our target is to insert 900,000 records which will be devided into 100,000 batch each.
Can you give us any hint how can we make it perfect and faster?
PS. Tried with Dapper insert too but it is not faster than BulkCopy.
First Covert your list into XML something like
List<int> Branches = new List<int>();
Branches.Add(1);
Branches.Add(2);
Branches.Add(3);
XElement xmlElements = new XElement("Branches", Branches.Select(i => new
XElement("branch", i)));
Then pass the xml to a SP as parameter and insert it directly to your table, Example :
DECLARE #XML XML
SET #XML = '<Branches>
<branch>1</branch>
<branch>2</branch>
<branch>3</branch>
</Branches>'
DECLARE #handle INT
DECLARE #PrepareXmlStatus INT
EXEC #PrepareXmlStatus= sp_xml_preparedocument #handle OUTPUT, #XML
SELECT * FROM OPENXML(#handle, '/Branches/branch', 2)
WITH (
branch varchar
)
EXEC sp_xml_removedocument #handle
Bach Size
From what I understand, you try to insert with a BatchSize of 100000. Higher is not always better.
Try to lower this amount to 5,000 instead and check for the performance difference.
You increase the amount of database round-trip but it may also go faster (Too much factor such as the row size are involved here)
TableLock
Using the SqlBulkCopyOptions.TableLock will improve your insert performance.
using (var sqlBulkCopy = new SqlBulkCopy(_connection, SqlBulkCopyOptions.KeepIdentity))
For inserting a huge amount of data in a database, I used to collect all the inserting information into a list and convert this list into a DataTable. I then insert that list to a database via SqlBulkCopy.
Where I send my generated list LiMyList which contain information of all bulk data which I want to insert to database and pass it to my bulk insertion operation
InsertData(LiMyList, "MyTable");
Where InsertData is
public static void InsertData<T>(List<T> list,string TableName)
{
DataTable dt = new DataTable("MyTable");
clsBulkOperation blk = new clsBulkOperation();
dt = ConvertToDataTable(list);
ConfigurationManager.OpenExeConfiguration(ConfigurationUserLevel.PerUserRoamingAndLocal);
using (SqlBulkCopy bulkcopy = new SqlBulkCopy(ConfigurationManager.ConnectionStrings["SchoolSoulDataEntitiesForReport"].ConnectionString))
{
bulkcopy.BulkCopyTimeout = 660;
bulkcopy.DestinationTableName = TableName;
bulkcopy.WriteToServer(dt);
}
}
public static DataTable ConvertToDataTable<T>(IList<T> data)
{
PropertyDescriptorCollection properties = TypeDescriptor.GetProperties(typeof(T));
DataTable table = new DataTable();
foreach (PropertyDescriptor prop in properties)
table.Columns.Add(prop.Name, Nullable.GetUnderlyingType(prop.PropertyType) ?? prop.PropertyType);
foreach (T item in data)
{
DataRow row = table.NewRow();
foreach (PropertyDescriptor prop in properties)
row[prop.Name] = prop.GetValue(item) ?? DBNull.Value;
table.Rows.Add(row);
}
return table;
}
Now I want to do an update operation, is there any way as for inserting data is done by SqlBulkCopy for Updating data to DataBase From C#.Net
What I've done before is perform a bulk insert from the data into a temp table, and then use a command or stored procedure to update the data relating the temp table with the destination table. The temp table is an extra step, but you can have a performance gain with the bulk insert and massive update if the amount of rows is big, compared to updating the data row by row.
Example:
public static void UpdateData<T>(List<T> list,string TableName)
{
DataTable dt = new DataTable("MyTable");
dt = ConvertToDataTable(list);
using (SqlConnection conn = new SqlConnection(ConfigurationManager.ConnectionStrings["SchoolSoulDataEntitiesForReport"].ConnectionString))
{
using (SqlCommand command = new SqlCommand("", conn))
{
try
{
conn.Open();
//Creating temp table on database
command.CommandText = "CREATE TABLE #TmpTable(...)";
command.ExecuteNonQuery();
//Bulk insert into temp table
using (SqlBulkCopy bulkcopy = new SqlBulkCopy(conn))
{
bulkcopy.BulkCopyTimeout = 660;
bulkcopy.DestinationTableName = "#TmpTable";
bulkcopy.WriteToServer(dt);
bulkcopy.Close();
}
// Updating destination table, and dropping temp table
command.CommandTimeout = 300;
command.CommandText = "UPDATE T SET ... FROM " + TableName + " T INNER JOIN #TmpTable Temp ON ...; DROP TABLE #TmpTable;";
command.ExecuteNonQuery();
}
catch (Exception ex)
{
// Handle exception properly
}
finally
{
conn.Close();
}
}
}
}
Notice that a single connection is used to perform the whole operation, in order to be able to use the temp table in each step, because the scope of the temp table is per connection.
In my personal experience, the best way to handled this situation is utilizing a Stored Procedure with a Table-Valued Parameter and a User-Defined Table Type. Just set up the type with the columns of the data table, and pass in said-data table as a parameter in the SQL command.
Within the Stored Procedure, you can either join directly on some unique key (if all rows you are updating exist), or - if you might run into a situation where you are having to do both updates and inserts - use the SQL Merge command within the stored procedure to handle both the updates and inserts as applicable.
Microsoft has both syntax reference and an article with examples for the Merge.
For the .NET piece, it's a simple matter of setting the parameter type as SqlDbType.Structured and setting the value of said-parameter to the Data Table that contains the records you want to update.
This method provides the benefit of both clarity and ease of maintenance. While there may be ways that offer performance improvements (such as dropping it into a temporary table then iterating over that table), I think they're outweighed by the simplicity of letting .NET and SQL handle transferring the table and updating the records itself. K.I.S.S.
Bulk Update:
Step 1: put the data which you want to update and primary key in a list.
Step 2: pass this list and ConnectionString to BulkUpdate Method As shown below
Example:
//Method for Bulk Update the Data
public static void BulkUpdateData<T>(List<T> list, string connetionString)
{
DataTable dt = new DataTable("MyTable");
dt = ConvertToDataTable(list);
using (SqlConnection conn = new SqlConnection(connetionString))
{
using (SqlCommand command = new SqlCommand("CREATE TABLE
#TmpTable([PrimaryKey],[ColumnToUpdate])", conn))
{
try
{
conn.Open();
command.ExecuteNonQuery();
using (SqlBulkCopy bulkcopy = new SqlBulkCopy(conn))
{
bulkcopy.BulkCopyTimeout = 6600;
bulkcopy.DestinationTableName = "#TmpTable";
bulkcopy.WriteToServer(dt);
bulkcopy.Close();
}
command.CommandTimeout = 3000;
command.CommandText = "UPDATE P SET P.[ColumnToUpdate]= T.[ColumnToUpdate] FROM [TableName Where you want to update ] AS P INNER JOIN #TmpTable AS T ON P.[PrimaryKey] = T.[PrimaryKey] ;DROP TABLE #TmpTable;";
command.ExecuteNonQuery();
}
catch (Exception ex)
{
// Handle exception properly
}
finally
{
conn.Close();
}
}
}
}
Step 3: put The ConvertToDataTable Method as shown Below.
Example:
public static DataTable ConvertToDataTable<T>(IList<T> data)
{
PropertyDescriptorCollection properties = TypeDescriptor.GetProperties(typeof(T));
DataTable table = new DataTable();
foreach (PropertyDescriptor prop in properties)
table.Columns.Add(prop.Name, Nullable.GetUnderlyingType(prop.PropertyType) ?? prop.PropertyType);
foreach (T item in data)
{
DataRow row = table.NewRow();
foreach (PropertyDescriptor prop in properties)
row[prop.Name] = prop.GetValue(item) ?? DBNull.Value;
table.Rows.Add(row);
}
return table;
}
Notes: WhereEver SquareBracket[] is there, put your own value.
Try out SqlBulkTools available on Nuget.
Disclaimer: I'm the author of this library.
var bulk = new BulkOperations();
var records = GetRecordsToUpdate();
using (TransactionScope trans = new TransactionScope())
{
using (SqlConnection conn = new SqlConnection(ConfigurationManager
.ConnectionStrings["SqlBulkToolsTest"].ConnectionString))
{
bulk.Setup<MyTable>()
.ForCollection(records)
.WithTable("MyTable")
.AddColumn(x => x.SomeColumn1)
.AddColumn(x => x.SomeColumn2)
.BulkUpdate()
.MatchTargetOn(x => x.Identifier)
.Commit(conn);
}
trans.Complete();
}
Only 'SomeColumn1' and 'SomeColumn2' will be updated. More examples can be found here
I would insert new values in a temporary table and then do a merge against the destination table, something like this:
MERGE [DestTable] AS D
USING #SourceTable S
ON D.ID = S.ID
WHEN MATCHED THEN
UPDATE SET ...
WHEN NOT MATCHED
THEN INSERT (...)
VALUES (...);
You could try to build a query that contains all data. Use a case. It could look like this
update your_table
set some_column = case when id = 1 then 'value of 1'
when id = 5 then 'value of 5'
when id = 7 then 'value of 7'
when id = 9 then 'value of 9'
end
where id in (1,5,7,9)
I'd go for a TempTable approach because that way you aren't locking anything. But if your logic needs to be only in the front end and you need to use bulk copy, I'd try a Delete/Insert approach but in the same SqlTransaction to ensure integrity which would be something like this:
// ...
dt = ConvertToDataTable(list);
using (SqlConnection cnx = new SqlConnection(myConnectionString))
{
using (SqlTranscation tran = cnx.BeginTransaction())
{
DeleteData(cnx, tran, list);
using (SqlBulkCopy bulkcopy = new SqlBulkCopy(cnx, SqlBulkCopyOptions.Default, tran))
{
bulkcopy.BulkCopyTimeout = 660;
bulkcopy.DestinationTableName = TabelName;
bulkcopy.WriteToServer(dt);
}
tran.Commit();
}
}
Complete answer, disclaimer: arrow code; this is mine built from research; Published in SqlRapper. It uses custom attributes over properties to determine whether a key is primary. Yes, super complicated. Yes super reusable. Yes, needs to be refactored. Yes, it is a nuget package. No, the documentation isn't great on github, but it exists. Will it work for everything? Probably not. Will it work for simple stuff? Oh yeah.
How easy is it to use after setup?
public class Log
{
[PrimaryKey]
public int? LogId { get; set; }
public int ApplicationId { get; set; }
[DefaultKey]
public DateTime? Date { get; set; }
public string Message { get; set; }
}
var logs = new List<Log>() { log1, log2 };
success = db.BulkUpdateData(logs);
Here's how it works:
public class PrimaryKeyAttribute : Attribute
{
}
private static bool IsPrimaryKey(object[] attributes)
{
bool skip = false;
foreach (var attr in attributes)
{
if (attr.GetType() == typeof(PrimaryKeyAttribute))
{
skip = true;
}
}
return skip;
}
private string GetSqlDataType(Type type, bool isPrimary = false)
{
var sqlType = new StringBuilder();
var isNullable = false;
if (Nullable.GetUnderlyingType(type) != null)
{
isNullable = true;
type = Nullable.GetUnderlyingType(type);
}
switch (Type.GetTypeCode(type))
{
case TypeCode.String:
isNullable = true;
sqlType.Append("nvarchar(MAX)");
break;
case TypeCode.Int32:
case TypeCode.Int64:
case TypeCode.Int16:
sqlType.Append("int");
break;
case TypeCode.Boolean:
sqlType.Append("bit");
break;
case TypeCode.DateTime:
sqlType.Append("datetime");
break;
case TypeCode.Decimal:
case TypeCode.Double:
sqlType.Append("decimal");
break;
}
if (!isNullable || isPrimary)
{
sqlType.Append(" NOT NULL");
}
return sqlType.ToString();
}
/// <summary>
/// SqlBulkCopy is allegedly protected from Sql Injection.
/// Updates a list of simple sql objects that mock tables.
/// </summary>
/// <typeparam name="T"></typeparam>
/// <param name="rows">A list of rows to insert</param>
/// <param name="tableName">a Table name if your class isn't your table name minus s.</param>
/// <returns>bool success</returns>
public bool BulkUpdateData<T>(List<T> rows, string tableName = null)
{
var template = rows.FirstOrDefault();
string tn = tableName ?? template.GetType().Name + "s";
int updated = 0;
using (SqlConnection con = new SqlConnection(ConnectionString))
{
using (SqlCommand command = new SqlCommand("", con))
{
using (SqlBulkCopy sbc = new SqlBulkCopy(con))
{
var dt = new DataTable();
var columns = template.GetType().GetProperties();;
var colNames = new List<string>();
string keyName = "";
var setStatement = new StringBuilder();
int rowNum = 0;
foreach (var row in rows)
{
dt.Rows.Add();
int colNum = 0;
foreach (var col in columns)
{
var attributes = row.GetType().GetProperty(col.Name).GetCustomAttributes(false);
bool isPrimary = IsPrimaryKey(attributes);
var value = row.GetType().GetProperty(col.Name).GetValue(row);
if (rowNum == 0)
{
colNames.Add($"{col.Name} {GetSqlDataType(col.PropertyType, isPrimary)}");
dt.Columns.Add(new DataColumn(col.Name, Nullable.GetUnderlyingType(col.PropertyType) ?? col.PropertyType));
if (!isPrimary)
{
setStatement.Append($" ME.{col.Name} = T.{col.Name},");
}
}
if (isPrimary)
{
keyName = col.Name;
if (value == null)
{
throw new Exception("Trying to update a row whose primary key is null; use insert instead.");
}
}
dt.Rows[rowNum][colNum] = value ?? DBNull.Value;
colNum++;
}
rowNum++;
}
setStatement.Length--;
try
{
con.Open();
command.CommandText = $"CREATE TABLE [dbo].[#TmpTable]({String.Join(",", colNames)})";
//command.CommandTimeout = CmdTimeOut;
command.ExecuteNonQuery();
sbc.DestinationTableName = "[dbo].[#TmpTable]";
sbc.BulkCopyTimeout = CmdTimeOut * 3;
sbc.WriteToServer(dt);
sbc.Close();
command.CommandTimeout = CmdTimeOut * 3;
command.CommandText = $"UPDATE ME SET {setStatement} FROM {tn} as ME INNER JOIN #TmpTable AS T on ME.{keyName} = T.{keyName}; DROP TABLE #TmpTable;";
updated = command.ExecuteNonQuery();
}
catch (Exception ex)
{
if (con.State != ConnectionState.Closed)
{
sbc.Close();
con.Close();
}
//well logging to sql might not work... we could try... but no.
//So Lets write to a local file.
_logger.Log($"Failed to Bulk Update to Sql: {rows.ToCSV()}", ex);
throw ex;
}
}
}
}
return (updated > 0) ? true : false;
}
The follow code will insert some values in my database. It gets 6 random values, puts the stuff in an array and then inserts it in the database.
public void LottoTest(object sender, EventArgs e)
{
Dictionary<int, int> numbers = new Dictionary<int, int>();
Random generator = new Random();
while (numbers.Count < 6)
{
numbers[generator.Next(1, 49)] = 1;
}
string[] lotto = numbers.Keys.OrderBy(n => n).Select(s => s.ToString()).ToArray();
foreach (String _str in lotto)
{
Response.Write(_str);
Response.Write(",");
}
var connectionstring = "Server=C;Database=lotto;User Id=lottoadmin;Password=password;";
using (var con = new SqlConnection(connectionstring)) // Create connection with automatic disposal
{
con.Open();
using (var tran = con.BeginTransaction()) // Open a transaction
{
// Create command with parameters (DO NOT PUT VALUES IN LINE!!!!!)
string sql =
"insert into CustomerSelections(val1,val2,val3,val4,val5,val6) values (#val1,#val2,#val3,#val4,#val5,#val6)";
var cmd = new SqlCommand(sql, con);
cmd.Parameters.AddWithValue("val1", lotto[0]);
cmd.Parameters.AddWithValue("val2", lotto[1]);
cmd.Parameters.AddWithValue("val3", lotto[2]);
cmd.Parameters.AddWithValue("val4", lotto[3]);
cmd.Parameters.AddWithValue("val5", lotto[4]);
cmd.Parameters.AddWithValue("val6", lotto[5]);
cmd.Transaction = tran;
cmd.ExecuteNonQuery(); // Insert Record
tran.Commit(); // commit transaction
Response.Write("<br />");
Response.Write("<br />");
Response.Write("Ticket has been registered!");
}
}
}
What is the best way to loop and insert MASS entries into the database. Lets say, 100,000 records via C#? I want to be able to generate the random numbers by my method and utilize the insert which i have too..
For true large scale inserts, SqlBulkCopy is your friend. The easy but inefficient way to do this is just to fill a DataTable with the data, and throw that at SqlBulkCopy, but it can be done twice as fast (trust me, I've timed it) by spoofing an IDataReader. I recently moved this code into FastMember for convenience, so you can just do something like:
class YourDataType {
public int val1 {get;set;}
public string val2 {get;set;}
... etc
public DateTime val6 {get;set;}
}
then create an iterator block (i.e. a non-buffered forwards only reader):
public IEnumerable<YourDataType> InventSomeData(int count) {
for(int i = 0 ; i < count ; i++) {
var obj = new YourDataType {
... initialize your random per row values here...
}
yield return obj;
}
}
then:
var data = InventSomeData(1000000);
using(var bcp = new SqlBulkCopy(connection))
using(var reader = ObjectReader.Create(data))
{ // note that you can be more selective with the column map
bcp.DestinationTableName = "CustomerSelections";
bcp.WriteToServer(reader);
}
You need Sql bulk insert. There is a nice tutorial on msdn http://blogs.msdn.com/b/nikhilsi/archive/2008/06/11/bulk-insert-into-sql-from-c-app.aspx
MSDN Table Value Parameters
Basically, you fill a datatable with the data you want to put into SqlServer.
DataTable tvp = new DataTable("LottoNumbers");
forach(var numberSet in numbers)
// add the data to the dataset
Then you pass the data through ADO using code similar to this...
command.Parameters.Add("#CustomerLottoNumbers", SqlDbType.Structured);
command.Parameters["CustomerLottoNumbers"].Value = tvp;
Then you could use sql similar to this...
INSERT CustomerSelections
SELECT * from #CustomerLottoNumbers
How can I get the field names of an MS Access database table?
Is there an SQL query I can use, or is there C# code to do this?
Use IDataReader.GetSchemaTable()
Here's an actual example that accesses the table schema and prints it plain and in XML (just to see what information you get):
class AccessTableSchemaTest
{
public static DbConnection GetConnection()
{
return new OleDbConnection("Provider=Microsoft.Jet.OLEDB.4.0;Data Source=..\\Test.mdb");
}
static void Main(string[] args)
{
using (DbConnection conn = GetConnection())
{
conn.Open();
DbCommand command = conn.CreateCommand();
// (1) we're not interested in any data
command.CommandText = "select * from Test where 1 = 0";
command.CommandType = CommandType.Text;
DbDataReader reader = command.ExecuteReader();
// (2) get the schema of the result set
DataTable schemaTable = reader.GetSchemaTable();
conn.Close();
}
PrintSchemaPlain(schemaTable);
Console.WriteLine(new string('-', 80));
PrintSchemaAsXml(schemaTable);
Console.Read();
}
private static void PrintSchemaPlain(DataTable schemaTable)
{
foreach (DataRow row in schemaTable.Rows)
{
Console.WriteLine("{0}, {1}, {2}",
row.Field<string>("ColumnName"),
row.Field<Type>("DataType"),
row.Field<int>("ColumnSize"));
}
}
private static void PrintSchemaAsXml(DataTable schemaTable)
{
StringWriter stringWriter = new StringWriter();
schemaTable.WriteXml(stringWriter);
Console.WriteLine(stringWriter.ToString());
}
}
Points of interest:
Don't return any data by giving a where clause that always evaluates to false. Of course this only applies if you're not interested in the data :-).
Use IDataReader.GetSchemaTable() to get a DataTable with detailed info about the actual table.
For my test table the output was:
ID, System.Int32, 4
Field1, System.String, 50
Field2, System.Int32, 4
Field3, System.DateTime, 8
--------------------------------------------------------------------------------
<DocumentElement>
<SchemaTable>
<ColumnName>ID</ColumnName>
<ColumnOrdinal>0</ColumnOrdinal>
<ColumnSize>4</ColumnSize>
<NumericPrecision>10</NumericPrecision>
<NumericScale>255</NumericScale>
<DataType>System.Int32, mscorlib, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089</DataType>
<ProviderType>3</ProviderType>
<IsLong>false</IsLong>
<AllowDBNull>true</AllowDBNull>
<IsReadOnly>false</IsReadOnly>
<IsRowVersion>false</IsRowVersion>
<IsUnique>false</IsUnique>
<IsKey>false</IsKey>
<IsAutoIncrement>false</IsAutoIncrement>
</SchemaTable>
[...]
</DocumentElement>
this will work on sql server 2005 and up:
select * from INFORMATION_SCHEMA.COLUMNS
where TABLE_Name='YourTableName'
order by ORDINAL_POSITION
Run this query:
select top 1 *
From foo
and then walk the list fields (and returned values) in the result set to get the field names.
Are you asking how you can get the column names of a table in a Database?
If so it completely depends on the Database Server you are using.
In SQL 2005 you can select from the INFORMATION_SCHEMA.Columns View
SELECT *
FROM INFORMATION_SCHEMA.COLUMNS
WHERE TABLE_NAME = 'MyTable'
IN SQL 2000 you can join SysObjects to SysColumns to get the info
SELECT
dbo.sysobjects.name As TableName
, dbo.syscolumns.name AS FieldName
FROM
dbo.sysobjects
INNER JOIN dbo.syscolumns
ON dbo.sysobjects.id = dbo.syscolumns.id
WHERE
dbo.sysobjects.name = 'MyTable'
Use the DAO automation classes. You may already have an interop library for it in your Visual Studio installation. If not, it's easy enough to create one; just add a reference to the DAO COM library.
using dao;
...
DBEngineClass dbengine = new DBEngineClass();
dbengine.OpenDatabase(path, null, null, null);
Database database = dbengine.Workspaces[0].Databases[0];
List<string> fieldnames = new List<string>();
TableDef tdf = database.TableDefs[tableName];
for (int i = 0; i < tdf.Fields.Count; i++)
{
fieldnames.Add(tdf.Fields[i].Name);
}
database.Close();
dbengine.Workspaces[0].Close();
This is just as easy as querying a system table (which I've found to be problematic in Access), and you can get a lot of additional information this way.
EDIT:
I've modified the code from what I posted yesterday, which I had just translated from VB.NET, and which was missing a couple of pieces. I've rewritten it and tested it in C# in VS2008.
This Code will print all column name of a table as a class with getter property of all column names which can be then used in c# code
declare #TableName sysname = '<EnterTableName>'
declare #Result varchar(max) = 'public class ' + #TableName + '
{'
select #Result = #Result + '
public static string ' + ColumnName + ' { get { return "'+ColumnName+'"; } }
'
from
(
select
replace(col.name, ' ', '_') ColumnName,
column_id ColumnId
from sys.columns col
join sys.types typ on
col.system_type_id = typ.system_type_id AND col.user_type_id = typ.user_type_id
where object_id = object_id(#TableName)
) t
order by ColumnId
set #Result = #Result + '
}'
print #Result
Output:
public class tblPracticeTestSections
{
public static string column1 { get { return "column1"; } }
public static string column2{ get { return "column2"; } }
public static string column3{ get { return "column3"; } }
public static string column4{ get { return "column4"; } }
}
Depending on the DB engine your using you can easily query the DB system tables for that information
For access i can't find the answer i know you can see the sys tables in access and from there you could try and determine where that information is but im not really sure how to do this part. tried using an example but got nowwhere
for microsoft SQL in c# you can do the following:
Dictionary<string, int> map =
(from DataRow row in Schema.Rows
let columnName = (string)row["ColumnName"]
select columnName)
.Distinct(StringComparer.InvariantCulture)
.Select((columnName, index) => new { Key = columnName, Value = index })
.ToDictionary(pair => pair.Key, pair => pair.Value);
thus creates a map of column name into its index which can be used as follows:
internal sealed class ColumnToIndexMap
{
private const string NameOfColumn = "ColumnName";
private DataTable Schema { get; set; }
private Dictionary<string, int> Map { get; set; }
public ColumnToIndexMap(DataTable schema)
{
if (schema == null) throw new ArgumentNullException("schema");
Schema = schema;
Map = (from DataRow row in Schema.Rows
let columnName = (string)row[NameOfColumn]
select columnName)
.Distinct(StringComparer.InvariantCulture)
.Select((columnName, index) => new { Key = columnName, Value = index })
.ToDictionary(pair => pair.Key, pair => pair.Value);
}
int this[string name]
{
get { return Map[name]; }
}
string this[int index]
{
get { return Schema.Rows[index][NameOfColumn].ToString(); }
}
}
I have had good luck with the GetSchema property of the OleDb.Connection:
A class to provide column data. This returns ALL columns in the database. The resulting DataTable can then be filtered by column names which correspond (mostly) to those found in a standard INFORMATION_SCHEMA (which MS Access does NOT provide for us):
class JetMetaData
{
/// <summary>
/// Returns a datatable containing MetaData for all user-columns
/// in the current JET Database.
/// </summary>
/// <returns></returns>
public static DataTable AllColumns(String ConnectionString)
{
DataTable dt;
using (OleDbConnection cn = new OleDbConnection(ConnectionString))
{
cn.Open();
dt = cn.GetSchema("Columns");
cn.Close();
}
return dt;
}
}
Then, Consuming the class in a rather crude and not-so-elegant example, and filtering on TABLE_NAME:
private void Form1_Load(object sender, EventArgs e)
{
DataTable dt = JetMetaData.AllColumns("", Properties.Settings.Default.JetConnection);
String RowFilter = "TABLE_NAME = 'YourTableName'";
DataView drv = dt.DefaultView;
drv.RowFilter = RowFilter;
DataGridView dgv = this.dataGridView1;
dgv.DataSource = drv;
}
Note that I do not pretend that this is all well-though out code. It is only an example. But I have used something like this on a number of occasions, and in fact even created an application to script out an entire MS Access database (contraints and all) using similar methods.
While I have seen others in this thread mention the get Schema, it seem slike some of the implementation was overly complicated . . .
Hope that helps!