I'm building a system that reads 5 CSV files each month. These files are supposed to follow a certain format and ordering. I have one master table and 5 temporary tables. Each CSV file is read first and then bulk inserted into its corresponding temporary table. After bulk inserting the 5 csv files into their respective temporary tables I once again insert all the records from the temporary table to the master table. This makes sure that all files are uploaded first before inserting the data to the master table.
I built this system using ASP.net and during debugging and testing everything went fine. The problem occurs whenever I deploy the application to a production server. After I deployed the application I used the same csv files I uploaded during development and testing and the system shows a data conversion error from string to date time format.
I tried many things to fix this but it seems the problem still persist. I tried changing the collation of the production database to the same one I used during development. I also tried changing some regional settings in the production server but it still doesn't work.
I thought maybe I can handle this programmatically and instead of bulk inserting from the temporary tables to the master table I would write some kind of a for loop that would insert each record manually to the master table, but then I suppose it would create a performance issue since I'll be inserting around 100,000 records each time.
I wonder if anyone has faced a similar issue during deployment. It still seems weird to me that the behaviour of the application changed after deployment.
following is a portion of the code where it uploads the inventory.csv file to the server and then bulk inserting the csv into a temporary table TB_TEMP_INVENTORY then inserting the records from temp to the master table TB_CATTLE. this is done to 4 other files and is almost identical to this.
OleDbConnection conn = new OleDbConnection(ConfigurationManager.AppSettings["LivestockConnectionString"]);
OleDbCommand comm;
OleDbDataAdapter adapter;
DataTable table = new DataTable();
string file = string.Empty;
string content = string.Empty;
StreamReader reader;
StreamWriter writer;
string month = monthDropDownList.SelectedValue;
string year = yearDropDownList.SelectedItem.Text;
// upload inventory file
file = System.IO.Path.GetFileName(inventoryFileUpload.PostedFile.FileName);
inventoryFileUpload.PostedFile.SaveAs("C://LivestockCSV//" + file);
// clean inventory file
file = "C://LivestockCSV//" + file;
reader = new StreamReader(file);
content = reader.ReadToEnd();
reader.Close();
writer = new StreamWriter(file);
writer.Write(content.Replace("\"", "")); // remove quotation
writer.Close();
writer = new StreamWriter(file);
writer.Write(content.Replace(",NULL,", ",,")); // remove NULL
writer.Close();
writer = new StreamWriter(file);
writer.Write(content.Replace(",0,", ",,")); // remove 0 dates
writer.Close();
writer = new StreamWriter(file);
writer.Write(content.Replace(",0", ",")); // remove 0 dates at eol
writer.Close();
try
{
conn.Open();
comm = new OleDbCommand("TRUNCATE TABLE TB_TEMP_INVENTORY", conn); // clear temp table
comm.ExecuteNonQuery();
// bulk insert from csv to temp table
comm = new OleDbCommand(#"SET DATEFORMAT DMY;
BULK INSERT TB_TEMP_INVENTORY
FROM '" + file + "'" +
#" WITH
(
FIELDTERMINATOR = ',',
ROWTERMINATOR = '\n'
)", conn);
comm.ExecuteNonQuery();
// check if data for same month exists in cattle table
comm = new OleDbCommand(#"SELECT *
FROM TB_CATTLE
WHERE Report='Inventory' AND Month=" + month + " AND Year=" + year, conn);
if (comm.ExecuteScalar() != null)
{
comm = new OleDbCommand(#"DELETE
FROM TB_CATTLE
WHERE Report='Inventory' AND Month=" + month + " AND Year=" + year, conn);
comm.ExecuteNonQuery();
}
// insert into master cattle table
comm = new OleDbCommand(#"SET DATEFORMAT MDY;
INSERT INTO TB_CATTLE(ID, Sex, BirthDate, FirstCalveDate, CurrentUnit, OriginalCost, AccumulatedDepreciation, WrittenDownValue, NetRealizableValue, CapitalGainLoss, Month, Year, Report, Locked, UploadedBy, UploadedAt)
SELECT DISTINCT ID, Sex, BirthDate, FirstCalveDate, CurrentUnit, 0, 0, 0, 0, 0, " + month + ", " + year + #", 'Inventory', 0, 'Admin', '" + DateTime.Now + #"'
FROM TB_TEMP_INVENTORY", conn);
comm.ExecuteNonQuery();
conn.Close();
}
catch (Exception ex)
{
ClientScript.RegisterStartupScript(typeof(string), "key", "<script>alert('" + ex.Message + "');</script>");
return;
}
You don't specify how you are doing the insert, but a reasonable option here would be something like SqlBulkCopy, which can take either a DataTable or an IDataReader as input; this would give you ample opportunity to massage the data - either in-memory (DataTable), or via the streaming API (IDataReader), while still using an efficient import. CsvReader is a good option for loading the CSV.
The other option is to use a very basic insert into the staging table, and massage the data via TSQL code.
Re why has it changed between dev/production; the most likely answers are:
the data you used in dev was not representative
there is an environmental/configuration difference between the two
1) Check SQL Server LANGUAGE and DATEFORMAT settings for dev/testing & production env.:
DBCC USEROPTIONS
2) What date format is used in CSV files (source) ?
3) What data type is used for date/time field (destination) ?
DECLARE #v VARCHAR(10) = '2010-08-23';
SET DATEFORMAT mdy;
SELECT CAST(#v AS DATETIME)
,CAST(#v AS DATE)
,YEAR(CAST(#v AS DATETIME))
,MONTH(CAST(#v AS DATETIME))
,DAY(CAST(#v AS DATETIME));
SET DATEFORMAT dmy;
SELECT CAST(#v AS DATETIME)
,CAST(#v AS DATE);
Related
I've got a segment of C# code that restores a SQL Server database. This works well. The issue is running it again on another .bak file. Even though the second backup has a different name, it needs to write to the same directory as the first backup and also has the same naming scheme for the .mdf and .ldf files.
I'm just curious if there is a way to modify the naming scheme of the .mdf and .ldf files, or if there is some other method to create subdirs under the initial SQL Server directory for these files to be restored to.
Error message that I'm currently getting:
Additional information: The file XXXXXX.MDF cannot be overwritten. It is being used by database XAXAXAXAX
I figure I could use a move statement, but I am trying to keep from needing all of the directory values hardcoded or logged in a config somewhere.
string sql = "SELECT database_id FROM sys.databases WHERE Name = '"+yuy+"'";
SqlConnection con = new SqlConnection(#"" + singleconn.Replace(#"\\", #"\"));
SqlCommand command = new SqlCommand(sql, con);
con.Open();
object resultObj = command.ExecuteScalar();
con.Close();
if (resultObj == null)
{
string sql2 = "Restore Database " + yuy + " FROM DISK = '" + #"\" + localdir.Replace(#"\\", #"\") + #"\" + FileName + "'";
SqlCommand command2 = new SqlCommand(sql2, con);
con.Open();
command2.ExecuteNonQuery();
con.Close();
con.Dispose();
File.Delete(#"\" + localdir.Replace(#"\\", #"\") + #"\" + FileName);
MessageBox.Show("Database recovered successfully!");
}
else
{
Random rnd = new Random();
int card = rnd.Next(52);
MessageBox.Show("There is already a database under this name; renaming the DB to " + yuy + card.ToString());
string sql2 = "Restore Database " + yuy + card.ToString() + " FROM DISK = '" + #"\" + localdir.Replace(#"\\", #"\") + #"\" + FileName + "'";
SqlCommand command2 = new SqlCommand(sql2, con);
con.Open();
command2.ExecuteNonQuery();
con.Close();
con.Dispose();
File.Delete(#"\" + localdir.Replace(#"\\", #"\") + #"\" + FileName);
MessageBox.Show("Database Recovered Successfully!");
}
Figured most of this out thanks to scsimon, for now the last thing I am getting in terms of errors is this.
Additional information: Logical file 'XXXXXX.mdf' is not part of database 'Databasename'. Use RESTORE FILELISTONLY to list the logical file names.
The thing is I am pulling this straight from the Databasename properties.
any help would be appreciated.
I'm just curious if there is a way to modify the naming scheme of the
.mdf and .ldf files, or if there is some other method to create
subdirs under the initial SQL Server directory for these files to be
restored to.
You can use the MOVE clause as you suggested with the RESTORE command to rename and/or move your data files. It looks like this:
RESTORE DATABASE myDatabase FROM DISK = '\\somedir\someSubDir\mybackup.bak'
WITH
MOVE 'datafile1' TO 'E:\somedir\new_datafile2.mdf',
MOVE 'logfile' TO 'E\somedir\new_log.ldf'
This was created to move the files from the default location in the backup to another directory (which you can do) but it can also be done to rename them too. Naturally you'd do it for all your .ndf too.
I am trying to keep from needing all of the directory values hardcoded
or logged in a config somewhere.
No worries there, just restore the database with the HEADERONLY to view the contents of the backup like name, date, and other useful info. For the file paths specifically, you'd use FILELISTONLY. This would prevent you from hard coding them. More on that here.
CREATE TABLE #DataFiles (LogicalName nvarchar(128)
,PhysicalName nvarchar(260)
,[Type] char(1)
,FileGroupName nvarchar(128) null
,Size numeric(20,0)
,MaxSize numeric(20,0)
,FileID bigint
,CreateLSN numeric(25,0)
,DropLSN numeric(25,0)
,UniqueID uniqueidentifier
,ReadOnlyLSN numeric(25,0) null
,ReadWriteLSN numeric(25,0) null
,BackupSizeInBytes bigint
,SourceBlockSize int
,FileGroupID int
,LogGroupGUID uniqueidentifier null
,DifferentialBaseLSN numeric(25,0) null
,DifferentialBaseGUID uniqueidentifier null
,IsReadOnly bit
,IsPresent bit
,TDEThumbprint varbinary(32) null
,SnapshotURL nvarchar(360) null
)
INSERT INTO #DataFiles
EXEC('RESTORE FILELISTONLY FROM DISK = ''E:\DB Backups\YourBackup.bak''')
SELECT LogicalName, PhysicalName FROM #DataFiles
DROP TABLE #DataFiles
I'm completely new to C#, so I'm sure I'm going to get a lot of comments about how my code is formatted - I welcome them. Please feel free to throw any advice or constructive criticisms you might have along the way.
I'm building a very simple Windows Form App that is eventually supposed to take data from an Excel file of varying size, potentially several times per day, and insert it into a table in SQL Server 2005. Thereafter, a stored procedure within the database takes over to perform various update and insert tasks depending on the values inserted into this table.
For this reason, I've decided to use the SQL Bulk Insert method, since I can't know if the user will only insert 10 rows - or 10,000 - at any given execution.
The function I'm using looks like this:
public void BulkImportFromExcel(string excelFilePath)
{
excelApp = new Excel.Application();
excelBook = excelApp.Workbooks.Open(excelFilePath);
excelSheet = excelBook.Worksheets.get_Item(sheetName);
excelRange = excelSheet.UsedRange;
excelBook.Close(0);
try
{
using (SqlConnection sqlConn = new SqlConnection())
{
sqlConn.ConnectionString =
"Data Source=" + serverName + ";" +
"Initial Catalog=" + dbName + ";" +
"User id=" + dbUserName + ";" +
"Password=" + dbPassword + ";";
using (OleDbConnection excelConn = new OleDbConnection())
{
excelQuery = "SELECT InvLakNo FROM [" + sheetName + "$]";
excelConn.ConnectionString = "Provider=Microsoft.ACE.OLEDB.12.0;Data Source=" + excelFilePath + ";Extended Properties='Excel 8.0;HDR=Yes'";
excelConn.Open();
using (OleDbCommand oleDBCmd = new OleDbCommand(excelQuery, excelConn))
{
OleDbDataReader dataReader = oleDBCmd.ExecuteReader();
using (SqlBulkCopy bulkImport = new SqlBulkCopy(sqlConn.ConnectionString))
{
bulkImport.DestinationTableName = sqlTable;
SqlBulkCopyColumnMapping InvLakNo = new SqlBulkCopyColumnMapping("InvLakNo", "InvLakNo");
bulkImport.ColumnMappings.Add(InvLakNo);
sqlQuery = "IF OBJECT_ID('ImportFromExcel') IS NOT NULL BEGIN SELECT * INTO [" + DateTime.Now.ToString().Replace(" ", "_") + "_ImportFromExcel] FROM ImportFromExcel; DROP TABLE ImportFromExcel; END CREATE TABLE ImportFromExcel (InvLakNo INT);";
using (SqlCommand sqlCmd = new SqlCommand(sqlQuery, sqlConn))
{
sqlConn.Open();
sqlCmd.ExecuteNonQuery();
while (dataReader.Read())
{
bulkImport.WriteToServer(dataReader);
}
}
}
}
}
}
}
catch(Exception ex)
{
MessageBox.Show(ex.ToString());
}
finally
{
excelApp.Quit();
}
}
The function runs without errors or warnings, and if I replace the WriteToServer with manual SQL commands, the rows are inserted; but the bulkImport isn't inserting anything.
NOTE: There is only one field in this example, and in the actual function I'm currently running to test; but in the end there will be dozens and dozens of fields being inserted, and I'll be doing a ColumnMapping for all of them.
Also, as stated, I am aware that my code is probably horrible - please feel free to give me any pointers you deem helpful. I'm ready and willing to learn.
Thanks!
I think it would be a very long and messy answer if I commented on your code and also gave pointer sample codes in the same message, so I decided to divide then into two messages. Comments first:
You are using automation to get what? You already have the sheet name as I see it and worse you are doing app.Quit() at the end. Completely remove that automation code.
If you needed some information from excel (like sheet names, column names) then you could use OleDbConnecton's GetOleDbSchemaTable method.
You might do the mapping basically in 2 ways:
Excel column ordinal to SQL table column name
Excel column name to SQL table column name
both would do. In a generic code, assuming you have column names same in both sources, but their ordinal and count may differ, you could get the column names from OleDbConnection schema table and do the mapping in a loop.
You are dropping and creating a table named "ImportFromExcel" for the purpose of temp data insertion, then why not simply create a temp SQL server table by using a # prefix in table name? OTOH that code piece is a little weird, it would do an import from "ImportFromExcel" if it is there, then drop and create a new one and attempt to do bulk import into that new one. In first run, SqlBulkCopy (SBC) would fill ImportFromExcel and on next run it would be copied to a table named (DateTime.Now ...) and then emptied via drop and create again. BTW, naming:
DateTime.Now.ToString().Replace(" ", "_") + "_ImportFromExcel"
doesn't feel right. While it looks tempting, it is not sortable, probably you would want something like this instead:
DateTime.Now.ToString("yyyyMMddHHmmss") + "_ImportFromExcel"
Or better yet:
"ImportFromExcel_" +DateTime.Now.ToString("yyyyMMddHHmmss")
so you would have something that is sorted and selectable for all the imports as a wildcard or looping for some reason.
Then you are writing to server inside a reader.Read() loop. That is not the way WriteToServer works. You wouldn't do reader.Read() but simply:
sbc.WriteToServer(reader);
In my next message e I will give simple schema reading and a simple SBC sample from excel into a temp table, as well as a suggestion how you should do that instead.
Here is the sample for reading schema information from Excel (here we read the tablenames - sheet names with tables in them):
private IEnumerable<string> GetTablesFromExcel(string dataSource)
{
IEnumerable<string> tables;
using (OleDbConnection con = new OleDbConnection("Provider=Microsoft.ACE.OLEDB.12.0;" +
string.Format("Data Source={0};", dataSource) +
"Extended Properties=\"Excel 12.0;HDR=Yes\""))
{
con.Open();
var schemaTable = con.GetOleDbSchemaTable(OleDbSchemaGuid.Tables, null);
tables = schemaTable.AsEnumerable().Select(t => t.Field<string>("TABLE_NAME"));
con.Close();
}
return tables;
}
And here is a sample that does SBC from excel into a temp table:
void Main()
{
string sqlConnectionString = #"server=.\SQLExpress;Trusted_Connection=yes;Database=Test";
string path = #"C:\Users\Cetin\Documents\ExcelFill.xlsx"; // sample excel sheet
string sheetName = "Sheet1$";
using (OleDbConnection cn = new OleDbConnection(
"Provider=Microsoft.ACE.OLEDB.12.0;Data Source="+path+
";Extended Properties=\"Excel 8.0;HDR=Yes\""))
using (SqlConnection scn = new SqlConnection( sqlConnectionString ))
{
scn.Open();
// create temp SQL server table
new SqlCommand(#"create table #ExcelData
(
[Id] int,
[Barkod] varchar(20)
)", scn).ExecuteNonQuery();
// get data from Excel and write to server via SBC
OleDbCommand cmd = new OleDbCommand(String.Format("select * from [{0}]",sheetName), cn);
SqlBulkCopy sbc = new SqlBulkCopy(scn);
// Mapping sample using column ordinals
sbc.ColumnMappings.Add(0,"[Id]");
sbc.ColumnMappings.Add(1,"[Barkod]");
cn.Open();
OleDbDataReader rdr = cmd.ExecuteReader();
// SqlBulkCopy properties
sbc.DestinationTableName = "#ExcelData";
// write to server via reader
sbc.WriteToServer(rdr);
if (!rdr.IsClosed) { rdr.Close(); }
cn.Close();
// Excel data is now in SQL server temp table
// It might be used to do any internal insert/update
// i.e.: Select into myTable+DateTime.Now
new SqlCommand(string.Format(#"select * into [{0}]
from [#ExcelData]",
"ImportFromExcel_" +DateTime.Now.ToString("yyyyMMddHHmmss")),scn)
.ExecuteNonQuery();
scn.Close();
}
}
While this would work, thinking in the long run, you need column names, and maybe their types differ, it might be an overkill to do this stuff using SBC and you might instead directly do it from MS SQL server's OpenQuery:
SELECT * into ... from OpenQuery(...)
the WriteToServer(IDataReader) is intended to do internally the IDataReader.Read()operation.
using (SqlCommand sqlCmd = new SqlCommand(sqlQuery, sqlConn))
{
sqlConn.Open();
sqlCmd.ExecuteNonQuery();
bulkImport.WriteToServer(dataReader);
}
You can check the MSDN doc on that function, has a working example: https://msdn.microsoft.com/en-us/library/434atets(v=vs.110).aspx
This is my comma separated value file conversion code.
string deliminator = ",";
string tablename = "Converted";
//string filelink = FileName.Text;
string fileintro = FileName.Text;
DataSet DS = new DataSet();
StreamReader SR = new StreamReader(fileintro.Trim());
DS.Tables.Add(tablename);
DS.Tables[tablename].Columns.Add("A");
DS.Tables[tablename].Columns.Add("B");
DS.Tables[tablename].Columns.Add("C");
DS.Tables[tablename].Columns.Add("D");
DS.Tables[tablename].Columns.Add("E");
DS.Tables[tablename].Columns.Add("F");
DS.Tables[tablename].Columns.Add("G");
DS.Tables[tablename].Columns.Add("H");
DS.Tables[tablename].Columns.Add("I");
...
SQLiteConnectionconn=newSQLiteConnection(ConfigurationManager.ConnectionStrings["dbcon"].ConnectionString);
SQLiteCommand cmd = new SQLiteCommand ("INSERT INTO ConvertData (a,b,c,d,e,f) values ('"+a+"'...), con);
cmd.ExecuteNonQuery();
co.Close();
FileName.Text = "";
}
it saves data in database table like "1,2,3,4,5,6,7" so how can i save it normally like 1,2,3,4,5,6,into database columns.'
The data from comma separate files are exported as string. So before inserting into the database tables convert them into required data types. For examples, if the data type in the table is int, use the Parse function and convert the string to integer before inserting, You need to repeat this for all columns
int Id = Int32.Parse(ExcelCol1.Text);
If you are looking for a detailed approach, the below link will help you
Uploading an Excel sheet and importing the data into SQL Server database
I am exporting datatable to dbf file.
If I have big column name, dbf file can export till first 10 characters.
Why it is like that?
using (var dBaseConnection = new OleDbConnection(
#"Provider=Microsoft.Jet.OLEDB.4.0; " +
#" Data Source=C:\Users\RobertWray\Documents\dBase; " +
#"Extended Properties=dBase IV"))
{
dBaseConnection.Open();
string createTableSyntax =
"Create Table Person " +
"(Name char(50), CustomerReference char(50), Phone char(20))";
var cmd = new OleDbCommand(createTableSyntax, dBaseConnection);
cmd.ExecuteNonQuery();
}
Is there any setting which I can do to export more length in column names.?
No.
DBF files have a limit on column length, and that limit is 10. If you abstract away all access to the files, you can do the mapping in a different layer (making the actual column names in the DBF just shortened IDs); if you simply write "SQL" directly, you're out of luck.
Why the limit is there? DBFs are old. Why are you using DBFs for anything today? Maybe there's a better embedded database for your usecase?
I'm creating an application that loads data from SQL Database once a day and saves it into a text file.
The main table is a "Transactions" table, which holds data about all transactions made on that day. One of the columns represents a middle-man call sign.
My program saves the data in a DataTable first and then with a StringBuilder I give it the proper form and finally save it into a text file with StreamWriter.
My question is, how or on which stage of the process can I distinguish one table entry from another. I want to create two files: one with transactions made by middle-man A and B.
This is my code so far:
// Query for Data
row = new SqlDataAdapter("SELECT [MSISDN], [Amount], [Transaction_ID], POS.[Name], MNO.[Call Sign] FROM"
+ "[Transactions] join [POS] "
+ "on Transactions.POS_ID = POS.idPOS "
+ "join [MNO] on Transactions.MNO_ID = MNO.idMNO "
+ "where [Status] = '1'", con);
row.Fill(Row);
// Save Data in StringBuilder
for (int i = 0; i < Row.Rows.Count; i++)
{
sb.Append(Row.Rows[i].ItemArray[0].ToString()).Append(",");
double amount = Convert.ToDouble(Row.Rows[i].ItemArray[1].ToString());
sb.Append(Math.Round(amount, 2).ToString().Replace(",", ".")).Append(",");
sb.Append(Row.Rows[i].ItemArray[2].ToString()).Append(",");
sb.Append(Row.Rows[i].ItemArray[3].ToString()).Append(",");
sb.Append(Row.Rows[i].ItemArray[4].ToString()).Append(",").Append(Environment.NewLine);
}
// Create a file from StringBuilder
mydocpath = #"C:\Transactions\" + fileDate.ToString(format) + ".txt";
FileStream fsOverwrite = new FileStream(mydocpath, FileMode.Create);
using (StreamWriter outfile = new StreamWriter(fsOverwrite))
{
outfile.WriteAsync(sb.ToString());
}
Hope I was clear enough. English isn't my strong side. As well as coding for what it seems...
One option.
Put all your data into a DataSet. And then do Xsl transformations against the ds.GetXml().
Here is kind of an example:
http://granadacoder.wordpress.com/2007/05/15/xml-to-xml-conversion/
But what I would do is eliminate the DataTable altogether. Use an IDataReader.
Loop over the data. Maybe do the original query as "Order By Middle-Man-Identifer", and then when the middleManIdentifer "makes a jump", close the previous file and write a new one.
Something like that.
You may be able to learn something from this demo:
http://granadacoder.wordpress.com/2009/01/27/bulk-insert-example-using-an-idatareader-to-strong-dataset-to-sql-server-xml/
Here is a couple of IDataReader helpers:
http://kalit-codesnippetsofnettechnology.blogspot.com/2009/05/write-textfile-from-sqldatareader.html
and
How to efficiently write to file from SQL datareader in c#?