EDIT: I am using SharpDevelop
I am new to C# so the answer may be an easy one...I have some code (below) and the WHILE loop runs just fine. The problem is that once the processing in the WHILE loop has finished, no more code is executed. If I put a breakpoint on my 'cn.Open(); line and run the program, I never hit that breakpoint. If I put a breakpoint on the curly bracket '}' just above the 'cn.Open();' line, the code will stop each time I hit that breakpoint. I am not sure how to get my additional code to run.
void MainFormLoad(object sender, EventArgs e)
{
DataTable dt = new DataTable();
string line = null;
int i = 0;
SqlConnection cn = new SqlConnection("Integrated Security=SSPI;Persist Security Info=False;Initial Catalog=Sandbox;Data Source=test");
StreamReader sr = File.OpenText(#"C:\Users\rl\Desktop\TEST_I~1.CSV");
while ((line = sr.ReadLine()) != null)
{
string[] data = line.Split(',');
if (data.Length > 0)
{
if (i == 0)
{
foreach (var item in data)
{
dt.Columns.Add(item.ToString());
}
i++;
}
DataRow row = dt.NewRow();
row.ItemArray = data;
dt.Rows.Add(row);
}
}
cn.Open();
SqlBulkCopy copy = new SqlBulkCopy(cn);
{
// copy.ColumnMappings.Add(0, 0);
// copy.ColumnMappings.Add(1, 1);
// copy.ColumnMappings.Add(2, 2);
// copy.ColumnMappings.Add(3, 3);
// copy.ColumnMappings.Add(4, 4);
copy.DestinationTableName = "Member2";
copy.WriteToServer(dt);
}
You have a few items you may want to address. These may or may not be related to whatever issue you're having debugging with #develop.
Declaring things long before you use them (style guidelines)
Not disposing of things that implement IDisposable (use using statements!)
Inner scope block; the copy variable is being used in it's own scope for no apparently good reason (I may be wrong, but it could be what's throwing #develop's debugger for a loop)
Instead, your code should be closer to this:
void MainFormLoad(object sender, EventArgs e)
{
var dt = new DataTable();
// You may want to pass other parameters to OpenText for read mode, etc.
using (var sr = File.OpenText(#"C:\Users\rl\Desktop\TEST_I~1.CSV"))
{
var first = true;
string line = null;
while ((line = sr.ReadLine()) != null)
{
string[] data = line.Split(',');
if (data.Length > 0)
{
if (first)
{
foreach (var item in data)
{
dt.Columns.Add(item.ToString());
}
first = false;
// Don't add the first row's data in the table (headers?)
continue;
}
var row = dt.NewRow();
row.ItemArray = data;
dt.Rows.Add(row);
}
}
}
using (var cn = new SqlConnection("<connection string>"))
{
cn.Open();
using (var copy = new SqlBulkCopy(cn))
{
// copy.ColumnMappings.Add(0, 0);
// copy.ColumnMappings.Add(1, 1);
// copy.ColumnMappings.Add(2, 2);
// copy.ColumnMappings.Add(3, 3);
// copy.ColumnMappings.Add(4, 4);
copy.DestinationTableName = "Member2";
copy.WriteToServer(dt);
}
}
}
The code is a bit odd but it looks like it should work. There's probably a file lock either making you run against old builds or hang on the .csv open line.
Cory's suggestions for tidying the code are rather good.
I think you have an infinite loop going on because your while check isn't quite right. You're asking if line = sr.ReadLine() is null, not if line is null. The result of setting line to the result of the read function will never return null.
Related
I am inputting a text file into a DataTable and then using SqlBulkCopy to copy to a Database. While BulkCopy is fast, inserting 50000+ lines into DataTable is not (around 5 mins). How do I make it efficient?
Can I insert data into the DataTable quickly?
If not, is there a way to save the inserted data permanently into the DataTable so I don't have to insert it every time I run the program?
for (; i < fares.Length; )
{
k = i;
Console.WriteLine("Inserting " + k + " out of " + (fares.Length));
for (; i <= (k + 3); i++)
{
if (i % 4 == 0)
{
for (int j = 0; j < fares.Length - 1; j++)
{
{
int space = fares[i].IndexOf(" ");
startStation = fares[i].Substring(0, space);
endStation = fares[i].Substring(space + 1, fares[i].Length - space - 1);
}
}
}
else if (i % 4 == 1)
{
valueFare = fares[i];
}
else if (i % 4 == 2)
{
standardFare = fares[i];
}
else if (i % 4 == 3)
{
time = int.Parse(fares[i]);
}
}
faresDT.Rows.Add(startStation, endStation, valueFare, standardFare, time);
If what you want is to optimize your load to the database, I suggest that you get rid of the DataTable completely. By making use of Marc Gravell's FastMember (and anyone who's using SqlBulkCopy should be using FastMember IMHO) you can get a DataReader directly from any IEnumerable.
I would use some variation of the below code whenever writing from a file directly to a database. The below code will stream the contents of the file directly to the SqlBulkCopy operation thru the clever use of yield returns and lazy load of IEnumerable.
using System;
using System.Collections.Generic;
using System.Data.SqlClient;
using System.IO;
using System.Text;
using FastMember;
namespace BulkCopyTest
{
public class Program
{
public static void Main(string[] args)
{
const string filePath = "SOME FILE THAT YOU WANT TO LOAD TO A DB";
WriteData(GetData<dynamic>(filePath));
}
private static void WriteData<T>(IEnumerable<T> data)
{
using (var bcp = new SqlBulkCopy(GetConnection(), SqlBulkCopyOptions.TableLock, null))
using (var reader = ObjectReader.Create(data))
{
SetColumnMappings<T>(bcp.ColumnMappings);
bcp.BulkCopyTimeout = 300;
bcp.BatchSize = 150000;
bcp.DestinationTableName = ""; //TODO: Set correct TableName
bcp.WriteToServer(reader);
}
}
private static void SetColumnMappings<T>(SqlBulkCopyColumnMappingCollection mappings)
{
//Setup your column mappings
}
private static IEnumerable<T> GetData<T>(string filePath)
{
using (var fileStream = File.OpenRead(filePath))
using (var reader = new StreamReader(fileStream, Encoding.UTF8))
{
string line;
while ((line = reader.ReadLine()) != null)
{
//TODO: Add actual parsing logic and whatever else is needed to create an instance of T
yield return Activator.CreateInstance<T>();
}
}
}
private static SqlConnection GetConnection()
{
return new SqlConnection(new SqlConnectionStringBuilder
{
//TODO: Set Connection information here
}.ConnectionString);
}
}
}
In this case I think you should take advantage of the BeginLoadData, LoadDataRow and EndLoadData methods provided in the DataTable class, you could use them like this:
try
{
faresDT.BeginLoadData();
// Your for loop...
{
// Logic defining the value of startStation, endStation, valueFare, standardFare and time removed for briefness.
faresDT.LoadDataRow(new object[] {startStation, endStation, valueFare, standardFare, time}, true);
}
}
finally
{
faresDT.EndLoadData();
}
What BeginLoadData() does is turning off some processing that happens every time you add a row, and only does it once when you are done loading data by calling EndLoadData().
You can find more details about these APIs here:
https://learn.microsoft.com/en-us/dotnet/api/system.data.datatable.loaddatarow?view=netframework-4.7.2
i am trying to bind xrefs in a sideload drawing database. the program is halting at this line ' if(!xNode.Database.Filename.Equals(NewDb.Filename))'. i am also receiving this error 'System.NullReferenceException: Object reference not set to an instance of an object.at XBind.RecursiveFileProcessor.ProcessFile(String path).' i've done some reaserch and found VB.NET code to attach a xref and tried to extrapolate that with no success. i'd appreciate someone pointing me in the right direction on this.
using (Database NewDb = new Database(false, true))
{
NewDb.ReadDwgFile(path, FileOpenMode.OpenForReadAndWriteNoShare, true, "");
NewDb.CloseInput(true);
using (Transaction tr = NewDb.TransactionManager.StartTransaction())
{
ObjectIdCollection xrefCollection = new ObjectIdCollection();
XrefGraph xg = NewDb.GetHostDwgXrefGraph(false);
int numOfNodes = xg.NumNodes;
for (int cnt = 0; cnt < xg.NumNodes; cnt++)
{
XrefGraphNode xNode = xg.GetXrefNode(cnt) as XrefGraphNode;
if (!xNode.Database.Filename.Equals(NewDb.Filename))
{
if (xNode.XrefStatus == XrefStatus.Resolved)
{
xrefCollection.Add(xNode.BlockTableRecordId);
}
}
}
if (xrefCollection.Count != 0)
{
NewDb.BindXrefs(xrefCollection, true);
}
tr.Commit();
}
NewDb.SaveAs(path, DwgVersion.Current);
}
Actually, this will work in-memory. Winslow North missing the following line of code after the CloseInput()...
NewDb.ResolveXrefs(true, false);
But also, you do not need the Transaction for this. It's not necessary. I created my own sample and tested it. It works. If you need me to post that, let me know. The problem was that the xNode had a null database due to the fact that the Xref was not resolved. You have to do that manually with the line above.
Don't believe this will work for in-memory database, you may try this approach, see a pice of it below:
[CommandMethod("CHX")]
public void ChangeXref()
{
var doc = Application.DocumentManager.MdiActiveDocument;
if (doc == null) return;
var ed = doc.Editor;
var db = doc.Database;
// Get the database associated with each xref in the
// drawing and change all of its circles to be dashed
using (var tr = db.TransactionManager.StartTransaction())
{
var bt = (BlockTable)tr.GetObject(db.BlockTableId, OpenMode.ForRead);
var ms = (BlockTableRecord)tr.GetObject(bt[BlockTableRecord.ModelSpace], OpenMode.ForRead);
// Loop through the contents of the modelspace
foreach (var id in ms)
{
// We only care about BlockReferences
var br = tr.GetObject(id, OpenMode.ForRead) as BlockReference;
if (br != null)
{
// Check whether the associated BlockTableRecord is
// an external reference
var bd = (BlockTableRecord)tr.GetObject(br.BlockTableRecord, OpenMode.ForRead);
if (bd.IsFromExternalReference)
{
// If so, get its Database and call the function
// to change the linetype of its Circles
var xdb = bd.GetXrefDatabase(false);
if (xdb != null)
{
using (var xf = XrefFileLock.LockFile(xdb.XrefBlockId))
{
// Make sure the original symbols are loaded
xdb.RestoreOriginalXrefSymbols();
xdb.RestoreForwardingXrefSymbols();
}
}
}
}
}
tr.Commit();
}
}
I have this method to read from a .dbf file:
public DataTable ReadBulkDBF(string dbfFile, Dictionary<string, string> columnKeys, int maxRows, string dynamicValue, int nextId)
{
long start = DateTime.Now.Ticks;
DataTable dt = new DataTable();
BinaryReader recReader;
string number;
string year;
string month;
string day;
long lDate;
long lTime;
DataRow row;
int fieldIndex;
bool foundLastColumn = false;
List<string> keys = new List<string>(columnKeys.Keys);
List<string> values = new List<string>(columnKeys.Values);
// For testing purposes
int rowCount = 0;
// If there isn't even a file, just return an empty DataTable
if ((!File.Exists(dbfFile)))
{
return dt;
}
BinaryReader br = null;
try
{
// Will allow shared open as long as the other application using it allows it too.
// Read the header into a buffer
br = new BinaryReader(File.Open(dbfFile, FileMode.Open, FileAccess.Read, FileShare.ReadWrite));
byte[] buffer = br.ReadBytes(Marshal.SizeOf(typeof(DBFHeader)));
// Marshall the header into a DBFHeader structure
GCHandle handle = GCHandle.Alloc(buffer, GCHandleType.Pinned);
DBFHeader header = (DBFHeader)Marshal.PtrToStructure(handle.AddrOfPinnedObject(), typeof(DBFHeader));
handle.Free();
// Read in all the field descriptors. Per the spec, 13 (0D) marks the end of the field descriptors
ArrayList fields = new ArrayList();
while ((13 != br.PeekChar()))
{
buffer = br.ReadBytes(Marshal.SizeOf(typeof(FieldDescriptor)));
handle = GCHandle.Alloc(buffer, GCHandleType.Pinned);
fields.Add((FieldDescriptor)Marshal.PtrToStructure(handle.AddrOfPinnedObject(), typeof(FieldDescriptor)));
handle.Free();
}
// Read in the first row of records, we need this to help determine column types below
((FileStream)br.BaseStream).Seek(header.headerLen + 1, SeekOrigin.Begin);
buffer = br.ReadBytes(header.recordLen);
recReader = new BinaryReader(new MemoryStream(buffer));
// Create the columns in our new DataTable
DataColumn col = null;
dt.Columns.Add(new DataColumn("updateId", typeof(int)));
if (!dbfFile.Contains("con_compania")) { dt.Columns.Add(new DataColumn("dynamic", typeof(string))); }
dt.Columns.Add(new DataColumn("fechasync", typeof(DateTime)));
foreach (FieldDescriptor field in fields)
{
// Adds columns to DataTable dt
}
// Skip past the end of the header.
((FileStream)br.BaseStream).Seek(header.headerLen, SeekOrigin.Begin);
// Read in all the records
for (int counter = 0; counter < header.numRecords && dt.Rows.Count < maxRows; counter++)
{
// First we'll read the entire record into a buffer and then read each field from the buffer
// This helps account for any extra space at the end of each record and probably performs better
buffer = br.ReadBytes(header.recordLen);
recReader = new BinaryReader(new MemoryStream(buffer));
// All dbf field records begin with a deleted flag field. Deleted - 0x2A (asterisk) else 0x20 (space)
if (recReader.ReadChar() == '*')
{
continue;
}
// Loop through each field in a record
fieldIndex = 2;
rowCount = dt.Rows.Count;
row = dt.NewRow();
foreach (FieldDescriptor field in fields)
{
switch (field.fieldType)
{
// Casts field's value according to its type and saves it in the dt.
}
fieldIndex++;
}
// Looks for key-value combination in every row until
// it finds it to know where to start reading the new rows.
if (!foundLastColumn && columnKeys.Keys.Count > 0)
{
foundLastColumn = true;
int i = 3;
if (dbfFile.Contains("con_compania")) { i = 2; }
for (; i < keys.Count && foundLastColumn; i++)
{
if (!row[keys[i]].ToString().Equals(values[i]))
{
foundLastColumn = false;
}
}
}
else
{
dt.Rows.Add(row);
nextId++;
}
}
}
catch (Exception e)
{
throw e;
}
finally
{
if (null != br)
{
br.Close();
br.Dispose();
}
}
long count = DateTime.Now.Ticks - start;
return dt;
}
The problem is somewhere I am leaving some kind of reference to this, so I'm getting OOM.
The method is called with something like:
DataTable dt = new ParseDBF().ReadBulkDBF(...);
//Use dt
dt.Dispose();
dt = null;
If I only call Dispose() it keeps the reference and if I call null dt becomes null, but the reference to the ParseDBF object is still there somewhere.
Any idea where the leak might be? I have looked all over the internet for ideas and tried calling Dispose() and Close(), and setting as null everything I can think of after I use it and it keeps happening.
I notice that recreader may not be getting freed.
I would strongly suggest making use of using blocks within this code to ensure that IDisposable objects are cleaned when execution leaves the using scope.
I have a csv file delimited with pipe(|). I am reading it using the following line of code:
IEnumerable<string[]> lineFields = File.ReadAllLines(FilePath).Select(line => line.Split('|'));
Now, I need to bind this to a GridView. So I am creating a dynamic DataTable as follows:
DataTable dt = new DataTable();
int i = 0;
foreach (string[] order in lineFields)
{
if (i == 0)
{
foreach (string column in order)
{
DataColumn _Column = new DataColumn();
_Column.ColumnName = column;
dt.Columns.Add(_Column);
i++;
//Response.Write(column);
//Response.Write("\t");
}
}
else
{
int j = 0;
DataRow row = dt.NewRow();
foreach (string value in order)
{
row[j] = value;
j++;
//Response.Write(column);
//Response.Write("\t");
}
dt.Rows.Add(row);
}
//Response.Write("\n");
}
This works fine. But I want to know if there is a better way to convert IEnumerable<string[]> to a DataTable. I need to read many CSVs like this, so I think the above code might have performance issues.
Starting from .Net 4:
use ReadLines.
DataTable FileToDataTable(string FilePath)
{
var dt = new DataTable();
IEnumerable<string[]> lineFields = File.ReadLines(FilePath).Select(line => line.Split('|'));
dt.Columns.AddRange(lineFields.First().Select(i => new DataColumn(i)).ToArray());
foreach (var order in lineFields.Skip(1))
dt.Rows.Add(order);
return dt;
}
(edit: instead this code, use the code of #Jodrell answer, This prevents double charging of the Enumerator).
Before .Net 4:
use streaming:
DataTable FileToDataTable1(string FilePath)
{
var dt = new DataTable();
using (var st = new StreamReader(FilePath))
{
// first line procces
if (st.Peek() >= 0)
{
var order = st.ReadLine().Split('|');
dt.Columns.AddRange(order.Select(i => new DataColumn(i)).ToArray());
}
while (st.Peek() >= 0)
dt.Rows.Add(st.ReadLine().Split('|'));
}
return dt;
}
since, in your linked example, the file has a header row.
const char Delimiter = '|';
var dt = new DataTable;
using (var m = File.ReadLines(filePath).GetEnumerator())
{
m.MoveNext();
foreach (var name in m.Current.Split(Delimiter))
{
dt.Columns.Add(name);
}
while (m.MoveNext())
{
dt.Rows.Add(m.Current.Split(Delimiter));
}
}
This reads the file in one pass.
private void SaveButton_Click(object sender, EventArgs e)
{
try
{
if (CashPaymentGridView.Rows.Count > 1)
{
CashPaymentandReceivedE cashpament =new CashPaymentandReceivedE();
List<CashPaymentandReceivedE> cashpaymentList = new List<CashPaymentandReceivedE>();
foreach ( DataGridViewRow rows in CashPaymentGridView.Rows)
{
if (rows.IsNewRow )
{
break ;
}
cashpament.VR_NO= Convert.ToInt16(VoucherNoTextBox.Text);
cashpament.VR_DATE = VrDate.Value ;
cashpament.ETYPE = "CPV";
cashpament.USER_ID = "1";
cashpament.PARTY_ID= Convert.ToString (rows.Cells[2].Value) ;
cashpament.DESCRIPTION = Convert.ToString ( rows.Cells[3].Value);
cashpament.INVOICE = Convert.ToString(rows.Cells[4].Value);
cashpament.DEBIT = Convert.ToInt32(rows.Cells[5].Value);
cashpament.CREDIT = 0;
cashpaymentList.Add(cashpament);
cashpament = new CashPaymentandReceivedE();
cashpament.VR_NO =Convert.ToInt16(VoucherNoTextBox.Text);
cashpament.VR_DATE = VrDate.Value;
cashpament.ETYPE = "CPV";
cashpament.USER_ID = "1";
cashpament.PARTY_ID = NewAccountsDAL.Get_Id_Name ("CASH");
cashpament.DESCRIPTION = Convert.ToString(rows.Cells[3].Value);
cashpament.INVOICE = Convert.ToString(rows.Cells[4].Value);
cashpament.CREDIT = Convert.ToInt32(rows.Cells[5].Value);
cashpament.DEBIT = 0;
cashpaymentList.Add(cashpament);
}
if (CashPaymentandReceivedDAL.Save(cashpaymentList))
{
MessageBox.Show("SAVE SUCCESSFULLY...............");
ResetForm();
}
}
else
{
MessageBox.Show ("Please select atleast one record.....");
}
}
catch (Exception ex)
{
MessageBox.Show(ex.Message );
}
}
Stored Procedure for saving data is given as.
public static bool Save(List <CashPaymentandReceivedE> cashreceivedpayment)
{
bool blnResult = false;
SqlConnection objSqlConnection = new SqlConnection(ConnectionString.Connection);
//SqlTransaction objSqlTransaction = null;
try
{
objSqlConnection.Open();
//objSqlTransaction = objSqlConnection.BeginTransaction();
int R = 0;
while (R < cashreceivedpayment.Count )
{
SqlCommand objSqlCommand = new SqlCommand("CASHRECEIVED_Save", objSqlConnection);
objSqlCommand.CommandType = CommandType.StoredProcedure;
//SqlParameter objIdentityParameter = objSqlCommand.Parameters.Add("#PLED_ID", SqlDbType.BigInt);
//objIdentityParameter.Direction = ParameterDirection.Output;
//objSqlCommand.Parameters.AddWithValue("#PLED_ID", cashreceivedpayment[R].PLED_ID);
objSqlCommand.Parameters.AddWithValue("#COMPANY_ID", "1");
objSqlCommand.Parameters.AddWithValue("#PARTY_ID", cashreceivedpayment[R].PARTY_ID);
objSqlCommand.Parameters.AddWithValue("#VR_NO", cashreceivedpayment[R].VR_NO);
objSqlCommand.Parameters.AddWithValue("#ETYPE", cashreceivedpayment[R].ETYPE);
objSqlCommand.Parameters.AddWithValue("#VR_DATE", cashreceivedpayment[R].VR_DATE);
objSqlCommand.Parameters.AddWithValue("#DESCRIPTION", cashreceivedpayment[R].DESCRIPTION);
objSqlCommand.Parameters.AddWithValue("#DEBIT", cashreceivedpayment[R].DEBIT);
objSqlCommand.Parameters.AddWithValue("#CREDIT", cashreceivedpayment[R].CREDIT);
objSqlCommand.Parameters.AddWithValue("#USER_ID", cashreceivedpayment[R].USER_ID);
//objSqlCommand.Parameters.AddWithValue("#COMPNAY_ID", cashreceivedpayment[R].COMPANY_ID);
objSqlCommand.Parameters.AddWithValue("#DESCRIPTION2", "DESCRIPTION2");
objSqlCommand.Parameters.AddWithValue("#INVOICE", cashreceivedpayment[R].INVOICE);
objSqlCommand.ExecuteNonQuery();
R++;
blnResult = true;
}
}
catch (Exception ex)
{
//objSqlTransaction.Rollback();
MessageBox.Show(ex.Message);
}
finally
{
//objSqlTransaction.Commit();
objSqlConnection.Close();
}
return blnResult;
}
When i save the record, one record should be of Party_id and one should be of cash.
but when i select more than one record just one entry saving to cash. when when i load the record jst one record is loaded.plz help if u understand.....
You're creating one CashPaymentandReceivedE object, and adding the reference to it on each iteration of the list. You're then also changing all the data within that single object on each iteration. Just move this line:
CashPaymentandReceivedE cashpament =new CashPaymentandReceivedE();
... inside your foreach statement and the problem should be resolved.
Before you do so though, make sure you understand why your code is behaving like this. It's really important to understand that the list doesn't contain objects - it contains references to objects. In your case, it would contain several references to a single object, until you fix it.
I'd also strongly suggest using a foreach in your Save method - or if you really need the index for some reason, use a for loop instead of a while.