I have a function that inserts SQL data based on a spreadsheet. I have had the issue of certain rows triggering the exception due to being truncated, and I am trying to figure out the rows that are causing the issue. (I have to query 3 different tables so I am using a function passing in SQL/command parameters/values instead of writing the same function 3 times)
The function works to insert the SQL data, except for the few rows that throws the ex message:
String or Binary data would be truncated. The statement has been
terminated
My question is how do I print out the row number that causes the above message to troubleshoot the data in the excel sheet. The size of the sheet is 100+ thousand rows, so I don't want to go through it row by row.
The function I have:
public static void insert_data(string[] cols, string[] vals, string sql)
{
int exception_count = 0;
List<string> rows = new List<string>();
string connectionString = "Server = ; Database = ; User Id = ; Password = ";
using (SqlConnection connection = new SqlConnection(connectionString))
using (SqlCommand command = new SqlCommand(sql, connection))
{
connection.Open();
for(int x = 0; x<cols.Length; x++)
command.Parameters.AddWithValue(cols[x], vals[x]);
try
{
command.ExecuteNonQuery();
}
catch (Exception ex)
{
exception_count =+ 1;
Console.WriteLine(ex.Message);
//rows.Add(--rows number--);
}
}
}
Like I can see, the rows come from array, so first use a lambda expression to find the rows that have more than the length that you want, or viceverse find rows than have less or equal length, depends of you.
public void test2()
{
//ensure that there is no empty rows in the array... of will thrown an exception
string[] vals = new string[7];
int myMaxColumnDatabaseLenght = 7;
vals[0] = "length7";
vals[1] = "length7";
vals[2] = "length7";
vals[3] = "length7";
vals[4] = "length_8";
vals[5] = "length__9";
vals[6] = "length__10";
Debug.WriteLine(vals.Count());
vals = vals.Where(x => x.Length <= myMaxColumnDatabaseLenght).ToArray();
Debug.WriteLine(vals.Count());
}
If you want make this dynamic, maybe you need query the length definition of the specific columns in SQL, if you want this part I will try to find for add information about that... .
Related
I have labels "Hardwork" and 1 datagirdview display when load form. I use the code below to do the quantity comparison in the column "TotalTime". I want if the value is in column "TotalTime"> = 30 then labels "Harwork" + 1
but not run.the result is: specified cast is not valid.
Please, help me fix it
public void BtnSearch_Click(object sender, EventArgs e)
{
db = new DbConnector();
lbTotal.Text = "00";
db.fillDataGridView("select *from tbl_WorkLoad where TimeComplete Between'" + dateTimePicker1.Value.ToString("dd-MM-yy| HH:mm:tt") + "' and '" + dateTimePicker2.Value.ToString("dd-MM-yy| HH:mm:tt") + "'", dataGridView1);
const string HardWorkLabelText = "Hard Work Count: {0}";
const int HardWorkThreshold = 30;
try
{
IEnumerable<DataGridViewRow> rows = dataGridView1.Rows.Cast<DataGridViewRow>().Where(r => ((Int32)r.Cells["TotalTime"].Value) >= HardWorkThreshold);
lbHardwork.Text = string.Format(HardWorkLabelText, rows.Count());
{
for (int i = 0; i < dataGridView1.Rows.Count; i++)
{
lbTotaltime.Text = (Convert.ToString(double.Parse(lbTotaltime.Text) + double.Parse(dataGridView1.Rows[i].Cells[7].Value.ToString())));
}
catch (Exception ex)
{
MessageBox.Show(ex.Message.ToString());
}
}
You have apart from the exception, you have few other issues in the code. But let us focus on the issue which you reported.
From the code, I understand that, you want to display the number of hardwork and also you want to display the total hardwork time.
Since you are looping thru the rows of gridview, you can calculate both of these in the for loop.
var hardworkCount = 0;
var totalTimeSum = 0.0;
for (int i = 0; i < dataGridView1.Rows.Count; i++)
{
double totalTime = double.Parse(dataGridView1.Rows[i].Cells[7].Value.ToString());
if(totalTime >= HardWorkThreshold)
{
hardworkCount++;
}
totalTimeSum += totalTime;
}
lbHardwork.Text = hardworkCount.ToString();
lbTotaltime.Text = totalTimeSum.ToString();
You life would be a lot simpler if you stop pulling your data out of the datagridview, and just have your datagridview based off a datatable that holds the data. It might be this way already, but we can't see the definition of your filldatagridview method
Here's how I'd do it:
//assuming sql sevrer, use correct dataadapter for your db
using(var da = new SqlDataAdapter "select * from tbl_WorkLoad where TimeComplete Between #a and #b", "YOUR CONNECTION STRING HERE")){
da.SelectedCommand.Parameters.AddWithValue("#a", dateTimePicker1.Value);
da.SelectedCommand.Parameters.AddWithValue("#b", dateTimePicker2.Value);
var dt = new DataTable();
da.Fill(dt);
dataGridViewX.DataSource = dt;
lbHardwork.Text = dt.Compute("Count([TotalTime])", "[TotalTime] >= 30").ToString();
lbTotalTime.Text = dt.Compute("Sum([TotalTime])", "[TotalTime] >= 30").ToString();
}
The way you wrote your SQL is a pattern that risks SQL injection if used with strings - always use parameters in SQL. Always. There is never a reason not to when dealing with data values. See why, and also here
You should read this blog before using AddWithValue with strings in SQL Server. There are better ways to do parameters than AWV, but at the very least it would be preferable to use AddWithValue than what you're doing now
Don't access data in a gridview via the grid; bind the grid to a datatable and if you want data, access the datatable. If you want to get the datatable out of a grid at any point, you can use var dt = datagridviewX.DataSource as DataTable
Is It possible to create a Linq for retrieving longest string values of DataReader columns ? Column data should be converted to string and then return longest string.
What I know so far is how to get same thing with DataTable which is Enumerable (as I asked here), example:
string maxString = dt
.AsEnumerable()
.Select(row => row[mycolumn].ToString())
.OrderByDescending(st => st.Length)
.FirstOrDefault();
I've tried combining upper solution for DataReader like this:
var enumerable = reader
.Cast<IDataRecord>();
string max_string = enumerable
.Select(record => record.GetName(col).ToString())
.Aggregate((s, a) => a.Length > s.Length ? a : s);
Unfortunally this doesn't work, I get
Sequence contains no elements in at
System.Linq.Enumerable.Aggregate
error. Thanks for help in advance :)
EDIT: I'm looking for a solution without loading data into Datatable etc., just directly from DataReader object. I'm trying to avoid "Out of memory exception", because data is large.
Latest attempt, suggested by Power Mouse (It returns correct value, but from column 1 only):
for (int col = 0; col < reader.FieldCount; col++)
{
string col_name = reader.GetName(col).ToString();
var enumerable = reader.Cast<IDataRecord>();
string max_string = enumerable.Where(x => enumerable.Max(y => y[col_name].ToString()
.Length) == x[col_name].ToString().Length)
.FirstOrDefault()?[col_name].ToString();
Console.WriteLine("max string of column is : " + max_string);
}
so according to your original request: you need to find the longest text in specific column when you utilizing a DataReader on the fly.
please review example
string storedString = String.Empty;
SqlConnection connection = new SqlConnection(this.Connection.ConnectionString);
using (connection)
{
string SQLcommand = "select * FROM (VALUES(1, 'xxx' ), (2, 'really long string xxxxxx'), (3, 'short string'), (4, 'another string')) t (id, fName)";
SqlCommand command = new SqlCommand(SQLcommand, connection);
connection.Open();
SqlDataReader reader = command.ExecuteReader();
while (reader.Read())
{
storedString = reader.Cast<IDataRecord>()
.Where(w=> w.GetOrdinal("fName").ToString().Length == reader.Cast<IDataRecord>()
.Max(m => m.GetOrdinal("fName")
.ToString().Length))
.Select(s=> s.GetString(1))
.FirstOrDefault();
}
}
Console.WriteLine($"The longest string: {storedString}. charcount= {storedString.Length}");
the result would be :
The longest string: really long string xxxxxx. charcount= 25
as you explained that you need to check multiple columns:
string storedNameString = String.Empty;
string storedName2String = String.Empty;
SqlConnection connection = new SqlConnection(this.Connection.ConnectionString);
using (connection)
{
string SQLcommand = "select * FROM (VALUES(1, 'xxx', 'dddddd' ), (2, 'really long string xxxxxx','dfghdt'), (3, 'short string', 'anothercolumn long string'), (4, 'another string', 'test')) t (id, fName, fName2)";
SqlCommand command = new SqlCommand(SQLcommand, connection);
connection.Open();
SqlDataReader reader = command.ExecuteReader();
while (reader.Read())
{
string fName = reader.GetString(reader.GetOrdinal("fName")).ToString();
if(fName.Length >= storedNameString.Length)
storedNameString = fName;
string fName2 = reader.GetString(reader.GetOrdinal("fName2")).ToString();
if (fName2.Length >= storedName2String.Length)
storedName2String = fName2;
}
}
Console.WriteLine($"The longest string: {storedNameString}. charcount= {storedNameString.Length}");
Console.WriteLine($"The longest string: {storedName2String}. charcount= {storedName2String.Length}");
I solved my problem, unfortunally without LINQ. Problem is that with DataReader you cannot just simply loop through rows & columns as you can with DataTable once stored in memory, but you must perfom somekind of same logic while reader.Read() method is running.
So, best thing I could came up with is to store column indexes and their string values into Dictionary while .Read() method is running.
Doing that, you must be careful about string blank spaces & null values. Here is my solution, which runs good for me:
Dictionary<int, string> col_values = new Dictionary<int, string>();
using (OracleDataReader reader = cmd.ExecuteReader())
{
for (int i = 0; i < reader.FieldCount; i++)
{
//First add indexes to Dictionary
// I add column names here - didn't test for string.Empty !!
col_values.Add(i, string.Empty);
}
//Then read row by row and modify Dictionary - If text is larger than string.Empty
//Dictionary must be .ToArray(), or else you'll have an error for modifying collection
while (reader.Read())
{
foreach (var item in col_values.ToArray())
{
string rdr_text = reader[item.Key].ToString().Trim()??string.Empty;
if (item.Value.Length<rdr_text.Length)
{
col_values[item.Key] = rdr_text;
}
}
}
foreach (var item in col_values)
{
//...And here we have all longest strings stored, for each column...Job done
}
}
For my purpuses this Iteration reads around 2.3 mio rows with 12 columns in 4 minutes. It's not fast, but at least It works. If anyone has better/faster idea please provide answer.
I am since recently struggling with a new problem. I've got a database table that contains multiple fields (columns) with a few rows containing (money - decimal) values that are related to the fieldnames.
For example:
table = money
Field names: Rent A, Rent B, Rent C
Values: $10, $20, $30.
What I want to do is getting these values from the database, add them together and displaying the total amount in a label.
Up till now I used the OleDbDataReader for all my outputting/storing value needs. Though I have absolutely no clue how I can add the read values since the reader usually expects an pre defined field name to read.
In my project however, a user can add a custom new field name (so a pre defined field name in the reader isn't possible because you don't know which custom field names will be added by the user in the DB. These custom added fields names and their values need to be added in the total amount as well though..
Does anyone have a clue how I can fix this?
I've tried multiple things like storing it in an array, defining decimal variables and using something like x = x + (decimal)reader[0] but this all didn't work so I think I am way off.
The code (and query) I am using for the reading part is as follows:
try
{
connection.Open();
OleDbCommand command = new OleDbCommand();
command.Connection = connection;
string query = "select * from money where [Month]='January'";
command.CommandText = query;
OleDbDataReader reader = command.ExecuteReader();
while (reader.Read())
{
//Something I tried
//x[0] = (decimal)reader[0];
//x[1] = (decimal)reader[1];
//And so on...
//Another thing I tried
//list.Add(reader.GetInt32(0));
}
//list.ToArray();
//int sum = list.Sum();
// y = x[0] + x[1] + ...;
connection.Close();
}
catch (Exception ex)
{
MessageBox.Show("Error" + ex);
}
You should just be able to declare a variable, then add up all the columns. If you don't know the number of columns in the reader, you can get that using reader.FieldCount.
You do not need to know the column name to get data from the reader. You can access it by column index as you started to do, e.g. reader[0], or using the helper methods such as GetDecimal(0).
try
{
connection.Open();
OleDbCommand command = new OleDbCommand();
command.Connection = connection;
string query = "select * from money where [Month]='January'";
command.CommandText = query;
OleDbDataReader reader = command.ExecuteReader();
// start the total at 0
int total = 0.0m;
while (reader.Read())
{
// loop through all the fields in the reader
for(int f = 0; f < reader.FieldCount; f++) {
// read as a decimal and add to the total
total += reader.GetDecimal(f);
}
}
connection.Close();
}
catch (Exception ex)
{
MessageBox.Show("Error" + ex);
}
Hope this helps -
decimal total = 0M;
while (dr.Read())
{
for (int i = 0; i < dr.FieldCount; i++)
{
total+= (decimal) (dr[i] != DBNull.Value ? dr[i] : 0.0);
}
}
this will add all the column's value for each row.
Datareader has property called field count which can give number of columns.. so you can use it something like below
double num=0.0m;
for (int i = 0; i < rdr.FieldCount; i++)
num += rdr[i];
The follow code will insert some values in my database. It gets 6 random values, puts the stuff in an array and then inserts it in the database.
public void LottoTest(object sender, EventArgs e)
{
Dictionary<int, int> numbers = new Dictionary<int, int>();
Random generator = new Random();
while (numbers.Count < 6)
{
numbers[generator.Next(1, 49)] = 1;
}
string[] lotto = numbers.Keys.OrderBy(n => n).Select(s => s.ToString()).ToArray();
foreach (String _str in lotto)
{
Response.Write(_str);
Response.Write(",");
}
var connectionstring = "Server=C;Database=lotto;User Id=lottoadmin;Password=password;";
using (var con = new SqlConnection(connectionstring)) // Create connection with automatic disposal
{
con.Open();
using (var tran = con.BeginTransaction()) // Open a transaction
{
// Create command with parameters (DO NOT PUT VALUES IN LINE!!!!!)
string sql =
"insert into CustomerSelections(val1,val2,val3,val4,val5,val6) values (#val1,#val2,#val3,#val4,#val5,#val6)";
var cmd = new SqlCommand(sql, con);
cmd.Parameters.AddWithValue("val1", lotto[0]);
cmd.Parameters.AddWithValue("val2", lotto[1]);
cmd.Parameters.AddWithValue("val3", lotto[2]);
cmd.Parameters.AddWithValue("val4", lotto[3]);
cmd.Parameters.AddWithValue("val5", lotto[4]);
cmd.Parameters.AddWithValue("val6", lotto[5]);
cmd.Transaction = tran;
cmd.ExecuteNonQuery(); // Insert Record
tran.Commit(); // commit transaction
Response.Write("<br />");
Response.Write("<br />");
Response.Write("Ticket has been registered!");
}
}
}
What is the best way to loop and insert MASS entries into the database. Lets say, 100,000 records via C#? I want to be able to generate the random numbers by my method and utilize the insert which i have too..
For true large scale inserts, SqlBulkCopy is your friend. The easy but inefficient way to do this is just to fill a DataTable with the data, and throw that at SqlBulkCopy, but it can be done twice as fast (trust me, I've timed it) by spoofing an IDataReader. I recently moved this code into FastMember for convenience, so you can just do something like:
class YourDataType {
public int val1 {get;set;}
public string val2 {get;set;}
... etc
public DateTime val6 {get;set;}
}
then create an iterator block (i.e. a non-buffered forwards only reader):
public IEnumerable<YourDataType> InventSomeData(int count) {
for(int i = 0 ; i < count ; i++) {
var obj = new YourDataType {
... initialize your random per row values here...
}
yield return obj;
}
}
then:
var data = InventSomeData(1000000);
using(var bcp = new SqlBulkCopy(connection))
using(var reader = ObjectReader.Create(data))
{ // note that you can be more selective with the column map
bcp.DestinationTableName = "CustomerSelections";
bcp.WriteToServer(reader);
}
You need Sql bulk insert. There is a nice tutorial on msdn http://blogs.msdn.com/b/nikhilsi/archive/2008/06/11/bulk-insert-into-sql-from-c-app.aspx
MSDN Table Value Parameters
Basically, you fill a datatable with the data you want to put into SqlServer.
DataTable tvp = new DataTable("LottoNumbers");
forach(var numberSet in numbers)
// add the data to the dataset
Then you pass the data through ADO using code similar to this...
command.Parameters.Add("#CustomerLottoNumbers", SqlDbType.Structured);
command.Parameters["CustomerLottoNumbers"].Value = tvp;
Then you could use sql similar to this...
INSERT CustomerSelections
SELECT * from #CustomerLottoNumbers
I've got a short piece of code that originally created an SqlDataAdapter object over and over.
Trying to streamline my calls a little bit, I replaced the SqlDataAdapter with an SqlCommand and moved the SqlConnection outside of the loop.
Now, whenever I try to edit rows of data returned to my DataTable, I get a ReadOnlyException thrown that was not thrown before.
NOTE: I have a custom function that retrieves the employee's full name based on their ID. For simplicity here, I used "John Doe" in my example code below to demonstrate my point.
ExampleQueryOld works with the SqlDataAdapter; ExampleQueryNew fails with the ReadOnlyException whenever I try to write to an element of the DataRow:
ExampleQueryOld
This works and has no issues:
public static DataTable ExampleQueryOld(string targetItem, string[] sqlQueryStrings) {
DataTable bigTable = new DataTable();
for (int i = 0; i < sqlQueryStrings.Length; i++) {
string sqlText = sqlQueryStrings[i];
DataTable data = new DataTable(targetItem);
using (SqlDataAdapter da = new SqlDataAdapter(sqlText, Global.Data.Connection)) {
try {
da.Fill(data);
} catch (Exception err) {
Global.LogError(_CODEFILE, err);
}
}
int rowCount = data.Rows.Count;
if (0 < rowCount) {
int index = data.Columns.IndexOf(GSTR.Employee);
for (int j = 0; j < rowCount; j++) {
DataRow row = data.Rows[j];
row[index] = "John Doe"; // This Version Works
}
bigTable.Merge(data);
}
}
return bigTable;
}
ExampleQueryNew
This example throws the ReadOnlyException:
public static DataTable ExampleQueryNew(string targetItem, string[] sqlQueryStrings) {
DataTable bigTable = new DataTable();
using (SqlConnection conn = Global.Data.Connection) {
for (int i = 0; i < sqlQueryStrings.Length; i++) {
string sqlText = sqlQueryStrings[i];
using (SqlCommand cmd = new SqlCommand(sqlText, conn)) {
DataTable data = new DataTable(targetItem);
try {
if (cmd.Connection.State == ConnectionState.Closed) {
cmd.Connection.Open();
}
using (SqlDataReader reader = cmd.ExecuteReader()) {
data.Load(reader);
}
} catch (Exception err) {
Global.LogError(_CODEFILE, err);
} finally {
if ((cmd.Connection.State & ConnectionState.Open) != 0) {
cmd.Connection.Close();
}
}
int rowCount = data.Rows.Count;
if (0 < rowCount) {
int index = data.Columns.IndexOf(GSTR.Employee);
for (int j = 0; j < rowCount; j++) {
DataRow row = data.Rows[j];
try {
// ReadOnlyException thrown below: "Column 'index' is read only."
row[index] = "John Doe";
} catch (ReadOnlyException roErr) {
Console.WriteLine(roErr.Message);
}
}
bigTable.Merge(data);
}
}
}
}
return bigTable;
}
Why can I write to the DataRow element in one case, but not in the other?
Is it because the SqlConnection is still open or is the SqlDataAdapter doing something behind the scene?
using DataAdapter.Fill does not load the database schema, which includes whether a column is a primary key or not, and whether a column is read-only or not. To load the database schema, use DataAdapter.FillSchema, but then that's not your questions.
using DataReader to fill a table loads the schema. So, the index column is read-only (probably because it's the primary key) and that information is loaded into the DataTable. Thereby preventing you from modifying the data in the table.
I think #k3b got it right; by setting ReadOnly = false, you should be able to write to the data table.
foreach (System.Data.DataColumn col in tab.Columns) col.ReadOnly = false;
I kept getting the same exception while trying different approaches. What finally worked for me was to set the column's ReadOnly property to false and change the value of the Expression column instead of row[index] = "new value";
In VB, don't pass a read-only DataRow Item by reference
The likelihood that you'll run into this is low, but I was working on some VB.NET code and got the ReadOnlyException.
I ran into this issue because the code was passing the DataRow Item to a Sub ByRef. Just the act of passing-byref triggers the exception.
Sub Main()
Dim dt As New DataTable()
dt.Columns.Add(New DataColumn With {
.ReadOnly = True,
.ColumnName = "Name",
.DataType = GetType(Integer)
})
dt.Rows.Add(4)
Try
DoNothing(dt.Rows(0).Item("Name"))
Console.WriteLine("All good")
Catch ex As Exception
Console.WriteLine(ex.Message)
End Try
End Sub
Sub DoNothing(ByRef item As Object)
End Sub
Output
Column 'Name' is read only
C-sharp
You can't even write code like this in C# . DoNothing(ref dt.Rows[0].Item["Name"]) gives you a compile time error.
open the yourdataset.xsd file of your data set. click on the table or object and click on the specific column which readonly property need to be changed.
its simple solutions.