I am using SqlBulkCopy to upload lots of data to an SQL table. It works very well apart from one thing (always the way).
So in my c# app, I have a function. It receives a variable myObj of type object (from Matlab). The object is actually an array.
I create a DataTable where I specify the column type & read in the data from myObj. One of the columns (let's call it salary) in the table is of type double.
The problem
If one of the rows has a NaN value for salary the upload won't work, it returns the message below.
OLE DB provider 'STREAM' for linked server '(null)' returned invalid data for column
What I need in the database is the row to be uploaded but with the Salary, column to have a value of null. Is there any way of doing this?
The only crude way I have come up with is testing for when the value is null ( or NaN) in my c# app and assigning it a value of -999 and then after the upload updating any values from -999 to null. However, this seems like a poor workaround.
Related
I have a field in a sqlite database, we'll call it field1, on which I'm trying to iterate over each record (there's over a thousand records). The field type is string. The value of field1 in the first four rows are as follows:
DEPARTMENT
09:40:24
PARAM
350297
Here is some simple code I use to iterate over each row and display the value:
while (sqlite_datareader.Read())
{
strVal = sqlite_datareader.GetString(0);
Console.WriteLine(strVal);
}
The first 3 values display correctly. However, when it gets to the numerical entry 350297 it errors out with the following exception on the .getString() method
An unhandled exception of type 'System.InvalidCastException' occurred in System.Data.SQLite.dll
I've tried casting to a string, and a bunch of other stuff. But I can't get to the bottom of why this is happening. For now, I'm forced to use getValue, which is of type object, then convert back to a string. But I'd like to figure out why getString() isn't working here.
Any ideas?
EDIT: Here's how I currently deal with the problem:
object objVal; // This is declared before the loop starts...
objVal = sqlite_datareader.IsDBNull(i) ? "" : sqlite_datareader.GetValue(i);
if (objVal != "")
{
strVal = (string)objVal;
}
What the question should have included is
The table schema, preferrably the CREATE TABLE statement used to define the table.
The SQL statement used in opening the sqlite_datareader.
Any time you're dealing with data type issues from a database, it is prudent to include such information. Otherwise there is much unnecessary guessing and floundering (as apparent in the comments), when so very useful, crucial information is explicitly defined in the schema DDL. The underlying query for getting the data is perhaps less critical, but it could very well be part of the issue if there are CAST statements and/or other expressions that might be affecting the returned types. If I were debugging the issue on my own system, these are the first thing I would have checked!
The comments contain good discussion, but a best solution will come with understanding how sqlite handles data types straight from the official docs. The key takeaway is that sqlite defines type affinities on a column and then stores actual values according to a limited set of storage classes. A type affinity is a type to which data will attempt to be converted before storing. But (from the docs) ...
The important idea here is that the type is recommended, not required. Any column can still store any type of data.
But now consider...
A column with TEXT affinity stores all data using storage classes NULL, TEXT or BLOB. If numerical data is inserted into a column with TEXT affinity it is converted into text form before being stored.
So even though values of any storage class can be stored in any column, the default behavior should have been to convert any numeric values, like 350297, as a string before storing the value... if the column was properly declared as a TEXT type.
But if you read carefully enough, you'll eventually come to the following at the end of section 3.1.1. Affinity Name Examples:
And the declared type of "STRING" has an affinity of NUMERIC, not TEXT.
So if the question details are taken literally and field1 was defined like field1 STRING, then technically it has NUMERIC affinity and so a value like 350297 would have been stored as an integer, not a string. And the behavior described in the question is precisely what one would expect when retrieving data into strictly-typed data model like System.Data.SQLite.
It is very easy to cuss at such an unintuitive design decisions and I won't defend the behavior, but
at least the results of "STRING" type are clearly stated so that the column can be redefined to TEXT in order to fix the problem, and
"STRING" is actually not a standard SQL data type. SQL strings are instead defined with TEXT, NTEXT, CHAR, NCHAR, VARCHAR, NVARCHAR, etc.
The solution is either to use code as currently implemented: Get all values as objects and then convert to string values... which should be universally possible with .Net objects since they should all have ToString() method defined.
Or, redefine the column to have TEXT affinity like
CREATE TABLE myTable (
...
field1 TEXT,
...
)
Exactly how to redefine an existing column filled with data is another question altogether. However, at least when doing the conversion from the original to the new column, remember to use a CAST(field1 AS TEXT) to ensure the storage class is changed for the existing data. (I'm not certain whether type affinity is "enforced" when simply copying/inserting data from an existing table into another or if the original storage class is preserved by default. That's why I suggest the cast to force it to a text value.)
in Oracle Database, I have got a table which contains as a number, varchar etc variables. in addition, it has sdo_geometry object column. the task I have wanted, I would like to fetch this sdo_geometry data type and assign inside appropriate data type using c#. I will be happy if you solve this issue.
Thanks for now
Best Regards...
I don't know if there's a .NET type you can map directly to that, but you can call Get_WKT to get the well known text.
SELECT ID, Name, myPolygon.Get_WKT() FROM YourSource
You can read that into a SqlGeometry object at run time if you need.
Environment: ASP.net, vb.net, SQL Server
I'm retrieving a dataset from a stored procedure. I need to iterate over every column, grab the column name, and the value of the associated row... I'm storing the column names and values in a custom class.
The problem is, this is an end user situation, therefore I have absolutely no idea what SQL query they will use, or stored procedure for that matter.
When testing, I first ran into an issue that I've seen before, and it was easy to deal with...
Consider the following:
For Each column As DataColumn In DynamicQuery.Tables(0).Columns
Dim columnName = column.ColumnName.ToString
Dim value = CType(row(column.Columnname), String)
' Other code...
Next column
My first problem was running into a NULL value. I solved that with:
CType(If(IsDBNull(row(column.ColumnName)), String.Empty, row(column.ColumnName)), String)
I should also add that I want every single value to be converted to a string...
Then I ran into another problem, and that was testing a table which had a uniqueidentifier datatype.
I received the error:
Error converting 'GUID' to type 'String'
After a bit of reading, I see it's suggested to cast it into a GUID struct and then convert it to a string.
This leads to my main question.
Is there a simpler way to do this? Or should I be using:
Select Case column.DataType
On every single column iteration? If GUID is the only problem, I can deal with that one... Just waiting for a bunch of gotcha's down the road where yet another datatype causes an issue.
I suppose I'm looking for the appropriate method of defensive programming here. Again, no matter what the rows are that are returned into the dataset, every single column is read, and it's value, and everything needs to be either a string with a length, or an empty string. Apologies if I'm missing something plain and simple. C# or vb.net examples welcomed with a smile.
I'm attempting to utilize the DataView.ToValue function, and while it is pulling the correct columns needed, I am still not getting the DISTINCT values. I have a DataValue that contains a gridview of all of the information on the website. When a client goes to download this data in a specific format, they need just the unique values to be processed into the file.
At present, I store all of the columns needed for the report within an array with the purpose of being dynamic. Once that string array has been created, I create my DataTable with
tblData = dvData.ToTable(true, arrColumns);
Despite this, the data is still coming back with all rows. Am I missing something? According to this from Microsoft's documentation I SHOULD be getting back distinct values.
Turns out I forgot that a step on the backend handles cleaning up NULLs into valid datapoints for the xml. So it was properly sorting, just the null fields were actually causing "copies".
When using OracleDataReader (Oracle.DataAccess.Client) in a c# program to read data from a recordset are there any situations where the data type in one of the fields could change from row to row ( except DBNull ).
I am thinking of situations like where the select uses a case statement that can return either a value from a column that is a number(9) or a value from a column that is defined as a number(14).
Are there any other gotcha's that might be relevant when up casting the value returned when using the indexer on the data reader e.g. int cityId = (int)dataReader["CityId"]; (except handling DBNull)
AFAIK, data type doesn't change on a row by row basis.
If the query uses a case statement that returns values from different data-types, the engine is responsible to pick the most appropriate to hold values for all, and it will describe the result-set using that data type, as the standard forces to use one data type per column in any result set.
Anyway, if you can't trust/know the source of the query, surround your cast in a try/catch sentence to get any unexpected result and act accordingly.
Each column in the result set will have the same data type and length/size/precision. JDBC connections
Using JDBC, these can actually be obtained from the result set metadata. Not sure whether there is an equivalent for C#
One oddity is a very special user defined type called SYS.ANYTYPE which is mostly used internally for replication. You can have three rows in a result set with one column. The data type of the column would be SYS.ANYTYPE but internally one could have a date value while another has a number and the last has a character. There are special packages for determining what is hidden inside any ANYTYPE value.