EDIT: I am now strongly suspecting this behavior is due to a bug in the OleDB.Oracle provider. Upon other testing, I was able to perform Select statements against other CAST column values with negative scale that did not cause the 'Decimal byte constructor...' exception. I also note that the provider is returning the absolute value of the scale when viewing the schema, eg scale of -2 is returned as 2. Additionally, this same test query does not cause an exception when run through the ODP.NET driver (rather than the Oracle OLEDB provider). Changing the numeric delimiter as suggested by Lalit (in comments) did not affect the results (but I thank him for his time nonetheless). I continue to research this problem and will advise if more information is realized.
I have a 64-bit C# application that fetches data from an Oracle database via the Oracle 11g OLEDB provider. When Oracle returns a numeric type defined or cast with negative scale (such as 'Select Cast(123.1 as Number(3,-1))', the mapped OleDB schema (from GetSchemaTable) is reporting that column as a Decimal with a scale of 255. The documentation indicates 255 is intended to represent an N/A or irrelevant value.
When OleDBDataReader.GetValues() is later called on the row containing such a column, an ArgumentException is thrown, advising that a 'Decimal byte array constructor...requires four valid decimal bytes," telling me that even though the OleDB Provider thinks its Decimal data, there's no valid Decimal data to read. I'm making an assumption that data is present, but not sure exactly what.
I have tried:
Explicitly getting the bytes from the column via calls to OleDbDataReader.GetBytes (even a call to size a buffer excepts), but doing so throws "Specified cast is not valid" ArgumentExceptions.
Written a chunk of test code to get every possible supported return data type, eg GetInt16, GetInt32, etc., etc., and each throws the same exception (invalid cast).
Does the Oracle OleDB provider not even return data to the caller when fetching a column defined with a negative scale? Is there some other mechanism to at least get the bytes "across the pond" and manipulate them on the receiving end?
Related
I have run into an issue where our decimal columns which have a column type of decimal(18,7) are causing the Sum operation to be evaluated locally.
From what I've been able to find this is because SQlite's numeric datatype's. I thought that was just an example, but after changing our decimals to be decimal(10,5) instead of decimal(18,7) the sum operation is no longer run locally. This confuses me because from what I've understood the datatypes/affinities don't actually effect how the data is stored.
The fact that decimal(10,5) works seems to also contradict the following statement they make.
Note that numeric arguments in parentheses that following the type name (ex: "VARCHAR(255)") are ignored by SQLite
We are using SQlite to store a copy of the data locally from a SQL Server Db, and the SQL Server uses decimal(18,7) so we can't change that for SQlite.
Is there any way to change this so we can use the linq Sum operation our decimal columns?
EDIT:
I found that if I cast the decimal to a double inside the Sum, it will work fine. Seems like it doesn't want to convert it to SQL because it will lose precision. At the moment I am calling ToList to pull the records in locally before doing the Sum, but this seems extremely wasteful
Does this mean that our decimal columns are actually stored as floating points in the database?
Overview
This question is a more specific version of this one:
sql server - performance hit when passing argument of C# type Int64 into T-SQL bigint stored procedure parameter
But I've noticed the same performance hit for other data types (and, in fact, in my case I'm not using any bigint types at all).
Here are some other questions that seem like they should cover the answer to this question, but I'm observing the opposite of what they indicate:
c# - When should "SqlDbType" and "size" be used when adding SqlCommand Parameters? - Stack Overflow
.net - What's the best method to pass parameters to SQLCommand? - Stack Overflow
Context
I've got some C# code for inserting data into a table. The code is itself data-driven in that some other data specifies the target table into which the data should be inserted. So, tho I could use dynamic SQL in a stored procedure, I've opted to generate dynamic SQL in my C# application.
The command text is always the same for row I insert so I generate it once, before inserting any rows. The command text is of the form:
INSERT SomeSchema.TargetTable ( Column1, Column2, Column3, ... )
VALUES ( SomeConstant, #p0, #p1, ... );
For each insert, I create an array of SqlParameter objects.
For the 'nvarchar' behavior, I'm just using the SqlParameter(string parameterName, object value) constructor method, and not setting any other properties explicitly.
For the 'degenerate' behavior, I was using the SqlParameter(string parameterName, SqlDbType dbType) constructor method and also setting the Size, Precision, and Scale properties as appropriate.
For both versions of the code, the value either passed to the constructor method or separately assigned to the Value property has a type of object.
The 'nvarchar' version of the code takes about 1-1.5 minutes. The 'degenerate' or 'type-specific' code takes longer than 9 minutes; so 6-9 times slower.
SQL Server Profiler doesn't reveal any obvious culprits. The type-specific code is generating what would seem like better SQL, i.e. a dynamic SQL command whose parameters contain the appropriate data type and type info.
Hypothesis
I suspect that, because I'm passing an object type value as the parameter value, the ADO.NET SQL Server client code is casting, converting, or otherwise validating the value before generating and sending the command to SQL Server. I'm surprised tho that the conversion from nvarchar to each of the relevant target table column types that SQL Server must be performing is so much faster than whatever the client code is doing.
Notes
I'm aware that SqlBulkCopy is probably the best-performing option for inserting large numbers of rows but I'm more curious why the 'nvarchar' case out-performs the 'type-specific' case, and my current code is fast enough as-is given the amount of data it routinely handles.
The answer does depend on the database you are running, but it has to do with the character encoding process. SQL Server introduced the NVarChar and NText field types to handle UTF encoded data. UTF also happens to be the internal string representation for the .NET CLR. NVarChar and NText don't have to be converted to another character encoding, which takes a very short but measurable amount of time.
Other databases allow you to define character encoding at the database level, and others let you define it on a column by column basis. The performance differences really depend on the driver.
Also important to note:
Inserting using a prepared statement emphasizes inefficiencies in converting to the database's internal format
This has no bearing on how efficient the database queries against a string--UTF-16 takes up more space than the default Windows-1252 encoding for Text and VarChar.
Of course, in a global application, UTF support is necessary
They're Not (but They're Almost as Fast)
My original discrepancy was entirely my fault. The way I was creating the SqlParameter objects for the 'degenerate' or 'type-specific' version of the code used an extra loop than the 'nvarchar' version of the code. Once I rewrote the type-specific code to use the same number of loops (one), the performance is almost the same. [About 1–2% slower now instead of 500-800% slower.]
A slightly modified version of the type-specific code is now a little faster; at least based on my (limited) testing – about 3-4% faster for ~37,000 command executions.
But it's still (a little) surprising that it's not even faster, as I'd expect SQL Server converting hundreds of nvarchar values to lots of other data types (for every execution) to be significantly slower than the C# code to add type info to the parameter objects. I'm guessing it's really hard to observe much difference because the time for SQL Server to convert the parameter values is fairly small relative to the time for all of the other code (including the SQL client code communicating with SQL Server).
One lesson I hope to remember is that it's very important to compare like with like.
Another seeming lesson is that SQL Server is pretty fast at converting text to its various other data types.
I've an Oracle table with a column of type NUMBER(38,0)
I need to fetch this column to my C# application.
I use the library System.Data.Odbc.OdbcDataReader to read the data from my Oracle table.
I've tried fetching the data by using the normal functions like:
var data = oracleReader["COLUMN"].ToString();
And
var data = oracleReader.GetString(0);
And even the oracleReader.GetBytes() function.
But I always get System.OverflowException, because the OdbcDataReader always try to get the column as decimal in the last step:
System.OverflowException: Value was either too large or too small for a Decimal.
at System.Data.Odbc.OdbcDataReader.internalGetDecimal(Int32 i)
at System.Data.Odbc.OdbcDataReader.GetValue(Int32 i, TypeMap typemap)
at System.Data.Odbc.OdbcDataReader.GetValue(Int32 i)
at System.Data.Odbc.DbCache.AccessIndex(Int32 i)
at System.Data.Odbc.OdbcDataReader.internalGetString(Int32 i)
at System.Data.Odbc.OdbcDataReader.GetString(Int32 i)
I'm happy if I can get this data as a String to my Application.
FYI, I can not change the datatype of the column, I need to work with this.
This data type is an alias for the NUMBER(38) data type, and is designed so that the OracleDataReader returns a System.Decimal or OracleNumber instead of an integer value. Using the .NET Framework data type can cause an overflow.
Come to think of it you actually need BigInteger to be able to represent the same number of significant digits as to what NUMBER defaults to. I've never seen anyone do that and I would suppose it's a very rare need. Also BigInteger still wouldn't cut it since NUMBER can be of positive and negative infinity.
You can use this list for future research.
I understand that collation can be set differently in different tables in a database. Collation is understood from What does character set and collation mean exactly?
There is a query that performs CAST from a char results as shown below. There are no tables involved. I guess, the encoding applied will be based on the collation in database level. Is this assumption correct?
SELECT CAST ( SSS.id_encrypt ('E','0000000{0}') AS CHAR(100) FOR BIT DATA)
AS ENCRYPT_ID FROM FFGLOBAL.ONE_ROW FETCH FIRST 1 ROW ONLY
QUESTION
In the question Get Byte[] from Db2 without Encoding answer given by #AlexFilipovici [.Net BlockCopy ] provides a different result when compared to CAST result. Why is it so if there is no codepage associated?
Based on National language support - Character conversion
Bit data (columns defined as FOR BIT DATA, or BLOBs, or binary strings) is not associated with any character set.
REFERENCE
Get Byte[] from Db2 without Encoding
Default code page for new databases is Unicode
National language support - Character conversion
To find out the collation at database level in SQL Server, try this:
SELECT DATABASEPROPERTYEX('databasename', 'Collation');
More: DATABASEPROPERTYEX
To answer your questions:
#1: Specifying FOR BIT DATA on a character-based data type (in DB2) means that DB2 stores / returns the raw data back with no codepage associated (i.e. it's just a string of bytes and will not go through any codepage conversion between client and server).
#2: In DB2 for Linux, UNIX and Windows, you can determine the database's collation by querying SYSIBMADM.DBCFG
select name,value
from sysibmadm.dbcfg
where name in ('codepage','codeset');
#3: Per #Iswanto San:
SELECT DATABASEPROPERTYEX('databasename', 'Collation');
I need to do some in-memory merging in C# of two sorted streams of strings coming from one or more SQL Server 2000 databases into a single sorted stream. These streams of data can be huge, so I don't want to pull both streams into memory. Instead, I need to keep one item at a time from each stream in memory and at each step, compare the current item from each stream, push the minimum onto the final stream, and pull the next item from the appropriate source stream. To do this correctly, though, the in-memory comparison has to match the collation of the database (consider the streams [A,B,C] and [A,B,C]: the correct merged sequence is [A,A,B,B,C,C], but if your in-memory comparison thinks C < B, your in-memory merge will yield A,A,B, at which point it will be looking at a B and a C, and will yield the C, resulting in an incorrectly sorted stream.)
So, my question is: is there any way to mimic any of the collations in SQL Server 2000 with a System.StringComparison enum in C# or vise-versa? The closest I've come is to use System.StringCompaison.Ordinal with the results of the database strings converted to VARBINARY with the standard VARBINARY ordering, which works, but I'd rather just add an "order by name collate X" clause to my SQL queries, where X is some collation that works exactly like the VARBINARY ordering, rather than converting all strings to VARBINARY as they leave the database and then back to strings as they come in memory.
Have a look at the StringComparer class. This provides for more robust character and string comparisons than you'll find with String.Compare. There are three sets of static instances (CurrentCulture, InvariantCulture, Ordinal) and case-insesitive versions of each. For more specialized cultures, you can use the StringComparer.Create() function to create a comparer tied to a particular culture.
With sql 2005 I know that the db engine does not make OS calls to do the sorting, the ordering rules are statically shipped with the db (may update with a service pack, but doesn't change with the OS). So I don't think you can safely say that a given set of application code can order the same way unless you have the same code as the db server, unless you use a binary collation.
But if you use a binary collation in the db and client code you should have no problem at all.
EDIT - any collation that ends in _BIN will give you binary sorting. The rest of the collation name will determine what code page is used for storing CHAR data, but will not affect the ordering. The _BIN means strictly binary sorting. See http://msdn.microsoft.com/en-us/library/ms143515(SQL.90).aspx