We Keep Coding

OLEDB Connection Truncates returned string

I am coming across an issue where using JObject.Parse(); truncates my JSON read from a excel file. It does not work on my machine, but the will work for the exact same data on another person machine using the same method and implementation.
Basically, we are calling a method part of a internal framework that reads an excel doc (data provider) based on the test method that is calling it. Once the row is selected, it then pulls the data stored in the columns cell. The format of this data is a JSON format. I have used 3 different JSON validators to ensure the JSON is valid.
JSON below this has just filler data as i cannot share the actual JSON
{
"columns": [
"column1",
"column2",
"column3",
"column4",
"column5",
"column6",
"column7",
"column8"
],
"data": []
}
when attempting to return the JSON as a JObject the below is done.
var data = JObject.Parse(MyObject.value.AColumn[0]);
Which returns the cell data as a JObject of the data in the cell specified in the above.
When debugging this, I have typed the JSON in the Excel cell and have gotten data to return, but at one point the data starts getting truncated as if there is a specific character limitation. But again, this works perfectly fine on someone else's machine.
I get this error because of the JSON being truncated:
Unterminated string. Expected delimiter: ". Path 'columns[10]' line 13, position 9.
We are using Newtonsoft to handle the JSON and Dapper for the connection.Query to execute a simple query against the xlsx spreadsheet.
What I am finding is that when executing the Query in the OLDB connection the returned string is maxing out only at 255 length. So this looks more like a Dapper / OLDBConnection issue where i need to set the max length higher.
Here is the code for that
// executing the query and retrieving the test data from the specific column
var query = string.Format("select * from [DataSet$] where TestName = '{0}'", testName);
var value = connection.Query(query).Select(x =>
{
var result = new MyObject{ TestName = x.testName };
foreach (var element in x)
{
if (element.Key.Contains(column))
{
result.CustomColumns.Add(element.Value.ToString());
}
}
return result;
}).FirstOrDefault();
Where x is a dynamic data type.
Has anyone come across this before? Is there some hidden character that is preventing this?

Answered in comments within the main question..
Issue was that the OLEDB connection was reading the fist 8 rows and determining the data type of subsequent rows.
The cells data being pulled was a JSON string. The OLEDB connection was reading the string, however when trying to Parse the string to a JObject, parsing was throwing exception.
Further debugging reviled that within the OLEDB connection when reading the row of data that the string was getting truncated at 255 characters. Formatting the columns did not fix then issue, nor did adding OLEDB settings when creating the connection.
What resolved this was updating the System Reg key used to determine how many Rows to read before determining data type/ length. This solution can be found in the comment section of the original Question.
Or here Data truncated after 255 bytes while using Microsoft.Ace.Oledb.12.0 provider

SqlDataReader and SQL Server 2016 FOR JSON splits json in chunks of 2k bytes

Recently I played around with the new for json auto feature of the Azure SQL database.
When I select a lot of records for example with this query:
Select
Wiki.WikiId
, Wiki.WikiText
, Wiki.Title
, Wiki.CreatedOn
, Tags.TagId
, Tags.TagText
, Tags.CreatedOn
From
Wiki
Left Join
(WikiTag
Inner Join
Tag as Tags on WikiTag.TagId = Tags.TagId) on Wiki.WikiId = WikiTag.WikiId
For Json Auto
and then do a select with the C# SqlDataReader:
var connectionString = ""; // connection string
var sql = ""; // query from above
var chunks = new List<string>();
using (var connection = new SqlConnection(connectionString))
using (var command = connection.CreateCommand()) {
command.CommandText = sql;
connection.Open();
var reader = command.ExecuteReader();
while (reader.Read()) {
chunks.Add(reader.GetString(0)); // Reads in chunks of ~2K Bytes
}
}
var json = string.Concat(chunks);
I get a lot of chunks of data.
Why do we have this limitation? Why don't we get everything in one big chunk?
When I read a nvarchar(max) column, I will get everything in one chunk.
Thanks for an explanation

From Format Query Results as JSON with FOR JSON:
Output of the FOR JSON clause
The result set contains a single column.
A small result set may contain a single row.
A large result set splits the long JSON string across multiple rows.
By default, SQL Server Management Studio (SSMS) concatenates the results into a single row when the output setting is Results to
Grid. The SSMS status bar displays the actual row count.
Other client applications may require code to recombine lengthy results into a single, valid JSON string by concatenating the
contents of multiple rows. For an example of this code in a C#
application, see Use FOR JSON output in a C# client app.
I would say it is strictly for performance reasons, similiar to XML. More SELECT FOR XML AUTO and return datatypes and What does server side FOR XML return?
In SQL Server 2000 the server side XML publishing - FOR XML (see http://msdn2.microsoft.com/en-us/library/ms178107(SQL.90).aspx) - was implemented in the layer of code between the query processor and the data transport layer. Without FOR XML a SELECT query is executed by the query processor and the resulting rowset is sent to the client side by the server side TDS code. When a SELECT statement contains FOR XML the query processor produces the result the same way as without FOR XML and then FOR XML code formats the rowset as XML. For maximum XML publishing performance FOR XML does steaming XML formatting of the resulting rowset and directly sends its output to the server side TDS code in small chunks without buffering whole XML in the server space. The chunk size is 2033 UCS-2 characters. Thus, XML larger than 2033 UCS-2 characters is sent to the client side in multiple rows each containing a chunk of the XML. SQL Server uses a predefined column name for this rowset with one column of type NTEXT - “XML_F52E2B61-18A1-11d1-B105-00805F49916B” – to indicate chunked XML rowset in UTF-16 encoding. This requires special handling of the XML chunk rowset by the APIs to expose it as a single XML instance on the client side. In ADO.Net, one needs to use ExecuteXmlReader, and in ADO/OLEDB one should use the ICommandStream interface.

As a workaround in the SQL code (i.e. if you don't want to change your querying code to put the chunks together), I found that wrapping the query in a CTE and then selecting form that gives me the results I expected:
--Note that I query from information_schema to just get a lot of data to replicate the problem.
--doing this query results in multiple rows (chunks) returned
SELECT * FROM information_schema.columns FOR JSON PATH, include_null_values
--doing this query results in a single row returned
;WITH SomeCTE(JsonDataColumn) AS
(
SELECT * FROM information_schema.columns FOR JSON PATH, INCLUDE_NULL_VALUES
)
SELECT JsonDataColumn FROM SomeCTE
The first query reproduces the problem for me (returns multiple rows, each a chunk of the total data), the second query gives one row with all the data. SSMS wasn't good for reproducing the issue, you have to try it out with other client code.

Strange Control Characters in C# DataTable results, not in SSMS or VB.NET results

I have a sql query, select distinct(name) from customers with (nolock) and it returns the text I want in SSMS, ie "Smith, John", etc.
However, when I get the string value from my DataTable in C#, I get back strange Control Characters at the beginning of my string, like \u001f\u001f\u001fSmith, John
Where is this coming from? Is it bad data in my database, or am I missing some steps related to character encoding or collation?
If anything culture or collation-related needs to be done, I'd prefer to do it from either from within the SQL query (without introducting a new SQL function) or from C#, since I can't control what values are placed in the database, I can only read from them.
UPDATE:
I have another VB.NET application which queries these names for a different purpose. This other program does NOT return the printing control characters in the DataTable. This leads me to believe there is something wrong with my SQLAdapter or SQLCommand implementation. Any ideas?
The table collation is SQL_Latin1_General_CP1_CI_AS.

MySql drivers for .NET doesn't support Polygon struct, does it?

I'm trying to fetch a Polygon data from MySQL into my C# application.
I have exactly defined in one table the Polygon field where geodata holds.
Proof:
SELECT GeometryType(GeomFromText(AsWKT(object))) as `type` FROM geo.data;
Returns:
So the object in the table is fine and correctly defined.
There is a source code in C#:
http://ideone.com/bn1urQ
And the main lines are (73-76):
//var polygon = (byte[])reader["object"];
//var obj = new MySqlGeometry(MySqlDbType.Blob, polygon);
var polygon = reader["object"].ToString();
var obj = MySqlGeometry.Parse(polygon);
I've commented, but this isn't an obstacle to tell you about both operations:
retrieving as BLOB with the next deserialization
parsing the fetched string via MySqlGeometry.Parse(System.String) method
BLOB retrieving
Well, I shall start with commented part of code, let's imagine, that these lines are uncommented and there aren't lines 75 and 76 with string parsing.
Also there is another correction, the SQL query which is sending to the MySQL server must look like:
SELECT AsWKB(object) as 'object' FROM geo.data
I've just changed in SQL query the function AsWKT() to the AsWKB() (from text to binary formatter at MySQL).
So, the result of this query would be:
At lines:
var polygon = (byte[])reader["object"];
var obj = new MySqlGeometry(MySqlDbType.Blob, polygon);
You are able to see that I'm fetching that BLOB object then converting it to the System.Byte[] array and only then I'm trying to build the MySqlGeomerty, but it's very pity and seems to be that MySQL libraries are identifying this object as a POINT, not a POLYGON.
Proof:
But!!! I have exactly the POLYGON object in MySQL, by the next SQL query:
SELECT GeometryType(GeomFromText(AsWKT(object))) as `type`,
AsWKT(object) as `data` FROM geo.data
Proof:
That was about the BLOB object.
Geometry parsing from System.String
Now... Let's imagine the original source with commented BLOB fetching.
Let's look at the lines:
var polygon = reader["object"].ToString();
var obj = MySqlGeometry.Parse(polygon);
And shall change the SQL query in the C# app source code to the:
SELECT AsWKT(object) as 'object' FROM geo.data
Yes... MySQL libraries for .NET are providing allegedly another style of geomerty bulding, from the System.String.
But, when I'm trying to parse the var polygon, which is retrieving correctly as you have seen above, I'm getting the next error:
System.FormatException: String does not contain a valid geometry value
Proof:
All these do look like, that MySQL libraries don't provide a full structures for the data binding from MySQL server.

I also have asked the similar question at the MySQL official forum, it's here:
http://forums.mysql.com/read.php?38,596620,596620
One man from Oracle team... Roberto Ezequiel Garcia Ballesteros has answered me that MySQL .NET driver doesn't support such a type of geometry currently:
http://forums.mysql.com/read.php?38,596620,596746#msg-596746
Hi Oleg,
I'm afraid we are supporting only Point and not Polygon. I'm sorry for
the inconvenience.
Cheers, Roberto

using c# datetime in mysql update query

I'm trying to run a query from C# to MySQL (version 5.5.27) using the mysql connector for .net from the MySQL website.
The ultimate goal of the .dll I'm developing is to keep track of the rows that I've read.
The query I use to read the database is:
string strSQL = "SELECT date,ask,bid,volume FROM gbpjpy where `read` = 0";
To read in the date, I have:
DateTime dateTime = mysqlReader.GetDateTime(0);
That works fine.
The table has 5 columns, the last column is called "read". Right after the select query and parsing is done, I execute the following query:
string sqlFormattedDate = dateTime.ToString("yyyy/MM/dd HH:mm:ss");
string query = "UPDATE gbpjpy SET `read` = 1 WHERE `date` = " + sqlFormattedDate;
However, when I check my log, I have the following error:
You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '01:20:08' at line 1.
The first date read in is 2012/08/30 01:20:08 (that's how it appears in the MySQL table, so that's where it gets the 01:20:08).
I've tried var sqlFormattedDate, changing the ToString overload to yyyy-MM-dd (using dashes and not forward slashes) and dateTime.ToString() all to no avail.
Why is MySQL not picking up the entire string?

Basically you should avoid including values in your query directly.
No doubt you could put quotes around the value... but you shouldn't. Instead, you should use paramterized SQL, and put the value in the parameter. That way you don't an error-prone string conversion, you avoid SQL injection attacks (for string parameters), and you separate code from data.
(As an example of how subtly-broken this can be, your current code will use the "current culture"'s date and time separators - which may not be / and :. You could fix this by specifying CultureInfo.InvariantCulture... but it's best not to do the conversion at all.)
Look for documentation of a Parameters property on whatever Command type you're using (e.g. MySqlCommand.Parameters) which will hopefully give you examples. There may even be a tutorial section in the documentation for parameterized SQL. For example, this page may be what you're after.

I suppose you have to put the whole value for the date in quotes. If you actually concatenate your query, it would look like
UPDATE gbpjpy SET `read` = 1 WHERE `date` = yyyy/MM/dd HH:mm:ss
That equal sign will only take the value until the first space.
Instead, it should look like
UPDATE gbpjpy SET `read` = 1 WHERE `date` = 'yyyy/MM/dd HH:mm:ss'
This is the particular reason in this case, however, concatenating queries like this leads to a real possibility of SQL injection. As a rule of thumb, you shouldn't do it. You can use parameterized queries and there's probably an API of the .NET connector you are using to do that.

Putting the info in a parameter allows the code to format as it needs. Likely, your original issue may have stemmed from using slashes instead of dashes in your date format. I would assume that slashes can work, but most all of the documentation I've seen has dashes separating dates with MySqlDateTimes.