We Keep Coding

SQLiteDataReader OverflowException using SQLiteDataReader on decimal.MaxValue

Problem
This is partially me being my own worst enemy. I have unit tests that verify my ability to write and then retrieve all different base data types to/from a SQLite database. Among my tests I verify several different values for each data type including (but not limited to) <datatype>.MinValue and <datatype>.MaxValue.
When I write a decimal.MaxValue to the database, then try to retrieve it, I get an Overflow Exception (thanks to rounding within the database itself).
Note: I have stripped my actual classes to the bare-bones and placed them inside a test method so I could show everything more easily.
private static SQLiteConnection connection;
[TestMethod()]
public void WriteDecimal()
{
using (var cmd = new SQLiteCommand(connection))
{
cmd.CommandText = $"INSERT INTO foo(name, value) VALUES('bar', {decimal.MaxValue})";
cmd.ExecuteNonQuery();
cmd.CommandText = "SELECT * FROM foo;";
using (SQLiteDataReader rdr = cmd.ExecuteReader())
{
while (rdr.Read())
{
Console.WriteLine($"{rdr.GetInt32(0)} {rdr.GetString(1)} {rdr.GetValue(2)}");
}
}
}
}
#region Setup/Cleanup
[ClassInitialize()]
public static void Setup(TestContext context)
{
FileInfo dbFile = new FileInfo(Path.Combine(Environment.GetEnvironmentVariable("temp"), #"\Sqlite\myDb.db"));
dbFile.Directory.Create();
dbFile.Delete();
string connectionString = $"Data Source={dbFile?.FullName ?? ":memory:"}";
connection = new SQLiteConnection(connectionString);
connection.Open();
using (var cmd = new SQLiteCommand(connection))
{
cmd.CommandText = #"CREATE TABLE foo(id INTEGER PRIMARY KEY, name TEXT, value Number)";
cmd.ExecuteNonQuery();
};
}
[ClassCleanup()]
public static void Cleanup()
{
connection.Close();
}
#endregion
Output:
Message:
Test method WriteDecimal threw exception:
System.OverflowException: Value was either too large or too small for a Decimal.
Stack Trace:
Number.ThrowOverflowException(TypeCode type)
DecCalc.VarDecFromR8(Double input, DecCalc& result)
IConvertible.ToDecimal(IFormatProvider provider)
Convert.ChangeType(Object value, Type conversionType, IFormatProvider provider)
SQLite3.GetValue(SQLiteStatement stmt, SQLiteConnectionFlags flags, Int32 index, SQLiteType typ)
SQLiteDataReader.GetValue(Int32 i)
DatabaseDirect.WriteDecimal() line 54
Workaround
I found a workaround (I just don't like it). Essentially, I let it fail, then go back and try to grab it as a Double; then convert it to what I need; because it overflowed I know it has to either be the max value or the min value:
using (SQLiteDataReader rdr = cmd.ExecuteReader())
{
while (rdr.Read())
{
decimal newVal;
try
{
newVal = (decimal)rdr.GetValue(2);
}
catch (OverflowException)
{
double val = rdr.GetDouble(2);
Type t = rdr.GetFieldType(2);
newVal = val > 0 ? decimal.MaxValue : decimal.MinValue;
}
Console.WriteLine($"{rdr.GetInt32(0)} {rdr.GetString(1)} {newVal}");
}
}
Bigger Issue (as I see it)
This isn't the only place I encounter this issue. It also happens with decimal.MinValue and ulong.MaxValue. I'm not exactly a fan of my solution simply because I just assume that if there's an overflow I need the max/min value. I'd also like to generalize it so it doesn't hard-code the min/max values I may need. Again, I found a solution; but again, it is ugly (a function that passes in the type to convert the value to and then do a switch on it...yucky).

You might not be able to do this but when I got a decimal overflow error, I tried changing my sqlite column type from decimal to real and that eliminated the error.

SQL query result to Array

I have written the following in an attempt to execute a sql query and store the result in an Array:
public static ArrayList DbQueryToArry()
{
string SqlCString = myConnString;
SqlConnection connection = null;
ArrayList valuesList = new ArrayList();
connection = new SqlConnection(SqlCString);
connection.Open();
SqlCommand command = new SqlCommand("Select CLIENTNO, ACCOUNT_Purpose from audit.ACCOUNTS_AUDIT", connection);
SqlDataReader reader = command.ExecuteReader();
while (reader.Read())
{
valuesList.Add(Convert.ToInt32(reader[0].ToString()));
}
return valuesList;
}
But running the following
var myArray = DbQueryToArry();
Console.WriteLine(myArray.ToString());
Does not return the query result..

You will need to join them manually with string.Join or something similar:
Concatenates the elements of a specified array or the members of a
collection, using the specified separator between each element or
member.
Console.WriteLine(string.Join(",",myArray.ToArray()));
The reason why your version doesnt work is because, Console.Writeline has a bunch of overloads for different types, however it falls back to WriteLine(Object) when it can't find a specific resolution match.
The source code to WriteLine(Object value) is as follows (which can be found here).
public virtual void WriteLine(Object value) {
if (value==null) {
WriteLine();
}
else {
// Call WriteLine(value.ToString), not Write(Object), WriteLine().
// This makes calls to WriteLine(Object) atomic.
IFormattable f = value as IFormattable;
if (f != null)
WriteLine(f.ToString(null, FormatProvider));
else
WriteLine(value.ToString());
}
}
Notice how it calls value.ToString() ?
Object.ToString Method
Returns a string that represents the current object.
Remarks
Object.ToString is the major formatting method in the .NET Framework.
It converts an object to its string representation so that it is
suitable for display.
An ArrayList has no overload to ToString() that would be able to anticipate what you want to show, so it relies on the default.
The default Object.ToString() method
The default implementation of the ToString method returns the fully
qualified name of the type of the Object
Which brings me to my next point, don't use ArrayList, Use a generic array int[] or List<int> you will find it much more fun and rewarding (fun level 100)

Get Integer as a string so I can parse it to integer?

When I try to get an Integer from my SQLite db I can only get it working by reading it as a string and then run int.Parse on it.
Is this right, I read something about this having to do with ExeculeScalar possibly giving back null?
Here is my current code SendSQLExecScalar() sends the command string etc. and return an object
public object SendSQLExecScalar(string C)
{
OpenConnection();
SQLiteCommand SQLCommand = new SQLiteCommand(C, DbConnection);
try
{
object Output = SQLCommand.ExecuteScalar();
CloseConnection();
return Output;
}
catch (Exception X)
{
MessageBox.Show(X.Message);
return null;
}
}
And:
int ID = int.Parse(SendSQLExecScalar(C).ToString());
EDIT :
Specified cast is not valid.
public static int GetImageID(string Path)
{
string C = "SELECT ID FROM Images WHERE Path LIKE '" + Path + "' LIMIT 1";
return ConvertFromDBVal<int>(SendSQLExecScalar(C));
}
public static T ConvertFromDBVal<T>(object obj)
{
if (obj == null || obj == DBNull.Value)
{
return default(T);
}
else
{
return (T)obj; //breaks here saying this cast is invalid
}
}

I read something about this having to do with execuleScalar possibly
giving back null?
Yes, if there is no data that your sql query returns, ExecuteScalar returns null reference.
If you are 100% sure the return value on the first column of the first row is already int, you can just cast it like;
int ID = (int)SendSQLExecScalar(C);
To prevent null cases on this method, I almost always uses rein's generic method as;
public static T ConvertFromDBVal<T>(object obj)
{
if (obj == null || obj == DBNull.Value)
{
return default(T); // returns the default value for the type
}
else
{
return (T)obj;
}
}

Use TryParse instead of Parse, this allows you to test whether something is parseable.
If you use int.Parse() with an invalid int, you'll get an exception while in the TryParse, it returns a boolean letting you know whether the parse succeeded or not.
In short use Parse if you are sure the value will be valid; otherwise use TryParse.
int number = int.Parse(someString);
int number;
int.TryParse(someString, out number);

(int)(long)SendSQLExecScalar(C);
Solved with this, looks Like SQLite integer will return a long object which I needed to unpack before casting it to an int.

"Object cannot be cast from DBNull to other types"

When my website gets to the following bit of code, it falls down with an exception as follows:
System.InvalidCastException: Object cannot be cast from DBNull to other types.
For the interests of brevity, I'm showing only the relevant code (it's a 4000+ LOC file I've been given).
if (dr["STAGE"] is DBNull)
{
dto.Stage = 1; // This is the line throwing the exception, according to stack trace
}
else
{
dto.Stage = Convert.ToInt32(dr["STAGE"]);
}
Here, dr is a DataRow object that is the result of a query to a database, dto is a basic class that just holds some properties, of which dto.Stage is an int member.
I've looked at other questions with the same error message, but most of them seem to suggest "Check if it's DBNull", which I'm already doing.
So can someone suggest a solution?

Use == instead of is
if (dr["STAGE"] == DBNull.Value)
{
}

Use this slightly more efficient approach
int stageOrdinal = dr.GetOrdinal("STAGE");
while (dr.Read()) {
dto = new DataTransferObject();
if (dr.IsDBNull(stageOrdinal)) {
dto.Stage = 1;
} else {
dto.Stage = dr.GetInt32(stageOrdinal);
}
//TODO: retrieve other columns.
dtoList.Add(dto);
}
Accessing the columns by their index is faster than accessing them by name. The index of a column can be retrieved with the GetOrdinal method of the DataReader. This is best done before the loop.

Use the System.Data.DataRow.IsNull function instead.
if(dr.IsNull("Stage"))
{
...
}

Below is an example of a nullable data type which you can use to avoid DBNull errors.
The example below is not a genuine solution for the issue you have, but is an example of how you can go about solving it. Think of it as learning to fish, instead of being given a fish.
I pulled this from http://msdn.microsoft.com/en-us/library/1t3y8s4s.aspx
class NullableExample
{
static void Main()
{
int? num = null;
if (num.HasValue == true)
{
System.Console.WriteLine("num = " + num.Value);
}
else
{
System.Console.WriteLine("num = Null");
}
// y is set to zero
int y = num.GetValueOrDefault();
// num.Value throws an InvalidOperationException if num.HasValue is false
try
{
y = num.Value;
}
catch (System.InvalidOperationException e)
{
System.Console.WriteLine(e.Message);
}
}
}

#MarcGravell had the right of it in his comment:
Ignore the exact line number in the stack-trace; the numbers can be
slightly off - I see that all the time. Run it in a debugger with
break-points instead, or just add extra logging while you nail it down
#Sergey's answer is just plain wrong.

Explicit Int Conversion Overflow

Problem
When I pass the value to an int variable where the value exceed the maximum value for int, the value of the int variable become 0.
Background
I use the following steps to retrieve my data. I do not use any try-catch block without throwing an exception.
Step 1
In my WCF Service, I retrieve a DataTable using IBM.Data.DB2.iSeries.iDB2DataAdapter.Fill(DataSet)
Step 2
Then I convert the DataTable to List<T> using the code :
public static List<T> DataTableToList<T>(DataTable dt)
{
List<T> tableEntity = new List<T>();
foreach (DataRow row in dt.Rows)
{
T rowEntity = Activator.CreateInstance<T>();
rowEntity.GetType().GetProperties().Where(o => dt.Columns.OfType<DataColumn>()
.Select(p => p.ColumnName).Contains(o.Name)).ToList()
.ForEach(o => o.SetValue(rowEntity, row[o.Name], null));
tableEntity.Add(rowEntity);
}
return tableEntity;
}
with the type :
public class Client
{
public int ID { get; set; }
public string Name { get; set; }
}
Step 3
I return it using the WCF service method :
[OperationContract]
public string GetClients()
{
List<Client> clients = new DB().RetrieveClients();
return Tools.SerializeObjectToXML<List<Client>>(clients);
}
with the helper method for serializing :
public static string SerializeObjectToXML<T>(T item)
{
XmlSerializer xs = new XmlSerializer(typeof(T));
using (StringWriter writer = new StringWriter())
{
xs.Serialize(writer, item);
return writer.ToString();
}
}
Step 4
Then in the Client Application, I retrieve it from the Service Reference with the default binding of WSHttpBinding with the code :
List<Client> clients = Tools.DeserializeXMLToObject<List<Client>>(new SomeServiceClient().GetClients());
with the helper method for deserializing :
public static T DeserializeXMLToObject<T>(string xmlText)
{
if (string.IsNullOrEmpty(xmlText)) return default(T);
XmlSerializer xs = new XmlSerializer(typeof(T));
using (MemoryStream memoryStream = new MemoryStream(new UnicodeEncoding().GetBytes(xmlText)))
using (XmlTextReader xsText = new XmlTextReader(memoryStream))
{
xsText.Normalization = true;
return (T)xs.Deserialize(xsText);
}
}
Question
The msdn says that :
When you convert from a double or float value to an integral type, the value is truncated. If the resulting integral value is outside the range of the destination value, the result depends on the overflow checking context. In a checked context, an OverflowException is thrown, while in an unchecked context, the result is an unspecified value of the destination type.
Why do the value of Client.ID becomes 0 when the value from the database exceeds the maximum allowed for int?
Is it because it is an unchecked context? If it is, how would I know?
My code is in C#, framework 4, build in VS2010 Pro.
Please help, thanks in advance.

There is something fishy going on here. SetValue doesn't do cast/conversions (not even from uint to int, just tested). You would be much better enlarging the ForEach code in a real foreach and debugging it.
For example:
foreach (var o in rowEntity.GetType().GetProperties().Where(o => dt.Columns.OfType<DataColumn>()
.Select(p => p.ColumnName).Contains(o.Name)))
{
if (o.Name == "ID")
{
// Put a breakpoint here
}
o.SetValue(rowEntity, row[o.Name], null));
}

By default, arithmetic overflow compiler switch is turned off in Visual Studio. To enable it:
Right-click on the project and click Properties.
Click on the Build tab.
Click on the Advanced… button.
Check the Check for arithmetic overflow/underflow checkbox.

I do not know exactly where you can find if you are checked or unchecked. But it depends on your compilers options as stated in http://msdn.microsoft.com/en-us/library/khy08726(v=vs.100).aspx
And as Thorsten Dittmar said, you can better use the DataContract to easily send the clients.
And if it overflows you can better use a long for the id instead of an int. That way it has way more positions (long == Int64). And if all of your Id's are > 0 you can use a unsigned int or unsigned long as your Id. Unsigned means that instead of also having numbers lower then zero it is only positive and therefor twice the positive positions.
Int32
Uint32
Int64 (long)
Uint64 (ulong)

Just a question: Why do you transmit the list as a string instead of doing the following:
[OperationContract]
public Client[] GetClients()
{
List<Client> clients = new DB().RetrieveClients();
return clients.ToArray();
}
and declare the Client class like:
[DataContract]
public class Client
{
[DataMember]
public int ID;
[DataMember]
public string Name;
}
Also: If the type of ID in the database allows for values larger than Int32.MaxValue, why would you use an Int32 at all and not an Int64?
As for overflow checking: I know that using Convert.ToInt32(Int64.MaxValue); throws an exception while int i = (int)Int64.MaxValue does not.