Double precision error when getting data from SQL server - c#

I'm trying to get an real number from an SQL view.
But after executing the query it will round the number from 6728873.3 to 6728873.5
Why is this happening and is there any way around this?
I have tried getting the data with a DataReader, Linq2SQL and ServiceStack.ORMLite but none of them work.
var lConnection = new SqlConnection(ConnectionString);
const string cmd = "SELECT [Origin],[X],[Y] FROM [SpiderDB61].[dbo].[View_Addresses_AddressNames]" +
" where Origin = 5 and X = -26262.5 and Y = 6728873.3";
var lSQLCommand = new SqlCommand(cmd, lConnection);
lConnection.Open();
SqlDataReader lReader = lSQLCommand.ExecuteReader();
object a;
object b;
object c;
while (lReader.Read())
{
a = lReader[0];
b = lReader[1];
c = lReader[2]; // <-- this will be 6728873.5, not 6728873.3
}

real is float(24), which gives you approximately 7 digits of precision. 6728873.3 is 8 digits. Basically, you are at the limit of what is reliable / precise for real. You should consider a different data type; perhaps float (which defaults to float(53)), perhaps decimal(x,y) - depending on whether this value is meant to be discreet or continuous.

Related

How to get the average of a column in MySQL

I'm trying to find a way to calculate the average of a column (as a double/float or decimal). I've found the AVG function in MySQL documentation but I can't get it to work. Currently I've got the following code:
public double GetRating(string username)
{
double average;
MySqlConnection databaseConnection = new MySqlConnection(connectionString);
string sumQuery = "select sum('rating') from " + username;
string countQuery = "SELECT COUNT(*) FROM " + username;
using (MySqlCommand sumCommand = new MySqlCommand(sumQuery, databaseConnection))
using (MySqlCommand countCommand = new MySqlCommand(countQuery, databaseConnection))
try
{
databaseConnection.Open();
double sum = (Double)sumCommand.ExecuteScalar();
double columnLength = (Double)countCommand.ExecuteScalar();
average = sum / columnLength;
return average;
}
finally
{
databaseConnection.Close();
}
return average;
}
Now for some reason this does not work. It returns "Unable to cast object of type 'System.Int64' to type 'System.Double'."
The stored data in the database is an int but i'm trying to cast them to double. Any suggestions or solutions? Once again; double/float or decimal are usable for me.
The built-in AVG function (an aggregate function) could be used like so:
select avg(rating) from table_name
Note that, like most aggregate functions, the average will exclude null values (the average of 1, 2, null is 1.5 instead of 1.0). Also, in MySQL the return datatype will be decimal if you're averaging decimal or integer columns so use the appropriate C# datatype.
A couple of things:
What is the data type of the rating column in the table? If it is an integer and not floating-point, please change the data type of sum accordingly to avoid the type-cast error.
As you already know, you may use avg() instead of sum() and then division by count(). Your SQL will look like:
select avg(rating) from table_name

How to convert from decimal to double in Linq to Entity

Suppose we have table T which has two columns A and B with float and money types respectively. I want to write a linq query like following T-SQL statement:
Select A, B, A * B as C
From SomeTable
Where C < 1000
I tried to cast like following
var list = (from row in model.Table
where ((decimal)row.A) * row.B < 1000
select new { A = row.A,
B = row.B ,
C = ((decimal)row.A) * row.B}
).ToList();
but it does not allow the cast operation. It throw an exception:
Casting to Decimal is not supported in Linq to Entity queries, because
the required precision and scale information cannot be inferred.
My question is how to convert double to decimal in Linq? I don't want to fetch data from database.
Update:
I notice the converting decimal to double works but reverse operation throws the exception. So,
Why can't we convert double to decimal? Does Sql server do the same mechanism in t-sql too? Doesn't it affect precision?
The difference between a float (double) and a decimal, is that a float is decimal precise. If you give the float a value of 10.123, then internally it could have a value 10.1229999999999, which is very near to 10.123, but not exactly.
A decimal with a precision of x decimals will always be accurate until the x-th decimal.
The designer of your database thought that type A didn't need decimal accuracy (or he was just careless). It is not meaningful to give the result of a calculation more precision than the input parameters.
If you really need to convert your result into a decimal, calculate your formula as float / double, and cast to decimal after AsEnumerable:
(I'm not very familiar with your syntax, so I'll use the extension method syntax)
var list = model.Table.Where(row => row.A * row.B < 1000)
.Select(row => new
{
A = row.A,
B = row.B,
})
.AsEnumerable()
.Select(row => new
{
A = row.A,
B = row.B,
C = (decimal)row.A * (decimal)row.B,
});
Meaning:
From my Table, take only rows that have values such that row.A * row.B
< 1000.
From each selected row, select the values from columns A and B.
Transfer those two values to local memory (= AsEnumerable),
for every transferred row create a new object with three properties:
A and B have the transferred values.
C gets the the product of the decimal values of transferred A and B
You can avoid AsEnumerable() explaining to Entity how many fractional digits you want.
var list = (from row in model.Table
where ((decimal)row.A) * row.B < 1000
select new { A = row.A,
B = row.B ,
C = (((decimal)((int)row.A)*100))/100) * row.B}
).ToList();

Getting numeric precision in case of int datatype of SQL Server

I am fetching following details of columns from SQL Server:
Size
Precision
Scale
But I have noticed that in case of int I am getting 10 as precision but when i did some research, I couldn't find any such thing related to int, and that int datatype have precision of 10 or in SQL Server Management Studio.
So I don't know how 10 is coming as precision in case of int data type.
Table:
Screenshot:
Code:
String[] columnRestrictions = new String[4];
columnRestrictions[0] = 'MyDb';
columnRestrictions[1] = 'dbo';
columnRestrictions[2] = 'Employee';
using (SqlConnection con = new SqlConnection("MyConnectionString"))
{
con.Open();
var columns = con.GetSchema("Columns", columnRestrictions).AsEnumerable()
.Select
(
t => new
{
Name = t[3].ToString(),
Datatype = t.Field<string>("DATA_TYPE"),
IsNullable = t.Field<string>("is_nullable"),
Size = t.Field<Int32?>("character_maximum_length"),
NumericPrecision = t.Field<int?>("NUMERIC_PRECISION"),
NumericScale = t.Field<Int32?>("NUMERIC_SCALE")
}).ToList();
Precision refers to the number of significant decimal digits a number can represent.
The int datatype, like the int datatype in C#, can range from -2147483648 to 2147483647, so it can have up to 10 significant decimal digits.
The precision of int is therefore always 10.

Rounding a field's value in a SELECT statement

What I need to do is to round a field to 2 decimals, but not in the usual way. I have a dropdown that's always rounded to 2 decimals (CIT_NBR). However, in the database table, it's sometimes rounded to 1 decimal. So now I'm trying to create a SELECT statement based on this field, but my front end stores it as 2 decimals and my back end can be stored as either 1 or 2 decimals. Don't ask, it's complicated. :o)
So, what I want to do in "aircode" is something like:
SELECT * FROM VW_MOS_DPL_AccountValidation WHERE CUST_NUM = #CNum
AND Format(CIT_NBR, 2 decimals) = #CITNum
This way, it forces the data in the table to use 2 decimals, so it can be compared to my dropdown.
Here's my code block:
using (SqlConnection con2 = new SqlConnection(str2))
{
using (SqlCommand cmd2 = new SqlCommand(#"SELECT * FROM VW_MOS_DPL_AccountValidation WHERE CUST_NUM = #CNum AND CIT_NBR = #CITNum", con2))
{
con2.Open();
cmd2.Parameters.AddWithValue("#CNum", TBAccountNum.Text);
string ddlCITVal2 = ddlCIT.SelectedValue;
cmd2.Parameters.AddWithValue("#CITNum", ddlCITVal2);
using (SqlDataReader DT2 = cmd2.ExecuteReader())
{
// If the SQL returns any records, process the info
if (DT2.HasRows)
{
while (DT2.Read())
{
.
.
.
etc
How could I go about doing this?
Cast the varchar to a decimal
SELECT SUM(Cast(CitNum as decimal(8,2))) as CitNum FROM table
There is a more performant approach, but this is easiest to read and maintain unless it causes a real performance problem.
SELECT *
FROM VW_MOS_DPL_AccountValidation
WHERE CUST_NUM = #CNum
AND (CIT_NBR=#CITNUM OR CIT_NBR+'0'=#CITNUM)
Unless you really meant rounded to one decimal instead of the example you gave which is just truncating a trailing zero, in which case a different approach would need to be used.

Newbie C# Question about float/int/text type formatting

I'm a total C# newb with a light (first year CS) background in Python. I wrote a console program in Python for doing marathon pace running calculations and I'm trying to figure out the syntax for this in C# using Visual Studio 2010. Here's a chunk of what I've got so far:
string total_seconds = ((float.Parse(textBox_Hours.Text) * 60 * 60) + (float.Parse(textBox_Minutes.Text) * 60) + float.Parse(textBox_Seconds.Text)).ToString();
float secs_per_unit = ((float)(total_seconds) / (float)(textBox_Distance.Text));
float mins_per_unit = (secs_per_unit / 60);
string pace_mins = (int)mins_per_unit.ToString();
string pace_secs = (float.Parse(mins_per_unit) - int.Parse(mins_per_unit) * 60).ToString();
textBox_Final_Mins.Text = pace_mins;
textBox_Final_Secs.Text = pace_mins;
Imagine you have a running pace of 8 minutes and 30 seconds per mile. secs_per_unit would be 510, mins_per_unit would be 8.5. pace_mins would simply be 8 and pace_secs would be 30. In Python I'd just convert variables from a float to a string to get 8 instead of 8.5, for example; hopefully the rest of the code gives you an idea of what I've been doing.
Any input would be appreciated.
For float to string if you want to cut off the fraction
.ToString("F0")
It would be better if you rephrase your question.
Hours and minutes should only take integers as you're already taking seconds (doesn't make sense to have have 1.5 hours and 30 minutes instead of just 2 hours 0 minutes).
var numHours = Convert.ToInt32(textBox_Hours.Text);
var numMinutes = Convert.ToInt32(textBox_Minutes.Text);
var numSeconds = Convert.ToDouble(textBox_Seconds.Text);
var totalDistance = Convert.ToDouble(textBox_Distance.Text);
var totalSeconds = ((numHours)*60) + numMinutes)*60 + numSeconds;
var secsPerUnit = totalSeconds/totalDistance;
var minsPerUnit = secsPerUnit/60;
var paceMinsStr = Math.Floor(minsPerUnit).ToString();
var paceSeconds = minsPerUnit - Math.Floor(minsPerUnit);
var paceSecondsStr = (paceSeconds/ 100 * 60).ToString();
Written quickly, haven't tested it.. but something like this should work, at least with very minor tweaks/typo fixes.
Try this. Overall, store things as integers more, rather than storing as floats and converting to integers multiple times. And don't convert to a string until the last moment.
// I'm assuming that the text boxes aren't intended to hold a fraction,
// "8.5", for example. Therefore, use 'int' instead of 'float', and don't
// convert to a string at the end.
int total_seconds = int.Parse(textBox_Hours.Text) * 60 * 60 +
int.Parse(textBox_Minutes.Text) * 60 +
int.Parse(textBox_Seconds.Text);
// you missed a Parse here.
// Use two separate variables for seconds per unit:
// one for the total (510, in your example), one for just the seconds
// portion of the Minute:Second display (30).
int total_secs_per_unit = (int)(total_seconds / float.Parse(textBox_Distance.Text));
int mins_per_unit = total_secs_per_unit / 60;
int secs_per_unit = total_secs_per_unit % 60;
string pace_mins = mins_per_unit.ToString();
string pace_secs = secs_per_unit.ToString();
textBox_Final_Mins.Text = pace_mins;
textBox_Final_Secs.Text = pace_secs;
You could use the cast operators and conversion functions.
This would be a cast:
double d = 1.2;
int i = (int)d;
This would be a conversion:
string s = "1";
int i = Convert.ToInt32(s);

Categories