We Keep Coding

Best way to record the following data

I am creating a SQL Server database that records vehicle movements, cleans, repairs etc.
I am struggling with how to record the following data.
I need to record each time the vehicle is cleaned. I also need to extract the amount of time that vehicle is cleaned over the past 28 days and make an average
So far I have a master table with the following columns:
Callsign, Registration, Location, Date Last Cleaned.
Then, as a work around I have "vehicle specific" tables which is where I record it each time it is cleaned.
however as I am working the ASP.net, the only way I can find to access the clean dates is by doing a foreach loop through each of the vehicle tables and returning the count of dates in the past 28 days. the problems with this I cannot work out how do this in 1 data connection instead of multiple requests to the server.
Here is a snippet of the code, but you can see that it runs through each of the check box list, and if "true" then will add to the vehicle specific table
string constr = myconstring;
SqlConnection cnn = new SqlConnection(constr);
SqlTransaction transaction;
for (int i = 0; i < checkboxlistAM.Items.Count; i++)
{
string callsignString = checkboxlistAM.Items[i].Text;
if (checkboxlistAM.Items[i].Selected == true)
{
string declare = "declare #morning datetime declare #date datetime declare #counting int ";
string setting = " set #date = DATEADD(d, DATEDIFF(d, 0, getdate()), 0) set #morning = dateadd(hh, 7, #date) set #counting = (select count([made ready]) from["+callsignString+"] where[Made Ready] = #morning) ";
string insertmorning = " if #counting <>1 insert into ["+callsignString+"] ([made ready]) values (#morning) ";
string QueryVehicleSpecificTable = declare + setting + insertmorning;
string QueryMasterTable = "update Shropshire SET[last made Ready AM] = GETDATE() where Callsign = '"+callsignString+"'";
cnn.Open();
transaction = cnn.BeginTransaction();
SqlCommand cmd1 = new SqlCommand(QueryVehicleSpecificTable, cnn);
cmd1.CommandType = CommandType.Text;
SqlCommand cmd2 = new SqlCommand(QueryMasterTable, cnn);
transaction.Commit();
cmd1.ExecuteNonQuery();
cmd2.ExecuteNonQuery();
cnn.Close();
}
else if (checkboxlistAM.Items[i].Selected == false)
{
string declare = "declare #morning datetime declare #date datetime declare #counting int ";
string setting = " set #date = DATEADD(d, DATEDIFF(d, 0, getdate()), 0) set #morning = dateadd(hh, 7, #date) set #counting = (select count([made ready]) from[" + callsignString + "] where[Made Ready] = #morning) ";
string deletemorning = " delete from ["+callsignString+"] where [Made Ready] = #morning";
string queryDeleteRecordfromVehicleSpecific = declare + setting + deletemorning;
string QueryMasterTable = "update Shropshire SET[last made Ready AM] = null where Callsign = '" + callsignString + "'";
cnn.Open();
transaction = cnn.BeginTransaction();
SqlCommand cmd1 = new SqlCommand(QueryMasterTable, cnn);
SqlCommand cmd2 = new SqlCommand(queryDeleteRecordfromVehicleSpecific, cnn);
cmd1.CommandType = CommandType.Text;
cmd2.CommandType = CommandType.Text;
transaction.Commit();
cmd1.ExecuteNonQuery();
cmd2.ExecuteNonQuery();
cnn.Close();
}
}
As you can see this loop has approx. 42 iterations in the morning and afternoon (84 altogether) and opening and closing the connection each time is going to slow everything down.
Does anyone have any ideas of how I can make this better.
Either by making a new "cleaning" table, but then how can I keep track over 28 days instead of forever.
Or by copying each of the "vehicle specific tables" to 1 DataTable and then somehow updating them again after altering them?!?
Any ideas or questions are welcomed.
Thanks in advance

Before I answer your question, I have to point this out:
where Callsign = '"+callsignString+"'"
This is called SQL concatenation, and it allows for SQL injection, which is the #1 security vulnerability in software today. Never do this. Use parametrized SQL queries all the time.
Now to answer your question.
opening and closing the connection each time is going to slow everything down.
That's probably true, but you should obtain proof that it is a performance problem before you decide to optimize it.
Like #PaulF mentions, you don't need to close the connection every time. Move the code that opens and close the connection outside of your loop.
Another technique is to create a stored procedure in your database that would do all this logic in TSQL. That way you can provide only the callsign to the stored proc and it would execute all 84 operations inside a single SQL command. The disadvantage of this approach is that stored procedures are generally a little more expensive to maintain and refactor.

Copying data from one database to another - Out of Memory Exception

I have a task where i want to copy all data from one database to another database & skipping 2 tables. There are more than 200 tables.
I have table structure ready for my 2nd databas.
So as a solution i created a page & on a button click i have below code :-
DataSet ds = new DataSet();
string connectionString = "Data Source=COMP112\\MSSQLSERVER2014;Initial Catalog=HCMBL;Integrated Security=True;Persist Security Info=True";
SqlConnection con = new SqlConnection(connectionString);
//render table name from database
string sqlTable = "SELECT * FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_TYPE='BASE TABLE' and TABLE_Schema='" + Session["SchemaName"].ToString() + "' and TABLE_NAME!='ENTRY' and TABLE_NAME!='OT' and TABLE_NAME!='BL_ENTRY' and TABLE_NAME!='BL_OT'";
con.Open();
SqlDataAdapter da = new SqlDataAdapter();
SqlCommand cmd = new SqlCommand(sqlTable, con);
cmd.CommandType = CommandType.Text;
da.SelectCommand = cmd;
da.Fill(ds);
con.Close();
//render connection string from WebConfig file
string strcon = ConfigurationManager.ConnectionStrings["SPSchema"].ConnectionString;
for (int i = 0; i < ds.Tables[0].Rows.Count; i++)
{
if (!(ds.Tables[0].Rows[i]["TABLE_NAME"].ToString().Contains("Asp")))
{
string deleteQuery = "Truncate table " + Session["SchemaName"].ToString() + "." + ds.Tables[0].Rows[i]["TABLE_NAME"];
con.Open();
SqlCommand cmdDelete = new SqlCommand(deleteQuery, con);
cmdDelete.ExecuteNonQuery();
con.Close();
DataSet dataSet = new DataSet();
SqlConnection conn = new SqlConnection(strcon);
conn.Open();
string selectData = "select * from " + Session["SchemaName"].ToString() + "." + ds.Tables[0].Rows[i]["TABLE_NAME"];
SqlCommand command = new SqlCommand(selectData, conn);
DataTable dataTable = new DataTable();
SqlDataAdapter dataAdapter = new SqlDataAdapter(selectData, conn);
dataAdapter.FillSchema(dataSet, SchemaType.Mapped);
dataAdapter.Fill(dataSet);
dataTable = dataSet.Tables[0];
conn.Close();
if (dataSet.Tables[0].Rows.Count > 0)
{
//Connect to second Database and Insert row/rows.
SqlConnection conn2 = new SqlConnection(connectionString);
conn2.Open();
SqlBulkCopy bulkCopy = new SqlBulkCopy(conn2);
bulkCopy.DestinationTableName = Session["SchemaName"].ToString() + "." + ds.Tables[0].Rows[i]["TABLE_NAME"].ToString();
bulkCopy.WriteToServer(dataTable);
conn2.Close();
}
}
}
As i run the above code after inserting data in less than 10 tables, it gives out of memory exception & program crashes.
How to handle this? I tried increasing the memory capacity of SQL Server but still same error.
Is there any other way to achieve the task?

What you are doing is very far from the best solution. You are using an ASP.NET MVC process to get all data of your entire database into memory, and then outputting it to another database. If your database is anything more than small and trivial, that will most definitely fill your process's alotted memory.
This type of task should never be done through the memory of a process, but rather using some form of Backup/Restore pattern.
You should look into SSIS projects and create an extract, transfer, and load (ETL) solution, which can be triggered from your ASP.NET MVC solution asynchronously.
An SSIS solution can be triggered from C# code in this way:
var app = new Application();
var package = app.LoadPackage("compiled-package.dtsx", null);
var results = package.Execute();
See this question for a little more information (not specifically about duplicating databases, but has information about triggering SSIS packages from code): How to execute an SSIS package from .NET?
Alternatively
You also have the option of running a query against both databases at once, however this requires some additional plumbing to be done. The user account of your ASP.NET MVC solution needs to have access to both databases. If your databases are hosted on different servers, you also need to link one server to the other: Create linked servers
To perform an insert directly from the output of a select, consider this:
string source = "NAME_OF_SOURCE_DATABASE";
string target = "NAME_OF_TARGET_DATABASE";
string schema = Session["SchemaName"].ToString();
string table = ds.Tables[0].Rows[i]["TABLE_NAME"];
// Uncomment this if you need to deal with autoincrement columns
/*string idInsQuery = $"SET IDENTITY_INSERT {target}.{schema}.{table} ON";
var idInsCommand = new SqlCommand(idInsQuery, conn);
idInsCommand.ExecuteNonQuery();*/
string insQuery = $"INSERT INTO {target}.{schema}.{table} SELECT * FROM {source}.{schema}.{table}";
var insCommand = new SqlCommand(insQuery, conn);
insCommand.ExecuteNonQuery();
// Uncomment this if you need to deal with autoincrement columns
/*string idInsQuery2 = $"SET IDENTITY_INSERT {target}.{schema}.{table} OFF";
var idInsCommand2 = new SqlCommand(idInsQuery2, conn);
idInsCommand2.ExecuteNonQuery();*/
This will only work if the table structures are identical. There might be problems with autoincrement ids or columns with default values, too.

This will copy data from a table in database 1 to a table in database 2
Insert into db2.dbo.table2 (col1,col2)
Select col1,col2 from db1.dbo.table1
Run this sql statement and the data will be copied without a round trip to your app.

Let me know if you find my approach is useful.
First of all, why you want to write down one whole application to do this job while SQL Server have inherited property to do it.
My approach would be configure an Linked Server and configure it which tables you want to copy and which one not.
https://learn.microsoft.com/en-us/sql/relational-databases/linked-servers/create-linked-servers-sql-server-database-engine
Secondly, You can just write down simple stored procedure and schedule that in your sql server to push into another server database as per your schedule. In this way you can control it in N number of ways. I mean about controlling any dependencies(Table level or Business level).
To do this in t-sql, you can use the following system stored procedures to schedule a daily job. This example schedules daily at 1:00 AM. See Microsoft help for details on syntax of the individual stored procedures and valid range of parameters.
DECLARE #job_name NVARCHAR(128), #description NVARCHAR(512), #owner_login_name NVARCHAR(128), #database_name NVARCHAR(128);
SET #job_name = N'Some Title';
SET #description = N'Periodically do something';
SET #owner_login_name = N'login';
SET #database_name = N'Database_Name';
-- Delete job if it already exists:
IF EXISTS(SELECT job_id FROM msdb.dbo.sysjobs WHERE (name = #job_name))
BEGIN
EXEC msdb.dbo.sp_delete_job
#job_name = #job_name;
END
-- Create the job:
EXEC msdb.dbo.sp_add_job
#job_name=#job_name,
#enabled=1,
#notify_level_eventlog=0,
#notify_level_email=2,
#notify_level_netsend=2,
#notify_level_page=2,
#delete_level=0,
#description=#description,
#category_name=N'[Uncategorized (Local)]',
#owner_login_name=#owner_login_name;
-- Add server:
EXEC msdb.dbo.sp_add_jobserver #job_name=#job_name;
-- Add step to execute SQL:
EXEC msdb.dbo.sp_add_jobstep
#job_name=#job_name,
#step_name=N'Execute SQL',
#step_id=1,
#cmdexec_success_code=0,
#on_success_action=1,
#on_fail_action=2,
#retry_attempts=0,
#retry_interval=0,
#os_run_priority=0,
#subsystem=N'TSQL',
#command=N'EXEC my_stored_procedure; -- OR ANY SQL STATEMENT',
#database_name=#database_name,
#flags=0;
-- Update job to set start step:
EXEC msdb.dbo.sp_update_job
#job_name=#job_name,
#enabled=1,
#start_step_id=1,
#notify_level_eventlog=0,
#notify_level_email=2,
#notify_level_netsend=2,
#notify_level_page=2,
#delete_level=0,
#description=#description,
#category_name=N'[Uncategorized (Local)]',
#owner_login_name=#owner_login_name,
#notify_email_operator_name=N'',
#notify_netsend_operator_name=N'',
#notify_page_operator_name=N'';
-- Schedule job:
EXEC msdb.dbo.sp_add_jobschedule
#job_name=#job_name,
#name=N'Daily',
#enabled=1,
#freq_type=4,
#freq_interval=1,
#freq_subday_type=1,
#freq_subday_interval=0,
#freq_relative_interval=0,
#freq_recurrence_factor=1,
#active_start_date=20170101, --YYYYMMDD
#active_end_date=99991231, --YYYYMMDD (this represents no end date)
#active_start_time=010000, --HHMMSS
#active_end_time=235959; --HHMMSS
Let me know in case you need more details on this.
Thanks,
Ayan

There is already an open DataReader associated with this Command which must be closed first. please help me out

public void StockUpdate()
{
cmd5 = new SqlCommand("select * from SupplierBillSelection where purordentryid=" + txtPurEntryID.Text + "", con);
var dr1 = cmd5.ExecuteReader();
if (dr1.HasRows)
{
while (dr1.Read())
{
cmd2 = new SqlCommand("select * from Stock where ItemName='" + dr1[2].ToString() + "'", con);
dr5 = cmd2.ExecuteReader();
if (dr1.HasRows)
{
if (dr5.HasRows)
{
dr5.Read();
string insert = "Update Stock set Quantity=" + (Convert.ToSingle(dr5[13]) + Convert.ToSingle(dr1[15])) + " ,TotalPrice=" + (Convert.ToSingle(dr5[14])+Convert.ToSingle(dr1[16]))+ " where ItemName='" + dr1[1].ToString() + "'and CompanyName='" + dr1[2].ToString() +"'";
cmd3 = new SqlCommand(insert, con);
Console.WriteLine(insert);

The reason you're getting this exception is because you end up having multiple resultsets over a single connection. In other words, you open up second SqlDataReader while the first one is still active.
To fix this you will either have to rewrite your query to use joins or, if available, enable MARS. I'm more than sure, though, that it is quite possible to rewrite this entire code block as a single UPDATE FROM SELECT.
Plus, you have the usual set slew of problems with not disposing of disposable objects properly, using select *, concatenating SQL statements and trusting user input.

Add This "MultipleActiveResultSets=True;" In Connection String.

I think you didn't post all the code, but in general if you call ExecuteReader like you do, you have to call Close() on the reader. The only exception is if you use Using-Blocks. The Using-Block calls dispose internally and auto-closes the reader.

You should work with your DataReaders this way:
using(DataReader dr1 = cmd5.ExecuteReader(CommandBehavior.CloseConnection)
{
//do stuff here
}

Sangram Kakade's answer should allow you to run your code, but I advice against coding your queries like this. Each row in the first query will generate an extra insert or update in the second reader and you will flood the server.
Consider using MERGE, as it does the job much faster and atomically (all inserts and updates are executed by default in an implicit transaction).

ODAC seems to be caching table schema?

I'm using Oracle's ODAC.NET for a .NET 3.5 project against an Oracle 11 Express database, and I'm seeing behavior that I can't explain (and can't seem to work around).
ODAC should be the latest, I just pulled it 3 days ago, but the versions are as follows:
Oracle.DataAccess.dll version 2.112.3.0 (release 5)
oci.dll (instant client) version 11.2.0.1
I have a Table, People, that has 3 columns:
ID
FirstName
LastName
In code I run an ALTER TABLE command, using OracleCommand.ExecuteNonQuery, to add a new column named "MIDDLE_NAME" to the table. That command succeeds. If I look at the table with Oracle SQL Developer, the columns shows up. All well and good.
Now if I run use OracleCommand.ExecuteReader with a command text of SELECT * FROM People right after I do the alter table, I get back data with only 3 columns, not 4!
Here is code that reproduces the problem:
public void FieldTest()
{
var sql1 = "CREATE TABLE People (" +
"ID NUMBER PRIMARY KEY, " +
"FirstName NVARCHAR2 (200), " +
"LastName NVARCHAR2 (200) NOT NULL)";
var sql2 = "ALTER TABLE People " +
"ADD Middle_Name NUMBER";
var sql3 = "SELECT * FROM People";
var sql4 = "SELECT column_name FROM all_tab_cols WHERE table_name = 'PEOPLE'";
var cnInfo = new OracleConnectionInfo("192.168.10.246", 1521, "XE", "system", "password");
var connectionString = BuildConnectionString(cnInfo);
using (var connection = new OracleConnection(connectionString))
{
connection.Open();
using (var create = new OracleCommand(sql1, connection))
{
create.ExecuteNonQuery();
}
using (var get = new OracleCommand(sql3, connection))
{
using (var reader = get.ExecuteReader())
{
Debug.WriteLine("Columns: " + reader.FieldCount);
// outputs 3, which is right
}
}
using (var alter = new OracleCommand(sql2, connection))
{
alter.ExecuteNonQuery();
}
using (var get = new OracleCommand(sql3, connection))
{
using (var reader = get.ExecuteReader())
{
Debug.WriteLine("Columns: " + reader.FieldCount);
// outputs 3, which is *wrong* <---- Here's the problem
}
}
using (var cols = new OracleCommand(sql4, connection))
{
using (var reader = cols.ExecuteReader())
{
int count = 0;
while (reader.Read())
{
count++;
Debug.WriteLine("Col: " + reader.GetString(0));
}
Debug.WriteLine("Columns: " + count.ToString());
// outputs 4, which is right
}
}
}
}
I've tried some things to prevent the behavior, and none of them give me back the 4th column:
I close the connection and re-open it
I use a new OracleConnection for the SELECT than for the ALTER
I use the same OracleConnection for the SELECT and for the ALTER
I use a new OracleCommand for the SELECT than for the ALTER
I use the same OracleCommand for the SELECT and for the ALTER
I call PurgeStatementCache on the connection between the ALTER and SELECT
I call FlushCache on the connection between the ALTER and SELECT
I explicitly Close and Dispose the OracleCommand and OracleConnection (as opposed to the using block) used for the ALTER and SELECT
Restarted the calling PC and the PC hosting the Oracle database.
If I look at the column list by doing a SELECT * FROM all_tab_cols, the new column is there.
The only thing that seems to work reliably is closing the app and re-starting it (well it's from a unit test, but it's a shutdown and restart of the test host). Then I get that 4th column. Sometimes I can use breakpoints and re-execute queries and the 4th column will appear, but nothing that is specifically repeatable with straight execution of code (meaning without setting a break point and moving the execution point back up).
Something in the bowels of ODAC seems to be caching the schema of that table, but I can figure out what, why or how to prevent it. Anyone have any experience with this, or ideas how I might prevent it?

I know this answer comes years later but if new readers run into problems with caching try setting:
Metadata Pooling = false, Self Tuning = False and Statement Cache Size = 0
...in the connection string. Keep in mind that there are performance implications for doing so.
https://docs.oracle.com/database/122/ODPNT/featConnecting.htm#GUID-0CFEB161-68EF-4BC2-8943-3BDFFB878602

Maybe post some of your C# code. The following is a test that behaves as expected, meaning I can see the new column immediately after adding it. This is using odp 11.2 rel 5 hitting an 11g db, using 4.0 framework:
The test table is:
CREATE TABLE T1
(
DTE DATE default sysdate
);
Drop and recreate it after each run of the following C# code (a bit dirty but anyway):
string connStr = "User Id=xxx;Password=yyy;Data Source=my11gDb;";
using (OracleConnection con = new OracleConnection(connStr))
{
string s = "ALTER TABLE T1 ADD (added_col VARCHAR2(10))";
using (OracleCommand cmd = new OracleCommand(s, con))
{
con.Open();
cmd.ExecuteNonQuery();
string s2 = "select column_name from all_tab_columns where table_name = 'T1'";
//con.FlushCache(); // doesn't seem to matter, works with or without
using (OracleCommand cmd2 = new OracleCommand(s2, con))
{
OracleDataReader rdr = cmd2.ExecuteReader();
for (int i = 0; rdr.Read(); i++)
{
Console.WriteLine("Column {0} => {1}",i+1,rdr.GetString(0));
}
rdr.Close();
}
}
}
Output:
Column 1 => DTE
Column 2 => ADDED_COL
Edit:
Ah, ok, I see what you're saying, it looks like statement caching. I played around with changing the cache size to 0 (in conn string, use "Statement Cache Size=0"), and also tried cmd.AddToStatementCache = false, but these did not work.
One thing that does work is to use a slightly different string, like adding a space. I know its a hack, but this is all I can get to work for me anyway.
Try your example with:
var sql3 = "SELECT * FROM People";
var sql5 = "SELECT * FROM People "; // note extra space
And use sql3 before adding column, and sql5 after adding a column.
Hope that helps

Too much connections to db using DataReader

Every time i get a value from a response from a odbcconnection by datareader, i made a connection to the database (if i have a query that return 9 fields, i have 9 connection to db), and i want to do only 1 connection and retrieve all information. it's possible with datareader ? I need to use other method of connection ?
Best Regards.
Code:
string strSql = "SELECT G.COMPANY_ID, U.USER_ID, U.GROUP_ID, U.NAME, U.DISPLAY_NAME, U.EMAIL, U.IS_CORPORATE, U.CALL_PARK, U.CALL_PICKUP, U.PCHUNTING, U.OUT_OF_OFFICE, U.DND, U.HOTLINE, U.PIN, U.FORCE_PIN_CHECKED, U.PCHUNTING_TYPE, U.DND_END_TIMESTAMP, U.DND_CONTACT, U.OUT_OF_OFFICE_TYPE, U.LANGUAGE, U.AVAILABLE_TIMESTAMP, U.LAST_DIALLED_NUMBER, U.LAST_INCOMING_CALL, U.LAST_MISSED_CALL, U.CALL_PICKUP_GROUP_ID, U.HOTLINE_NUMBER, U.PORTAL_PASSWORD, U.PROFILE, U.MAIN_NUMBER, U.DUAL_OUTGOING_CTRANSFER, U.MY_CALL_PICKUP, U.VM_RECONNECT_NOTIFY, U.SPARE_STRING1, U.INSERT_DATE, U.INSERT_USER, U.UPDATE_DATE, U.UPDATE_USER " +
"FROM {0}_TT_USER U LEFT OUTER JOIN {0}_TT_GROUP G ON U.GROUP_ID = G.GROUP_ID " +
"WHERE USER_ID = :USER_ID ";
conn = new OdbcConnection(GetIpCntrxTimestenConnString(opCode));
cmd = new OdbcCommand(
string.Format(strSql
, config.GetIpCntrxEsmViewName(opCode))
, conn);
cmd.Parameters.AddWithValue(":USER_ID", user_id);
cmd.CommandType = CommandType.Text;
conn.Open();
dataReader = cmd.ExecuteReader();
object[] meta = new object[dataReader.FieldCount];
int NumberOfColums = dataReader.GetValues(meta);

No, you need another way to query.
Instead of one query per field, SELECT all 9 at once and close the connection right away.
Another problem with this approach is that there's no layering whatsoever. I wouldn't like mingling UI and database code together. There's no abstraction your way.

Try using a StringBuilder and AppendLine your select queries. Loop through your dbReader and store in an List thats the best i can come up with. I do understand you want one call with 9 different queries but either way there is going to be overhead on your sql machine or program machine.