I'm trying to use SqlBulkCopy to import data into a temp table.
private bool CreateTempTable(IDbConnection conn, byte[] fileByteArray, string tempTableName)
{
try
{
Stream stream = new MemoryStream(fileByteArray);
using (var reader = new CsvReader(stream, false, System.Text.Encoding.UTF8))
{
// Bulk insert the data into a temporary table.
using (SqlBulkCopy bulkCopy = new SqlBulkCopy((SqlConnection)conn))
{
// Import the data into the temp table
bulkCopy.DestinationTableName = tempTableName;
bulkCopy.EnableStreaming = true;
bulkCopy.BatchSize = 20000;
bulkCopy.BulkCopyTimeout = 600;
bulkCopy.WriteToServer(reader);
}
}
}
catch (Exception ex)
{
Logger.Error(ex, $"CreateTempTable(): Exception while creating the TempTable \"{tempTableName}\" - {ex.Message}");
return false;
}
return true;
}
This method is called within another method that runs as async. The connection is passed and maintained by code that looks similar to this:
private async Task<bool> ConsumeAsyn(byte[] fileByteArray, string tempTableName)
{
using (var conn = (SqlConnection)OpenConnection())
{
if (CreateTempTable(conn, fileByteArray, tempTableName))
{
// success
}
else
{
// fail
}
}
}
This is basically how the ConsumeAsync is being called:
public async Task<bool> ProcessNextAsync2()
{
try
{
isIdle = false;
string filePath = ImportFilePickupPath + "\\" + Filename;
byte[] inStream = await LoadImportFileAsync(filePath);
bool consumeSuccess = await Consume(inStream, "##TempTable");
if (consumeSuccess)
{
// Delete the file
DeleteImportFile(filePath);
}
else
{
// Rescedule job
}
isIdle = true;
}
catch (Exception ex)
{
isIdle = true;
}
return isIdle;
}
If I use a small record set, of about a 100 rows in my csv, then everything works. The problem is if I have thousands of records, then the connection gets terminated during the bulk import process.
How can I pass the connection to the method and ensure that it remains open till the bulk import completes?
Maybe an answer, maybe not, but perhaps need to keep your IDisposables (at least/especially the SqlConnection) outside of the async tasks.
This answer over here makes it sound like .NET considers the IDiposables to be out of scope and so disposes them while you're still trying to use them:
SqlBulkCopy.WriteToServer() keep getting "connection is closed"
I were able to fix my problem by dropping support on a third party nuget lib that were dropping my connections. While it's unclear as to why it was happening, getting rid of all async calls and managing the connections myself seemed to have fixed my issue. Bulk imports was working fine however processing the data from the ##Temp table was slow. I decided to keep it all in memory and process it from there.
private bool Consume(byte[] fileByteArray, IDataProcess dataConsumer)
{
try
{
using (var conn = OpenConnection())
{
// Convert byte Array to a stream
Stream stream = new MemoryStream(fileByteArray);
// Create a reader from the stream
using (var reader = new CsvReader(stream, false, System.Text.Encoding.UTF8))
{
RecordEnumerator enumerator = reader.GetEnumerator();
enumerator.MoveNext();
do
{
// Proccess enumerator.Current with dataConsumer
} while (enumerator.MoveNext());
}
}
}
catch (Exception ex)
{
return false;
}
return true;
}
We run the risk of running out of memory but until we get a faster approach, this will have to do.
Related
So I am using c# windows form with visual studio to query an access database.
When I run with debugger and stop the application from within visual studio there is no problem, however when I run WITHOUT debugger, query the database and then close using X, the process which appears under "Apps" in Task manager becomes a background process. I can have multiple instances of this process if I run the application numerous times.
I would appreciate any information on this, Thanks!
Here is the code I am using.
private void BtnSendQuery_Click(object sender, EventArgs e)
{
ReadDatabase();
}
private void Form1_FormClosing(object sender, FormClosingEventArgs e)
{
var x = MessageBox.Show("Are you sure you want to exit? ", "Exit", MessageBoxButtons.YesNo, MessageBoxIcon.Question);
if (x == DialogResult.No)
{
e.Cancel = true;
}
else
{
e.Cancel = false;
}
}
private void ReadDatabase()
{
string CONNECTION_STR = #"Provider=Microsoft.ACE.OLEDB.12.0;
Data Source = C:\\Users\\***\\Documents\\db_folder\\access_db.accdb;
Persist Security Info = False";
string query = ""; // query string
OleDbConnection DB_CONNECTION = null;
try
{
DB_CONNECTION = new OleDbConnection(CONNECTION_STR);
DB_CONNECTION.Open();
query = TbInputQuery.Text;
var command = new OleDbCommand(query, DB_CONNECTION);
var str = new StringBuilder();
using (OleDbDataReader reader = command.ExecuteReader())
{
while (reader.Read())
{
str.AppendLine(reader["ID"].ToString());
}
TbOutputTable.Text = str.ToString();
}
DB_CONNECTION.Close();
}
catch (Exception ex)
{
MessageBox.Show(ex.Message);
}
finally
{
if (DB_CONNECTION != null)
{
DB_CONNECTION.Close();
}
}
}
}
As general rule, both your connection and cmdSQL or reader should be in a using block.
While your reader is in a using block, the ALL important connection object is not.
In fact, once beyond the using block for connection? You could get away not even having using blocks for the command and reader object.
And even if a trapped error, or UN-trapped error occurs? The using block WILL ALWAYS clean up the connection.
So, for command and reader - not end of world for using block.
But, for connection? yes, always do that.
Project->settings - I would use the connection builder for the connection string - not put in code.
eg this one:
Then use advanced, and make sure you choose ACE (for accdb) or JET (for mdb)
So this:
So, with above setting, then we have ONE spot in the system - never typing connecting string by hand or having to place in the code (makes change of connection very hard).
Also, don't use "any cpu" force the project to x86 for using Access x32
(or if using x64, then force project to that).
So, code say like this:
private void ReadDatabase()
{
string CONNECTION_STR = Properties.Settings.Default.AccessDB;
string query = ""; // query string
try
{
using (OleDbConnection DB_CONNECTION = new OleDbConnection(CONNECTION_STR))
{
using (OleDbCommand command = new OleDbCommand(query, DB_CONNECTION))
{
DB_CONNECTION.Open();
var str = new StringBuilder();
using (OleDbDataReader reader = command.ExecuteReader())
{
while (reader.Read())
{
str.AppendLine(reader["ID"].ToString());
}
TbOutputTable.Text = str.ToString();
}
}
}
}
catch (Exception ex)
{
MessageBox.Show(ex.Message);
}
finally
{
}
}
Note in above - don't really care about the catch block - as long as the using block for the connection is built - it gets cleaned up no matter what - and even if no try/catch, or if in fact you have one!!
And if a error trigger - still again, note how we do NOT have to clean up, or close the connection.
I have a code to asynchronously update multiple rows in SQL Server's table. I tested it on updating 540 rows and 144 rows are updated in the table instanly, then it waits for about 5 minutes and then the rest is updated. At least this is how it looks when I check for updated rows with SELECT.. I'm wondering why is that.
The whole thing is triggered by button's click:
DialogResult res = MessageBox.Show($"Znaleziono {num} pasujących maszyn. Czy chcesz zaktualizować priorytet maszyny danymi z pliku?", "Potwierdź", MessageBoxButtons.YesNo, MessageBoxIcon.Question);
if(res == DialogResult.Yes)
{
await UpdatePriority();
MessageBox.Show("Updated!");
Here's UpdatePriority method that asynchronously call place.Edit() method for all places in the list of items:
public async Task<string> UpdatePriority()
{
List<Task<string>> UpdateTasks = new List<Task<string>>();
try
{
foreach (Place p in Items.Where(i => i.IsUpdated==true))
{
UpdateTasks.Add(Task.Run(()=> p.Edit()));
}
string response = "OK";
IEnumerable<string> res = await Task.WhenAll<string>(UpdateTasks);
}
catch (Exception ex)
{
throw;
}
return "Nie udało się zaktualizować danych żadnego zasobu..";
}
And here is Edit() method of place object. It basically updates place data in SQL server table:
public async Task<string> Edit()
{
string iSql = #"UPDATE JDE_Places
SET Priority=#Priority
WHERE PlaceId=#PlaceId";
string msg = "OK";
using (SqlCommand command = new SqlCommand(iSql, Settings.conn))
{
command.Parameters.AddWithValue("#PlaceId", PlaceId);
command.Parameters.AddWithValue("#Priority", Priority);
int result = -1;
try
{
result = await command.ExecuteNonQueryAsync();
IsUpdated = false;
}
catch (Exception ex)
{
msg = $"Wystąpił błąd przy edycji zasobu {Name}. Opis błędu: {ex.Message}";
}
}
return msg;
}
And here's Settings conn property that serves as reusable connection object:
public static class Settings
{
private static SqlConnection _conn { get; set; }
public static SqlConnection conn
{
get
{
if (_conn == null)
{
_conn = new SqlConnection(Static.Secrets.ConnectionString);
}
if (_conn.State == System.Data.ConnectionState.Closed || _conn.State == System.Data.ConnectionState.Closed)
{
try
{
_conn.Open();
}
catch (Exception ex)
{
MessageBox.Show("Nie udało się nawiązać połączenia z bazą danych.. " + ex.Message);
}
}
return _conn;
}
}
}
I realize it's probably better to keep the connection within using statement (instead of reusing it), but when I added it to place.Edit() method it worked even slower (and unreliably).
UPDATE: I ran few tests more and the time they took to add 540 rows varied from 15 seconds to 400 seconds.. Then I just changed result = await command.ExecuteNonQueryAsync() to result = command.ExecuteNonQuery() in Edit() of place object, ran few tests more, and all finished under 10 seconds! I don't know why async version of ExecuteNonQuery() was so much worse than non-async one, though. Single Edit() method was taking around 0,1 sec with ExecuteNonQuery() and 1 - 400 seconds with ExecuteNonQueryAsync(). Here are logs: ExecuteNonQuery() ExecuteNonQueryAsync()
Your issue here is your Settings class. You're essentially trying to use the same SqlConnection object in multiple Sqlcommands. SqlConnection is not threadsafe when used like this. You end up with multiple commands because your code is non-blocking and async. That is what is causing your code the "wait" (or deadlock). This is why when you run it sync (without the ExecuteNonQueryAsync, etc.) it works correctly.
You don't need this object at all anyway. ADO.Net handles connection pooling for you, so there is no advantage in re-using the same SqlConnection. Just create a new one for each SqlCommand:
public async Task<string> Edit()
{
using (SqlConnection conn = new SqlConnection(...))
using (SqlCommand command = new SqlCommand(iSql, conn))
{
...
}
}
and you should find that your "wait" goes away.
I'm using sqlite to save log and meet write performance issue.
string log = "INSERT INTO Log VALUES ('2019-12-12 13:43:06','Error','Client','This is log message')"
public int WriteLog(string log)
{
return ExecuteNoQuery(log);
}
public int ExecuteNoQuery(string command)
{
int nResult = -1;
try
{
using (SQLiteConnection dbConnection = new SQLiteConnection(ConnectString))
{
dbConnection.Open();
using (SQLiteCommand dbCommand = dbConnection.CreateCommand())
{
dbCommand.CommandText = command;
nResult = dbCommand.ExecuteNonQuery();
}
}
}
catch (Exception e)
{
// Output error message
}
return nResult;
}
Search in google, transaction could improve the write performance significantly, but unfortunately I don't know when a log message will come, I could not combine the log message. Is there any other way to improve my log write performance?
I tried to add a timer to my code and commit transaction automatically. But I don't think it's a good way to speed up log write performance.
public class DatabaseManager : IDisposable
{
private static SQLiteTransaction transaction = null;
private SQLiteConnection dbConnection = null;
private static Timer transactionTimer;
private long checkInterval = 500;
private DatabaseManager(string connectionString)
{
dbConnection = new SQLiteConnection(connectionString);
dbConnection.Open();
StartTransactionTimer();
}
public void Dispose()
{
if(transaction != null)
{
transaction.Commit();
transaction = null;
}
dbConnection.Close();
dbConnection = null;
}
private void StartTransactionTimer()
{
transactionTimer = new Timer();
transactionTimer.Interval = checkInterval;
transactionTimer.Elapsed += TransactionTimer_Elapsed;
transactionTimer.AutoReset = false;
transactionTimer.Start();
}
private void TransactionTimer_Elapsed(object sender, ElapsedEventArgs e)
{
StartTransation();
transactionTimer.Enabled = true;
}
public void StartTransation()
{
try
{
if (dbConnection == null || dbConnection.State == ConnectionState.Closed)
{
return;
}
if (transaction != null)
{
transaction.Commit();
transaction = null;
}
transaction = dbConnection.BeginTransaction();
}
catch(Exception e)
{
LogError("Error occurs during commit transaction, error message: " + e.Message);
}
}
public int ExecuteNoQuery(string command)
{
int nResult = -1;
try
{
using (SQLiteCommand dbCommand = dbConnection.CreateCommand())
{
dbCommand.CommandText = command;
nResult = dbCommand.ExecuteNonQuery();
}
}
catch (Exception e)
{
LogError("Error occurs during execute sql no result query, error message: ", e.Message);
}
return nResult;
}
}
This started out as a comment, but it's evolving to an answer.
Get rid of the GC.Collect(); code line.
That's not your job to handle garbage collection - and you're probably degrading performance by using it.
No need to close the connection, you're disposing it in the next line anyway.
Why are you locking? Insert statements are usually thread safe - and this one doesn't seem to be an exception of that rule.
You are swallowing exceptions. That's a terrible habit.
Since you're only ever insert a single record, you don't need to return an int - you can simply return a bool (true for success, false for failure)
Why you don't use the entity framework to do the communications with the database?
For me is the easiest way. It's a Microsoft library so you can sure that the performance is very good.
I made some work with entity framework and sqlite db's and everything works very well.
Here an example of use:
var context = new MySqliteDatabase(new SQLiteConnection(#"DataSource=D:\\Mydb.db;cache=shared"));
var author = new Author {
FirstName = "William",
LastName = "Shakespeare",
Books = new List<Book>
{
new Book { Title = "Hamlet"},
new Book { Title = "Othello" },
new Book { Title = "MacBeth" }
}
};
context.Add(author);
context.SaveChanges();
The type of MySqliteDatabase can be created automatically using database first approach or with Code First approach. You have a lot of information and examples on the internet.
Here the link to the official documentation:
https://learn.microsoft.com/en-us/ef/ef6/
This is my code
using UnityEngine;
using System.Collections;
using System;
using System.IO;
using System.Net.Sockets;
public class s_TCP : MonoBehaviour {
internal Boolean socketReady = false;
TcpClient mySocket;
NetworkStream theStream;
StreamWriter theWriter;
StreamReader theReader;
String Host = "198.57.44.231";
Int32 Port = 1337;
string channel = "testingSona";
void Start () {
setupSocket();
//string msg = "__SUBSCRIBE__"+channel+"__ENDSUBSCRIBE__";
string msg = "Sending By Sona";
writeSocket(msg);
readSocket();
}
void Update () {
//readSocket();
}
public void setupSocket() {
try {
mySocket = new TcpClient(Host, Port);
theStream = mySocket.GetStream();
theWriter = new StreamWriter(theStream);
theReader = new StreamReader(theStream);
socketReady = true;
}
catch (Exception e) {
Debug.Log("Socket error: " + e);
}
}
public void writeSocket(string theLine) {
if (!socketReady)
return;
String foo = theLine + "\r\n";
theWriter.Write(foo);
theWriter.Flush();
}
public String readSocket() {
if (!socketReady)
return "";
if (theStream.DataAvailable){
string message = theReader.ReadLine();
print(message);print(12345);
return theReader.ReadLine();
}
else{print("no value");
return "";
}
}
public void closeSocket() {
if (!socketReady)
return;
theWriter.Close();
theReader.Close();
mySocket.Close();
socketReady = false;
}
}
Connection created. But message not writing into server and reading
How can i do it
I think you have taken this code from http://answers.unity3d.com/questions/15422/unity-project-and-3rd-party-apps.html, but I think there is an error in this code. I'll repeat here what I posted there.
The following code does not work correctly:
public String readSocket() {
if (!socketReady)
return "";
if (theStream.DataAvailable)
return theReader.ReadLine();
return "";
}
This caused me a headache for quite few hours. I think that checking DataAvailable on the stream is not a reliable way to check if there is data to be read on the streamreader. So you do not want to check for DataAvailable. However, if you just remove that, then the code will block on ReadLine when there is no more to read. So instead, you need to set a timeout for reading from the stream, so that you won't wait longer than (say) a millisecond:
theStream.ReadTimeout = 1;
And then, you can use something like:
public String readSocket() {
if (!socketReady)
return "";
try {
return theReader.ReadLine();
} catch (Exception e) {
return "";
}
}
This code isn't perfect, I still need to improve it (e.g., check what kind of exception was raised, and deal with it appropriately). And maybe there's a better way overall to do this (I experimented with using Peek(), but the -1 it returns I suspect is for when the socket closes, and not just when there is no more data to read for now). However, this should solve problems with the posted code, like those I was having. If you're finding data is missing from the server, then it's probably sitting in your reader stream, and won't be read until new data is sent from the server and stored in the stream such that theStream.DataAvailable returns true.
I've Got two Programs (Server / Client)
I'm trying to setup IPC for them (They both run on the same box)
Using System.IO.Pipes & Net 3.5
When I call ComOpen, it opens the Pipe correctly, sends the Process ID to the server, but then the Pipe closes and I get an error when it tries to send "Second Write Test"
So Question is.
How do I keep the Pipe open for the Life of the Program?
(I use the Process ID on the server to close everything down if the Client crashes)
private static StreamWriter MyWriter;
private static StreamReader MyReader;
private static NamedPipeClientStream IPCPipe = new NamedPipeClientStream(".", "MyPipe", PipeDirection.InOut);
public static bool MyWrite(string DataOut)
{
bool ValidPipeOut = false;
if(ValidComPort)
try
{
// Send Data
using (QstWriter = new StreamWriter(IPCPipe))
{
QstWriter.AutoFlush = true;
QstWriter.WriteLine(QstDataOut);
QstWriter.Close();
QstWriter.Dispose();
}
ValidPipeOut = true;
}
catch
{
ValidPipeOut = false;
}
return ValidPipeOut;
}
public static bool ComOpen()
{
ValidComPort = true;
try { IPCPipe.Connect(1000); }
catch (Exception ex)
{
string Erroris;
Erroris = ex.Message;
if (Erroris == "Already in a connected state.")
{
// We're Already Connected, Ignore this error.
ValidComPort = true;
}
else
{
ValidComPort = false;
MessageBox.Show(Erroris);
}
}
if (ValidComPort)
{
string ClientProcessID = System.Diagnostics.Process.GetCurrentProcess().Id.ToString();
MyReader = new StreamReader(IPCPipe);
ValidComPort = MyWrite(ClientProcessID);
ValidComPort = MyWrite("Second Write Test");
}
return ValidComPort;
}
The problem is the following line:
using (QstWriter = new StreamWriter(IPCPipe))
At the end of the using statement, the StreamWriter will be disposed and that will in turn dispose the IPCPipe. You are also explicitly calling Dispose and Close on QstWriter, which will close the pipe too.
To fix this, remove the using statement and the calls to Dispose and Close on QstWriter. And assign+initialize QstWriter only once.