I have an app to move data from a MS SQL server into a MySQL server using async query execution; the data moves fine but Task.WaitAll(tasks) call in the RunAllTasks() method never completes.
The async tasks all follow the pattern of PumpLocsAsync() where the call to MS SQL is invoked asynchronously via BeginExecuteReader; when the reader returns with results, then MySQL inserts normally.
..
async Task PumpLocsAsync()
{
using (var conn = new SqlConnection(SqlConnStr))
using (var cn = new MySqlConnection(MysqlConnStr))
using (var cmd = new SqlCommand())
{
cmd.Connection = conn;
cmd.CommandText = "SELECT HERE";
conn.Open();
var handle = cmd.BeginExecuteReader();
await Task.Factory.FromAsync(handle, (ar) =>
{
var rdr = cmd.EndExecuteReader(ar);
var qry = #"INSERT HERE";
cn.Open();
using (var cmdm = new MySqlCommand(qry, cn))
{
cmdm.CommandTimeout = MysqlCmdtimeout;
<PARAM SETUP HERE>
<COLUMN MAPPING HERE>
while (RetryUtility.RetryMethod<bool>(() => rdr.Read(), 3, 1000))
{
<LOADING PARAMS WITH BITS HERE>
RetryUtility.RetryAction(() => cmdm.ExecuteNonQuery(), 3, 1000);
}
}
Console.WriteLine("Finished Locs!");
cn.Close();
});
conn.Close();
}
}
...
void RunAllTasks()
{
Task[] tasks = { PumpLocsAsync(), PumpPicsAsync() };
try
{
Task.WaitAll(tasks);
Console.WriteLine("Finished with all tasks...");
foreach (var task in tasks)
{
Console.WriteLine("Id: {0}, Status: {1}", task.Id, task.Status);
}
}
....
You misunderstood how to call Task.Factory.FromAsync(), you pass to it the result of the BeginXxx method and a delegate to the EndXxx method:
var rdr = await Task.Factory.FromAsync(
cmd.BeginExecuteReader,
(Func<IAsyncResult, SqlDataReader>)cmd.EndExecuteReader,
null);
If you want to do something after the operation completes, simply put it below this line, await will make sure it executes at the right time.
Related
I'm currently testing out C# 8's async streams, and it seems that when I try to run the application using the old pattern of of using async/await and returning Task> it seems to be faster. (I measured it using a stopwatch and tried running it multiple times, and the result was that the old pattern I mentioned seems somewhat faster than using IAsyncEnumerable).
Here's a simple Console App that I wrote (I'm also thinking perhaps I'm loading the data from database the wrong way)
class Program
{
static async Task Main(string[] args)
{
// Using the old pattern
//Stopwatch stopwatch = Stopwatch.StartNew();
//foreach (var person in await LoadDataAsync())
//{
// Console.WriteLine($"Id: {person.Id}, Name: {person.Name}");
//}
//stopwatch.Stop();
//Console.WriteLine(stopwatch.ElapsedMilliseconds);
Stopwatch stopwatch = Stopwatch.StartNew();
await foreach (var person in LoadDataAsyncStream())
{
Console.WriteLine($"Id: {person.Id}, Name: {person.Name}");
}
stopwatch.Stop();
Console.WriteLine(stopwatch.ElapsedMilliseconds);
Console.ReadKey();
}
static async Task<IEnumerable<Person>> LoadDataAsync()
{
string connectionString = "Server=localhost; Database=AsyncStreams; Trusted_Connection = True;";
var people = new List<Person>();
using (SqlConnection connection = new SqlConnection(connectionString))
{
//SqlDataReader
await connection.OpenAsync();
string sql = "Select * From Person";
SqlCommand command = new SqlCommand(sql, connection);
using (SqlDataReader dataReader = await command.ExecuteReaderAsync())
{
while (await dataReader.ReadAsync())
{
Person person = new Person();
person.Id = Convert.ToInt32(dataReader[nameof(Person.Id)]);
person.Name = Convert.ToString(dataReader[nameof(Person.Name)]);
person.Address = Convert.ToString(dataReader[nameof(Person.Address)]);
person.Occupation = Convert.ToString(dataReader[nameof(Person.Occupation)]);
person.Birthday = Convert.ToDateTime(dataReader[nameof(Person.Birthday)]);
person.FavoriteColor = Convert.ToString(dataReader[nameof(Person.FavoriteColor)]);
person.Quote = Convert.ToString(dataReader[nameof(Person.Quote)]);
person.Message = Convert.ToString(dataReader[nameof(Person.Message)]);
people.Add(person);
}
}
await connection.CloseAsync();
}
return people;
}
static async IAsyncEnumerable<Person> LoadDataAsyncStream()
{
string connectionString = "Server=localhost; Database=AsyncStreams; Trusted_Connection = True;";
using (SqlConnection connection = new SqlConnection(connectionString))
{
//SqlDataReader
await connection.OpenAsync();
string sql = "Select * From Person";
SqlCommand command = new SqlCommand(sql, connection);
using (SqlDataReader dataReader = await command.ExecuteReaderAsync())
{
while (await dataReader.ReadAsync())
{
Person person = new Person();
person.Id = Convert.ToInt32(dataReader[nameof(Person.Id)]);
person.Name = Convert.ToString(dataReader[nameof(Person.Name)]);
person.Address = Convert.ToString(dataReader[nameof(Person.Address)]);
person.Occupation = Convert.ToString(dataReader[nameof(Person.Occupation)]);
person.Birthday = Convert.ToDateTime(dataReader[nameof(Person.Birthday)]);
person.FavoriteColor = Convert.ToString(dataReader[nameof(Person.FavoriteColor)]);
person.Quote = Convert.ToString(dataReader[nameof(Person.Quote)]);
person.Message = Convert.ToString(dataReader[nameof(Person.Message)]);
yield return person;
}
}
await connection.CloseAsync();
}
}
I would like to know whether IAsyncEnumerable is not best suited for this kind of scenario or there was something wrong with how I queried the data while using IAsyncEnumerable? I might be wrong but I actually expect using IAsyncEnumerable would be faster. (by the way...the difference are usually in hundreds of milliseconds)
I tried the application with a sample data of 10,000 rows.
Here's also the code for populating the data just in case...
static async Task InsertDataAsync()
{
string connectionString = "Server=localhost; Database=AsyncStreams; Trusted_Connection = True;";
using (SqlConnection connection = new SqlConnection(connectionString))
{
string sql = $"Insert Into Person (Name, Address, Birthday, Occupation, FavoriteColor, Quote, Message) Values";
for (int i = 0; i < 1000; i++)
{
sql += $"('{"Randel Ramirez " + i}', '{"Address " + i}', '{new DateTime(1989, 4, 26)}', '{"Software Engineer " + i}', '{"Red " + i}', '{"Quote " + i}', '{"Message " + i}'),";
}
using (SqlCommand command = new SqlCommand(sql.Remove(sql.Length - 1), connection))
{
command.CommandType = CommandType.Text;
await connection.OpenAsync();
await command.ExecuteNonQueryAsync();
await connection.CloseAsync();
}
}
}
IAsyncEnumerable<T> is not inherently faster or slower than Task<T>. It depends on the implementation.
IAsyncEnumerable<T> is about asynchronously retrieving data providing individual values as soon as possible.
IAsyncEnumerable<T> allows batch producing values which will make some of the invocations of MoveNextAsync synchronous, as in the next example:
async Task Main()
{
var hasValue = false;
var asyncEnumerator = GetValuesAsync().GetAsyncEnumerator();
do
{
var task = asyncEnumerator.MoveNextAsync();
Console.WriteLine($"Completed synchronously: {task.IsCompleted}");
hasValue = await task;
if (hasValue)
{
Console.WriteLine($"Value={asyncEnumerator.Current}");
}
}
while (hasValue);
await asyncEnumerator.DisposeAsync();
}
async IAsyncEnumerable<int> GetValuesAsync()
{
foreach (var batch in GetValuesBatch())
{
await Task.Delay(1000);
foreach (var value in batch)
{
yield return value;
}
}
}
IEnumerable<IEnumerable<int>> GetValuesBatch()
{
yield return Enumerable.Range(0, 3);
yield return Enumerable.Range(3, 3);
yield return Enumerable.Range(6, 3);
}
Output:
Completed synchronously: False
Value=0
Completed synchronously: True
Value=1
Completed synchronously: True
Value=2
Completed synchronously: False
Value=3
Completed synchronously: True
Value=4
Completed synchronously: True
Value=5
Completed synchronously: False
Value=6
Completed synchronously: True
Value=7
Completed synchronously: True
Value=8
Completed synchronously: True
I think the answer to the question of "I would like to know whether IAsyncEnumerable is not best suited for this kind of scenario" got a bit lost in #Bizhan's example of batching and the ensuing discussion, but to reiterate from that post:
IAsyncEnumerable<T> is about asynchronously retrieving data providing individual values as soon as possible.
The OP is measuring the total time to read all records and ignoring how quickly the first record is retrieved and ready to be used by the calling code.
If "this kind of scenario" means reading all the data into memory as fast as possible, then IAsyncEnumerable is not best suited for that.
If it is important to start processing the initial records before waiting for all of the records to be read, that is what IAsyncEnumerable is best suited for.
However, in the real world, you should really be testing the performance of the total system, which would include actual processing of the data as opposed to simply outputting it to a console. Particularly in a multithreaded system, maximum performance could be gained by starting to process multiple records simultaneously as quickly as possible, while at the same time reading in more data from the database. Compare that to waiting for a single thread to read all of the data up front (assuming you could fit the entire dataset into memory) and only then being able to start processing it.
I just wanted to create a dummy ASP.NET MVC project where i wanted to explore more about async & await.I created async methods in DB access Layer and Repository Layer and i also make sure that the action is also written in async way. I noticed that a single method takes around 7ms to execute, so logically if i call the method 3 times i also should take 7 ms or around 7ms ! But its taking about 20-23 ms. I am sure that i am doing something terribly wrong. My Code snippet goes below:
DATA ACCESS LAYER:
public async Task<DataTable> DtGetAllCustomers()
{
await Task.Run(() =>
{
_dtCustomer = new DataTable();
_connectionString = Repo.GetConnString();
_spName = "spGetCus";
_spParameters = new SqlParameter[]
{
new SqlParameter("#QryOption",1)
};
using (var conn = new SqlConnection(_connectionString))
{
using (var cmd = new SqlCommand())
{
cmd.Connection = conn;
cmd.CommandText = _spName;
cmd.Parameters.AddRange(_spParameters);
cmd.CommandType = CommandType.StoredProcedure;
conn.Open();
using (SqlDataAdapter da = new SqlDataAdapter(cmd))
{
da.Fill(_dtCustomer);
conn.Close();
da.Dispose();
}
}
}
});
return _dtCustomer;
}
REPOSITORY:
public async Task<List<CustomerModel>> GetAllCustomers()
{
_dtCustomer = await _customerDal.DtGetAllCustomers();
List<CustomerModel> allCustomers = new List<CustomerModel>();
return allCustomers = (from DataRow row in _dtCustomer.Rows
select new CustomerModel
{
CustomerId = (int)row["CustomerId"],
CustomerName = (string)row["CustomerName"]
}).ToList();
}
ACTION:
public async Task<ActionResult> Index()
{
var watch = System.Diagnostics.Stopwatch.StartNew();
List<CustomerModel> model = new List<CustomerModel>();
CustomerRepo2 aa = new CustomerRepo2();
await aa.GetAllCustomers();
await aa.GetAllCustomers();
await aa.GetAllCustomers();
watch.Stop();
var elapsedMs = watch.ElapsedMilliseconds;
ViewBag.time = elapsedMs;
return View();
}
You're awaiting each call, so before it can move onto the next, it will pause execution. Consider creating three tasks, and awaiting them all with Task.WhenAll():
var task1 = aa.GetAllCustomers();
var task2 = aa.GetAllCustomers();
var task2 = aa.GetAllCustomers();
await Task.WhenAll(task1, task2, task3);
Here I need to export some data from a local server into Excel. I am using an async call for that method. But still, it is blocking the UI while exporting the data.
May I know the reason why the UI is being blocking?
Also, I need some clarification
1)await keyword is using to wait some time until the process is
completed, then what is the difference b/w synchronous and
asynchronous process.
2)If one method is declared as Async task, then all inner methods
are performing as asynchronous?
3)While Executing inner methods(method1,method2,method3), method3 will
be depends upon method1. So, put await keyword for method1. Am I
right?
Code snippet:
private async void ConvertExcel(object sender)
{
var excelEngine = new ExcelEngine();
var application = excelEngine.Excel;
var newbookWorkbook = application.Workbooks.Create(1);
var work = newbookWorkbook.Worksheets[0];
var openDialog = new SaveFileDialog
{
FilterIndex = 2,
Filter = "Excel 97 to 2003 Files(*.xls)|*.xls|Excel 2007 to 2013 Files(*.xlsx)|*.xlsx"
};
if (openDialog.ShowDialog() == true)
{
newbookWorkbook.Version = openDialog.FilterIndex == 1 ? ExcelVersion.Excel97to2003 : ExcelVersion.Excel2013;
}
else
return;
try
{
// This method is used to get the data from server.
var table = GetFullDataAsync();
work.ImportDataTable(table, true, 1, 1, true);
using (Stream stream = openDialog.OpenFile())
{
newbookWorkbook.SaveAs(stream);
}
newbookWorkbook.Close();
excelEngine.Dispose();
}
catch (Exception exception)
{
Console.WriteLine("Exception message: {0}",ex.Message);
}
}
internal async Task<DataTable> GetFullDataAsync()
{
DataTable dataTable = new DataTable();
dataTable = GetDataFromServer(DataEngine.Query,DataEngine.Connection);
dataTable.Locale = CultureInfo.InvariantCulture;
return dataTable;
}
public DataTable GetDataFromServer(string query, DbConnection connection)
{
if (connection.State != ConnectionState.Open)
{
connection.Open();
}
var command = connection.CreateCommand();
command.CommandText = query;
command.Connection = connection;
var reader = command.ExecuteReader();
var dataTable = new DataTable
{
Locale = System.Globalization.CultureInfo.InvariantCulture
};
dataTable.Load(reader);
return dataTable;
}
1) await keyword is using for wait some time until the process is completed, then what is the difference b/w synchronous and asynchronous process.
synchronous: waiting blocks the thread, so locks the ui
asynchronous: waiting frees up the thread, so does not block the ui
2) If one method is declared as Async task, then all inner methods are performing as asynchronous?
Don't assume that is true. You probably want that to be the case, but it is up to you as the developer to code it that way.
The complier does not enforce this for you.
In your code above, the answer is no; your code is not asynchronous. Notice that you never use the await keyword.
You could need to access the data in an asynchronous way first (returning a Task), otherwise async/await keywords can't help you.
3) While Executing inner methods(method1,method2,method3), method3 will be depends upon method1. So, put await keyword for method1. Am i right?
Yes. If each method is returning a Task, then await them all.
Typically when you starting using await in one place, it will bubble up through the rest of your code, so you keep adding more awaits as you have descibed.
have you tried
GetFullDataAsync().ContinueWith((table)=>{
work.ImportDataTable(table, true, 1, 1, true);
using (Stream stream = openDialog.OpenFile())
{
newbookWorkbook.SaveAs(stream);
}
newbookWorkbook.Close();
excelEngine.Dispose();
})
You can change your code to use async communication with Sql server like that:
private async Task ConvertExcel(object sender)
{
var excelEngine = new ExcelEngine();
var application = excelEngine.Excel;
var newbookWorkbook = application.Workbooks.Create(1);
var work = newbookWorkbook.Worksheets[0];
var openDialog = new SaveFileDialog
{
FilterIndex = 2,
Filter = "Excel 97 to 2003 Files(*.xls)|*.xls|Excel 2007 to 2013 Files(*.xlsx)|*.xlsx"
};
if (openDialog.ShowDialog() == true)
{
newbookWorkbook.Version = openDialog.FilterIndex == 1 ? ExcelVersion.Excel97to2003 : ExcelVersion.Excel2013;
}
else
return;
try
{
// This method is used to get the data from server.
var table = await GetFullDataAsync();
work.ImportDataTable(table, true, 1, 1, true);
var saveWorkbookTask = Task.Run(() => {
using (Stream stream = openDialog.OpenFile())
{
newbookWorkbook.SaveAs(stream);
}
});
await saveWorkbookTask;
newbookWorkbook.Close();
excelEngine.Dispose();
}
catch (Exception exception)
{
Console.WriteLine("Exception message: {0}",ex.Message);
}
}
internal async Task<DataTable> GetFullDataAsync()
{
DataTable dataTable = new DataTable();
dataTable = await GetDataFromServer(DataEngine.Query,DataEngine.Connection);
dataTable.Locale = CultureInfo.InvariantCulture;
return dataTable;
}
public async Task<DataTable> GetDataFromServer(string query, DbConnection connection)
{
if (connection.State != ConnectionState.Open)
{
connection.Open();
}
var command = connection.CreateCommand();
command.CommandText = query;
command.Connection = connection;
var reader = await command.ExecuteReaderAsync();
var loadTableTask = Task.Run(() => {
var dataTable = new DataTable
{
Locale = System.Globalization.CultureInfo.InvariantCulture
};
dataTable.Load(reader);
return dataTable;
});
return await loadTableTask;
}
But you still have blocking IO operations. So if your file is large than the UI will be blocked when the file will be written to disk.
I'm having a really difficult time wrapping my head around asynchronous. I've tried reading numerous questions/answers, forums etc.
I have to make about 50 calls to the database(I can't make a change to the database to return all in one call). So the answer is to use asynchronous so that I only have to wait for essentially the longest call. I am trying to do a simple logging of the time for 10 round trips.
My theory is that because I'm using two asynchronous tasks one in one method, and one in the calling method. The tasks look like they are completing but have not completed yet. here is my code.
I'm just logging the time it takes to complete the round trips for now, but the idea is to return the List<ResultList> in the future.
protected async void Page_Load(object sender, EventArgs e)
{
List<Buildings> bldgList = new List<Buildings>();
//lots of buildings here now.
Stopwatch GetByOrg = new Stopwatch();
lblorg.Text = await RunOrg(GetByOrg, bldgList);
}
async Task<string> RunOrg(Stopwatch getByOrg, List<Buildings> retVal)
{
getByOrg.Start();
List<Task> tasks = new List<Task>();
for (int i = 0; i < 10; i++)
{
foreach (Buildings b in bldgList)
{
tasks.Add(Task.Run(() => ExecuteOrg(b)));
}
// var tasky = new Task()
}
Task.WaitAll(tasks.ToArray());
getByOrg.Stop();
return String.Format("{0:00}:{1:00}:{2:00}.{3:00}", getByOrg.Elapsed.Hours, getByOrg.Elapsed.Minutes, getByOrg.Elapsed.Seconds, getByOrg.Elapsed.Milliseconds / 10);
}
async Task<List<ResultSet>> ExecuteOrg(Buildings b)
{
List<ResultSet> resulty = new List<ResultSet>();
var asyncconn = new SqlConnectionStringBuilder("That Server Connection") { AsynchronousProcessing = true }.ToString();
using (SqlConnection conn = new SqlConnection(asyncconn))
{
using (SqlCommand cmd = new SqlCommand("rptTotalConcentrators", conn))
{
cmd.CommandType = System.Data.CommandType.StoredProcedure;
cmd.Parameters.Add("#school_yr", System.Data.SqlDbType.SmallInt).Value = 2016;
conn.Open();
using (var reader = await cmd.ExecuteReaderAsync())
{
while (await reader.ReadAsync())
{
ResultSet testy = new ResultSet();
testy.bldg_no = reader["bldg_no"].ToString();
testy.bldg_name = reader["bldg_name"].ToString();
testy.school_year = reader["school_year"].ToString();
testy.concentrator = reader["conc"].ToString();
resulty.Add(testy);
}
}
}
}
return resulty;
}
the problem is that lblorg.Text is showing about .06 for completed milliseconds. Either .WhenAll doesn't work how i think, or something else is wrong. because that should take a lot longer.
I posted a previous question about this, I found a sort of working method using threads - but it was causing reliability issues when variables were rewritten between threads. I was pointed towards using ping async and tasks, but I'm getting the following exception:
Error: System.InvalidOperationException: An asynchronous call is already in progress. It must be completed or canceled before you can call this method.
That points specifically to the "await PingAsync(HostList)" and "await Task.WhenAll(tasks)"
Heres the code:
MySqlConnection Connection = new MySqlConnection(ConnectionString);
Connection.Open();
MySqlCommand query = Connection.CreateCommand();
query.CommandText = "SELECT obj_id AS id,ip FROM objects LIMIT 15";
MySqlDataReader Nodes = query.ExecuteReader();
// Record in log that a NEW iteration is starting - for tracking issues with pings
Library.WriteErrorLog("######################## New Iteration ########################");
var HostList = new List<HostObject>();
int i = 1;
while (Nodes.Read())
{
var Host = new HostObject();
Host.HostName = Nodes["ip"].ToString();
Host.HostID = Nodes["id"].ToString();
HostList.Add(Host);
i++;
}
Connection.Close();
var results = await PingAsync(HostList);
bool WaitingOnPings = false;
while (WaitingOnPings == false)
{
if (results != null)
{
foreach (PingReply result in results)
{
Library.WriteErrorLog(result.Status.ToString());
}
WaitingOnPings = true;
}
else
{
Thread.Sleep(500);
}
}
and :
private async Task<List<PingReply>> PingAsync(List<HostObject> HostList)
{
Ping pingSender = new Ping();
var tasks = HostList.Select(HostName => pingSender.SendPingAsync(HostName.ToString(), 2000));
var results = await Task.WhenAll(tasks);
return results.ToList();
}
Thanks for the help!
Ping was probably not intended to be used for multiple concurrent requests. Try creating a new Ping object for each host.
private async Task<List<PingReply>> PingAsync(List<HostObject> HostList)
{
var tasks = HostList.Select(HostName => new Ping().SendPingAsync(HostName.ToString(), 2000));
var results = await Task.WhenAll(tasks);
return results.ToList();
}