We Keep Coding

Why is using IAsyncEnumerable slower than returning async/await Task<T>?

I'm currently testing out C# 8's async streams, and it seems that when I try to run the application using the old pattern of of using async/await and returning Task> it seems to be faster. (I measured it using a stopwatch and tried running it multiple times, and the result was that the old pattern I mentioned seems somewhat faster than using IAsyncEnumerable).
Here's a simple Console App that I wrote (I'm also thinking perhaps I'm loading the data from database the wrong way)
class Program
{
static async Task Main(string[] args)
{
// Using the old pattern
//Stopwatch stopwatch = Stopwatch.StartNew();
//foreach (var person in await LoadDataAsync())
//{
// Console.WriteLine($"Id: {person.Id}, Name: {person.Name}");
//}
//stopwatch.Stop();
//Console.WriteLine(stopwatch.ElapsedMilliseconds);
Stopwatch stopwatch = Stopwatch.StartNew();
await foreach (var person in LoadDataAsyncStream())
{
Console.WriteLine($"Id: {person.Id}, Name: {person.Name}");
}
stopwatch.Stop();
Console.WriteLine(stopwatch.ElapsedMilliseconds);
Console.ReadKey();
}
static async Task<IEnumerable<Person>> LoadDataAsync()
{
string connectionString = "Server=localhost; Database=AsyncStreams; Trusted_Connection = True;";
var people = new List<Person>();
using (SqlConnection connection = new SqlConnection(connectionString))
{
//SqlDataReader
await connection.OpenAsync();
string sql = "Select * From Person";
SqlCommand command = new SqlCommand(sql, connection);
using (SqlDataReader dataReader = await command.ExecuteReaderAsync())
{
while (await dataReader.ReadAsync())
{
Person person = new Person();
person.Id = Convert.ToInt32(dataReader[nameof(Person.Id)]);
person.Name = Convert.ToString(dataReader[nameof(Person.Name)]);
person.Address = Convert.ToString(dataReader[nameof(Person.Address)]);
person.Occupation = Convert.ToString(dataReader[nameof(Person.Occupation)]);
person.Birthday = Convert.ToDateTime(dataReader[nameof(Person.Birthday)]);
person.FavoriteColor = Convert.ToString(dataReader[nameof(Person.FavoriteColor)]);
person.Quote = Convert.ToString(dataReader[nameof(Person.Quote)]);
person.Message = Convert.ToString(dataReader[nameof(Person.Message)]);
people.Add(person);
}
}
await connection.CloseAsync();
}
return people;
}
static async IAsyncEnumerable<Person> LoadDataAsyncStream()
{
string connectionString = "Server=localhost; Database=AsyncStreams; Trusted_Connection = True;";
using (SqlConnection connection = new SqlConnection(connectionString))
{
//SqlDataReader
await connection.OpenAsync();
string sql = "Select * From Person";
SqlCommand command = new SqlCommand(sql, connection);
using (SqlDataReader dataReader = await command.ExecuteReaderAsync())
{
while (await dataReader.ReadAsync())
{
Person person = new Person();
person.Id = Convert.ToInt32(dataReader[nameof(Person.Id)]);
person.Name = Convert.ToString(dataReader[nameof(Person.Name)]);
person.Address = Convert.ToString(dataReader[nameof(Person.Address)]);
person.Occupation = Convert.ToString(dataReader[nameof(Person.Occupation)]);
person.Birthday = Convert.ToDateTime(dataReader[nameof(Person.Birthday)]);
person.FavoriteColor = Convert.ToString(dataReader[nameof(Person.FavoriteColor)]);
person.Quote = Convert.ToString(dataReader[nameof(Person.Quote)]);
person.Message = Convert.ToString(dataReader[nameof(Person.Message)]);
yield return person;
}
}
await connection.CloseAsync();
}
}
I would like to know whether IAsyncEnumerable is not best suited for this kind of scenario or there was something wrong with how I queried the data while using IAsyncEnumerable? I might be wrong but I actually expect using IAsyncEnumerable would be faster. (by the way...the difference are usually in hundreds of milliseconds)
I tried the application with a sample data of 10,000 rows.
Here's also the code for populating the data just in case...
static async Task InsertDataAsync()
{
string connectionString = "Server=localhost; Database=AsyncStreams; Trusted_Connection = True;";
using (SqlConnection connection = new SqlConnection(connectionString))
{
string sql = $"Insert Into Person (Name, Address, Birthday, Occupation, FavoriteColor, Quote, Message) Values";
for (int i = 0; i < 1000; i++)
{
sql += $"('{"Randel Ramirez " + i}', '{"Address " + i}', '{new DateTime(1989, 4, 26)}', '{"Software Engineer " + i}', '{"Red " + i}', '{"Quote " + i}', '{"Message " + i}'),";
}
using (SqlCommand command = new SqlCommand(sql.Remove(sql.Length - 1), connection))
{
command.CommandType = CommandType.Text;
await connection.OpenAsync();
await command.ExecuteNonQueryAsync();
await connection.CloseAsync();
}
}
}

IAsyncEnumerable<T> is not inherently faster or slower than Task<T>. It depends on the implementation.
IAsyncEnumerable<T> is about asynchronously retrieving data providing individual values as soon as possible.
IAsyncEnumerable<T> allows batch producing values which will make some of the invocations of MoveNextAsync synchronous, as in the next example:
async Task Main()
{
var hasValue = false;
var asyncEnumerator = GetValuesAsync().GetAsyncEnumerator();
do
{
var task = asyncEnumerator.MoveNextAsync();
Console.WriteLine($"Completed synchronously: {task.IsCompleted}");
hasValue = await task;
if (hasValue)
{
Console.WriteLine($"Value={asyncEnumerator.Current}");
}
}
while (hasValue);
await asyncEnumerator.DisposeAsync();
}
async IAsyncEnumerable<int> GetValuesAsync()
{
foreach (var batch in GetValuesBatch())
{
await Task.Delay(1000);
foreach (var value in batch)
{
yield return value;
}
}
}
IEnumerable<IEnumerable<int>> GetValuesBatch()
{
yield return Enumerable.Range(0, 3);
yield return Enumerable.Range(3, 3);
yield return Enumerable.Range(6, 3);
}
Output:
Completed synchronously: False
Value=0
Completed synchronously: True
Value=1
Completed synchronously: True
Value=2
Completed synchronously: False
Value=3
Completed synchronously: True
Value=4
Completed synchronously: True
Value=5
Completed synchronously: False
Value=6
Completed synchronously: True
Value=7
Completed synchronously: True
Value=8
Completed synchronously: True

I think the answer to the question of "I would like to know whether IAsyncEnumerable is not best suited for this kind of scenario" got a bit lost in #Bizhan's example of batching and the ensuing discussion, but to reiterate from that post:
IAsyncEnumerable<T> is about asynchronously retrieving data providing individual values as soon as possible.
The OP is measuring the total time to read all records and ignoring how quickly the first record is retrieved and ready to be used by the calling code.
If "this kind of scenario" means reading all the data into memory as fast as possible, then IAsyncEnumerable is not best suited for that.
If it is important to start processing the initial records before waiting for all of the records to be read, that is what IAsyncEnumerable is best suited for.
However, in the real world, you should really be testing the performance of the total system, which would include actual processing of the data as opposed to simply outputting it to a console. Particularly in a multithreaded system, maximum performance could be gained by starting to process multiple records simultaneously as quickly as possible, while at the same time reading in more data from the database. Compare that to waiting for a single thread to read all of the data up front (assuming you could fit the entire dataset into memory) and only then being able to start processing it.

UI will be blocked while using async and await in c#

Here I need to export some data from a local server into Excel. I am using an async call for that method. But still, it is blocking the UI while exporting the data.
May I know the reason why the UI is being blocking?
Also, I need some clarification
1)await keyword is using to wait some time until the process is
completed, then what is the difference b/w synchronous and
asynchronous process.
2)If one method is declared as Async task, then all inner methods
are performing as asynchronous?
3)While Executing inner methods(method1,method2,method3), method3 will
be depends upon method1. So, put await keyword for method1. Am I
right?
Code snippet:
private async void ConvertExcel(object sender)
{
var excelEngine = new ExcelEngine();
var application = excelEngine.Excel;
var newbookWorkbook = application.Workbooks.Create(1);
var work = newbookWorkbook.Worksheets[0];
var openDialog = new SaveFileDialog
{
FilterIndex = 2,
Filter = "Excel 97 to 2003 Files(*.xls)|*.xls|Excel 2007 to 2013 Files(*.xlsx)|*.xlsx"
};
if (openDialog.ShowDialog() == true)
{
newbookWorkbook.Version = openDialog.FilterIndex == 1 ? ExcelVersion.Excel97to2003 : ExcelVersion.Excel2013;
}
else
return;
try
{
// This method is used to get the data from server.
var table = GetFullDataAsync();
work.ImportDataTable(table, true, 1, 1, true);
using (Stream stream = openDialog.OpenFile())
{
newbookWorkbook.SaveAs(stream);
}
newbookWorkbook.Close();
excelEngine.Dispose();
}
catch (Exception exception)
{
Console.WriteLine("Exception message: {0}",ex.Message);
}
}
internal async Task<DataTable> GetFullDataAsync()
{
DataTable dataTable = new DataTable();
dataTable = GetDataFromServer(DataEngine.Query,DataEngine.Connection);
dataTable.Locale = CultureInfo.InvariantCulture;
return dataTable;
}
public DataTable GetDataFromServer(string query, DbConnection connection)
{
if (connection.State != ConnectionState.Open)
{
connection.Open();
}
var command = connection.CreateCommand();
command.CommandText = query;
command.Connection = connection;
var reader = command.ExecuteReader();
var dataTable = new DataTable
{
Locale = System.Globalization.CultureInfo.InvariantCulture
};
dataTable.Load(reader);
return dataTable;
}

1) await keyword is using for wait some time until the process is completed, then what is the difference b/w synchronous and asynchronous process.
synchronous: waiting blocks the thread, so locks the ui
asynchronous: waiting frees up the thread, so does not block the ui
2) If one method is declared as Async task, then all inner methods are performing as asynchronous?
Don't assume that is true. You probably want that to be the case, but it is up to you as the developer to code it that way.
The complier does not enforce this for you.
In your code above, the answer is no; your code is not asynchronous. Notice that you never use the await keyword.
You could need to access the data in an asynchronous way first (returning a Task), otherwise async/await keywords can't help you.
3) While Executing inner methods(method1,method2,method3), method3 will be depends upon method1. So, put await keyword for method1. Am i right?
Yes. If each method is returning a Task, then await them all.
Typically when you starting using await in one place, it will bubble up through the rest of your code, so you keep adding more awaits as you have descibed.

have you tried
GetFullDataAsync().ContinueWith((table)=>{
work.ImportDataTable(table, true, 1, 1, true);
using (Stream stream = openDialog.OpenFile())
{
newbookWorkbook.SaveAs(stream);
}
newbookWorkbook.Close();
excelEngine.Dispose();
})

You can change your code to use async communication with Sql server like that:
private async Task ConvertExcel(object sender)
{
var excelEngine = new ExcelEngine();
var application = excelEngine.Excel;
var newbookWorkbook = application.Workbooks.Create(1);
var work = newbookWorkbook.Worksheets[0];
var openDialog = new SaveFileDialog
{
FilterIndex = 2,
Filter = "Excel 97 to 2003 Files(*.xls)|*.xls|Excel 2007 to 2013 Files(*.xlsx)|*.xlsx"
};
if (openDialog.ShowDialog() == true)
{
newbookWorkbook.Version = openDialog.FilterIndex == 1 ? ExcelVersion.Excel97to2003 : ExcelVersion.Excel2013;
}
else
return;
try
{
// This method is used to get the data from server.
var table = await GetFullDataAsync();
work.ImportDataTable(table, true, 1, 1, true);
var saveWorkbookTask = Task.Run(() => {
using (Stream stream = openDialog.OpenFile())
{
newbookWorkbook.SaveAs(stream);
}
});
await saveWorkbookTask;
newbookWorkbook.Close();
excelEngine.Dispose();
}
catch (Exception exception)
{
Console.WriteLine("Exception message: {0}",ex.Message);
}
}
internal async Task<DataTable> GetFullDataAsync()
{
DataTable dataTable = new DataTable();
dataTable = await GetDataFromServer(DataEngine.Query,DataEngine.Connection);
dataTable.Locale = CultureInfo.InvariantCulture;
return dataTable;
}
public async Task<DataTable> GetDataFromServer(string query, DbConnection connection)
{
if (connection.State != ConnectionState.Open)
{
connection.Open();
}
var command = connection.CreateCommand();
command.CommandText = query;
command.Connection = connection;
var reader = await command.ExecuteReaderAsync();
var loadTableTask = Task.Run(() => {
var dataTable = new DataTable
{
Locale = System.Globalization.CultureInfo.InvariantCulture
};
dataTable.Load(reader);
return dataTable;
});
return await loadTableTask;
}
But you still have blocking IO operations. So if your file is large than the UI will be blocked when the file will be written to disk.

Database Calls Asynchronously

I'm having a really difficult time wrapping my head around asynchronous. I've tried reading numerous questions/answers, forums etc.
I have to make about 50 calls to the database(I can't make a change to the database to return all in one call). So the answer is to use asynchronous so that I only have to wait for essentially the longest call. I am trying to do a simple logging of the time for 10 round trips.
My theory is that because I'm using two asynchronous tasks one in one method, and one in the calling method. The tasks look like they are completing but have not completed yet. here is my code.
I'm just logging the time it takes to complete the round trips for now, but the idea is to return the List<ResultList> in the future.
protected async void Page_Load(object sender, EventArgs e)
{
List<Buildings> bldgList = new List<Buildings>();
//lots of buildings here now.
Stopwatch GetByOrg = new Stopwatch();
lblorg.Text = await RunOrg(GetByOrg, bldgList);
}
async Task<string> RunOrg(Stopwatch getByOrg, List<Buildings> retVal)
{
getByOrg.Start();
List<Task> tasks = new List<Task>();
for (int i = 0; i < 10; i++)
{
foreach (Buildings b in bldgList)
{
tasks.Add(Task.Run(() => ExecuteOrg(b)));
}
// var tasky = new Task()
}
Task.WaitAll(tasks.ToArray());
getByOrg.Stop();
return String.Format("{0:00}:{1:00}:{2:00}.{3:00}", getByOrg.Elapsed.Hours, getByOrg.Elapsed.Minutes, getByOrg.Elapsed.Seconds, getByOrg.Elapsed.Milliseconds / 10);
}
async Task<List<ResultSet>> ExecuteOrg(Buildings b)
{
List<ResultSet> resulty = new List<ResultSet>();
var asyncconn = new SqlConnectionStringBuilder("That Server Connection") { AsynchronousProcessing = true }.ToString();
using (SqlConnection conn = new SqlConnection(asyncconn))
{
using (SqlCommand cmd = new SqlCommand("rptTotalConcentrators", conn))
{
cmd.CommandType = System.Data.CommandType.StoredProcedure;
cmd.Parameters.Add("#school_yr", System.Data.SqlDbType.SmallInt).Value = 2016;
conn.Open();
using (var reader = await cmd.ExecuteReaderAsync())
{
while (await reader.ReadAsync())
{
ResultSet testy = new ResultSet();
testy.bldg_no = reader["bldg_no"].ToString();
testy.bldg_name = reader["bldg_name"].ToString();
testy.school_year = reader["school_year"].ToString();
testy.concentrator = reader["conc"].ToString();
resulty.Add(testy);
}
}
}
}
return resulty;
}
the problem is that lblorg.Text is showing about .06 for completed milliseconds. Either .WhenAll doesn't work how i think, or something else is wrong. because that should take a lot longer.

How do I use Dapper in an asynchronous controller?

This is my first time using async/await with the Dapper ORM. I keep getting the message that my controller lacks the await method. How can I correctly insert the await method in my code?
public async Task<ActionResult> index(int? page)
{
if (page == null)
{
page = 1;
ViewBag.page = page;
}
else
{
ViewBag.page = page;
}
string Connectionstring = ConfigurationManager.ConnectionStrings["mycontext"].ConnectionString;
using (System.Data.SqlClient.SqlConnection sqlConnection = new System.Data.SqlClient.SqlConnection(Connectionstring))
{
sqlConnection.Open();
var sqlM = #"SELECT * from threads order by activities desc";
var resultM = sqlConnection.Query<thread>(sqlM).ToList();
await Task.WhenAll(resultM);
return View(new homepage { panel = resultM });
}
}
I have tried await Task.WhenAll(resultM); but it didn't change anything.

You're getting the warning because you aren't using any asynchronous methods in the controller. As it is, there's no reason to mark it as async.
In order to actually make the method asynchronous, you need to use Dapper's async methods and await them. See Using Async Await keywords with Dapper for another example of this.
In your code, it would look like:
public async Task<ActionResult> index(int? page)
{
if (page == null)
{
page = 1;
ViewBag.page = page;
}
else
{
ViewBag.page = page;
}
string Connectionstring = ConfigurationManager.ConnectionStrings["mycontext"].ConnectionString;
using (System.Data.SqlClient.SqlConnection sqlConnection = new System.Data.SqlClient.SqlConnection(Connectionstring))
{
await sqlConnection.OpenAsync();
var sqlM = #"SELECT * from threads order by activities desc";
var resultM = await sqlConnection.QueryAsync<thread>(sqlM);
return View(new homepage { panel = resultM });
}
}
If resultM must be a List<thread> instead of IEnumerable<T>, you can call ToList - but make sure it's after awaiting the asynchronous call.

Async SQL query execution - Task.WaitAll(tasks) never completes

I have an app to move data from a MS SQL server into a MySQL server using async query execution; the data moves fine but Task.WaitAll(tasks) call in the RunAllTasks() method never completes.
The async tasks all follow the pattern of PumpLocsAsync() where the call to MS SQL is invoked asynchronously via BeginExecuteReader; when the reader returns with results, then MySQL inserts normally.
..
async Task PumpLocsAsync()
{
using (var conn = new SqlConnection(SqlConnStr))
using (var cn = new MySqlConnection(MysqlConnStr))
using (var cmd = new SqlCommand())
{
cmd.Connection = conn;
cmd.CommandText = "SELECT HERE";
conn.Open();
var handle = cmd.BeginExecuteReader();
await Task.Factory.FromAsync(handle, (ar) =>
{
var rdr = cmd.EndExecuteReader(ar);
var qry = #"INSERT HERE";
cn.Open();
using (var cmdm = new MySqlCommand(qry, cn))
{
cmdm.CommandTimeout = MysqlCmdtimeout;
<PARAM SETUP HERE>
<COLUMN MAPPING HERE>
while (RetryUtility.RetryMethod<bool>(() => rdr.Read(), 3, 1000))
{
<LOADING PARAMS WITH BITS HERE>
RetryUtility.RetryAction(() => cmdm.ExecuteNonQuery(), 3, 1000);
}
}
Console.WriteLine("Finished Locs!");
cn.Close();
});
conn.Close();
}
}
...
void RunAllTasks()
{
Task[] tasks = { PumpLocsAsync(), PumpPicsAsync() };
try
{
Task.WaitAll(tasks);
Console.WriteLine("Finished with all tasks...");
foreach (var task in tasks)
{
Console.WriteLine("Id: {0}, Status: {1}", task.Id, task.Status);
}
}
....

You misunderstood how to call Task.Factory.FromAsync(), you pass to it the result of the BeginXxx method and a delegate to the EndXxx method:
var rdr = await Task.Factory.FromAsync(
cmd.BeginExecuteReader,
(Func<IAsyncResult, SqlDataReader>)cmd.EndExecuteReader,
null);
If you want to do something after the operation completes, simply put it below this line, await will make sure it executes at the right time.