I have a DB table with 7 columns and 10rows. Each row is provided an input parameter to a web api call, and the response returned by the api, is inserted into a table. My problem is, the Parallel.Foreach is not producing the same result as the regular ForEach.
Specifically, if 1st row has address as "123 Jump Street Arizona Us", I get a response from web api with the standardized address as "123 Jump Street Arizona USA", like that I have 10 different rows with 10 different input address. However, the output response I get from Parallel.Foreach is for the same address repeated 5 times. And the next time i run it, it is a different result altogether
Could someone please point out why this is happening and the potential solution?
Here is my code:
public void Main()
{
// TODO: Add your code here
string query = "SELECT ADDR_LINE_ONE,ADDR_LINE_TWO,ADDR_LINE_THREE,COUNTRY,PROVINCE,CITY_NAME,POSTAL_CODE FROM Addresstestpoc";
try
{
using (OleDbConnection connection = new OleDbConnection(conn))
{
OleDbCommand command = new OleDbCommand(query, connection);
connection.Open();
OleDbDataReader reader = command.ExecuteReader();
int i = reader.FieldCount;
bool b = reader.HasRows;
Parallel.ForEach(GetFromReader(reader), record =>
{
//AddrOne = record[0].ToString();
string AddrOne = record.GetString(0);
string AddrTwo = record.GetString(1);
string AddrThree = record.GetString(2);
string Country = record.GetString(3);
string Province = record.GetString(4);
string City = record.GetString(5);
string PostalCode = record.GetString(6);
string Sender = "G";
//sqlk = (string)Dts.Variables["User::sqlconn"].Value;
standardizeAddressReturn result;
string data = string.Empty;
string queri;
MDMStandardizeAddressService web = new MDMStandardizeAddressService();
try
{
result = web.standardizeAddress(AddrOne, AddrTwo, AddrThree, City, Province, PostalCode, Country, Sender);
data = SerializeToXml(result);
queri = "insert into [CPM].[dbo].[AddressResponsetest_new] values ('" + data + "')";
//MessageBox.Show(data);
insertintosql(queri);
}
catch (Exception ex)
{
MessageBox.Show(ex.Message);
}
});
}
}
catch (Exception ex)
{
string msg = ex.Message;
}
}
IEnumerable<IDataRecord> GetFromReader(IDataReader reader)
{
while (reader.Read()) yield return reader;
}
I think you need another approach. Command and DataReader are not thread safe. Even if DataReader was thread safe it is a forward only cursor so it is not going to be faster.
I recommend a producer consumer pattern (.e.g BlockingCollection). In the consumer is where you can parallel process the
MDMStandardizeAddressService web = new MDMStandardizeAddressService();
try
{
You could probably use tasks, await, asynch for producer consumer.
An easier approach might be to create a class for properties and put that in a List and then just Read that List. This will happen if a fraction of a second.
You can then parallel process the List.
Stub code:
public class WebMailer
{
public void process()
{
List<Addr> Addrs = new List<Addr>();
SqlCommand command = new SqlCommand();
using (var reader = command.ExecuteReader())
{
using (SqlDataReader r = command.ExecuteReader())
{
Addrs.Add(new Addr(r.GetString(0), r.GetString(1)));
}
}
foreach(Addr addr in Addrs) // can use parallel here
{ }
}
}
s
Related
I created a service in windows Worker service that connects to the database at regular intervals and sends the first newest row in the table to the API
But how to make the service send the same row only once and wait until the next new one appears.
ID cannot have the same value, is it possible to create a condition when dbResult[0] (id) of the already sent row != dbResult[0] can be sent ?
My part of code:
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
while (!stoppingToken.IsCancellationRequested)
{
try
{
string connectionString = "User=SYSDBA;" +
"Password=masterkey;" +
"Database=test.DTB;" +
"DataSource=localhost;" +
"Port:3050";
FbConnection mConnection = new FbConnection(connectionString);
mConnection.Open();
FbTransaction mTransaction = mConnection.BeginTransaction();
string SQLCommandText = "select first 1 * from TABLE where NAME = 449 order by DATE desc ";
FbCommand mCommand = new FbCommand(SQLCommandText, mConnection, mTransaction);
FbDataReader mReader = mCommand.ExecuteReader();
if (mReader.Read())
{
var values = new object[mReader.FieldCount];
{
mReader.GetValues(values);
var dbResult = values.Distinct().ToArray();
var dbResults = (string.Join("|", dbResult));
var result = new
{
ID = dbResult[0],
NAME = dbResult[1],
DATE = dbResult[2],
};
var jsonString = Newtonsoft.Json.JsonConvert.SerializeObject(result);
}
}
}
catch (Exception ex)
{
Log.Fatal(ex, "Problem reading the database.");
}
await Task.Delay(10000, stoppingToken);
}
}
}
UPDATE after get nice clue from Barr J
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
while (!stoppingToken.IsCancellationRequested)
{
try
{
string connectionString = "User=SYSDBA;" +
"Password=masterkey;" +
"Database=test.DTB;" +
"DataSource=localhost;" +
"Port:3050";
FbConnection mConnection = new FbConnection(connectionString);
mConnection.Open();
FbTransaction mTransaction = mConnection.BeginTransaction();
string SQLCommandText = "select NAME, DATE from TABLE where NAME = 449 and ISSENT = 0";
FbCommand mCommand = new FbCommand(SQLCommandText, mConnection, mTransaction);
FbDataReader mReader = mCommand.ExecuteReader();
while (mReader.Read())
{
var values = new object[mReader.FieldCount];
{
mReader.GetValues(values);
var dbResult = values.Distinct().ToArray();
var dbResults = (string.Join("|", dbResult));
var result = new
{
ID = dbResult[0],
NAME = dbResult[1],
DATE = dbResult[2],
};
var jsonString = Newtonsoft.Json.JsonConvert.SerializeObject(result);
using FbConnection UpdateConnection = new(connectionString);
UpdateConnection.Open();
FbCommand writeCommand = new("update TABLE set ISSENT = #isSentValue where ID= #idValue", UpdateConnection);
writeCommand.Parameters.Add("#isSentValue", 1);
writeCommand.Parameters.Add("#idValue", dbResult[0]);
writeCommand.ExecuteNonQuery();
}
}
}
catch (Exception ex)
{
Log.Fatal(ex, "Problem reading the database.");
}
await Task.Delay(10000, stoppingToken);
}
}
}
Because you are using a worker and running async operation, I would not recommend to mix conditions that will complicate the already cumbersome debugging process.
I would add a column to your table as bit and use it as a flag column called IsSent
and then your query would look like:
select...... where name = X and IsSent = Y ...."
this way you will get only rows that are not sent.
you can otherwise query the row in the code and check if the ID was sent.
cleaner, better, easier.
Make your code maintainable.
I would suggest to implement MemCache or simple memory caching on your worker service, by caching your new data. Either the whole or just the key identifier field value. Then, you can pass that value the next time you query the Db where the Id > CachedId or Id != CachedId order by desc.
At any point, when you retrieve the data, you cache it/upsert your cache with that Id.
These are some examples/references for your knowledge
https://qawithexperts.com/article/c-sharp/in-memory-cache-c-explanation-with-example/302
I am creating a simple inventory system using c#.
When I am generating the invoice number, the form is loaded but it doesn't show anything.
It is an auto-incremented invoice number; order is completed incrementally by 1.
For example, starting at E-0000001, after order we expect E-0000002. I don't understand why it is blank.
No error displayed. I tried to debug the code but I couldn't find what's wrong.
public void invoiceno()
{
try
{
string c;
sql = "SELECT MAX(invoid) FROM sales";
cmd = new SqlCommand(sql, con);
var maxInvId = cmd.ExecuteScalar() as string;
if (maxInvId == null)
{
label4.Text = "E-000001";
}
else
{
int intVal = int.Parse(maxInvId.Substring(2, 6));
intVal++;
label4.Text = String.Format("E-{0:000000}", intVal);
}
}
catch (Exception ex)
{
Console.WriteLine(ex.ToString());
Console.Write(ex.StackTrace);
}
}
Let's extract a method - NextInvoiceId - we
Open connection
Execute query
Obtain next invoice number
Code:
private int NextInvoiceNumber() {
//TODO: put the right connection string here
using(var conn = new SqlConnection(ConnectionStringHere)) {
conn.Open();
string sql =
#"SELECT MAX(invoid)
FROM sales";
using (var cmd = new SqlCommand(sql, conn)) {
var raw = cmd.ExecuteScalar() as string;
return raw == null
? 1 // no invoces, we start from 1
: int.Parse(raw.Trim('e', 'E', '-')) + 1;
}
}
}
Then we can easily call it:
public void invoiceno() {
label4.Text = $"E-{NextInvoiceNumber():d6}";
}
Edit: You should not swallow exceptions:
try
{
...
}
// Don't do this!
catch (Exception ex) // All the exceptions will be caught and...
{
// printed on the Console...
// Which is invisible to you, since you develop Win Forms (WPF) application
Console.WriteLine(ex.ToString());
Console.Write(ex.StackTrace);
}
let system die and inform you that something got wrong
Am trying to use Web Api to output some json within an ASP.Net web forms web application.
Below is my get method:
public string Get()
{
String paramOne = "paramOneValue";
using (var conn = new SqlConnection(sqlConStr))
{
try
{
var command = new SqlCommand("getMyList", conn);
command.CommandType = CommandType.StoredProcedure;
command.Parameters.AddWithValue("#paramOne", paramOne);
conn.Open();
ArrayList result = new ArrayList();
var reader = command.ExecuteReader();
while (reader.Read())
{
result.Add(new
{
id = reader[0],
val= reader[1],
val1= reader[2]
});
}
return JsonConvert.SerializeObject(result);
}
catch (SqlException sxp)
{
string msg = Resource.fetchError + " " + sxp.Message;
return "error";
}
}
}
and below is the Response from firebug when the method is called within an aspx page:
"[{\"id\":1,\"val\":\"valFromDB\",\"val1\":\"valOneFromDB\"},{\"id\":2,\"val\":\"row2ValFromDB\",\"val1\":\"row2ValOneFromDB\"}]"
but the weird thing is under JSON tab individual characters are being returned as in below:
0 "["
1 "{"
2 """
3 "i"
4 "d"
Anything am missing?
I am new to MySQL database, I am using Visual Studio C# to connect to my database. I have got a following select method. How can I run it to check if it is working?
EDITED The open and close connection methods
//Open connection to database
private bool OpenConnection()
{
try
{
// connection.open();
return true;
}
catch (MySqlException ex)
{
//When handling errors, your application's response based
//on the error number.
//The two most common error numbers when connecting are as follows:
//0: Cannot connect to server.
//1045: Invalid user name and/or password.
switch (ex.Number)
{
case 0:
MessageBox.Show("Cannot connect to server.");
break;
case 1045:
MessageBox.Show("Invalid username/password, please try again");
break;
}
return false;
}
}
//Close connection
private bool CloseConnection()
{
try
{
connection.Close();
return true;
}
catch (MySqlException ex)
{
MessageBox.Show(ex.Message);
return false;
}
}
Select method which is in the same class as the close and open connection as shown above
public List<string>[] Select()
{
string query = "SELECT * FROM Questions";
//Create a list to store the result
List<string>[] list = new List<string>[3];
list[0] = new List<string>();
list[1] = new List<string>();
list[2] = new List<string>();
list[3] = new List<string>();
list[4] = new List<string>();
list[5] = new List<string>();
list[6] = new List<string>();
list[7] = new List<string>();
//Open connection
if (this.OpenConnection() == true)
{
//Create Command
MySqlCommand cmd = new MySqlCommand(query, connection);
//Create a data reader and Execute the command
MySqlDataReader dataReader = cmd.ExecuteReader();
//Read the data and store them in the list
while (dataReader.Read())
{
list[0].Add(dataReader["id"] + "");
list[1].Add(dataReader["difficulty"] + "");
list[2].Add(dataReader["qustions"] + "");
list[3].Add(dataReader["c_answer"] + "");
list[4].Add(dataReader["choiceA"] + "");
list[5].Add(dataReader["choiceB"] + "");
list[6].Add(dataReader["choiceC"] + "");
list[7].Add(dataReader["choiceD"] + "");
}
//close Data Reader
dataReader.Close();
//close Connection
this.CloseConnection();
//return list to be displayed
return list;
}
else
{
return list;
}
}
This method is in a separate class which has got all the database connection settings. Now that I want to call this method from my main class to test it to see if it's working, how can I do this?
You should create an object instance of that DB class and then call the Select() method.
So, supposing that this DB class is named QuestionsDB you should write something like this:
QuestionDB questionDAL = new QuestionDB();
List<string>[] questions = questionDAL.Select();
However, before this, please correct this line
List<string>[] list = new List<string>[8]; // you need 8 lists for your db query
You could check if you have any record testing if the first list in your array list has more than zero elements.
if(questions[0].Count > 0)
... // you have read records.
However, said that, I will change your code adding a specific class for questions and using a list(of Question) instead of an array of list
So, for example, create a class like this
public class Question
{
public string ID;
public string Difficulty;
public string Question;
public string RightAnswer;
public string AnswerA;
public string AnswerB;
public string AnswerC;
public string AnswerD;
}
and change your select to return a List(of Question)
List<Question> list = new List<Question>;
......
while (dataReader.Read())
{
Question qst = new Question();
qst.ID = dataReader["id"] + "";
qst.Difficulty = dataReader["difficulty"] + "";
qst.Question = dataReader["qustions"] + "";
qst.RightAnswer = dataReader["c_answer"] + "";
qst.AnswerA = dataReader["choiceA"] + "";
qst.AnswerB = dataReader["choiceB"] + "";
qst.AnswerC = dataReader["choiceC"] + "";
qst.AnswerD = dataReader["choiceD"] + "";
list.Add(qst);
}
return list;
You can test whether the method works by writing a unit test for it. A good unit testing frame work is Nunit. Before you call this you must create and open a connection to the DB:
//Open connection
if (this.OpenConnection() == true)
{
as the other person said, you will want to fix the lists up.
I am attempting to read a MySQL database from my C# project using the MySQL drivers for .net off the MySQL site.
Though I did a bit of research on this (including this), I am still flummoxed why this is happening. I later ran a spike and I still get the same error. (Prior to running this I populated the database with some default values.) Here's the spike code in toto.
class Program {
static void Main (string[] args) {
Console.WriteLine (GetUserAge ("john")); // o/p's -1
}
static int GetUserAge (string username) {
string sql = "select age from users where name=#username";
int val = -1;
try {
using (MySqlConnection cnn = GetConnectionForReading ()) {
cnn.Open ();
MySqlCommand myCommand = new MySqlCommand (sql, cnn);
myCommand.Parameters.AddWithValue ("#username", username);
using (MySqlDataReader reader = myCommand.ExecuteReader ()) {
DataTable dt = new DataTable ();
dt.Load (reader);
if (reader.Read ()) {
val = reader.GetInt32 (0);
}
}
}
} catch (Exception ex) {
Console.WriteLine (ex.Message);
} finally {
}
return val;
}
private static MySqlConnection GetConnectionForReading () {
string conStr = "Data Source=localhost;Database=MyTestDB;User ID=testuser;Password=password";
return new MySqlConnection (conStr);
}
}
The code above gives me the exception: "Invalid attempt to Read when reader is closed."
Later I modified the if-condition like so:
if (reader.HasRows && reader.Read ()) {
val = reader.GetInt32 (0);
}
And now the o/p is -1. (The data's in there in the table.) If for some reason the result set had zero rows, the reader should not have got into the if-block in the first place. I mean, the whole point of the Read() method is to check if there are any rows in the result set in the first place.
At my wit's end here... just cannot figure out where I'm going wrong.
Thank you for your help! :)
I think using DataTable.Load will "consume" the reader and, at the very least, position it at the end. It may even account for the closed connection (but I'm just guessing here). What if you remove that line? I don't think it makes any sense here.