We Keep Coding

More efficient way to check DataTable values?

I want to iterate through my table, check to see if the Quantity received is higher than the quantity expected for each row, and if so execute some code.
The way I've done it below feels amateurish.
bool allRecieved = true;
foreach (DataRow row in SubVwr.Tables[0].Tbl.Rows)
{
if(row["QtyRcvd"] < row["QtyPer"]) allRecieved = false;
}
if(allRecieved) // execute code

You can use LINQ, for better readability (presuming those are int-columns):
bool allRecieved = SubVwr.Tables[0].Tbl.AsEnumerable()
.All(row => row.Field<int>("QtyRcvd") >= row.Field<int>("QtyPer"));
An advantage over your current loop is that this will stop as soon as one record doesn't match this condition. Your loop will continue until end without break.

This is a bit radical, but I'd start by not using DataTable, and instead use a simple model:
public class SomeType {
// I'd probably name these QuantityReceived etc, but... meh
public int QtyRcvd {get;set;}
public int QtyPer {get;set;}
// ... etc
}
Then I can very conveniently check properties etc. For example, to mirror Tim Schmelter's LINQ answer:
List<SomeType> someData = ...
var allReceived = someData.All(x => x.QtyRcvd >= x.QtyPer);
so now all we need is to load SomeType from the DB, which is where ORMs and micro-ORMs excel. For example, with "Dapper", this would be:
string region = "North"; // just to show parameter usage
var someData = connection.Query<SomeType>(
#"select * from SomeTable where Region=#region", new { region }).AsList();

SSIS Script Component Input0Buffer method no GetName()?

I am looking for a way to obtain my property names in a SSIS data flow task Script Component. I have been searching high and low only finding this. I have been trying to get this code to work, but I am too novice to understand what is happening here and I don't feel it is explained very well(no offense).
The source before this component is using a SQL query joining two tables. Inside the component, I would like to compare column to column. Then call an update method I created to use SqlConnection to perform the update.
public override void Input0_ProcessInputRow(Input0Buffer Row)
{
if (Row.TableALastName != Row.TableBLastName)
// Call the update method if the last name did not match.
this.UpdateRecord("TableBLastName", Row.TableALastName.ToString(), Row.TableAAssociateId.ToString());
}
}
private void UpdateRecord(string columnName, string change, string associateId)
{
SqlConnection sqlConnection;
sqlConnection = new SqlConnection(this.Variables.Connection);
string updateQuery = "UPDATE [SomeDataBase].[dbo].[TableB] SET " + columnName + " = " + change + " WHERE [Associate_ID] = " + associateId;
using (SqlCommand cmd2 = new SqlCommand(updateQuery, sqlConnection))
{
sqlConnection.Open();
cmd2.ExecuteNonQuery();
sqlConnection.Close();
}
}
I would like to somehow get the PropertyName of Row.TableBLastName instead of having to hard code "TableBLastName" for each test I am doing, which will be a lot.
The problem is that the input buffer class does not have Property.GetName() This also means I can't add a method to the class to get the property names, as it is regenerated each run.

public Input0_ProcessInputRow(Input0Buffer Row)
{
Dictionary<string, List<string>> list = new Dictionary<string, List<string>>();
List<string> propertyList = new List<string>();
Type myType = typeof(Input0Buffer);
PropertyInfo[] allPropInfo = myType.GetProperties();
List<PropertyInfo> SqlPropInfo = allPropInfo.Where(x => !x.Name.Contains("AM_")).ToList();
// Loop through all the Sql Property Info so those without AM_
for (int i = 0; i < SqlPropInfo.Count(); i++)
{
List<string> group = new List<string>();
foreach (var propInfo in allPropInfo)
{
if (propInfo.Name.Contains(SqlPropInfo[i].Name))
{
// Group the values based on the property
// ex. All last names are grouped.
group.Add(propInfo.GetValue(Row, null).ToString());
}
}
// The Key is the Sql's Property Name.
list.Add(SqlPropInfo[i].Name, group);
}
foreach (var item in list)
{
// Do a check if there are two values in both SQL and Oracle.
if (item.Value.Count >= 2)
{
if (item.Value.Count() != item.Value.Distinct().Count())
{
// Duplicates exist do nothing.
}
else
{
// The values are different so update the value[0]. which is the SQL Value.
UpdateRecord(item.Key, item.Value[0], Row.AssociateId);
}
}
}
}
I separated the values from the two tables so there are two lists values from TableA and TableB. You can prefix the values from TableA with "AM_" or something distinct so you can use reflection to to get the properties with and without the prefix and find out which values belong to which table. Then I just loop through the properties and group the values with the properties from the target value (so those without the prefix "AM_") I then loop through the grouped list and compare the two values and if it's different, update TableA with the TableB values to match them

You are already in SSIS so I will propose using that (no matter how quick I usually jump to C# to solve problems)
This is a classic conditional split scenario:
Do your test then run the results into a SQL Update statement.

Getting Id during sql query and using for another Insert Statment

So I have been creating a library that uses dapper and allows user to manipulate a database.
I need some help with finding the best way to achieve the following.
Lets say I have an "order" table and I have a "transaction" table and an "order_line" table.
I want to take the Increment Id of table "order" when inserting and use it to store it in a column in "transaction" and "order_line" table and I want all of this done in a SQL transaction so that I can roll back in case of any issue.
Now since my library is dynamic to any type and action, I am not sure on how to approach something like this.
Here is the code on how you would insert:
I have 2 global variables
private string connectionString { get; set; }
public void newConnection(string connection)
{
if (string.IsNullOrWhiteSpace(connectionString))
{
connectionString = connection;
}
}
private List<KeyValuePair<string, object>> transactions = new List<KeyValuePair<string, object>>();
Here is how you call to have a class to be saved to the database:
public void Add(object item)
{
string propertyNames = "";
string propertyParamaters = "";
Type itemType = item.GetType();
System.Reflection.PropertyInfo[] properties = itemType.GetProperties();
for (int I = 0; I < properties.Count(); I++)
{
if (properties[I].Name.Equals("Id", StringComparison.CurrentCultureIgnoreCase) || properties[I].Name.Equals("AutoId", StringComparison.CurrentCultureIgnoreCase))
{
continue;
}
if (I == properties.Count() - 1)
{
propertyNames += "[" + properties[I].Name + "]";
propertyParamaters += "#" + properties[I].Name;
}
else
{
propertyNames += "[" + properties[I].Name + "],";
propertyParamaters += "#" + properties[I].Name + ",";
}
}
string itemName = itemType.Name;
KeyValuePair<string, object> command = new KeyValuePair<string, object>($"Insert Into[{ itemName}] ({ propertyNames}) Values({ propertyParamaters})", item);
transactions.Add(command);
}
There are more methods and like edit, remove, edit list, remove list etc. but are not relevant in this case.
When you want to commit changes to the database you call:
public void SaveChanges()
{
using (SqlConnection sqlConnection = new SqlConnection(connectionString))
{
sqlConnection.Open();
using (SqlTransaction sqlTransaction = sqlConnection.BeginTransaction())
{
try
{
foreach (KeyValuePair<string, object> command in transactions)
{
sqlConnection.Execute(command.Key, command.Value, sqlTransaction);
}
sqlTransaction.Commit();
}
catch
{
sqlTransaction.Rollback();
throw;
}
finally
{
sqlConnection.Close();
transactions.Clear();
}
}
sqlConnection.Close();
}
transactions.Clear();
}
You can find my library at github.com
https://github.com/pietercdevries/Bamboo.Net

Can it be done... yes... should we be trying to do this ourselves... I wouldn't :) but lets try it any way.
Some ideas that can make this code simpler:
Define helper interfaces and force the data classes to implement them or use attribute declarations to specify id fields and foreign key references
Investigate Injection or code generation techniques so that you can get some of this 'dynamic' coding and lookup executed at compile time, not runtime.
I don't use Dapper and your SqlConnection.Execute() is an extension method I am not familiar with but I assume that it generates DbParameters from the passed in object and applies them to the SqlCommand when it gets executed. Hopefully dapper has some functions to extract the parameters, so that they can be used in this code example, or perhaps you can use some of these concepts and adapt them to your dapper code. I just want to acknowledge that upfront and that I have omitted any code example here that parameterises the objects when executing the commands.
This is the journey that the following snippets will go down
Prepare the generated SQL to capture the Id field
Output the Id value when we save changes
Iterate over all remaining objects in the array and set the foreign key values
Note: these code changes are not tested or exception handled for production, nor would I call this "best practice" its just to prove the concept and help out a fellow coder :)
You have already devised a convention for Id field tracking, lets extend that idea by preparing the sql statement to set the value of an output parameter:
NOTE: in MS SQL, please use SCOPE_IDENTITY() in preference to ##Identity.
What is the difference between Scope_Identity(), Identity(), ##Identity, and Ident_Current?
NOTE: because the generated statements are using parameters, and we are not yet reading the parameter values, we will not need to regenerate the saved SQL statements later after we have found an Id value to insert to other objects... phew...
public void Add(object item)
{
List<string> propertyNames = new List<string>();
Type itemType = item.GetType();
System.Reflection.PropertyInfo[] properties = itemType.GetProperties();
for (int I = 0; I < properties.Count(); I++)
{
if (properties[I].Name.Equals("Id", StringComparison.CurrentCultureIgnoreCase) || properties[I].Name.Equals("AutoId", StringComparison.CurrentCultureIgnoreCase))
{
continue;
}
propertyNames.Add(properties[I].Name);
}
string itemName = itemType.Name;
KeyValuePair<string, object> command = new KeyValuePair<string, object>
($"Insert Into[{itemName}] ({String.Join(",", propertyNames.Select(p => $"[{p}]"))}) Values({String.Join(",", propertyNames.Select(p => $"#{p}"))}); SET #OutId = SCOPE_IDENTITY();", item);
transactions.Add(command);
// Simply append your statement with a set command on an #id parameter we will add in SaveChanges()
}
In Save Changes, implement output parameter to capture the created Id, and if the Id was captured, save it back into the object that the command is associated to.
NOTE: this code snippet shows the references to the solution in item 3.
And the foreach was replaced with a for so we could do forward iterations from the current index
public void SaveChanges()
{
using (SqlConnection sqlConnection = new SqlConnection(connectionString))
{
sqlConnection.Open();
using (SqlTransaction sqlTransaction = sqlConnection.BeginTransaction())
{
try
{
for (int i = 0; i < transactions.Count; i++)
{
KeyValuePair<string, object> command = transactions[i];
// 1. Execute the command, but use an output parameter to capture the generated id
var cmd = sqlConnection.CreateCommand();
cmd.Transaction = sqlTransaction;
cmd.CommandText = command.Key;
SqlParameter p = new SqlParameter()
{
ParameterName = "#OutId",
Size = 4,
Direction = ParameterDirection.Output
};
cmd.Parameters.Add(p);
cmd.ExecuteNonQuery();
// Check if the value was set, non insert operations wil not set this parameter
// Could optimise by not preparing for the parameter at all if this is not an
// insert operation.
if (p.Value != DBNull.Value)
{
int idOut = (int)p.Value;
// 2. Stuff the value of Id back into the Id field.
string foreignKeyName = null;
SetIdValue(command.Value, idOut, out foreignKeyName);
// 3. Update foreign keys, but only in commands that we haven't execcuted yet
UpdateForeignKeys(foreignKeyName, idOut, transactions.Skip(i + 1));
}
}
sqlTransaction.Commit();
}
catch
{
sqlTransaction.Rollback();
throw;
}
finally
{
sqlConnection.Close();
transactions.Clear();
}
}
sqlConnection.Close();
}
transactions.Clear();
}
/// <summary>
/// Update the Id field of the specified object with the provided value
/// </summary>
/// <param name="item">Object that we want to set the Id for</param>
/// <param name="idValue">Value of the Id that we want to push into the item</param>
/// <param name="foreignKeyName">Name of the expected foreign key fields</param>
private void SetIdValue(object item, int idValue, out string foreignKeyName)
{
// NOTE: There are better ways of doing this, including using interfaces to define the key field expectations.
// This logic is consistant with existing code so that you are familiar with the concepts
Type itemType = item.GetType();
foreignKeyName = null;
System.Reflection.PropertyInfo[] properties = itemType.GetProperties();
for (int I = 0; I < properties.Count(); I++)
{
if (properties[I].Name.Equals("Id", StringComparison.CurrentCultureIgnoreCase) || properties[I].Name.Equals("AutoId", StringComparison.CurrentCultureIgnoreCase))
{
properties[I].SetValue(item, idValue);
foreignKeyName = $"{item.GetType().Name}_{properties[I].Name}";
break;
}
}
}
So now your objects have their Id's updated as they are inserted.
Now for the fun part... After updating the Id, you should now iterate through the other objects and update their foreign key fields.
How you go about this in reality depends a lot on what kind of assumptions/conventions you are ready enforce over the data that you are updating. For simplicity sake, lets say that all of the foreign keys that we need to update are named with the convention {ParentClassName}_{Id}.
That means that if in our example we just inserted a new 'Widget', then we can try to forcibly update all other objects in this transaction scope that have a field 'Widget_Id' (or 'Widget_AutoId')
private void UpdateForeignKeys(string foreignKeyName, int idValue, IEnumerable<KeyValuePair<string, object>> commands)
{
foreach(var command in commands)
{
Type itemType = command.Value.GetType();
var keyProp = itemType.GetProperty(foreignKeyName);
if(keyProp != null)
{
keyProp.SetValue(command.Value, idValue);
}
}
}
This is a very simplistic example of how you could go about updating foreign (or reference) keys in OPs data persistence library.
You have probably observed in reality that relational key fields are rarely consistently named using any convention, but even when conventions are followed, my simple convention would not support a table that had multiple references to parents of the same type, for example a Manifest in one of my client's apps has 3 links back to a user table:
public class Manifest
{
...
Driver_UserId { get; set; }
Sender_UserId { get; set; }
Receiver_UserId { get; set; }
...
}
You would need to evolve some pretty advanced logic to tackle all possible linkage combinations.
Some ORMs do this by setting the values as negative numbers, and decrementing the numbers each type a new type is added to the command collection. Then after an insert you only need to update key fields that held the faked negative number with the updated number. You still need to know which fields are key fields, but atleast you don't need to track the precise fields that form the ends of each relationship, we can track with the values.
I like how Entity Framework goes about it though, try inject this linkage information about the fields using attributes on the properties. You may have to invent your own, but it's a clean declarative concept that forces you to describe these relationships up front in the data model classes in a way that all sorts of logic can later take advantage of, not just for generating SQL statements.
I don't want tobe too critical of Dapper, but once you start to go down this path or manually managing referential integrity like this there is a point where you should consider a more enterprise ready ORM like Entity Framework or nHibernate. Sure they come with some baggage but those ORMs have really evolved into mature products that have been optimised by the community. I now have very little manually written or scripted code to customise any interactions with the RDBMS at all, which means much less code to test or maintain. (= less bugs)

It doesn't say which database you are using. If it is MSSQL you can do
var id = connection.Query<int?>("SELECT ##IDENTITY").SingleOrDefault();
after executing the Insert. That gives you the id of the last insert.

Best practice to check if DataRow contains a certain column

At the moment, when I iterate over the DataRow instances, I do this.
foreach(DataRow row in table)
return yield new Thingy { Name = row["hazaa"] };
Sooner of later (i.e. sooner), I'll get the table to be missing the column donkey and the poo will hit the fan. After some extensive googling (about 30 seconds) I discovered the following protection syntax.
foreach(DataRow row in table)
if(row.Table.Columns.Contains("donkey"))
return yield new Thingy { Name = row["hazaa"] };
else
return null;
Now - is this the simplest syntax?! Really? I was expecting a method that gets me the field if it exists or null otherwise. Or at least a Contains method directly on the row.
Am I missing something? I'll be mapping in many fields that way so the code will look dreadfully unreadable...

You can create an extension method to make it cleaner:
static class DataRowExtensions
{
public static object GetValue(this DataRow row, string column)
{
return row.Table.Columns.Contains(column) ? row[column] : null;
}
}
Now call it like below:
foreach(DataRow row in table)
return yield new Thingy { Name = row.GetValue("hazaa") };

As your DataTable table always has the same columns ( they won`t change for any row ) you only need to check for the columnname once.
if (table.Columns.Contains("donkey"))
{
foreach ...
}

I really liked the approach taken by #Varun K. So, having that as a departing point I just wanted to put my two cents, in case it helps someone else. I simply improved it making it generic instead of just using object as a return type.
static class Extensions
{
public static T Get<T>(this DataRow self, string column)
{
return self.Table.Columns.Contains(column)
? (T)self[column]
: default(T);
}
}
}

To build on the answer by Varun K, use a generic type parameter:
public static T GetValue<T>(this DataRow row, string column)
{
if (!row.Table.Columns.Contains(column))
return default(T);
object value = row[ColumnName];
if (value == DBNull.Value)
return default(T);
return (T)value;
}

foreach (DataColumn item in row.Table.Columns)
{
switch (item.ColumnName)
{
case "ID":
{
p.ID = Convert.ToInt32(row[item.ColumnName].ToString());
}
break;
case "firstName":
{
p.firstName = row[item.ColumnName].ToString();
}
break;
case "lastName":
{
p.lastName = row[item.ColumnName].ToString();
}
break;
default:
break;
};
}

Sometimes a column name might exist, but a row does not contain the data for that column; for example, after filling DataTable using ReadXML.
A simple, fast and secure solution would be to use type checking:
if(row["columnname"].GetType() != typeof(System.DBNull)){
//DataRow contains "columname"
}else{
//a safe scope to set default cell data
}

C# DataRow Empty-check

I got this:
DataTable dtEntity = CreateDataTable();
drEntity = dtEntity.NewRow();
Then I add data to the row (or not).
Lots of code, really don't know if there's anything inside the row.
Depends on the input (i am importing from some files).
I'd like to do something like:
if (drEntity`s EVERY CELL IS NOT EMPTY)
{
dtEntity.Rows.Add(drEntity);
}
else
{
//don't add, will create a new one (drEntity = dtEntity.NewRow();)
}
Is there some nice way to check if the DataRow's every cell is empty?
Or I should foreach, and check them one by one?

A simple method along the lines of:
bool AreAllColumnsEmpty(DataRow dr)
{
if (dr == null)
{
return true;
}
else
{
foreach(var value in dr.ItemArray)
{
if (value != null)
{
return false;
}
}
return true;
}
}
Should give you what you're after, and to make it "nice" (as there's nothing as far as I'm aware, in the Framework), you could wrap it up as an extension method, and then your resultant code would be:
if (datarow.AreAllColumnsEmpty())
{
}
else
{
}

I created an extension method (gosh I wish Java had these) called IsEmpty as follows:
public static bool IsEmpty(this DataRow row)
{
return row == null || row.ItemArray.All(i => i is DBNull);
}
The other answers here are correct. I just felt mine lent brevity in its succinct use of Linq to Objects. BTW, this is really useful in conjunction with Excel parsing since users may tack on a row down the page (thousands of lines) with no regard to how that affects parsing the data.
In the same class, I put any other helpers I found useful, like parsers so that if the field contains text that you know should be a number, you can parse it fluently. Minor pro tip for anyone new to the idea. (Anyone at SO, really? Nah!)
With that in mind, here is an enhanced version:
public static bool IsEmpty(this DataRow row)
{
return row == null || row.ItemArray.All(i => i.IsNullEquivalent());
}
public static bool IsNullEquivalent(this object value)
{
return value == null
|| value is DBNull
|| string.IsNullOrWhiteSpace(value.ToString());
}
Now you have another useful helper, IsNullEquivalent which can be used in this context and any other, too. You could extend this to include things like "n/a" or "TBD" if you know that your data has placeholders like that.

I prefer approach of Tommy Carlier, but with a little change.
foreach (DataColumn column in row.Table.Columns)
if (!row.IsNull(column))
return false;
return true;
I suppose this approach looks more simple and cleaner.

public static bool AreAllCellsEmpty(DataRow row)
{
if (row == null) throw new ArgumentNullException("row");
for (int i = row.Table.Columns.Count - 1; i >= 0; i--)
if (!row.IsNull(i))
return false;
return true;
}

I know this has been answered already and it's an old question, but here's an extension method to do the same:
public static class DataExtensions
{
public static bool AreAllCellsEmpty(this DataRow row)
{
var itemArray = row.ItemArray;
if(itemArray==null)
return true;
return itemArray.All(x => string.IsNullOrWhiteSpace(x.ToString()));
}
}
And you use it like so:
if (dr.AreAllCellsEmpty())
// etc

You could use this:
if(drEntity.ItemArray.Where(c => IsNotEmpty(c)).ToArray().Length == 0)
{
// Row is empty
}
IsNotEmpty(cell) would be your own implementation, checking whether the data is null or empty, based on what type of data is in the cell. If it's a simple string, it could end up looking something like this:
if(drEntity.ItemArray.Where(c => c != null && !c.Equals("")).ToArray().Length == 0)
{
// Row is empty
}
else
{
// Row is not empty
}
Still, it essentially checks each cell for emptiness, and lets you know whether all cells in the row are empty.

DataTable.NewRow will initialize each field to:
the default value for each DataColumn (DataColumn.DefaultValue)
except for auto-increment columns (DataColumn.AutoIncrement == true), which will be initialized to the next auto-increment value.
and expression columns (DataColumn.Expression.Length > 0) are also a special case; the default value will depend on the default values of columns on which the expression is calculated.
So you should probably be checking something like:
bool isDirty = false;
for (int i=0; i<table.Columns.Count; i++)
{
if (table.Columns[i].Expression.Length > 0) continue;
if (table.Columns[i].AutoIncrement) continue;
if (row[i] != table.Columns[i].DefaultValue) isDirty = true;
}
I'll leave the LINQ version as an exercise :)

AFAIK, there is no method that does this in the framework. Even if there was support for something like this in the framework, it would essentially be doing the same thing. And that would be looking at each cell in the DataRow to see if it is empty.

I did it like this:
var listOfRows = new List<DataRow>();
foreach (var row in resultTable.Rows.Cast<DataRow>())
{
var isEmpty = row.ItemArray.All(x => x == null || (x!= null && string.IsNullOrWhiteSpace(x.ToString())));
if (!isEmpty)
{
listOfRows.Add(row);
}
}

Maybe a better solution would be to add an extra column that is automatically set to 1 on each row. As soon as there is an element that is not null change it to a 0.
then
If(drEntitity.rows[i].coulmn[8] = 1)
{
dtEntity.Rows.Add(drEntity);
}
else
{
//don't add, will create a new one (drEntity = dtEntity.NewRow();)
}

To delete null and also empty entries Try this
foreach (var column in drEntitity.Columns.Cast<DataColumn>().ToArray())
{
if (drEntitity.AsEnumerable().All(dr => dr.IsNull(column) | string.IsNullOrEmpty( dr[column].ToString())))
drEntitity.Columns.Remove(column);
}