We Keep Coding

SSIS Script Component Input0Buffer method no GetName()?

I am looking for a way to obtain my property names in a SSIS data flow task Script Component. I have been searching high and low only finding this. I have been trying to get this code to work, but I am too novice to understand what is happening here and I don't feel it is explained very well(no offense).
The source before this component is using a SQL query joining two tables. Inside the component, I would like to compare column to column. Then call an update method I created to use SqlConnection to perform the update.
public override void Input0_ProcessInputRow(Input0Buffer Row)
{
if (Row.TableALastName != Row.TableBLastName)
// Call the update method if the last name did not match.
this.UpdateRecord("TableBLastName", Row.TableALastName.ToString(), Row.TableAAssociateId.ToString());
}
}
private void UpdateRecord(string columnName, string change, string associateId)
{
SqlConnection sqlConnection;
sqlConnection = new SqlConnection(this.Variables.Connection);
string updateQuery = "UPDATE [SomeDataBase].[dbo].[TableB] SET " + columnName + " = " + change + " WHERE [Associate_ID] = " + associateId;
using (SqlCommand cmd2 = new SqlCommand(updateQuery, sqlConnection))
{
sqlConnection.Open();
cmd2.ExecuteNonQuery();
sqlConnection.Close();
}
}
I would like to somehow get the PropertyName of Row.TableBLastName instead of having to hard code "TableBLastName" for each test I am doing, which will be a lot.
The problem is that the input buffer class does not have Property.GetName() This also means I can't add a method to the class to get the property names, as it is regenerated each run.

public Input0_ProcessInputRow(Input0Buffer Row)
{
Dictionary<string, List<string>> list = new Dictionary<string, List<string>>();
List<string> propertyList = new List<string>();
Type myType = typeof(Input0Buffer);
PropertyInfo[] allPropInfo = myType.GetProperties();
List<PropertyInfo> SqlPropInfo = allPropInfo.Where(x => !x.Name.Contains("AM_")).ToList();
// Loop through all the Sql Property Info so those without AM_
for (int i = 0; i < SqlPropInfo.Count(); i++)
{
List<string> group = new List<string>();
foreach (var propInfo in allPropInfo)
{
if (propInfo.Name.Contains(SqlPropInfo[i].Name))
{
// Group the values based on the property
// ex. All last names are grouped.
group.Add(propInfo.GetValue(Row, null).ToString());
}
}
// The Key is the Sql's Property Name.
list.Add(SqlPropInfo[i].Name, group);
}
foreach (var item in list)
{
// Do a check if there are two values in both SQL and Oracle.
if (item.Value.Count >= 2)
{
if (item.Value.Count() != item.Value.Distinct().Count())
{
// Duplicates exist do nothing.
}
else
{
// The values are different so update the value[0]. which is the SQL Value.
UpdateRecord(item.Key, item.Value[0], Row.AssociateId);
}
}
}
}
I separated the values from the two tables so there are two lists values from TableA and TableB. You can prefix the values from TableA with "AM_" or something distinct so you can use reflection to to get the properties with and without the prefix and find out which values belong to which table. Then I just loop through the properties and group the values with the properties from the target value (so those without the prefix "AM_") I then loop through the grouped list and compare the two values and if it's different, update TableA with the TableB values to match them

You are already in SSIS so I will propose using that (no matter how quick I usually jump to C# to solve problems)
This is a classic conditional split scenario:
Do your test then run the results into a SQL Update statement.

ASP.NET SqlDataReader and Lists

I am getting data via a SqlDataReader and now looping through the results and putting the results in the list. I am trying to add 2 columns to each list, but I am unsuccessful.
Here is my code:
for (int i = 0; i < reader.FieldCount; i++)
{
List<string> costs = new List<string>();
if (reader.GetName(i).ToString().Contains("TotalCost"))
{
costs.Add(reader.GetValue(i).ToString());
}
if (reader.GetName(i).ToString().Contains("SqftCost"))
{
costs.Add(reader.GetValue(i).ToString());
}
jobList.Add(costs);
}
But this puts the two columns in separate lists, I really need the 2 columns in one list.
The reason I am doing it like this is because I have columns that are called TotalCost101, SqftCost101, TotalCost102, SqftCost102, TotalCost104, SqftCost104. So each column that contains TotalCost and SqftCost should be in its own list. I hope this makes sense, anyone got any ideas on how to put these 2 columns in their own list. So at the end I will have a bunch of lists with 2 values.
I updated my code so I now use a Class instead of a List
for (int i = 0; i < reader.FieldCount; i++)
{
CostMatrix costs = new CostMatrix();
if (reader.GetName(i).ToString().Contains("TotalCost"))
{
costs.TotalCost = reader.GetValue(i).ToString();
}
if (reader.GetName(i).ToString().Contains("SqftCost"))
{
costs.sqftCost = reader.GetValue(i).ToString();
}
jobList.Add(costs);
}
Here is the current output:
<d3p1:CostMatrix>
<d3p1:TotalCost>550</d3p1:TotalCost>
<d3p1:sqftCost i:nil="true"/>
</d3p1:CostMatrix>
<d3p1:CostMatrix>
<d3p1:TotalCost i:nil="true"/>
<d3p1:sqftCost>0.41</d3p1:sqftCost>
</d3p1:CostMatrix>
What I am looking for is:
<d3p1:CostMatrix>
<d3p1:TotalCost>550</d3p1:TotalCost>
<d3p1:sqftCost>0.41</d3p1:sqftCost>
</d3p1:CostMatrix>

Honestly, I would use an object.
public class Price
{
public decimal Sqft { get; set; }
public decimal Total { get; set; }
}
You have an object that actually represents something tangible. You're clearly indicating what type of price is applicable. This will avoid confusion with other people working on the project and for you, with an expressive usage. Nothing is being obfuscated into a Tuple or string.
Then when you use the data reader, you could do something along these lines:
public static T GetValueOrNull<T>(this IDataReader reader, string column)
{
int ordinal;
if(!string.IsNullOrEmpty(column) && !reader.IsDBNull(reader.GetOrdinal(column)))
if(int.TryParse(reader.GetOrdinal(column).ToString(), out ordinal))
return (T)reader.GetValue(ordinal);
return default(T);
}
You can basically tell this, "which column" then assign it to that property. This could also be handled by some form of object relational mapper.
// Example:
List<Product> products = db.GetProducts();
var example = products.Where(o => o.Price.Sqft >= 5.00);
var sample = products.Where(o => o.Price.Total <= 5.00);
You can store Price inside of another object, allowing a web-site to filter a product based on how multiple types of price values, for instance. It has other benefits as well, it will also document your code nicely, to know how pricing may be implemented.
Not search for a collection of strings, how would that persist throughout your application? A List<string> would be hard to find all implementations for price, unless seeking a data attribute. But these are a bunch of reasons.
Hope this clarifies a bit.
Based on your update, you could do:
public class CostMatrix
{
public ConcurrentList<decimal> Total { get; set; }
public ConcurrentList<decimal> Sqft {get; set; }
}
Your object would have two separate list, then as you read through the table column by column and row by row, you could simply add. So if you used the above static method it would be:
using(var connection = new SqlConnection(dbConnection))
using(var command = new SqlCommand(query, dbConnection))
using(var reader = new SqlDataReader())
while(reader.Read())
{
Total.Add(GetValueOrNull<decimal>(reader, "TotalCost");
Sqft.Add(GetValueOrNull<decimal>(reader, "Sqft1");
}
I placed ConcurrentList because your implementation may use async. Wasn't sure, if not you can use a normal List. But a List isn't thread safe by default, you'd need a ConcurrentList.

Getting Id during sql query and using for another Insert Statment

So I have been creating a library that uses dapper and allows user to manipulate a database.
I need some help with finding the best way to achieve the following.
Lets say I have an "order" table and I have a "transaction" table and an "order_line" table.
I want to take the Increment Id of table "order" when inserting and use it to store it in a column in "transaction" and "order_line" table and I want all of this done in a SQL transaction so that I can roll back in case of any issue.
Now since my library is dynamic to any type and action, I am not sure on how to approach something like this.
Here is the code on how you would insert:
I have 2 global variables
private string connectionString { get; set; }
public void newConnection(string connection)
{
if (string.IsNullOrWhiteSpace(connectionString))
{
connectionString = connection;
}
}
private List<KeyValuePair<string, object>> transactions = new List<KeyValuePair<string, object>>();
Here is how you call to have a class to be saved to the database:
public void Add(object item)
{
string propertyNames = "";
string propertyParamaters = "";
Type itemType = item.GetType();
System.Reflection.PropertyInfo[] properties = itemType.GetProperties();
for (int I = 0; I < properties.Count(); I++)
{
if (properties[I].Name.Equals("Id", StringComparison.CurrentCultureIgnoreCase) || properties[I].Name.Equals("AutoId", StringComparison.CurrentCultureIgnoreCase))
{
continue;
}
if (I == properties.Count() - 1)
{
propertyNames += "[" + properties[I].Name + "]";
propertyParamaters += "#" + properties[I].Name;
}
else
{
propertyNames += "[" + properties[I].Name + "],";
propertyParamaters += "#" + properties[I].Name + ",";
}
}
string itemName = itemType.Name;
KeyValuePair<string, object> command = new KeyValuePair<string, object>($"Insert Into[{ itemName}] ({ propertyNames}) Values({ propertyParamaters})", item);
transactions.Add(command);
}
There are more methods and like edit, remove, edit list, remove list etc. but are not relevant in this case.
When you want to commit changes to the database you call:
public void SaveChanges()
{
using (SqlConnection sqlConnection = new SqlConnection(connectionString))
{
sqlConnection.Open();
using (SqlTransaction sqlTransaction = sqlConnection.BeginTransaction())
{
try
{
foreach (KeyValuePair<string, object> command in transactions)
{
sqlConnection.Execute(command.Key, command.Value, sqlTransaction);
}
sqlTransaction.Commit();
}
catch
{
sqlTransaction.Rollback();
throw;
}
finally
{
sqlConnection.Close();
transactions.Clear();
}
}
sqlConnection.Close();
}
transactions.Clear();
}
You can find my library at github.com
https://github.com/pietercdevries/Bamboo.Net

Can it be done... yes... should we be trying to do this ourselves... I wouldn't :) but lets try it any way.
Some ideas that can make this code simpler:
Define helper interfaces and force the data classes to implement them or use attribute declarations to specify id fields and foreign key references
Investigate Injection or code generation techniques so that you can get some of this 'dynamic' coding and lookup executed at compile time, not runtime.
I don't use Dapper and your SqlConnection.Execute() is an extension method I am not familiar with but I assume that it generates DbParameters from the passed in object and applies them to the SqlCommand when it gets executed. Hopefully dapper has some functions to extract the parameters, so that they can be used in this code example, or perhaps you can use some of these concepts and adapt them to your dapper code. I just want to acknowledge that upfront and that I have omitted any code example here that parameterises the objects when executing the commands.
This is the journey that the following snippets will go down
Prepare the generated SQL to capture the Id field
Output the Id value when we save changes
Iterate over all remaining objects in the array and set the foreign key values
Note: these code changes are not tested or exception handled for production, nor would I call this "best practice" its just to prove the concept and help out a fellow coder :)
You have already devised a convention for Id field tracking, lets extend that idea by preparing the sql statement to set the value of an output parameter:
NOTE: in MS SQL, please use SCOPE_IDENTITY() in preference to ##Identity.
What is the difference between Scope_Identity(), Identity(), ##Identity, and Ident_Current?
NOTE: because the generated statements are using parameters, and we are not yet reading the parameter values, we will not need to regenerate the saved SQL statements later after we have found an Id value to insert to other objects... phew...
public void Add(object item)
{
List<string> propertyNames = new List<string>();
Type itemType = item.GetType();
System.Reflection.PropertyInfo[] properties = itemType.GetProperties();
for (int I = 0; I < properties.Count(); I++)
{
if (properties[I].Name.Equals("Id", StringComparison.CurrentCultureIgnoreCase) || properties[I].Name.Equals("AutoId", StringComparison.CurrentCultureIgnoreCase))
{
continue;
}
propertyNames.Add(properties[I].Name);
}
string itemName = itemType.Name;
KeyValuePair<string, object> command = new KeyValuePair<string, object>
($"Insert Into[{itemName}] ({String.Join(",", propertyNames.Select(p => $"[{p}]"))}) Values({String.Join(",", propertyNames.Select(p => $"#{p}"))}); SET #OutId = SCOPE_IDENTITY();", item);
transactions.Add(command);
// Simply append your statement with a set command on an #id parameter we will add in SaveChanges()
}
In Save Changes, implement output parameter to capture the created Id, and if the Id was captured, save it back into the object that the command is associated to.
NOTE: this code snippet shows the references to the solution in item 3.
And the foreach was replaced with a for so we could do forward iterations from the current index
public void SaveChanges()
{
using (SqlConnection sqlConnection = new SqlConnection(connectionString))
{
sqlConnection.Open();
using (SqlTransaction sqlTransaction = sqlConnection.BeginTransaction())
{
try
{
for (int i = 0; i < transactions.Count; i++)
{
KeyValuePair<string, object> command = transactions[i];
// 1. Execute the command, but use an output parameter to capture the generated id
var cmd = sqlConnection.CreateCommand();
cmd.Transaction = sqlTransaction;
cmd.CommandText = command.Key;
SqlParameter p = new SqlParameter()
{
ParameterName = "#OutId",
Size = 4,
Direction = ParameterDirection.Output
};
cmd.Parameters.Add(p);
cmd.ExecuteNonQuery();
// Check if the value was set, non insert operations wil not set this parameter
// Could optimise by not preparing for the parameter at all if this is not an
// insert operation.
if (p.Value != DBNull.Value)
{
int idOut = (int)p.Value;
// 2. Stuff the value of Id back into the Id field.
string foreignKeyName = null;
SetIdValue(command.Value, idOut, out foreignKeyName);
// 3. Update foreign keys, but only in commands that we haven't execcuted yet
UpdateForeignKeys(foreignKeyName, idOut, transactions.Skip(i + 1));
}
}
sqlTransaction.Commit();
}
catch
{
sqlTransaction.Rollback();
throw;
}
finally
{
sqlConnection.Close();
transactions.Clear();
}
}
sqlConnection.Close();
}
transactions.Clear();
}
/// <summary>
/// Update the Id field of the specified object with the provided value
/// </summary>
/// <param name="item">Object that we want to set the Id for</param>
/// <param name="idValue">Value of the Id that we want to push into the item</param>
/// <param name="foreignKeyName">Name of the expected foreign key fields</param>
private void SetIdValue(object item, int idValue, out string foreignKeyName)
{
// NOTE: There are better ways of doing this, including using interfaces to define the key field expectations.
// This logic is consistant with existing code so that you are familiar with the concepts
Type itemType = item.GetType();
foreignKeyName = null;
System.Reflection.PropertyInfo[] properties = itemType.GetProperties();
for (int I = 0; I < properties.Count(); I++)
{
if (properties[I].Name.Equals("Id", StringComparison.CurrentCultureIgnoreCase) || properties[I].Name.Equals("AutoId", StringComparison.CurrentCultureIgnoreCase))
{
properties[I].SetValue(item, idValue);
foreignKeyName = $"{item.GetType().Name}_{properties[I].Name}";
break;
}
}
}
So now your objects have their Id's updated as they are inserted.
Now for the fun part... After updating the Id, you should now iterate through the other objects and update their foreign key fields.
How you go about this in reality depends a lot on what kind of assumptions/conventions you are ready enforce over the data that you are updating. For simplicity sake, lets say that all of the foreign keys that we need to update are named with the convention {ParentClassName}_{Id}.
That means that if in our example we just inserted a new 'Widget', then we can try to forcibly update all other objects in this transaction scope that have a field 'Widget_Id' (or 'Widget_AutoId')
private void UpdateForeignKeys(string foreignKeyName, int idValue, IEnumerable<KeyValuePair<string, object>> commands)
{
foreach(var command in commands)
{
Type itemType = command.Value.GetType();
var keyProp = itemType.GetProperty(foreignKeyName);
if(keyProp != null)
{
keyProp.SetValue(command.Value, idValue);
}
}
}
This is a very simplistic example of how you could go about updating foreign (or reference) keys in OPs data persistence library.
You have probably observed in reality that relational key fields are rarely consistently named using any convention, but even when conventions are followed, my simple convention would not support a table that had multiple references to parents of the same type, for example a Manifest in one of my client's apps has 3 links back to a user table:
public class Manifest
{
...
Driver_UserId { get; set; }
Sender_UserId { get; set; }
Receiver_UserId { get; set; }
...
}
You would need to evolve some pretty advanced logic to tackle all possible linkage combinations.
Some ORMs do this by setting the values as negative numbers, and decrementing the numbers each type a new type is added to the command collection. Then after an insert you only need to update key fields that held the faked negative number with the updated number. You still need to know which fields are key fields, but atleast you don't need to track the precise fields that form the ends of each relationship, we can track with the values.
I like how Entity Framework goes about it though, try inject this linkage information about the fields using attributes on the properties. You may have to invent your own, but it's a clean declarative concept that forces you to describe these relationships up front in the data model classes in a way that all sorts of logic can later take advantage of, not just for generating SQL statements.
I don't want tobe too critical of Dapper, but once you start to go down this path or manually managing referential integrity like this there is a point where you should consider a more enterprise ready ORM like Entity Framework or nHibernate. Sure they come with some baggage but those ORMs have really evolved into mature products that have been optimised by the community. I now have very little manually written or scripted code to customise any interactions with the RDBMS at all, which means much less code to test or maintain. (= less bugs)

It doesn't say which database you are using. If it is MSSQL you can do
var id = connection.Query<int?>("SELECT ##IDENTITY").SingleOrDefault();
after executing the Insert. That gives you the id of the last insert.

How to choose the Data Stucture

I have a table of data like this.
I want to perform numerous operations on it like
How many times was PLAY 'no' when it was sunny
How many times was WINDY 'true' when PLAY was 'yes'
What data structure should I use?
right now I have simple array which is coming out to be one heck of a task to control.
string[,] trainingData = new string[noOfRecords,5];
using (TextReader reader = File.OpenText("input.txt"))
{
int i =0;
string text = "";
while ((text = reader.ReadLine()) != null)
{
string[] bits = text.Split(' ');
for (int j = 0; j < noOfColumnsOfData; j++)
{
trainingData[i, j] = bits[j];
}
i++;
}
}

To widen #Doan cuong's anwer,
I would use an enumarable list of objects.
each object can be calle: Record and the collection can be called Table.
(Table is IEnumarable).
Here is a simple example:
static void Main(string[] args)
{
Table table = new Table();
int count1 = table.records.Where(r => r.Play == false && r.Outlook.ToLower() == "sunny").Count();
}
public class Record
{
public bool Play;
public string Outlook;
}
public class Table
{
//This should be private and Table should be IEnumarable
public List<Record> records = new List<Record>();
}

this is a highly uncomfortable question because it's about opinion more then a valid programing question. a lot of programmers would say this or that. for the table you are showing there is no problem using array as you did for simple queries as you mentioned. for more complex data and queries i would suggest you'll take the time and effort to study LINQ

Create a class and write the values to properties. I.e.:
public class Weather
{
public string Outlook {get;set;}
...
}
And then store them into a List<Weather> collection (during your loop). Like already said, you can run LINQ queries on it. The internet is full of example how to use LINQ.

problem in preparing where clause of an update method in c#

hey guys, i'm finding little difficult to prepare a where clause in following update method
i'm creating a update method, here am i doin' this in a rightway ? but still m confused that how do i know that which property is to use in where clause
or any better approach to create a generic update method ? i wud be very thankfull
EDIT
public bool UpdateData(object Entity, ref String error)
{
Type objectType = Entity.GetType();
PropertyInfo[] properties = objectType.GetProperties();
error = "";
string column = null;
int i = 0;
SqlConnection conn = OpenConnection();
SqlCommand sqlcommand=null;
foreach (PropertyInfo info in properties)
{
if (i == 0)
{
i++;
continue;
}
column += (i >= 0 && i < properties.Length - 1) ? string.Format(#"{0}=#{0},", info.Name) : string.Format(#"{0}=#{0}",info.Name);
i++;
}
try
{
string sqlQuery = string.Format(#"update {0} set {1}
where {2}='{3}'", objectType.Name, column,1,1);//see here m not getting how to prepare this where clause
sqlcommand = new SqlCommand(sqlQuery, conn);
i = 0;
foreach (PropertyInfo info in properties)
{
if (i == 0)
{
i++;
continue;
}
sqlcommand.Parameters.AddWithValue(string.Format("#{0}", info.Name), info.GetValue(Entity, null));
}
sqlcommand.ExecuteNonQuery();
sqlcommand = null;
return true;
}
catch (Exception ex)
{
return false;
}
finally
{
CloseConnection(conn);
}
}
EDIT
see in above code there is an if condition if (i == 0) in foreach loop i dont want to do this way coz here i'm assuming my objects property i.e employee_id is at 1st index in array of PropertyInfo[].. what if some one makes a class properties like public string employee_name{get,set} public string employee_add{get,set} public string employee_id{get,set} in this case the foreach loop will skip employee_name instead of 'employee_id' coz i used if(i=0), i want a way to skip only identity value i.e employee_id in foreach loop in my update function irresptive of its index in propertyInfo array..... did i explained well ?

You can use a custom attribute or an interface to determine the property that represent the id of your entity and use it in your where clause. Although this approach has a negative side that you have to change your entity classes (which is not always possible).
Here's an example :
[PrimaryKey]
public string EmployeeId
{
get;set;
}
or
public class Employee:IEntity
{
public object EntityId
{
get
{
return this.EmployeeId
}
}
}
another way might be to store the entities metadata somewhere else (a xml file or a dictionary for example)
for example:
<entities>
<entity type="Employee" primarykey="EmployeeId" />
</entities>
or
Hashtable EntityPrimaryKeys=new Hashtable{(typeof(Employee),"EmployeeId")};

You really should use SQLCOmmand.Parameters instead of concatenating strings int he sql.
That way it will be more readable, secure and more feature proof.
You have a classical query exploit in your code where somebody could send in a parameter containing '; drop database yourdatabase; select * from dual where ''=' or similar.
So please update your code and if you have the same issue we will see what's going on.