We Keep Coding

From child class, call base class method while using child fields

I have created a small ETL program with a single method that only takes a connection string for the source database and a connection string for a target database.
Basically, it has 5 steps:
Step 1. Select data from the source.
Step 2. Create a temporary table on the target.
Step 3. Load data from the source into the temporary on the target.
Step 4. Merge the data from temporary table into the actual target table.
Step 5. Drop the temporary table
This works great for a single transformation that needs to take place, but I have about 20 different ETL "jobs" that need to take place.
So instead of copying and pasting the same method verbatim 19 different times, I would like to define a base class that defines this single method one time, and then call this single method from each child class, with its own select, create, merge and drop statements.
Is this possible?
Base Class:
public class ETLBase
{
private static string Select;
private static string CreateTemp;
private static string Merge;
private static string CleanUp;
private static string DestinationTable;
public static void ExecuteJob(string sourceConnectionString, string destinationConnectionString)
{
using (OracleConnection sourceConnection = new OracleConnection(sourceConnectionString))
{
sourceConnection.Open();
OracleCommand selectCommand = new OracleCommand(Select, sourceConnection);
OracleDataReader reader = selectCommand.ExecuteReader();
using (SqlConnection destinationConnection = new SqlConnection(destinationConnectionString))
{
destinationConnection.Open();
SqlCommand createTempCommand = new SqlCommand(CreateTemp, destinationConnection);
createTempCommand.ExecuteNonQuery();
SqlCommand mergeCommand = new SqlCommand(Merge, destinationConnection);
SqlCommand dropCommand = new SqlCommand(CleanUp, destinationConnection);
using (SqlBulkCopy bulkCopy = new SqlBulkCopy(destinationConnection))
{
bulkCopy.DestinationTableName = DestinationTable;
try
{
bulkCopy.WriteToServer(reader);
mergeCommand.ExecuteNonQuery();
dropCommand.ExecuteNonQuery();
}
catch (Exception ex)
{
Console.WriteLine(ex.Message);
}
finally
{
reader.Close();
}
}
}
}
}
}
Child Class:
public class ChildETL: ETLBase
{
private static string Select = #"Select THIS DataStatement";
private static string CreateTemp = #"CREATE TABLE Statement";
private static string Merge = #"Merge Table statement";
private static string CleanUp = "DROP TABLE Statement";
private static string DestinationTable = "##TempTable";
}
And then execute it something like this, but where each child class uses its own defined fields, so it uses it's own SQL statements.
public class Program
{
public static void Main(string[] args)
{
IConfiguration config = new ConfigurationBuilder()
.AddJsonFile("appsettings.json", true, true)
.Build();
string schoolDBConnection = config["SchoolConnection"];
string courseDBConnection = config["CourseConnection"];
string teacherDBConnection = config["Connection"];
ChildETLA.ExecuteJob(schoolDBConnection, courseDBConnection);
ChildETLB.ExecuteJob(teacherDBConnection, courseDBConnection);
//...and so on for each child ETL class
}
}

First of all, you need to run "ExecuteJob()" polymorphically, which means behaviour of "ExeceuteJob" will be different based on source and destination connection string. You can't achieve polymorphism for static function or properties. First of all, you need to refactor it to instance method by taking out the "Static" keywords. Also, the behaviour of the base class is deciding by child classes so that nobody should be able to create an object of base class to make it an abstract class. Look at your code you have two behaviours one for school and another one for the teacher. So you have to create two different child classes which inherit the abstract base class. It is the responsibility of object creation to compose object with source and destination connection string so pass it to the constructor and set it while creating the object itself. Please find the refactored code,
public abstract class ETLBase
{
private readonly string sourceConnectionString;
private readonly string destinationConnectionString;
protected virtual string Select { get; set; } = #"Select THIS DataStatement";
protected virtual string CreateTemp { get; set; } = #"CREATE TABLE Statement";
protected virtual string Merge { get; set; } = #"Merge Table statement";
protected virtual string CleanUp { get; set; } = "DROP TABLE Statement";
protected virtual string DestinationTable { get; set; } = "##TempTable";
protected ETLBase(string sourceConnectionString, string destinationConnectionString)
{
this.sourceConnectionString = sourceConnectionString;
this.destinationConnectionString = destinationConnectionString;
}
public void ExecuteJob()
{
using (OracleConnection sourceConnection = new OracleConnection(sourceConnectionString))
{
sourceConnection.Open();
OracleCommand selectCommand = new OracleCommand(Select, sourceConnection);
OracleDataReader reader = selectCommand.ExecuteReader();
using (SqlConnection destinationConnection = new SqlConnection(destinationConnectionString))
{
destinationConnection.Open();
SqlCommand createTempCommand = new SqlCommand(CreateTemp, destinationConnection);
createTempCommand.ExecuteNonQuery();
SqlCommand mergeCommand = new SqlCommand(Merge, destinationConnection);
SqlCommand dropCommand = new SqlCommand(CleanUp, destinationConnection);
using (SqlBulkCopy bulkCopy = new SqlBulkCopy(destinationConnection))
{
bulkCopy.DestinationTableName = DestinationTable;
try
{
bulkCopy.WriteToServer(reader);
mergeCommand.ExecuteNonQuery();
dropCommand.ExecuteNonQuery();
}
catch (Exception ex)
{
Console.WriteLine(ex.Message);
}
finally
{
reader.Close();
}
}
}
}
}
And the child classes,
public class ChildETLSchool : ETLBase
{
public ChildETLSchool(string sourceConnectionString, string destinationConnectionString)
: base(sourceConnectionString, destinationConnectionString)
{
//Change values of below lines only if you want to override the values
//Select = #"Select THIS DataStatement";
//CreateTemp = #"CREATE TABLE Statement";
//Merge = #"Merge Table statement";
//CleanUp = "DROP TABLE Statement";
//DestinationTable = "##TempTable";
}
}
public class ChildETLTeacher : ETLBase
{
public ChildETLTeacher(string sourceConnectionString, string destinationConnectionString)
: base(sourceConnectionString, destinationConnectionString)
{
//Change values of below lines only if you want to override the values
//Select = #"Select THIS DataStatement";
//CreateTemp = #"CREATE TABLE Statement";
//Merge = #"Merge Table statement";
//CleanUp = "DROP TABLE Statement";
//DestinationTable = "##TempTable";
}
}
And the object creation in main function,
static void Main(string[] args)
{
IConfiguration config = new ConfigurationBuilder()
.AddJsonFile("appsettings.json", true, true)
.Build();
string schoolDBConnection = config["SchoolConnection"];
string courseDBConnection = config["CourseConnection"];
string teacherDBConnection = config["Connection"];
var childETLSchool = new ChildETLSchool(schoolDBConnection, courseDBConnection);
var childETLTeacher = new ChildETLTeacher(teacherDBConnection, courseDBConnection);
childETLSchool.ExecuteJob();
childETLTeacher.ExecuteJob();
}

Migrate data using FluentMigrator

I use FluentMigrator in my project.
And now I Add new column in table and how I can update data in this column by SQL query?
public override void Up()
{
Alter.Table("Images").AddColumn("Item_id").AsInt32().Nullable();
//do something like "Update Images img set img.Item_id=(Select i.Id
//from Items i where i.Image=img.Id)"
}
public override void Down()
{
Delete.Column("Item_id").FromTable("Images");
}

You can either use Insert.IntoTable:
Insert.IntoTable("Users").Row(new { Username = "CURRENT_USER" });
or Execute.WithConnectionto insert data using FluentMigrator

Hi I believe You can make it simpler for Your case, but I had a more complex issue. I added a new column of varbinary type, where I needed to keep a complex object and I needed to prepare it in c# code. So I couldn't just execute UPDATE combined with SELECT command.
I solved that by adding a static class where I execute SQL SELECT command, then I use retrieved data during migration to prepare UPDATE commands.
The important clue is to use ConnectionString which is accessible directly from FluentMigrator.Migration class and execute own SQL commands to get data, then process it in C# and then simply use Update.Table method of FluentMigrator.
Here is my code (simplified):
DatabaseHelper class:
/// <summary>
/// Helper class for executing SQL on a Database defined by given ConnectionString
/// </summary>
public static class DatabaseHelper
{
/// <summary>
/// Run SQL SELECT on a Database defined by given ConnectionString
/// </summary>
/// <param name="connectionString">Connection string</param>
/// <param name="sql">Full SQL SELECT command to be executed</param>
/// <returns>List of rows containing all columns as string[] array</returns>
public static string[][] SelectList(string connectionString, string sql)
{
using (SqlConnection sqlConnection = new SqlConnection($"{connectionString}"))
{
try
{
sqlConnection.Open();
SqlCommand command = new SqlCommand(sql, sqlConnection);
SqlDataReader reader = command.ExecuteReader();
if (!reader.HasRows)
return null;
List<string[]> rowsList = new List<string[]>();
//save all rows (including all columns) from the response to the list
while (reader.Read())
{
//every row has n columns
var row = new string[reader.FieldCount];
//fill every column
for (int i = 0; i < reader.FieldCount; i++)
{
row[i] = reader[i].ToString();
}
//add row to the list
rowsList.Add(row);
}
reader.Close();
sqlConnection.Close();
return rowsList.ToArray();
}
catch (Exception e)
{
if (sqlConnection.State == ConnectionState.Open)
sqlConnection.Close();
Console.WriteLine(e);
throw;
}
}
}
}
Migration class:
public class Migration123 : Migration
{
public override void Up()
{
Create.Column("NEW_COLUMN").OnTable("TABLE_NAME").AsCustom("NVARCHAR(255)").Nullable()
//migrate data from OLD_COLUMN to NEW_COLUMN
MigrateData();
...
}
private void MigrateData()
{
string[][] rowsArray = DatabaseHelper.SelectList(ConnectionString,"SELECT [ID],[OLD_COLUMN] FROM [TABLE_NAME]");
//if nothing to migrate then exit
if (rowsArray == null)
return;
foreach (var row in rowsArray)
{
//prepare a value which will be inserted into a new column, basing on old columns value
var someNewValueForNewColumn = row[1] + " (modified)";
//insert a value into a new column
Update.Table("TABLE_NAME").Set(new
{
NEW_COLUMN = someNewValueForNewColumn
}).Where(new
{
ID = row[0]
});
}
}
public override void Down()
{
...
}
}
Hope it helps somebody!

public class <MigrationClassName> : Migration
{
private readonly ScriptResourceManager _sourceManager;
public <MigrationClassName>(
ScriptResourceManager sourceManager)
{
_sourceManager = sourceManager;
}
public override void Up()
{
MigrateData();
}
private void MigrateData()
{
var script = _sourceManager
.Read("<FileName>.sql");
Execute.Sql(script);
}
}
using System.IO;
public class ScriptResourceManager
{
public string Read(string name)
{
var assembly = typeof(ScriptResourceManager).Assembly;
var resourcesBasePath = typeof(ScriptResourceManager).Namespace;
var resourcePath = $"{resourcesBasePath}.{name}";
using var stream = assembly.GetManifestResourceStream(resourcePath);
using var reader = new StreamReader(stream);
return reader.ReadToEnd();
}
}
SQL FILE
UPDATE table_name
SET column1 = value1, column2 = value2, ...
WHERE condition;

Reduce Multiple Calls To Database

I have a code a base that iterates through a list of records and does the follwing
'select * from Table to see if fields exist
then later in the interation
'select * from Table to retreive some data
then farther down in the interation
select field1, field2 from Table to get certain pieces of information
I need to do all of these functions for each record. Would the process speed up if I called the query once for each record and hold the data in a datatable and retreive what I need from there? Or is there another more efficient way to not have to make 3 db calls to the same table which will speed up the process?

You can cache query data to System.Data.DataTable. To simplify things I has written CMyDynaset class, which populates DataTable with data from DB. Here how to use it for example with MySQL:
using System;
using System.Data.Common;
using MySql.Data.MySqlClient;
namesapce MyProg
{
class Program
{
private const string strMyConnection = "Host=localhost;Database=mydb;User Id=myuser;Password=mypsw";
static void Main(string[] args)
{
using (MySqlConnection myConnection = new MySqlConnection(strMyConnection))
{
using (MyDb.CMyDynaset dyn = new MyDb.CMyDynaset(myConnection, MySqlClientFactory.Instance))
{
// populate dyn.Table (System.Data.DataTable)
dyn.Init("select * from Table");
dyn.Load();
// access fields
foreach (DataColumn column in dyn.Table.Columns)
{
// ...
}
// get data
long nCountAll = dyn.Table.Rows.Count; // rows count
foreach (DataRow row in dyn.Table.Rows)
{
Object val1 = row[1]; // acess by index
Object val2 = row["id"]; // acess by name
// ...
}
// update data
dyn.Table.Rows[0]["name"] = "ABC";
dyn.Update();
}
}
}
}
}
CMyDynaset class (CMyDynaset.cs):
// CMyDynaset.cs
using System;
using System.Data.Common;
namespace MyDb
{
/// <summary>
/// Summary description for CMyDynaset.
/// </summary>
public class CMyDynaset : IDisposable
{
public System.Data.DataTable Table = null;
// private
private DbConnection myConnection = null;
private DbProviderFactory myFactory = null;
private DbDataAdapter dataAdap = null;
private DbCommandBuilder cmdBld = null;
private bool bIsSchema = false;
public CMyDynaset(DbConnection conn, DbProviderFactory factory)
{
this.myConnection = conn;
this.myFactory = factory;
}
#region IDisposable Members
public void Dispose()
{
if (this.Table != null)
{
this.Table.Dispose();
this.Table = null;
}
if (this.cmdBld != null)
{
this.cmdBld.Dispose();
this.cmdBld = null;
}
if (this.dataAdap != null)
{
this.dataAdap.Dispose();
this.dataAdap = null;
}
// This object will be cleaned up by the Dispose method.
// Therefore, you should call GC.SupressFinalize to
// take this object off the finalization queue
// and prevent finalization code for this object
// from executing a second time.
GC.SuppressFinalize(this);
}
#endregion
// Init
public void Init(string strSelect)
{
DbCommand cmdSel = this.myConnection.CreateCommand();
cmdSel.CommandText = strSelect;
this.dataAdap = this.myFactory.CreateDataAdapter();
this.dataAdap.SelectCommand = cmdSel;
this.cmdBld = this.myFactory.CreateCommandBuilder();
this.cmdBld.DataAdapter = this.dataAdap;
this.Table = new System.Data.DataTable();
// schema
this.bIsSchema = false;
}
public void AddParameter(string name, object value)
{
DbParameter param = this.dataAdap.SelectCommand.CreateParameter();
param.ParameterName = name;
param.Value = value;
this.dataAdap.SelectCommand.Parameters.Add(param);
}
public void AddParameter(DbParameter param)
{
this.dataAdap.SelectCommand.Parameters.Add(param);
}
// Open, Close
private void Open(ref bool bClose)
{
if (this.myConnection.State == System.Data.ConnectionState.Closed)
{
this.myConnection.Open();
bClose = true;
}
if (!this.bIsSchema)
{ // schema
this.dataAdap.FillSchema(this.Table, System.Data.SchemaType.Mapped);
this.bIsSchema = true;
}
}
private void Close(bool bClose)
{
if (bClose)
this.myConnection.Close();
}
// Load, Update
public void Load()
{
bool bClose = false;
try
{
this.Table.Clear();
this.Open(ref bClose);
this.dataAdap.Fill(this.Table);
}
catch (System.Exception ex)
{
throw ex;
}
finally
{
this.Close(bClose);
}
}
public void Update()
{
bool bClose = false;
try
{
this.Open(ref bClose);
this.dataAdap.Update(this.Table);
}
catch (System.Exception ex)
{
throw ex;
}
finally
{
this.Close(bClose);
}
}
}
}

saving object to database c#

I'm having class Report and class Program, what I want to accomplish (no luck so far) is to send data from class Program and method SaveRep() to class Report method Save() and save it in this method.
I apologize if the question is badly formulated, I am really stuck at this, please help. Thanks
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.IO;
using System.Configuration;
using System.Data.SqlClient;
namespace Application
{
class Program
{
static void Main(string[] args)
{
//call method SaveRep
}
public void SaveRep(...)
{
int RepID = 1;
string Data1 = "SomeData1"
string Data2 = "SomeData2"
//This method should send the data above to method Save() in Report class
//data to this method will be provided from another method.
}
}
public class Report
{
private static int _repID;
public int RepID
{
get { return _repID; }
set { _repID = value; }
}
private static string _data1;
public string Data1
{
get { return _data1; }
set { _data1 = value; }
}
private static string __data2;
public string Data1
{
get { return _data2; }
set { _data2 = value; }
}
public void Save()
{
string strConnectionString = (#"server=(local)\sqlexpress;Integrated Security=True;database=DataBase");
SqlConnection connection = new SqlConnection(strConnectionString);
connection.Open();
// This method should save all data (RepID,Data1,Data2)
// to the DB, provided to her by SaveRep method from Program class.
// Properties are columns from a database
}
}
}

public void Save()
{
string yourConnString="Replace with Your Database ConnectionString";
using(SqlConnection connection = new SqlConnection(yourConnString))
{
string sqlStatement = "INSERT Table1(Data1) VALUES(#Data1)";
using(SqlCommand cmd = new SqlCommand(sqlStatement, connection))
{
cmd.CommandType = CommandType.Text;
cmd.Parameters.AddWithValue("#Data1",Data1);
//add more parameters as needed and update insert query
try
{
connection.Open();
cmd.ExecuteNonQuery();
}
catch(Exception ex)
{
//log error
}
}
}
}
Assuming your Data1 is the name of your Column name. Update the sqlStatement to have your relevant columns.
Now in SaveRep, you can call it like
public void SaveRep()
{
Report objReport=new Report();
objReport.Data1="Something somethinng";
objReport.Save();
}

string connectionString = ....; //adjust your connection string
string queryString = "INSERT INTO ....;"; //Adjust your query
using (SqlConnection connection = new SqlConnection(
connectionString))
{
using( SqlCommand command = new SqlCommand(
queryString, connection))
{
connection.Open();
command .ExecuteNonQuery();
}
}

One tiny thing that the previous speakers have not noticed, is that your Report class is severely broken. Why did you use static keywords on the variables? Do not do that! Well, unless you know what it does, and here I have the feeling you don't. Update your Report class to look like this:
public class Report
{
public int RepID {get;set;}
public string Data1 {get;set;}
public string Data2 {get;set;}
public void Save()
{
// what others said
}
}
'static' in your original code makes all te variables global/shared. That means that if you'd create 500 Report objects with new, they all would have the same one ID, Data1 and Data2. Any change to any single of those 500 objects would cause all the objects to change. But it's be misleading, the objects would not change. They would simply have the same data. Looking at the "ID" field, I think you'd rather want a separate objects with separate data for separate records..

LINQ Database

I have a system that supports multiple products. Each product has its own database with the same exact schema.
When I pass in the connection string as a parameter to my Data Context constructor it always uses the default database listed in the connection string, or the default database of the user connecting if I do not provide an Initial Catalog in the connection string.
I would like to be able to have the system utilize a database without having to change the connection string and by passing in the database name as a parameter.
Here is an example of the code I am using:
class Program
{
static void Main(string[] args)
{
var d = new Data("Data Source=(LOCAL);Initial Catalog=Database1;Integrated Security=true;");
var d1 = new Data("Data Source=(LOCAL);Initial Catalog=Database2;Integrated Security=true;");
Console.ReadLine();
}
}
internal class Data
{
public Data(string connection)
{
using (var ctx = new DataClassDataContext(connection))
{
var query = from c in ctx.MyTable select c;
try
{
Console.WriteLine(query.Count());
}
catch (Exception ex)
{
Console.WriteLine(ex.Message);
}
}
}
}
If this code gets executed, then the first result will pull from Database1 and the second result will pull from Database2. I would like it to have the ability to pull from a database that is not provided in the connection string. The reason for this is because the database could change based on a specific scenario but the connection string will remain the same.
Here is an example of what I am using to "fake" it, but I don't really think this is the best solution for this:
class oConnection
{
public string Server { get; set; }
public string Database { get; set; }
public bool IntegratedSecurity { get; set; }
public string UserName { get; set; }
public string Password { get; set; }
}
class Program
{
static void Main(string[] args)
{
var d = new Data(new oConnection
{
Database = "Database1",
Server = "(Local)",
IntegratedSecurity = true
});
var d1 = new Data(new oConnection
{
Database = "Database2",
Server = "(Local)",
IntegratedSecurity = true
});
Console.ReadLine();
}
}
internal class Data
{
private static string BuildConnection(oConnection connection)
{
var sb = new StringBuilder();
sb.Append("Data Source=" + connection.Server + ";Initial Catalog=" + connection.Database + ";");
if(connection.IntegratedSecurity)
{
sb.Append("Integrated Security=true;");
}
else
{
sb.Append("user id=" + connection.UserName + ";password=" + connection.Password);
}
return sb.ToString();
}
public Data(oConnection connection)
{
using (var ctx = new DataClassDataContext(BuildConnection(connection)))
{
var query = from c in ctx.MyTable select c;
try
{
Console.WriteLine(query.Count());
}
catch (Exception ex)
{
Console.WriteLine(ex.Message);
}
}
}
}
Another note: the goal of this is really to be able to support not having multiple different connection strings when running queries that will span across multiple databases. For example: If I want to query the account records from a database and then query some sort of lookup data from another database, I would have to create a new connection string for the context.
Any help would be appreciated.
Thanks

Use the constructor that receives System.Data.IDbConnection connection. You can use the same connection string, and call connection.ChangeDatabase("mydb") before passing it to the constructor. Alternatively you can add a new constructor on the partial class, so the calling call doesn't has to deal with that.

you can use the SqlConnectionStringBuilder
class