We Keep Coding

How can I handle mismatch between schema and model after transformation?

Exploring ML.Net and I want to predict employee turnover. I have a dataset available, with a mix between numeric and string values.
This is all just purely exploration in my attempt in getting to know ML.net. So my approach was to, simply step by step explore the options, so I really would understand each and every step as good as possible.
Load the data
Prepare the dataset and do a categorical transform on the string features
Display the dataset after applying the transformations
Then split the dataset into a train and test dataset
Train the model with a classification algorithm
Evaluate against the test dataset
Output the feature weights of the model
Do some cool stuff with it
The model is as follows and based on the open source attrition dataset from IBM. https://www.kaggle.com/pavansubhasht/ibm-hr-analytics-attrition-dataset
The model:
public class Employee
{
[LoadColumn(0)]
public int Age { get; set; }
[LoadColumn(1)]
//[ColumnName("Label")]
public string Attrition { get; set; }
[LoadColumn(2)]
public string BusinessTravel { get; set; }
[LoadColumn(3)]
public int DailyRate { get; set; }
[LoadColumn(4)]
public string Department { get; set; }
[LoadColumn(5)]
public int DistanceFromHome { get; set; }
[LoadColumn(6)]
public int Education { get; set; }
[LoadColumn(7)]
public string EducationField { get; set; }
[LoadColumn(8)]
public int EmployeeCount { get; set; }
[LoadColumn(9)]
public int EmployeeNumber { get; set; }
[LoadColumn(10)]
public int EnvironmentSatisfaction { get; set; }
[LoadColumn(11)]
public string Gender { get; set; }
[LoadColumn(12)]
public int HourlyRate { get; set; }
[LoadColumn(13)]
public int JobInvolvement { get; set; }
[LoadColumn(14)]
public int JobLevel { get; set; }
[LoadColumn(15)]
public string JobRole { get; set; }
[LoadColumn(16)]
public int JobSatisfaction { get; set; }
[LoadColumn(17)]
public string MaritalStatus { get; set; }
[LoadColumn(18)]
public int MonthlyIncome { get; set; }
[LoadColumn(19)]
public int MonthlyRate { get; set; }
[LoadColumn(20)]
public int NumCompaniesWorked { get; set; }
[LoadColumn(21)]
public string Over18 { get; set; }
[LoadColumn(22)]
public string OverTime { get; set; }
[LoadColumn(23)]
public int PercentSalaryHike { get; set; }
[LoadColumn(24)]
public int PerformanceRating{ get; set; }
[LoadColumn(25)]
public int RelationshipSatisfaction{ get; set; }
[LoadColumn(26)]
public int StandardHours{ get; set; }
[LoadColumn(27)]
public int StockOptionLevel{ get; set; }
[LoadColumn(28)]
public int TotalWorkingYears{ get; set; }
[LoadColumn(29)]
public int TrainingTimesLastYear{ get; set; }
[LoadColumn(30)]
public int WorkLifeBalance{ get; set; }
[LoadColumn(31)]
public int YearsAtCompany{ get; set; }
[LoadColumn(32)]
public int YearsInCurrentRole{ get; set; }
[LoadColumn(33)]
public int YearsSinceLastPromotion{ get; set; }
[LoadColumn(34)]
public int YearsWithCurrManager { get; set; }
}
The string properties are then transformed (as explained here https://learn.microsoft.com/en-us/dotnet/machine-learning/how-to-guides/prepare-data-ml-net#work-with-categorical-data)
var categoricalEstimator = mlContext.Transforms.Categorical.OneHotEncoding("Attrition")
.Append(mlContext.Transforms.Categorical.OneHotEncoding("BusinessTravel"))
.Append(mlContext.Transforms.Categorical.OneHotEncoding("EducationField"))
.Append(mlContext.Transforms.Categorical.OneHotEncoding("Gender"))
.Append(mlContext.Transforms.Categorical.OneHotEncoding("JobRole"))
.Append(mlContext.Transforms.Categorical.OneHotEncoding("MaritalStatus"))
.Append(mlContext.Transforms.Categorical.OneHotEncoding("Over18"))
.Append(mlContext.Transforms.Categorical.OneHotEncoding("OverTime"));
ITransformer categoricalTransformer = categoricalEstimator.Fit(dataView);
IDataView transformedData = categoricalTransformer.Transform(dataView);
Now I want to inspect what has changed (https://learn.microsoft.com/en-us/dotnet/machine-learning/how-to-guides/inspect-intermediate-data-ml-net#convert-idataview-to-ienumerable). The challenge I have now is that after applying a transformation on the string properties, the schema has changed and now contains the expected vectors.
So the following is happening. The Employee model schema does not match the schema from the transformedData object anymore and tries to fit a Vector property into a String property and throws the following error "Can't bind the IDataView column 'Attrition' of type 'Vector' to field or property 'Attrition' of type 'System.String'."
IEnumerable<Employee> employeeDataEnumerable =
mlContext.Data.CreateEnumerable<Employee>(transformedData, reuseRowObject: true);
The CreateEnumerable also has a SchemaDefinition argument, so my first guess was to extract the Schema from the transformedData, and supply that to the CreateEnumerable. However it expects a Microsoft.ML.DataViewSchema and the schema produced by the transform is a Microsoft.ML.Data.SchemaDefinition. So that didn't work either.
I hope someone can advice me on this. Should I do something different?
Full Controller Action:
public ActionResult Turnover()
{
MLContext mlContext = new MLContext();
var _appPath = AppDomain.CurrentDomain.BaseDirectory;
var _dataPath = Path.Combine(_appPath, "Datasets", "WA_Fn-UseC_-HR-Employee-Attrition.csv");
// Load data from file
IDataView dataView = mlContext.Data.LoadFromTextFile<Employee>(_dataPath, hasHeader: true);
// 0. Get the column name of input features.
string[] featureColumnNames =
dataView.Schema
.Select(column => column.Name)
.Where(columnName => columnName != "Label")
.ToArray();
// Define categorical transform estimator
var categoricalEstimator = mlContext.Transforms.Categorical.OneHotEncoding("Attrition")
.Append(mlContext.Transforms.Categorical.OneHotEncoding("BusinessTravel"))
.Append(mlContext.Transforms.Categorical.OneHotEncoding("EducationField"))
.Append(mlContext.Transforms.Categorical.OneHotEncoding("Gender"))
.Append(mlContext.Transforms.Categorical.OneHotEncoding("JobRole"))
.Append(mlContext.Transforms.Categorical.OneHotEncoding("MaritalStatus"))
.Append(mlContext.Transforms.Categorical.OneHotEncoding("Over18"))
.Append(mlContext.Transforms.Categorical.OneHotEncoding("OverTime"));
ITransformer categoricalTransformer = categoricalEstimator.Fit(dataView);
IDataView transformedData = categoricalTransformer.Transform(dataView);
// Inspect (fails because Employee (35 cols) cannot be mapped to new schema (52 cols)
IEnumerable<Employee> employeeDataEnumerable =
mlContext.Data.CreateEnumerable<Employee>(transformedData, reuseRowObject: true, schemaDefinition : transformedData.Schema);
// split the transformed dataset into training and a testing datasets
DataOperationsCatalog.TrainTestData dataSplit = mlContext.Data.TrainTestSplit(transformedData, testFraction: 0.2);
IDataView trainData = dataSplit.TrainSet;
IDataView testData = dataSplit.TestSet;
return View();
}

I ran into this recently and as a quick workaround, I simply created a new class that matches the transformed data schema. For example, you can create EmoloyeeTransformed class with the correct properties (i.e. vector instead of string) and use that as follows:
CreateEnumerable<EmployeeTransformed>
This isnt optimal if you are going to create various transformed schemas, but it works.
Hope that helps.

For debugging purposes you can also call transformedData.Preview() and look at the data and the resulting Schema.

How do I iteratively assign to a nested list within an object in C#?

I've been asked to replicate a complex xml structure we use within our internal systems, against data retrieved from another system. Unfortunately the xml structure is improvised and we have no specification for it.
I've been mapping out it's structure in C# classes so that I can assign those properties with values from the database and ultimately serialise it as xml.
I've hit a bit of a road block in terms of iteratively adding new list items to a nested list within an object that's already being intialised as a list and being looped through. To make matters more complicated I need to use a value from the iteration to filter down the dataset being used to instantiate the second loop round.
Sorry for the poor explanation - open to rewording... hopefully however the example I've written will demonstrate what I'm trying to do:
using System;
using System.Collections.Generic;
using System.Data;
public class TransactionModel
{
public int Id { get; set; }
public string Description { get; set; }
public DateTime SysDate { get; set; }
public List<TransactionItemModel> Trade { get; set; } = new List<TransactionItemModel>();
}
public class TransactionItemModel
{
public int ItemId { get; set; }
public int TransactionId { get; set; }
public string ItemDescription { get; set; }
public decimal ItemNetAmount { get; set; }
}
public class Program
{
public void Main()
{
DataTable tranResultSet = MethodToReturnResultsFromTranQuery();
DataTable itemResultSet = MethodToReturnResultsFromItemQuery();
var transactions = new List<TransactionModel>();
foreach (DataRow tran in tranResultSet.Rows)
{
transactions.Add(
new TransactionModel() {
Id = (dynamic)tran["Id"],
Description = (dynamic)tran["Description"],
SysDate = (dynamic)tran["SysDate"],
//Trade = <Stuck Here>
// Need to iterate through itemResultSet, adding to TransactionModel.Trade
// where item["TransactionId"] = tran["Id"]
}
);
}
}
}

This approach doesn't set the Trade collection initially, but populates it once you go through the Items. There's likely a lot of optimization that can be added, but this might get you started.
public class TransactionModel
{
public int Id { get; set; }
public string Description { get; set; }
public DateTime? SysDate { get; set; }
public List<TransactionItemModel> Trade { get; set; }
public TransactionModel(DataRow row)
{
if(row == null)
throw new ArgumentNullException(nameof(row));
Id = row.Field<int>("Id");
Description = row.Field<string>("Description");
SysDate = row.Field<DateTime?>("SysDate");
Trade = new List<TransactionItemModel>();
}
}
public class TransactionItemModel
{
public int ItemId { get; set; }
public int TransactionId { get; set; }
public string ItemDescription { get; set; }
public decimal? ItemNetAmount { get; set; }
public TransactionItemModel(DataRow row)
{
if(row == null)
throw new ArgumentNullException(nameof(row));
ItemId = row.Field<int>("Id");
TransactionId = row.Field<int>("TransactionId");
ItemDescription = row.Field<string>("ItemDescription");
ItemNetAmount = row.Field<decimal?>("ItemNetAmount");
}
}
public static class Program
{
private static void Main()
{
DataTable tranResultSet = MethodToReturnResultsFromTranQuery();
DataTable itemResultSet = MethodToReturnResultsFromItemQuery();
var transactions = tranResultSet.AsEnumerable()
.Select(r => new TransactionModel(r));
foreach(TransactionModel transaction in transactions)
{
var items = itemResultSet.AsEnumerable()
.Where(r => r.Field<int>("TransactionId") == transaction.Id)
.Select(r => new TransactionItemModel(r));
transaction.Trade.AddRange(items);
}
}
}
It's likely going to be ideal to query your ItemResultSet based on the current TransactionId instead of grabbing them all up front. You could implement a DataReader, or use Dapper.

Multiple .Include in Repository

At first, here what I have:
4 Classes have some relations between them as:
Project has many Bridges.
Bridge has many Foundations.
Foundation has many PierCaps and many PierColumns.
Project.cs:
public class Project
{
public Project()
{
Bridges = new HashSet<Bridge>();
StartDate = DateTime.Now;
EndDate = DateTime.Now;
}
// Fields
public DateTime StartDate { get; set; }
public DateTime EndDate { get; set; }
// Calculated Fields
public int BridgeCount => Bridges.Count;
public int PierFoundationCount = 0;
public int PierCapCount = 0;
public int PierColumnCount = 0;
// Relation To Children
public virtual ICollection<Bridge> Bridges { get; private set; }
}
Bridge.cs:
public class Bridge : BasicBaseEntityWithName
{
public Bridge()
{
PierFoundations = new HashSet<PierFoundation>();
}
// Fields
public int ProjectID { set; get; }
public Project Project { set; get; }
// Calculated Fields
public int PierFoundationCount => PierFoundations.Count;
public int PierCapCount => PierFoundations.Count;
public int PierColumnCount => PierFoundations.Count;
// Relation To Children
public ICollection<PierFoundation> PierFoundations { get; private set; }
}
PierFoundation.cs:
public class PierFoundation : BasicBaseEntityWithName
{
public PierFoundation()
{
PierCaps = new HashSet<PierCap>();
PierColumns = new HashSet<PierColumn>();
}
// Fields
public float Length { set; get; }
// Calculated Fields
public int PierColumnCount => PierColumns.Count;
public int PierCapCount => PierCaps.Count;
public float FoundationQty => (Length * Width * Height);
// Relations To Parents
public int BridgeID { set; get; }
public Bridge Bridge { set; get; }
// Relations To Children
public ICollection<PierCap> PierCaps { get; set; }
public ICollection<PierColumn> PierColumns { get; set; }
}
PierColumn.cs:
public class PierColumn : BaseEntityNoName
{
// Fields
public float Diameter { set; get; }
// Relations To Parents
public int PierFoundationID { set; get; }
public PierFoundation PierFoundation { set; get; }
}
PierCap.cs:
public class PierCap : BaseEntityNoName
{
// Fields
public float CapWidth { set; get; }
// Relations To Parents
public int PierFoundationID { set; get; }
public PierFoundation PierFoundation { set; get; }
}
BaseEntityNoName.cs:
public class BaseEntityNoName
{
public BaseEntityNoName()
{
AddedBy = Core.General.Global.CurrentUseId;
AddedDate = DateTime.Now;
}
public int Id { get; set; }
public DateTime AddedDate { get; set; }
public int AddedBy { get; set; }
// Becuase they will not appear at new, they will be nullable
public DateTime? ModifiedDate { get; set; }
public int? ModifiedBy { get; set; }
}
BasicBaseEntityWithName.cs:
public class BasicBaseEntityWithName : BaseEntityNoName
{
public string Name { get; set; }
}
Adding to that, I have this method in my Repository:
public IEnumerable<TEntity> Include(params Expression<Func<TEntity, object>>[] includeExpressions)
{
DbSet<TEntity> dbSet = Context.Set<TEntity>();
IEnumerable<TEntity> query = null;
foreach (var includeExpression in includeExpressions)
{
query = dbSet.Include(includeExpression);
}
return query ?? dbSet;
}
My questions are:
Q1. When I use Include the related entities (PierFoundation, PierCap, PierColumn), I don't get them all!. So, when I type:
PierFoundations = CurrentUnitOfWork.PierFoundationRepository.Include(x => x.PierColumns, x => x.PierCaps);
I received just PierFoundation and PierCap, but not PierColumn
What is the wrong here?. I want to retrieve all data in the all three of them.
Q2. In the Project class, I have four calculate fields:
BridgeCount which is solved by Bridges.Count
PierFoundationCount
PierCapCount
PierColumnCount
The last three, I don't know how to get them, because they don't have direct relation with Project
BTW: I am using EF Core 2

The two issues are unrelated, so answering only the main (as indicated by the post title) Q1:
When I use Include the related entities (PierFoundation, PierCap, PierColumn), I don't get them all!.
It's caused by the following line in your repository method implementation:
query = dbSet.Include(includeExpression);
As with many LINQ method, it's important to use the Include method return value when chaining with other calls, and as you can see, this code stores only the last Include call, hence the behavior you are observing.
It can be fixed by changing the implementation like this:
var query = Context.Set<TEntity>().AsQueryable();
foreach (var includeExpression in includeExpressions)
query = query.Include(includeExpression); // Note the usage of query variable
return query;
or simply
return includeExpresions.Aggregate(Context.Set<T>().AsQueryable(), (q, e) => q.Include(e));
But note that EF Core uses different approach for including nested data levels (ThenInclude) which cannot be expressed with Expression<Func<T, object>>, so you might consider exposing IQueryable<TEntity> returning method from your repository and use the EF Core provided Include / ThenInclude extension methods.
What about Q2, consider posting it in a separate SO question (post).

How to go about combining two objects, manipulate the data, then separate them?

I have two classes with some similar fields, some different, and a form that utilizes two different objects depending on what mode it's in (insert/edit).
Instead of using two different objects and if statements checking the form mode, I'd like to have one struct to be hydrated with either of the two objects fields so I can manipulate one object through the page life-cycle. Then separated the struct back to its respective object for insert/updating the DB.
Example of classes:
public partial class SomeClass
{
public Int32 B {get;set;}
public String C {get;set;}
public Boolean D {get;set;}
}
public class SomeOtherClass
{
public Int32 A {get;set;}
public Int32 B {get;set;}
public String C {get;set;}
}
Update with Solution Example:
public interface IInsertable
{
string SharedName { get; set; }
string SharedID { get; set; }
string editedFieldValue { get; set; }
long GetSuperSecreteInfo();
}
internal class InsertableImplementation : IInsertable
{
public string SharedName { get; set; }
public string SharedID { get; set; }
public string editedFieldValue { get; set; }
public long GetSuperSecreteInfo()
{
return -1;
}
}
public interface IUpdateable
{
string SharedName { get; set; }
string SharedID { get; set; }
string updatedFieldValue { get; set; }
Guid GenerateStevesMagicGuid();
}
internal class UpdateableImplementation : IUpdateable
{
public string SharedName { get; set; }
public string SharedID { get; set; }
public string updatedFieldValue { get; set; }
public Guid GenerateStevesMagicGuid()
{
return new Guid();
}
}
public static class WonderTwinFactory
{
public static WonderTwins GenerateWonderTwin(IUpdateable updateable, IInsertable insertable)
{
var wt = new WonderTwins();
// who will win?
wt.SharedID = updateable.SharedID;
wt.SharedID = insertable.SharedID;
// you decide?
wt.SharedName = updateable.SharedName;
wt.editedFieldValue = "stuff";
return wt;
}
}
public class WonderTwins : IInsertable, IUpdateable
{
public string SharedName { get; set; }
public string SharedID { get; set; }
public string editedFieldValue { get; set; }
public long GetSuperSecreteInfo()
{
return 1;
}
public string updatedFieldValue { get; set; }
public Guid GenerateStevesMagicGuid()
{
return new Guid();
}
}
class Program
{
static void Main(string[] args)
{
IUpdateable updateable = new UpdateableImplementation();
IInsertable insertable = new InsertableImplementation();
WonderTwins dualImplementatin = WonderTwinFactory.GenerateWonderTwin(updateable, insertable);
IUpdateable newUpdateable = dualImplementatin as IUpdateable;
IInsertable newInsertable = dualImplementatin as IInsertable;
}
}

Have both classes implement an interface that defines the operations common to each, including both the fields that are shared (assuming the view needs to access them) and also a method to actually perform the operation that they represent (insert/edit).

Other way of doing such things is using C# dynamic object and assign properties directly. It may help to avoid any new type or interface and directly utilizing new dynamic object any time, as much as required.
var newObject = new {
objectOfClass1 = x.prop1,
objectOfClass2 = x.prop2
}

jQuery datatable in MVC (server-side)

https://datatables.net/usage/server-side
On the page above, there are parameters that you need to receive to make server-side datatable work.
I have a helper class
public class TableParameter
{
public string sEcho { get; set; }
public int iDisplayStart { get; set; }
public int iDisplayLength { get; set; }
public int iSortingCols { get; set; }
}
But in order to sort columns I need to receive
string sSortDir_(int)
How do I do that? I know (int) represents column ID that needs to be sorted, but I just can't catch it in my controller.

The datatable will post one or more sSortDir_x parameters to your controller, depending on how many columns are sorted on simultaneously in the table.
The specific columns that the table is sorted by are sent in the iSortCol_ parameters (again, one or more).
public class TableParameter
{
public string sEcho { get; set; }
public int iDisplayStart { get; set; }
public int iDisplayLength { get; set; }
public int iSortingCols { get; set; }
public int iSortCol_0 { get; set; } // the first (and usually only) column to be sorted by
public string sSortDir_0 { get; set; } // the direction of the first column sort (asc/desc)
public int iSortCol_1 { get; set; } // the second column to be sorted by
public string sSortDir_1 { get; set; } // the direction of the second column sort
// etc
}

For receiveing a column name in action, that is used for one-column sorting:
public ActionResult SomeMethod(FormCollection coll)
{
var sortingColumnNumber = Convert.ToInt32(coll["iSortCol_0"]);
var sortingColumnName = coll[string.Format("mDataProp_{0}", sortingColumnNumber)];
var propertyInfo = typeof(SomeObject).GetProperty(sortingColumnName);
//..get List<SomeObject> sortedObjects
sortedObjects = sortedObjects.OrderBy(x => propertyInfo.GetValue(x, null)).ToList();
//...
}