We Keep Coding

FlatFile library, delimited layout, wrong parsing when multiple fields are empty at the end of the row

We use in some of our applications the FlatFile library (https://github.com/forcewake/FlatFile) to parse some files delimited with separator (";"), since a lot of time without problems.
We faced yesterday a problem receiving files having multiple fields empty at the end of the row.
I replicated the problem with short console application to show and permit you to verify in a simple way:
using FlatFile.Delimited;
using FlatFile.Delimited.Attributes;
using FlatFile.Delimited.Implementation;
using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
namespace FlatFileTester
{
class Program
{
static void Main(string[] args)
{
var layout = GetLayout();
var factory = new DelimitedFileEngineFactory();
using (MemoryStream ms = new MemoryStream())
using (FileStream file = new FileStream(#"D:\shared\dotnet\FlatFileTester\test.csv", FileMode.Open, FileAccess.Read))
{
byte[] bytes = new byte[file.Length];
file.Read(bytes, 0, (int)file.Length);
ms.Write(bytes, 0, (int)file.Length);
var flatFile = factory.GetEngine(layout);
ms.Position = 0;
List<TestObject> records = flatFile.Read<TestObject>(ms).ToList();
foreach(var record in records)
{
Console.WriteLine(string.Format("Id=\"{0}\" - DescriptionA=\"{1}\" - DescriptionB=\"{2}\" - DescriptionC=\"{3}\"", record.Id, record.DescriptionA, record.DescriptionB, record.DescriptionC));
}
}
Console.ReadLine();
}
public static IDelimitedLayout<TestObject> GetLayout()
{
IDelimitedLayout<TestObject> layout = new DelimitedLayout<TestObject>()
.WithDelimiter(";")
.WithQuote("\"")
.WithMember(x => x.Id)
.WithMember(x => x.DescriptionA)
.WithMember(x => x.DescriptionB)
.WithMember(x => x.DescriptionC)
;
return layout;
}
}
[DelimitedFile(Delimiter = ";", Quotes = "\"")]
public class TestObject
{
[DelimitedField(1)]
public int Id { get; set; }
[DelimitedField(2)]
public string DescriptionA { get; set; }
[DelimitedField(3)]
public string DescriptionB { get; set; }
[DelimitedField(4)]
public string DescriptionC { get; set; }
}
}
This is an example of file:
1;desc1;desc1;desc1
2;desc2;desc2;desc2
3;desc3;;desc3
4;desc4;desc4;
5;desc5;;
So the first 4 rows are parsed as expected:
All fields with values in the first and second row
empty string for third field of third row
empty string for fouth field of fourth row
in the fifth row we expect empty string on third and fourth field, like this:
Id=5
DescriptionA="desc5"
DescriptionB=""
DescriptionC=""
instead we receive this:
Id=5
DescriptionA="desc5"
DescriptionB=";" // --> THE SEPARATOR!!!
DescriptionC=""
We can't understand if is a problem of configuration, bug of the library, or some other problem in the code...
Anyone have some similar experiences with this library, or can note some problem in the code above not linked with the library but causing the error...?

I took a look and debug the source code of the open source library: https://github.com/forcewake/FlatFile.
It seems there's a problem, in particular in this case, in witch there are 2 empty fields, at the end of a row, the bug take effects on the field before the last of the row.
I opened an issue for this libray, hoping some contributor of the library could invest some time to investigate, and, if it is so, to fix: https://github.com/forcewake/FlatFile/issues/80
For now we decided to fix the wrong values of the list, something like:
string separator = ",";
//...
//...
//...
records.ForEach(x => {
x.DescriptionC = x.DescriptionC.Replace(separator, "");
});
For our case, anyway, it make not sense to have a character corresponding to the separator as value of that field...
...even if it would be better to have bug fixing of the library

Adding support for dynamic file loaders in c#

If I have a Console App in C# that reads in files of a certain format and converts them to business objects, I design this by having an IReader interface so that I can support different formats, eg XML, CSV, pipe delimited etc, and have different concrete classes for each file format.
If the requirement is to be able to load new file readers (new formats) in dynamically without having to recompile, is there a way I can accomplish that?
The only way I can think of is somehow using XSD or reg expressions but it seems to me there should be a better solution

This sounds like you want a plugin mechanism for loading your IReaders dynamically. There are plenty of examples out there.
Simple plugin mechanism sample
SO discussion

You could use reflection. Each implementation of IReader could go in a distinct DLL. You would also create an Attribute to tag each implementation of IReader that states which file format it handles.
public sealed class InputFormatAttribute : Attribute
{
private string _format;
public string Format
{
get { return format; }
}
public InputFormatAttribute(string format)
{
_format = format;
}
}
[InputFormat("CSV")]
public class CSVReader : IReader
{
// your CSV parsing code here
public BusinessObject Parse(string file)
{}
}
BusinessObject LoadFile(string fileName)
{
BusinessObject result = null;
DirectoryInfo dirInfo = new DirectoryInfo(Path.GetDirectoryName(System.Reflection.Assembly.GetExecutingAssembly().Location));
FileInfo[] pluginList = dirInfo.GetFiles("*.DLL");
foreach (FileInfo plugin in pluginList)
{
System.Reflection.Assembly assem = System.Reflection.Assembly.LoadFile(fileInfo.FullName);
Type[] types = assem.GetTypes();
Type type = types.First(t => t.BaseType == "IReader");
object[] custAttrib = type.GetCustomAttributes(typeof(InputFormatAttribute), false);
InputFormatAttribute at = (InputFormatAttribute)custAttrib[0];
if (at.Format.Equals(Path.GetExtension(fileName).Substring(1), StringComparison.CurrentCultureIgnoreCase))
{
IReader reader = (IReader)assem.CreateInstance(type.FullName);
return reader.Parse(fileName);
}
}
// got here because not matching plugin found
return null;
}

Depends on the complicity of Readers, you may decide to use CodeDom to let someone to write the code directly. Short example:
// create compiler
CodeDomProvider provider = CSharpCodeProvider.CreateProvider("C#");
CompilerParameters options = new CompilerParameters();
// add more references if needed
options.ReferencedAssemblies.Add("system.dll");
options.GenerateExecutable = false;
options.GenerateInMemory = true;
// compile the code
string source = "using System;namespace Bla {public class Blabla { public static bool Test { return false; }}}";
CompilerResults result = provider.CompileAssemblyFromSource(options, source);
if (!result.Errors.HasErrors)
{
Assembly assembly = result.CompiledAssembly;
// instance can be saved and then reused whenever you need to run the code
var instance = assembly.CreateInstance("Bla.Blabla");
// running some method
MethodInfo method = instance.GetType().GetMethod("Test"));
var result = (bool)method.Invoke(_formulas, new object[] {});
}
But probably you'll have to provide sort of editor or accomplish the task partially (so only necessary code have to be written).

csscript unloading assembly used with an interface

I am trying to execute cs scripts in a directory a loop. Every time the script changed (or if it's new) it gets loaded and executed. But I receive an error on trying to load the script a second time:
Access to the path 'C:\Users\Admin\AppData\Local\Temp\CSSCRIPT\Cache\647885655\hello.cs.compiled' is denied.
What I tried to do was:
static Dictionary<string, string> mFilePathFileHashes = new Dictionary<string, string>();
public static void LoadFromDir(string dir)
{
foreach (string filepath in Directory.GetFiles(dir))
{
string hash = GetMD5HashFromFile(filepath); //Generate file hash
if (mFilePathFileHashes.Contains(new KeyValuePair<string, string>(filepath, hash))) continue; //Skip if it hasn't changed
if (mFilePathFileHashes.ContainsKey(filepath))
{ //Hash changed
mFilePathFileHashes[filepath] = hash;
}
else //This is the first time this file entered the loop
mFilePathFileHashes.Add(filepath, hash);
//Load the script
IScript script = CSScript.Load(filepath)
.CreateInstance("Script")
.AlignToInterface<IScript>();
//Do stuff
script.AddUserControl();
}
protected static string GetMD5HashFromFile(string fileName)
{
FileStream file = new FileStream(fileName, FileMode.Open);
MD5 md5 = new MD5CryptoServiceProvider();
byte[] retVal = md5.ComputeHash(file);
file.Close();
StringBuilder sb = new StringBuilder();
for (int i = 0; i < retVal.Length; i++)
{
sb.Append(retVal[i].ToString("x2"));
}
return sb.ToString();
}
At the "Load the script" part it would throw the error. So I read up on it a bit and tried this:
//Load the script
string asmFile = CSScript.Compile(filepath, null, false);
using (AsmHelper helper = new AsmHelper(asmFile, "temp_dom_" + Path.GetFileName(filepath), true))
{
IScript script = helper.CreateAndAlignToInterface<IScript>("Script");
script.AddUserControl();
//helper.Invoke("Script.AddUserControl");
}
Because that page said Script is loaded in the temporary AppDomain and unloaded after the execution. To set up the AsmHelper to work in this mode instantiate it with the constructor that takes the assembly file name as a parameter
But that won't Align to the interface : Type 'Script' in Assembly 'hello.cs, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null' is not marked as serializable. What does that even mean, and why would it need to be serializable?
If I switch to the helper.Invoke line instead I get a NullReferenceException.
The script:
using System;
using System.Windows.Forms;
using CSScriptTest;
class Script : CSScriptTest.IScript
{
public void AddUserControl()
{
Form1.frm.AddUserControl1(this, "test_uc_1");
}
}
So that last error may be because I never actually Aligned to an interface, or because I am calling a static method from outside of the main AppDomain (I really wouldn't know).
Is there any way to get this working?

Well, it works by passing the object I want to operate on to the interface' method like this:
using (var helper = new AsmHelper(CSScript.Compile(filepath), null, false))
{
IScript script = helper.CreateAndAlignToInterface<IScript>("Script");
script.AddUserControl(Form1.frm);
}
With the script inheriting from MarshalByRefObject like so:
using System;
using System.Windows.Forms;
using CSScriptTest;
class Script : MarshalByRefObject, CSScriptTest.IScript
{
public void AddUserControl(CSScriptTest.Form1 host)
{
host.AddUserControl1(this, "lol2");
}
}
MSDN sais MarshalByRefObject Enables access to objects across application domain boundaries in applications that support remoting. So I guess that makes sense.. but is there any way for me to expose my main application's methods to the scripts?
It doesn't seem to be possible by inheriting from MarshalByRefObject in the main program, like so:
public class CTestIt : MarshalByRefObject
{
public static CTestIt Singleton;
internal static void SetSingleton()
{ //This method is successfully executed before we start loading scripts
Singleton = new CTestIt();
Console.WriteLine("CTestIt Singleton set");
}
public static void test()
{
//Null reference when a script calls CSScriptTest.CTestIt.test();
Singleton.test_member();
}
public void test_member()
{
Console.WriteLine("test");
}
}

.NET assemblies: understanding type visibility

I am trying to reproduce something that System.Xml.Serialization already does, but for a different source of data.
For now task is limited to deserialization only.
I.e. given defined source of data that I know how to read. Write a library that takes a random type, learns about it fields/properties via reflection, then generates and compiles "reader" class that can take data source and an instance of that random type and writes from data source into the object's fields/properties.
here is a simplified extract from my ReflectionHelper class
public class ReflectionHelper
{
public abstract class FieldReader<T>
{
public abstract void Fill(T entity, XDataReader reader);
}
public static FieldReader<T> GetFieldReader<T>()
{
Type t = typeof(T);
string className = GetCSharpName(t);
string readerClassName = Regex.Replace(className, #"\W+", "_") + "_FieldReader";
string source = GetFieldReaderCode(t.Namespace, className, readerClassName, fields);
CompilerParameters prms = new CompilerParameters();
prms.GenerateInMemory = true;
prms.ReferencedAssemblies.Add("System.Data.dll");
prms.ReferencedAssemblies.Add(Assembly.GetExecutingAssembly().GetModules(false)[0].FullyQualifiedName);
prms.ReferencedAssemblies.Add(t.Module.FullyQualifiedName);
CompilerResults compiled = new CSharpCodeProvider().CompileAssemblyFromSource(prms, new string[] {source});
if (compiled.Errors.Count > 0)
{
StringWriter w = new StringWriter();
w.WriteLine("Error(s) compiling {0}:", readerClassName);
foreach (CompilerError e in compiled.Errors)
w.WriteLine("{0}: {1}", e.Line, e.ErrorText);
w.WriteLine();
w.WriteLine("Generated code:");
w.WriteLine(source);
throw new Exception(w.GetStringBuilder().ToString());
}
return (FieldReader<T>)compiled.CompiledAssembly.CreateInstance(readerClassName);
}
private static string GetFieldReaderCode(string ns, string className, string readerClassName, IEnumerable<EntityField> fields)
{
StringWriter w = new StringWriter();
// write out field setters here
return #"
using System;
using System.Data;
namespace " + ns + #".Generated
{
public class " + readerClassName + #" : ReflectionHelper.FieldReader<" + className + #">
{
public void Fill(" + className + #" e, XDataReader reader)
{
" + w.GetStringBuilder().ToString() + #"
}
}
}
";
}
}
and the calling code:
class Program
{
static void Main(string[] args)
{
ReflectionHelper.GetFieldReader<Foo>();
Console.ReadKey(true);
}
private class Foo
{
public string Field1 = null;
public int? Field2 = null;
}
}
The dynamic compilation of course fails because Foo class is not visible outside of Program class. But! The .NET XML deserializer somehow works around that - and the question is: How?
After an hour of digging System.Xml.Serialization via Reflector I came to accept that I lack some kind of basic knowledge here and not really sure what am I looking for...
Also it is entirely possible that I am reinventing a wheel and/or digging in a wrong direction, in which case please do speak up!

You don’t need to create a dynamic assembly and dynamically compile code in order to deserialise an object. XmlSerializer does not do that either — it uses the Reflection API, in particular it uses the following simple concepts:
Retrieving the set of fields from any type
Reflection provides the GetFields() method for this purpose:
foreach (var field in myType.GetFields(BindingFlags.Instance | BindingFlags.Public | BindingFlags.NonPublic))
// ...
I’m including the BindingFlags parameter here to ensure that it will include non-public fields, because otherwise it will return only public ones by default.
Setting the value of a field in any type
Reflection provides the function SetValue() for this purpose. You call this on a FieldInfo instance (which is returned from GetFields() above) and give it the instance in which you want to change the value of that field, and the value to set it to:
field.SetValue(myObject, myValue);
This is basically equivalent to myObject.Field = myValue;, except of course that the field is identified at runtime instead of compile-time.
Putting it all together
Here is a simple example. Notice you need to extend this further to work with more complex types such as arrays, for example.
public static T Deserialize<T>(XDataReader dataReader) where T : new()
{
return (T) deserialize(typeof(T), dataReader);
}
private static object deserialize(Type t, XDataReader dataReader)
{
// Handle the basic, built-in types
if (t == typeof(string))
return dataReader.ReadString();
// etc. for int and all the basic types
// Looks like the type t is not built-in, so assume it’s a class.
// Create an instance of the class
object result = Activator.CreateInstance(t);
// Iterate through the fields and recursively deserialize each
foreach (var field in t.GetFields(BindingFlags.Instance | BindingFlags.Public | BindingFlags.NonPublic))
field.SetValue(result, deserialize(field.FieldType, dataReader));
return result;
}
Notice I had to make some assumptions about XDataReader, most notably that it can just read a string like that. I’m sure you’ll be able to change it so that it works with your particular reader class.
Once you’ve extended this to support all the types you need (including int? in your example class), you can deserialize an object by calling:
Foo myFoo = Deserialize<Foo>(myDataReader);
and you can do this even when Foo is a private type as it is in your example.

If I try to use sgen.exe (the standalone XML serialization assembly compiler), I get the following error message:
Warning: Ignoring 'TestApp.Program'.
- TestApp.Program is inaccessible due to its protection level. Only public types can be processed.
Warning: Ignoring 'TestApp.Program+Foo'.
- TestApp.Program+Foo is inaccessible due to its protection level. Only public types can be processed.
Assembly 'c:\...\TestApp\bin\debug\TestApp.exe' does not contain any types that can be serialized using XmlSerializer.
Calling new XmlSerializer(typeof(Foo)) in your example code results in:
System.InvalidOperationException: TestApp.Program+Foo is inaccessible due to its protection level. Only public types can be processed.
So what gave you the idea that XmlSerializer can handle this?
However, remember that at runtime, there are no such restrictions. Trusted code using reflection is free to ignore access modifiers. This is what .NET binary serialization is doing.
For example, if you generate IL code at runtime using DynamicMethod, then you can pass skipVisibility = true to avoid any checks for visibility of fields/classes.

I've been working a bit on this. I'm not sure if it will help but, anyway I think it could be the way. Recently I worked with Serialization and DeSerealization of a class I had to send over the network. As there were two different programs (the client and the server), at first I implemented the class in both sources and then used serialization. It failed as the .Net told me it had not the same ID (I'm not sure but it was some sort of assembly id).
Well, after googling a bit I found that it was because the serialized class was on different assemblies, so the solution was to put that class in a independent library and then compile both client and server with that library. I've used the same idea with your code, so I put both Foo class and FieldReader class in a independent library, let's say:
namespace FooLibrary
{
public class Foo
{
public string Field1 = null;
public int? Field2 = null;
}
public abstract class FieldReader<T>
{
public abstract void Fill(T entity, IDataReader reader);
}
}
compile it and add it to the other source (using FooLibrary;)
this is the code I've used. It's not exactly the same as yours, as I don't have the code for GetCSharpName (I used t.Name instead) and XDataReader, so I used IDataReader (just for the compiler to accept the code and compile it) and also change EntityField for object
public class ReflectionHelper
{
public static FieldReader<T> GetFieldReader<T>()
{
Type t = typeof(T);
string className = t.Name;
string readerClassName = Regex.Replace(className, #"\W+", "_") + "_FieldReader";
object[] fields = new object[10];
string source = GetFieldReaderCode(t.Namespace, className, readerClassName, fields);
CompilerParameters prms = new CompilerParameters();
prms.GenerateInMemory = true;
prms.ReferencedAssemblies.Add("System.Data.dll");
prms.ReferencedAssemblies.Add(Assembly.GetExecutingAssembly().GetModules(false)[0].FullyQualifiedName);
prms.ReferencedAssemblies.Add(t.Module.FullyQualifiedName);
prms.ReferencedAssemblies.Add("FooLibrary1.dll");
CompilerResults compiled = new CSharpCodeProvider().CompileAssemblyFromSource(prms, new string[] { source });
if (compiled.Errors.Count > 0)
{
StringWriter w = new StringWriter();
w.WriteLine("Error(s) compiling {0}:", readerClassName);
foreach (CompilerError e in compiled.Errors)
w.WriteLine("{0}: {1}", e.Line, e.ErrorText);
w.WriteLine();
w.WriteLine("Generated code:");
w.WriteLine(source);
throw new Exception(w.GetStringBuilder().ToString());
}
return (FieldReader<T>)compiled.CompiledAssembly.CreateInstance(readerClassName);
}
private static string GetFieldReaderCode(string ns, string className, string readerClassName, IEnumerable<object> fields)
{
StringWriter w = new StringWriter();
// write out field setters here
return #"
using System;
using System.Data;
namespace " + ns + ".Generated
{
public class " + readerClassName + #" : FieldReader<" + className + #">
{
public override void Fill(" + className + #" e, IDataReader reader)
" + w.GetStringBuilder().ToString() +
}
}";
}
}
by the way, I found a tiny mistake, you should use new or override with the Fill method, as it is abstract.
Well, I must admit that GetFieldReader returns null, but at least the compiler compiles it.
Hope that this will help you or at least it guides you to the good answer
regards

Return byte array from C# to VBScript via COM interop

I have C# method that returns a byte array I want to be able to access from VBScript. More or less:
namespace ClassLibrary7
{
[ClassInterface(ClassInterfaceType.AutoDual)]
[Guid("63A77D29-DB8C-4733-91B6-3CC9C2D1340E")]
[ComVisible(true)]
public class Class1
{
public void Create(
out byte[] BinaryData
)
{
// do some work and return BinaryData
BinaryData = new byte[] { 1, 2, 3, 4 };
}
}
}
and the vbscript to look like:
dim o
dim b
set o = wscript.CreateObject("ClassLibrary7.Class1")
o.Create b
MsgBox ubound(b)
I'm lost. Google doesn't want to cooperate... and I'm hoping someone here can help!
Rob

This should help:
COM Interop Part 2: C# Server Tutorial
http://msdn.microsoft.com/en-us/library/aa645738(VS.71).aspx
and this:
Creating a COM server with .NET. C#
http://codebetter.com/blogs/peter.van.ooijen/archive/2005/08/02/130157.aspx