We Keep Coding

Is there a better way to map multiple headers from an input file to a class model?

I receive a variety of input files (CSV, Excel) from clients that all feed into the same data model. The clients all use different headers for their data so I need to map the header options to a single field in a data model.
Example: I have a field Company in the class model and the possible headers I need to read from might be Firm, Company, CompanyName, or Business.
Right now I'm using a large repetitive switch statement to handle this based on the headers that I've seen. This is working but it's really ugly and repetitive.
var map = new Dictionary<string, string>();
var colKey = "";
switch (columnName.ToUpper())
{
case "FIRM":
case "COMPANY":
case "COMPANYNAME":
case "BUSINESS":
colKey = "COMPANY";
if (!map.ContainsKey( colKey ))
{
map.Add( colKey, columnName);
}
break;
case "ADDRESS1":
case "LINE1":
colKey = "LINE1";
if (!map.ContainsKey ( colKey ))
{
map.Add( colKey, columnName);
}
break;
}
Ultimately I need to read the data out of DataRows that may have a variety of headers into a data model. The above code gives me a dictionary I can use to access the data by a known name even though the data's header may not match my data model.
Is there a better way to map/parse these possible header options into a data model field?

As this is something you can't control I would put the mapping either in some config file or in the database. There you can change or add new mapping easily + you don't need to recompile your program every time when new header comes.
I had a similar problem and I solved it by storing mappings between my keys and client headers into the database:
MapTo MapFrom
-------------------------------------
COMPANY | FIRM,BUSINESS,COMPANYNAME |
-------------------------------------
LINE1 | LINE!,ADDRESS1 |
-------------------------------------
Now in your program you just query the database and store mappings into an Dictionary<string, string[]> where key holds MaptTo and value holds MapFrom.
You can also have some caching involved if you don't want to query the database every time you need the mapping.
When your dictionary is ready you can now easily find your match by using Linq (this is more like a pseudo code, but you will get the idea):
// load mappings - and cache them if possible
var mappings = _repo.GetMappings();
// find your match
var match = mappings.SingleOrDefault(
x => x.Value.Any(
v => v.Equals(columnName, StringComparison.OrdinalIgnoreCase)));
// take the key - you can do this also in the linq above
var colKey = match.Key;
Of course you will need to add here null checks and improve the code, but this is just for you to get the idea.
The benefit is that your clients will not ruin your code with their weird header column names and your code will be clean and maintainable.

since the columnMappings are predefined, we could have a static Dictionary with all possible column names as key, and the colKey as the value like:
static readonly Dictionary<string, string> _columnMappings = new Dictionary<string, string>();
static ClassName() //static constructor
{
_columnMappings.Add("FIRM", "COMPANY");
_columnMappings.Add("COMPANY", "COMPANY");
_columnMappings.Add("COMPANYNAME", "COMPANY");
_columnMappings.Add("BUSINESS", "COMPANY");
_columnMappings.Add("ADDRESS1", "LINE1");
_columnMappings.Add("LINE1", "LINE1");
}
And could be used like:
var columnName = "FIRM";
var map = new Dictionary<string, string>();
var colKey = string.Empty;
if (_columnMappings.TryGetValue(columnName, out colKey))
map.Add(colKey, columnName);

You can't avoid writing the mapping itself, but you can create a dictionary using dictionary initializer that will make it a bit more pretty (plus some constants):
const string companyColumnName = "COMPANY";
var map = new Dictionary<string, string>
{
{"FIRM", companyColumnName },
{"Company", companyColumnName },
// Etc.
};

Getting the values from this specific JSON

I'm trying to get all of the "last" values from this JSON here:
{"btc":{
"usd": {
"bitfinex": {
"last": "1191.60",
"volume": "1.99324e+7"
},
"bitstamp": {
"last": "1193.06",
"volume": "8.73693e+6"
},
"btce": {
"last": "1174.27",
"volume": "6.03521e+6"
}
}
}
But for some reason I can only access "btc" and "usd". I can't get anything out of it including the "last" values. Here is the code i'm using:
private string GetPrice()
{
WebClient wc = new WebClient();
var data = wc.DownloadString("http://preev.com/pulse/units:btc+usd/sources:bitfinex+bitstamp+btce");
JObject o = JObject.Parse(data);
string response = o["btc"].ToString();
return response;
}
If I change it to:
o["last"].ToString();
It just doesn't return anything. Can someone please provide me with a solution? I also tried making a key/value dict out of it and looping over each pair. Did not work.

The JObject structure is similar to a class with properties, so the first-level indexer ["btc"] returns another object that you have to query for its own properties ["usd"]
You can also opt for using JObject.SelectToken, generally not a bad idea. Other answers have shown how to chain the indexers but that's hard to read and maintain. Instead you can do:
jObj.SelectToken("btc[0].usd[0].bitstamp[0].last").ToString();
Further you can use the power of this syntax for other queries:
// a list o all the 'last' values
jObj.SelectTokens("btc.usd.*.last").Select(t=>t.ToString()).ToList();
Another advantage, if you're building a more complex system, is that you could put the queries in a config file or attributes etc to make them more manageable or deploy logic changes without rebuilding.
Yet another approach would be to build your own class structure and deserialize your json into it, so you have strongly typed values (double instead of string for the values for example)
public class btc {
public usd usd {get;set;}
}
public class usd....
var btcLoaded = JsonConvert.DeserializeObject<btc>(jsonString);
var lastBitstamp = btc.usd.bitstamp.last;

Use: o["btc"]["usd"]["bitfinex"]["last"].ToString() to get the 'last' value of 'bitfinex'.

After parsing the JSON whenever you index the o variable you are indexing from the root of the JSON. In order to access nested properties like 'last' you will need to index into the next level of the JSON as such:
var bitfinex = o["btc"]["usd"]["bitfinex"]["last"].ToString();
var bitstamp = o["btc"]["usd"]["bitstamp"]["last"].ToString();
var btce = o["btc"]["usd"]["btce"]["last"].ToString();
To reduce the repetition you could iterate over the properties under the btc.usd field.

if u want to all last values use this..
decimal[] lastValues = obj.SelectTokens("$..last").ToArray()
.Select(a => a.Parent.ToObject<decimal>()).ToArray();
if u want to dictionary, use this..
var dictionary = obj["btc"]["usd"].Select(a =>
new
{
Key = ((JProperty)a).Name,
Value = a.First["last"].ToObject<decimal>()
})
.ToDictionary(a => a.Key, a => a.Value);

How do I get the value in a generic Dictionary?

I have a json string like this:
{
"ipaddress": "xxx",
"hostname": "comcast.xxx",
"popup": {
"position": "1256",
"pagename": "home"
}
}
In my Windows Form code I've been using JavaScriptSerializer for phare those line to dictionary.
var obj = new JavaScriptSerializer().Deserialize<Dictionary<string, object>>(json);
It is working fine at the moment, but I don't know how to get value inside popup? Because it's another dictionary.
[7] = {[popup, System.Collections.Generic.Dictionary`2[System.String,System.Object]]}

The fastest (yet unsafe) way of doing it is like this is via the indexer:
First extract the first dictionary and cast, since the first dictionary will yield an object of type object:
var popup = (Dictionary<string, object>)obj["popup"];
Then, you extract the values based on keys:
var position = popup["position"];
var pagename = popup["pagename"];
If you're not sure both keys will exist in the result, you can use Dictionary.TryGetValue if they exist:
obj position;
if (!popup.TryGetValue("position", out position))
{
// Key isn't in the dictionary.
}

Use JSON .Net, then simply:
JObject dynJson = JObject.Parse(jsonString);
followed by:
string data = dynJson["popup"]["position"];
JObject.Parse

MongoDB Get names of all keys in collection using c#

How can I get names of all the keys in a MongoDB collection using c#.
I am using mongocsharpdriver.
I am able to get all the records using
var collection1 = database1.GetCollection("Games").FindAll();
Now I need the key names to display/use it. I need key names of collection that I have fetched.
e.g. If I have collection which
{ "_id" : ObjectId("c3"), "GameID" : 20, "GameName" : "COD5", "Cost" : 100}
{ "_id" : ObjectId("c4"), "GameID" : 21, "GameName" : "NFS", "Publisher" : "EA"}
{ "_id" : ObjectId("c5"), "GameID" : 22, "GameName" : "CS", "Cost" : 200}
So I should get list of keys like GameID, GameName, Cost, Publisher.
I also went through MongoDB Get names of all keys in collection but was not able to implement it, didnot understood it & got problem with mapreduce.

Inspired from the link in your question:
string map = #"function() {
for (var key in this) { emit(key, null); }
}";
string reduce = #"function(key, stuff) { return null; }";
string finalize = #"function(key, value){
return key;
}";
MapReduceArgs args = new MapReduceArgs();
args.FinalizeFunction = new BsonJavaScript(finalize);
args.MapFunction = new BsonJavaScript(map);
args.ReduceFunction = new BsonJavaScript(reduce);
var results = collection1.MapReduce(args);
foreach (BsonValue result in results.GetResults().Select(item => item["_id"]))
{
Console.WriteLine(result.AsString);
}

A very inefficient, but simple way to do this is
HashSet<string> keys = new HashSet<string>();
foreach (var rover in collection1.FindAll())
{
rover.Names.ToList().ForEach(p => keys.Add(p));
}
Keep in mind that finding the set of keys, no matter how it's implemented, will always have to iterate the entire collection, so it will be terribly slow on larger collections.
It makes sense to use Map/Reduce for this problem on larger collections, because that avoids all the data transfer and deserialization overhead that is incurred by the solution I posted above, but you should generally try to avoid doing something like this at all. At least, don't do it where it's needed synchronously.
If you somehow need to know the set of all fields quickly, you're better off keeping track of the fields during writes and store the list of fields in a separate collection somewhere.

Which approach to templating in C# should I take?

What I have
I have templates that are stored in a database, and JSON data that gets converted into a dictionary in C#.
Example: 
Template: "Hi {FirstName}"
Data: "{FirstName: 'Jack'}"
This works easily with one level of data by using a regular expression to pull out anything within {} in the template.
What I want
I would like to be able to go deeper in the JSON than the first layer.
Example:
Template: "Hi {Name: {First}}"
Data: "{Name: {First: 'Jack', Last: 'Smith'}}"
What approach should I be taking? (and some guidance on where to start with your pick)
A regular expression
Not use JSON in the template (in favor of xslt or something similar)
Something else
I'd also like to be able to loop through data in the template, but I have no idea at all where to start with that one!
Thanks heaps

You are in luck! SmartFormat does exactly as you describe. It is a lightweight, open-source string formatting utility.
It supports named placeholders:
var template = " {Name:{Last}, {First}} ";
var data = new { Name = new { First="Dwight", Last="Schrute" } };
var result = Smart.Format(template, data);
// Outputs: " Schrute, Dwight " SURPRISE!
It also supports list formatting:
var template = " {People:{}|, |, and} ";
var data = new { People = new[]{ "Dwight", "Michael", "Jim", "Pam" } };
var result = Smart.Format(template, data);
// Outputs: " Dwight, Michael, Jim, and Pam "
You can check out the unit tests for Named Placeholders and List Formatter to see plenty more examples!
It even has several forms of error-handling (ignore errors, output errors, throw errors).
Note: the named placeholder feature uses reflection and/or dictionary lookups, so you can deserialize the JSON into C# objects or nested Dictionaries, and it will work great!

Here is how I would do it:
Change your template to this format Hi {Name.First}
Now create a JavaScriptSerializer to convert JSON in Dictionary<string, object>
JavaScriptSerializer jss = new JavaScriptSerializer();
dynamic d = jss.Deserialize(data, typeof(object));
Now the variable d has the values of your JSON in a dictionary.
Having that you can run your template against a regex to replace {X.Y.Z.N} with the keys of the dictionary, recursively.
Full Example:
public void Test()
{
// Your template is simpler
string template = "Hi {Name.First}";
// some JSON
string data = #"{""Name"":{""First"":""Jack"",""Last"":""Smith""}}";
JavaScriptSerializer jss = new JavaScriptSerializer();
// now `d` contains all the values you need, in a dictionary
dynamic d = jss.Deserialize(data, typeof(object));
// running your template against a regex to
// extract the tokens that need to be replaced
var result = Regex.Replace(template, #"{?{([^}]+)}?}", (m) =>
{
// Skip escape values (ex: {{escaped value}} )
if (m.Value.StartsWith("{{"))
return m.Value;
// split the token by `.` to run against the dictionary
var pieces = m.Groups[1].Value.Split('.');
dynamic value = d;
// go after all the pieces, recursively going inside
// ex: "Name.First"
// Step 1 (value = value["Name"])
// value = new Dictionary<string, object>
// {
// { "First": "Jack" }, { "Last": "Smith" }
// };
// Step 2 (value = value["First"])
// value = "Jack"
foreach (var piece in pieces)
{
value = value[piece]; // go inside each time
}
return value;
});
}
I didn't handle exceptions (e.g. the value couldn't be found), you can handle this case and return the matched value if it wasn't found. m.Value for the raw value or m.Groups[1].Value for the string between {}.

Have you thought of using Javascript as your scripting language? I had great success with Jint, although the startup cost is high. Another option is Jurassic, which I haven't used myself.
If you happen to have a Web Application, using Razor maybe an idea, see here.
Using Regex or any sort of string parsing can certainly work for trivial things, but can get painful when you want logic or even just basic hierarchies. If you deserialize your JSON into nested Dictionaries, you can build a parser relatively easily:
// Untested and error prone, just to illustrate the concept
var parts = "parentObj.childObj.property".split('.');
Dictionary<object,object> current = yourDeserializedObject;
foreach(var key in parts.Take(parts.Length-1)){
current = current[key];
}
var value = current[parts.Last()];
Just whatever you do, don't do XSLT. Really, if XSLT is the answer then the question must have been really desperate :)

Why not us nvelocity or something?