We Keep Coding

How to convert some RAW data from an API and save it into variables so i can work with them in C#

I'm working on a personal project. Its a C# app that communicates with some web services using an API.
i finally got the first raw data with this few lines:
var client = new RestClient("https://api.abcd.com/token");
var request = new RestRequest(Method.POST);
request.AddParameter("username", usr);
request.AddParameter("password", pass);
request.AddParameter("grant_type", "password");
and in postman the response (JSON) looks like :
{"access_token":"aaaaaaa","token_type":"bearer","expires_in":899,"refresh_token":"bbbbbbb",".issued":"Fri,
01 May 2020 16:11:36 GMT",".expires":"Fri, 01 May 2020 16:26:36
GMT",".refreshexpires":"Fri, 01 May 2020 17:11:36 GMT"}
my next step is to find the way to separate those key/value pair into different variables in C# so i can work with them.
thank you so much for the help.

But I guess for small purpose no need to create a class rather use weakly typed data structure like this:
dynamic responseObject = JsonConvert.DeserializeObject(responseString);
//then use every property like this
responseObject.accessToken ...
responseObject.token_type.....
But you need to use Newtonsoft.Json for this too.

You want to use a JSON deserialiser to do this.
So you would create a class:
public class Response {
public string accessToken {get; set;)
public string token_type {get; set;)
.....
}
And then use something like Newtonsoft.Json (available from NuGet) to deserialise:
using Newtonsoft.Json;
.....
var response = JsonConvert.Deserialise<Response>([RAW TEXT FROM REST CLIENT]);

You can look into using Json.Net which will allow you to deserialize the JSON into an object like so. Note you'll need to download the package and then add using Newtonsoft.Json;
{
"varone":"valueone"
}
public class MyJsonClass
{
//JsonProperty isn't strictly required but I personally think it helps when trying to deserialize for sanity sake
[JsonProperty("varone")]
public string VarOneValue { get; set; } //The value will be "valueone" on deserialization
}
var myobj = JsonConvert.DeserializeObject<MyJsonObject>(JSONDATA);
Console.Write(myobj.VarOneValue); //Will be "valueone"
Nuget CLI: Install-Package Newtonsoft.Json
Page: https://www.newtonsoft.com/json

Kafka Producer Error: ' Value serializer not specified and there is no default serializer defined for type ...'

I just started using Kafka and hit the following rookie error:
'Value cannot be null.
Parameter name: Value serializer not specified and there is no default serializer defined for type ActMessage.'
It happens when trying to send a class object, ActMessage object, rather then the a simple string that comes with the example. The line of code that raises the erros is:
using (var p = new ProducerBuilder<Null, ActMessage>(config ).Build()
I am using the .net client.
My understanding is that i need to use one of the default serializes in the first type parameter, one that come with Kafka client, as explained here, but can't find them on this .net package.
I guess i could build one but that would be a waste of time.
Here a reproducible example:
public class ActMessage {
public int SomeId {get;set;}
public string SomeContent {get;set;}
}
class Tester {
void send(){
var config = new ProducerConfig { BootstrapServers = "localhost:9092" };
using (var p = new ProducerBuilder<Null, ActMessage>(config).Build()) //throws error here
{
var dr = p.ProduceAsync("news", new Message<Null, ActMessage>
{
Value = new ActMessage { SomeId = 1, SomeContent="hi" },
}
).Result;
}
}
}

I suggest checking out the working examples/ dir in that repo to see working code that you can copy into your own projects.
If you have your own class, you need to implement the ISerializer and IDeserializer interfaces.
Or you can use the built-in ones
However, an alternative is to use Avro
This requires writing an Avro schema file, then using avrogen to create your class, not manually write it. E.g.
dotnet tool install --global Apache.Avro.Tools
avrogen -s User.avsc .
Then you must always add some ValueSerializer in Kafka clients in order to send data

.NET assemblies: understanding type visibility

I am trying to reproduce something that System.Xml.Serialization already does, but for a different source of data.
For now task is limited to deserialization only.
I.e. given defined source of data that I know how to read. Write a library that takes a random type, learns about it fields/properties via reflection, then generates and compiles "reader" class that can take data source and an instance of that random type and writes from data source into the object's fields/properties.
here is a simplified extract from my ReflectionHelper class
public class ReflectionHelper
{
public abstract class FieldReader<T>
{
public abstract void Fill(T entity, XDataReader reader);
}
public static FieldReader<T> GetFieldReader<T>()
{
Type t = typeof(T);
string className = GetCSharpName(t);
string readerClassName = Regex.Replace(className, #"\W+", "_") + "_FieldReader";
string source = GetFieldReaderCode(t.Namespace, className, readerClassName, fields);
CompilerParameters prms = new CompilerParameters();
prms.GenerateInMemory = true;
prms.ReferencedAssemblies.Add("System.Data.dll");
prms.ReferencedAssemblies.Add(Assembly.GetExecutingAssembly().GetModules(false)[0].FullyQualifiedName);
prms.ReferencedAssemblies.Add(t.Module.FullyQualifiedName);
CompilerResults compiled = new CSharpCodeProvider().CompileAssemblyFromSource(prms, new string[] {source});
if (compiled.Errors.Count > 0)
{
StringWriter w = new StringWriter();
w.WriteLine("Error(s) compiling {0}:", readerClassName);
foreach (CompilerError e in compiled.Errors)
w.WriteLine("{0}: {1}", e.Line, e.ErrorText);
w.WriteLine();
w.WriteLine("Generated code:");
w.WriteLine(source);
throw new Exception(w.GetStringBuilder().ToString());
}
return (FieldReader<T>)compiled.CompiledAssembly.CreateInstance(readerClassName);
}
private static string GetFieldReaderCode(string ns, string className, string readerClassName, IEnumerable<EntityField> fields)
{
StringWriter w = new StringWriter();
// write out field setters here
return #"
using System;
using System.Data;
namespace " + ns + #".Generated
{
public class " + readerClassName + #" : ReflectionHelper.FieldReader<" + className + #">
{
public void Fill(" + className + #" e, XDataReader reader)
{
" + w.GetStringBuilder().ToString() + #"
}
}
}
";
}
}
and the calling code:
class Program
{
static void Main(string[] args)
{
ReflectionHelper.GetFieldReader<Foo>();
Console.ReadKey(true);
}
private class Foo
{
public string Field1 = null;
public int? Field2 = null;
}
}
The dynamic compilation of course fails because Foo class is not visible outside of Program class. But! The .NET XML deserializer somehow works around that - and the question is: How?
After an hour of digging System.Xml.Serialization via Reflector I came to accept that I lack some kind of basic knowledge here and not really sure what am I looking for...
Also it is entirely possible that I am reinventing a wheel and/or digging in a wrong direction, in which case please do speak up!

You don’t need to create a dynamic assembly and dynamically compile code in order to deserialise an object. XmlSerializer does not do that either — it uses the Reflection API, in particular it uses the following simple concepts:
Retrieving the set of fields from any type
Reflection provides the GetFields() method for this purpose:
foreach (var field in myType.GetFields(BindingFlags.Instance | BindingFlags.Public | BindingFlags.NonPublic))
// ...
I’m including the BindingFlags parameter here to ensure that it will include non-public fields, because otherwise it will return only public ones by default.
Setting the value of a field in any type
Reflection provides the function SetValue() for this purpose. You call this on a FieldInfo instance (which is returned from GetFields() above) and give it the instance in which you want to change the value of that field, and the value to set it to:
field.SetValue(myObject, myValue);
This is basically equivalent to myObject.Field = myValue;, except of course that the field is identified at runtime instead of compile-time.
Putting it all together
Here is a simple example. Notice you need to extend this further to work with more complex types such as arrays, for example.
public static T Deserialize<T>(XDataReader dataReader) where T : new()
{
return (T) deserialize(typeof(T), dataReader);
}
private static object deserialize(Type t, XDataReader dataReader)
{
// Handle the basic, built-in types
if (t == typeof(string))
return dataReader.ReadString();
// etc. for int and all the basic types
// Looks like the type t is not built-in, so assume it’s a class.
// Create an instance of the class
object result = Activator.CreateInstance(t);
// Iterate through the fields and recursively deserialize each
foreach (var field in t.GetFields(BindingFlags.Instance | BindingFlags.Public | BindingFlags.NonPublic))
field.SetValue(result, deserialize(field.FieldType, dataReader));
return result;
}
Notice I had to make some assumptions about XDataReader, most notably that it can just read a string like that. I’m sure you’ll be able to change it so that it works with your particular reader class.
Once you’ve extended this to support all the types you need (including int? in your example class), you can deserialize an object by calling:
Foo myFoo = Deserialize<Foo>(myDataReader);
and you can do this even when Foo is a private type as it is in your example.

If I try to use sgen.exe (the standalone XML serialization assembly compiler), I get the following error message:
Warning: Ignoring 'TestApp.Program'.
- TestApp.Program is inaccessible due to its protection level. Only public types can be processed.
Warning: Ignoring 'TestApp.Program+Foo'.
- TestApp.Program+Foo is inaccessible due to its protection level. Only public types can be processed.
Assembly 'c:\...\TestApp\bin\debug\TestApp.exe' does not contain any types that can be serialized using XmlSerializer.
Calling new XmlSerializer(typeof(Foo)) in your example code results in:
System.InvalidOperationException: TestApp.Program+Foo is inaccessible due to its protection level. Only public types can be processed.
So what gave you the idea that XmlSerializer can handle this?
However, remember that at runtime, there are no such restrictions. Trusted code using reflection is free to ignore access modifiers. This is what .NET binary serialization is doing.
For example, if you generate IL code at runtime using DynamicMethod, then you can pass skipVisibility = true to avoid any checks for visibility of fields/classes.

I've been working a bit on this. I'm not sure if it will help but, anyway I think it could be the way. Recently I worked with Serialization and DeSerealization of a class I had to send over the network. As there were two different programs (the client and the server), at first I implemented the class in both sources and then used serialization. It failed as the .Net told me it had not the same ID (I'm not sure but it was some sort of assembly id).
Well, after googling a bit I found that it was because the serialized class was on different assemblies, so the solution was to put that class in a independent library and then compile both client and server with that library. I've used the same idea with your code, so I put both Foo class and FieldReader class in a independent library, let's say:
namespace FooLibrary
{
public class Foo
{
public string Field1 = null;
public int? Field2 = null;
}
public abstract class FieldReader<T>
{
public abstract void Fill(T entity, IDataReader reader);
}
}
compile it and add it to the other source (using FooLibrary;)
this is the code I've used. It's not exactly the same as yours, as I don't have the code for GetCSharpName (I used t.Name instead) and XDataReader, so I used IDataReader (just for the compiler to accept the code and compile it) and also change EntityField for object
public class ReflectionHelper
{
public static FieldReader<T> GetFieldReader<T>()
{
Type t = typeof(T);
string className = t.Name;
string readerClassName = Regex.Replace(className, #"\W+", "_") + "_FieldReader";
object[] fields = new object[10];
string source = GetFieldReaderCode(t.Namespace, className, readerClassName, fields);
CompilerParameters prms = new CompilerParameters();
prms.GenerateInMemory = true;
prms.ReferencedAssemblies.Add("System.Data.dll");
prms.ReferencedAssemblies.Add(Assembly.GetExecutingAssembly().GetModules(false)[0].FullyQualifiedName);
prms.ReferencedAssemblies.Add(t.Module.FullyQualifiedName);
prms.ReferencedAssemblies.Add("FooLibrary1.dll");
CompilerResults compiled = new CSharpCodeProvider().CompileAssemblyFromSource(prms, new string[] { source });
if (compiled.Errors.Count > 0)
{
StringWriter w = new StringWriter();
w.WriteLine("Error(s) compiling {0}:", readerClassName);
foreach (CompilerError e in compiled.Errors)
w.WriteLine("{0}: {1}", e.Line, e.ErrorText);
w.WriteLine();
w.WriteLine("Generated code:");
w.WriteLine(source);
throw new Exception(w.GetStringBuilder().ToString());
}
return (FieldReader<T>)compiled.CompiledAssembly.CreateInstance(readerClassName);
}
private static string GetFieldReaderCode(string ns, string className, string readerClassName, IEnumerable<object> fields)
{
StringWriter w = new StringWriter();
// write out field setters here
return #"
using System;
using System.Data;
namespace " + ns + ".Generated
{
public class " + readerClassName + #" : FieldReader<" + className + #">
{
public override void Fill(" + className + #" e, IDataReader reader)
" + w.GetStringBuilder().ToString() +
}
}";
}
}
by the way, I found a tiny mistake, you should use new or override with the Fill method, as it is abstract.
Well, I must admit that GetFieldReader returns null, but at least the compiler compiles it.
Hope that this will help you or at least it guides you to the good answer
regards

OFX File parser in C#

I am looking for an OFX file parser library in C#. I have search the web but there seems to be none. Does anyone know of any good quality C# OFX file parser. I need to process some bank statements files which are in OFX format.
Update
I have managed to find a C# library for parsing OFX parser.
Here is the link ofx sharp. This codebase seems to be the best case to startup my solution.

I tried to use the ofx sharp library, but realised it doesn't work is the file is not valid XML ... it seems to parse but has empty values ...
I made a change in the OFXDocumentParser.cs where I first fix the file to become valid XML and then let the parser continue. Not sure if you experienced the same issue?
Inside of the method:
private string SGMLToXML(string file)
I added a few lines first to take file to newfile and then let the SqmlReader process that after the following code:
string newfile = ParseHeader(file);
newfile = SGMLToXMLFixer.Fix_SONRS(newfile);
newfile = SGMLToXMLFixer.Fix_STMTTRNRS(newfile);
newfile = SGMLToXMLFixer.Fix_CCSTMTTRNRS(newfile);
//reader.InputStream = new StringReader(ParseHeader(file));
reader.InputStream = new StringReader(newfile);
SGMLToXMLFixer is new class I added into the OFXSharp library. It basically scans all the tags that open and verifies it has a closing tag too.
namespace OFXSharp
{
public static class SGMLToXMLFixer
{
public static string Fix_SONRS(string original)
{ .... }
public static string Fix_STMTTRNRS(string original)
{ .... }
public static string Fix_CCSTMTTRNRS(string original)
{ .... }
private static string Fix_Transactions(string file, string transactionTag, int lastIdx, out int lastIdx_new)
{ .... }
private static string Fix_Transactions_Recursive(string file_modified, int lastIdx, out int lastIdx_new)
{ .... }
}
}

Try http://www.codeproject.com/KB/aspnet/Ofx_to_DataSet.aspx. The code uses Framework 3.5 and transforms an ofx into a dataset, this may help with what you're trying to do.

Using System.Json for non-Silverlight projects?

Any idea on how to do it? If not possible, what's a good JSON library for C#?

System.Json is now available in non-Silverlight projects via NuGet (.Net's package management system) and is hopefully going to be released as part of the core framework in vnext. The NuGet package is named JsonValue.
Imagine that we have the following JSON in the string variable json:
[{"a":"foo","b":"bar"},{"a":"another foo","b":"another bar"}]
We can get write the value "another bar" to the console using the following code:
using System.Json;
dynamic jsonObj = JsonValue.Parse(json);
var node = jsonObj[1].b;
System.Console.WriteLine(node.Value);

Here's an extenstion method to serialize any object instance to JSON:
public static class GenericExtensions
{
public static string ToJsonString<T>(this T input)
{
string json;
DataContractJsonSerializer ser = new DataContractJsonSerializer(input.GetType());
using (MemoryStream ms = new MemoryStream())
{
ser.WriteObject(ms, input);
json = Encoding.Default.GetString(ms.ToArray());
}
return json;
}
}
You'll need to add a reference to System.ServiceModel.Web to use the DataContractSerializer.

Scott Guthrie blogged about this
http://weblogs.asp.net/scottgu/archive/2007/10/01/tip-trick-building-a-tojson-extension-method-using-net-3-5.aspx

If you're just looking for JSON encoding/decoding, there is an official System.Web extension library from Microsoft that does it, odds are you probably already have this assembly (System.Web.Extensions):
http://msdn.microsoft.com/en-us/library/system.web.script.serialization.javascriptserializer.aspx
Example:
using System;
using System.Web.Script.Serialization;
class App
{
static void Main(string[] args = null)
{
JavaScriptSerializer jss = new JavaScriptSerializer();
String sJson = "{\"Name\": \"Your name\"}";
DesJson json = jss.Deserialize<DesJson>(sJson);
Console.WriteLine(json.Name);
}
}
class DesJson {
public string Name {get; set;}
}

Another option is to use Mono's implementation of System.Json,
I was able to backport it to C# 2.0 with a few minor changes.
You can simply download my C# 2.0 project from here.