I am considering using NO SQL databases such as MongoDb, RavenDb or any other ones recommend I would consider.
Can someone give me some advice, tutorials and useful links regarding my following question.
This system I want to write must be dynamic e.g. the model may change allot and should not be hard coded in C#.
For example if I had a JSON document saved holding ID, NAME, FULL NAME and then added a property called PHONENUMBER I would not want to rebuild the C# code or redeploy.
Is it possible to build C# models from a dynamic JSON? and then be able to manipulate it?
If so what library are most recommend for this type of system? What libraries work best with .NET?
This question is a step in to starting my university project.
Thanks for help & advice.
Yes, you can do that quite easily with RavenDB.
You can do it in one of two ways.
Either you will use a fully dynamic model, utilizing the C# dynamic keyword. That will let you do pretty much whatever you want, including adding properties at runtime, querying on runtime properties, etc.
However, a more common setup is that you'll use a lot of common properties (a customer has to have a name, for example). So you'll have a model that looks something like this:
public class Customer
{
public string Id {get;set;}
public string Name {get;set;}
public dynamic Props {get;set;}
}
The fixed properties are coded in C#, which helps you get into an easier, more consistent model and work with all the usual compiled tooling.
The dynamic stuff is in the Props property (which is usually initialized to ExpandoObject).
Note that you cannot do linq queries using dynamic. This is a limitation of C#, not RavenDB. You can still query dynamically using RavenDB, but you'll have to use the string based query API.
I implemented a Json.NET serializer wrapper that may help you:
https://github.com/welegan/RedisSessionProvider/blob/master/RedisSessionProvider/Serialization/RedisJSONSerializer.cs
I use it in my library which stores the contents of ASP.NET's Session object inside of Redis, which is a NoSQL option you did not mention. Still, given the typeless nature of Json, I imagine it will be applicable to your needs regardless of what NoSQL db you choose. The basic steps are:
Serialize:
Decide on a delimiter (I probably could have chosen a better one)
Store the type info (you can cache it for performance gains)
Store the object data after a delimiter
Deserialize:
Find the type info up to the delimiter
Deserialize the Type object
Pass Type as well as the object data to the library of your choosing. At the very least, Json.NET and ServiceStack.Json both expose serializers that will do the trick.
Edit
Seems I misunderstood part of your question. You want to be able to support adding json properties without redeploying your C#, and my example would strip out the extra properties during the serialize step back to the noSql db. You can use either a Dictionary<string, string> or ExpandoObject like ayende or mxmissile suggest, but keep in mind you will then have very few guarantees about the type of the properties of the object you get out.
In other words, you can freely add property names but as soon as you change the type of a property from int to long your code will break unexpectedly. Depending on your use case, that may or may not matter, just something to keep in mind.
Yes, using a Dictionary. However, I am not sure how those database systems handle dictionaries. Gracefully or not.
No, c# is compiled, so once that is done, there is no changing it without changing the source and compiling again. I think you should add some Javascript for that as it is a JS strong point.
Related
Using Protobuf-net, I want to know what properties of an object have been updated at the end of a merge operation so that I can notify interested code to update other components that may relate to those updated properties.
I noticed that there are a few different types of properties/methods I can add which will help me serialize selectively (Specified and ShouldSerialize). I noticed in MemberSpecifiedDecorator that the ‘read’ method will set the specified property to true when it reads. However, even if I add specified properties for each field, I’d have to check each one (and update code when new properties were added)
My current plan is to create a custom SerializationContext.context object, and then detect that during the desearalization process – and update a list of members. However… there are quite a few places in the code I need to touch to do that, and I’d rather do it using an existing system if possible.
It is much more desirable to get a list of updated member information. I realize that due to walking down an object graph that may result in many members, but in my use case I’m not merging complex objects, just simple POCO’s with value type properties.
Getting a delta log isn't an inbuilt feature, partly because of the complexity when it comes to complex models, as you note. The Specified trick would work, although this isn't the purpose it was designed for - but to avoid adding complexity to your own code,that would be something best handled via reflection, perhaps using the Expression API for performance. Another approach might be to use a ProtoReader to know in advance which fields will be touched, but that demands an understanding of the field-number/member map (which can be queried via RuntimeTypeModel).
Are you using habd-crafted models? Or are you using protogen? Yet another option would be to have code in the setters that logs changes somewhere. I don't think protogen currently emits partial method hooks, but it possibly could.
But let me turn this around: it isn't a feature that is built in right now, and it is somewhat limited due to complexity anyway, but: what would a "good" API for this look like to you?
As a side note: this isn't really a common features in serializers - you'd have very similar challenges in any mainstream serializer that I can think of.
I am using lookup tables for references. e.g. registration types, admin, moderator then using a factory to determine the type of registration. What is the easiest way to create a strongly typed way of comparing registrations. Sort of a similar behaviour to an enum. for example
pssudo code
class regfactory
{
case()
if(regType.Admin: return new adminReg()
}
The only way I can think of is a dictionary of magic strings generated from the database.
I bevieve the only way to to accomplish strongly typed enums for your situation would be code code generation. Anything not generated before compiletime would not serve for strong typing.
Robert Koritnik posted a very slick way to do this: T4 template to Generate Enums
Another way to 'generate' better readable enum names (in case you need them) is the HUmanizer project at https://github.com/MehdiK/Humanizer.
From a practical point of view it may seem a bit error-prone and you may feel like you're breaking some good practice rule by not centralizing access to that data and leaving it at risk of going out of synch during maintenance, however, from an architectural point of view, considering that we're talking about look-up data, it's ok to hard-code it as it's just part of your "static data contract", if you will.
If you do have lots of those, then maybe there could be a case for putting those constants in a format where a build or database patch script could update them when those values are changed, but 9 out of 10 just stuffing them in an enum works fine.
It's worth noting that some ORMs do have good support for enums, including EF which will allow for keeping those values in synch if you adopt a code first approach. However, we're talking about adding a whole new layer to your software so you gotta have more reasons than just wanting to keep your static look-up data in synch to implement that.
You can use reflection in c#. There is an excellent example on this answer to list the declared classes in a given namespace.
Then you would compare the name of your registration type with the available classes' names to decide which class to instantiate.
If I want to serialize an object I have to use [Serializable] attribute and all member variables will be written to the file. What I don't know how to do versioning e.g. if I add a new member variable (rename a variable or just remove a variable) and then I open (deserialize) the file how can I determine the object/file version so I can correctly set the new member or take some kind of migration? How can I determine that the variable was initialized during the load or not (ignored by deserializer).
I know that there are version tolerant approaches and I can mark variables with [OptionalField(VersionAdded = 1)] attribute. If I open an old file the framework will ignore this optional (new variable) and it will be just zero/null. But again how can I determine if the variable is initialized by load or it was ignored.
I can write the class/object version number to the stream. Use the ISerializable approach and in the constructor(SerializationInfo oInfo, StreamingContext context) method read this version number. This will exactly tell me what is the class version in the stream.
However I expected that such kind of versioning is already implemented by the streaming framework in C#. I tried to obtain the Assembly version from the SerializationInfo but it is always set to current version not to the version which was used when the object was saved.
What is the preferred approach? I found a lot of articles on the net, but I could not find a good solution for this which addresses versioning...
Any help is appreciated
Thanks,
Abyss
Forgive me if some of what I write is too obvious,
First of all, please! you must stop thinking that you are serializing an object...
That is simply incorrect as the methods which are part of your object are not being persisted.
You are persisting information - and so.. DATA only.
.NET serialization also serializing the type name of your object which contain the assembly name and its version, so when you deserialize - it compares the persisted assembly information with the type that will be manifested with the information - if they are not the same it will return an exception.
Beside the versioning problem - not everything can be serialized so easily.. try to serialize a System.Drawing.Color type and you will begin to understand the problems with the over simplistic mechanism of .NET serialization.
Unless you plan to serialize something really simple which has no plans to evolve I wouldn't use the serialization mechanism provided by .NET.
Getting the focus back to your question, you can read here about the versioning ignorance ability:
http://msdn.microsoft.com/en-us/library/ms229752(v=vs.80).aspx which is provided for BinaryFormatter.
You should also check XML Serialization which has some nice abilities, but the biggest benefit is that you getting an XML which is Human readable so your data will never be lost even if you had complication with the versioning of your types.
But finally, I recommend you either use Database with Entity Framework to persist your data or write your own flat file manager.. while EF is very good for most solutions, sometime you might want something lighter to persist something very simple.
(my imply is that I can no longer see a solution where .NET serialization can be relevant.)
I hope this helps, Good luck.
Inside our application (C# - .NET 4.5) we have a report generation tool. This tool must receive only the SQL command, verify it and from that, create a whole new report with available fields with the same name as specified by the SQL command and corresponding data types, similar to what an ORM tool would do.
Because of the nature of this tool we're using Reflection and Emit to create a whole new class. From fields provided by a dataReader (System.Data.SqlClient.SqlDataReader) we can create the type and populate it with the corresponding data.
The result of this is a IQueryable object that I can use on my reports.
This whole process is done and tested but to keep both the report, the generated class and the SQL command together we need to save this new type on the database and because or our database layout and system definitions, this requires me to provide an XML-like file or string to a method that will compress and convert to a Base64 string before saving it.
It would be a simple task if I were to save the report into a DLL file, just like shown HERE.
But since this new type must be transformed into an XML-like format I'm a little bit lost in here.
I have done the opposite on the past: fully create a type from an pure XML file, manually. I also know that I can do something similar but it would require me to loop into every detail/property/method/member/etc of the class to create the XML file.
Is there any way (like a helper from .NET framework) that could help me here?
Instead of doing it 100% manually I'd like to delegate the XML generation/parse to a tool probably with a better perforance too...
Edit:
People, READ THE TITLE BEFORE POSTING COMMENTS! I'm trying to save an XML for the type. TYPE. Not the object. The type.
#Mark Gravell
Thanks for the tip. I'll check that.
But about the schema: any way to save/load it automatically?
For saving the type, I would say either simply store the schema, and re-create a compatible type at runtime, or just use AssemblyBuilder etc and configure the dynamic-assembly as saveable, and write it as a .dll to disk (or elsewhere). Then just load the .dll at runtime and find the type. Either approach can work. If you already have the code to create a Type from a schema, the first may be easier.
For saving the data, my first instinct would be XmlSerializer, however that works via Assembly generation, so it might not like working against a fully-dynamic Type, from TypeBuilder. If XmlSerializer isn't happy, you could try protobuf-net; that also works in-memory (by default), so should be pretty happy.
However! I should note that you might also want to consider simply using a DataTable. While I don't have tons of love for DataTable, it is designed for exactly this scenario:
it can model fields that are defined only at runtime
it has inbuilt serialization of both schema and data
it implements the ComponentModel APIs for declaring runtime-models
which means most tools work with it for free
I often write my objects out to a database in xml form.
However, if I change the form of my objects, say by changing the name or by changing the fields, I can no longer read them from the database, which makes somewhat difficult the task of reading them, converting them to their new form, and writing them back out to the database.
I'd rather not have to rename my classes everytime I change something about them.
*Note: I am relying on C#'s XmlSerialization/Deserialization of objects for generating the Xml. Perhaps this is not desirable if I change the format of the objects.
If you implement the ISerializable interface on your objects, then you can implement custom serialization/deserialization routines that provide backwards compatibility with older versions of the objects.
See here for an example of how it can be done: ISerializable and backward compatibility
It ultimately depends on how you serialize your objects.
One convenient option is to store them as hash (key-value pairs). This way, if to class Dog having property name I add another property breed, existing objects won't be invalidated. They'll just have have breed = nil.
Exact storage format (xml, json or separate 'properties' table) is not important in this case, the important thing is how you convert objects to it and back.
But I don't think anyone could give specific suggestions without knowing specific platform.