I am attempting to optimise around a possible bottleneck.
I have a server application that is serving objects from a database to applications remotely, who can work with 1 - n objects of 1 - n different types (where n can be a relatively high number) that all implement a common interface but may contain many unique properties on different types.
The client applications store the server objects in a local cache, until they are ready to persist them back, through the server, to the database.
This is being done currently in WCF with each class defining a DataContract.
Due to the possibly large amount of objects that may need to be passed back to the server (it changes depending on implementation), I would prefer not to do these all as individual calls any more, rather wrap all the objects in a single serialized (or better still compressed) stream and send them through as one connection to the server.
I can quite simply roll my own, but would prefer to use a recommended approach, and hoping someone may suggest one. If you can convince me, I am also willing to accept that my approach is probably not the best idea.
How high is "relatively high"?
For example, one option that occurs is to use a wrapper object:
[DataContract]
public class Wrapper {
[DataMember(Order = 1)]
public List<Foo> Foos {get {...}}
[DataMember(Order = 2)]
public List<Bar> Bars {get {...}}
[DataMember(Order = 3)]
public List<Blop> Blops {get {...}}
}
Then you should be able to send a single message with any number of Foo, Bar and/or Blop records. My inclusion of the Order attribute was deliberate - if you want to reduce the size of the stream, you might consider protobuf-net - with the above layout, protobuf-net can hook into WCF simply by including [ProtoBehavior] on the method (in the operation-contract interface) that you want to attack (at both client and server). This switches the transfer to use google's "protocol buffers" binary format, using base-64 to encode. If you are using the basic-http binding, this can also use MTOM if enabled, so even the base-64 isn't an issue. Using this, you can get significant data transfer savings (~1/5th the space based on the numbers shown).
(edit 1 - protobuf-net assumes that Foo, Bar and Blop also use the Order attribute)
(edit 2 - note that you could always break up the request into a number of mid-size Wrapper messages, then call a method to apply all the changes once you have them at the server (presumably in a staging table in the database))
Related
I have a class that stores and manipulates some entities. Depending on the number of inputs, I may not be able to store entities in memory, so I'm trying to serialize my objects to be written on hard disk using protocol buffers. I'm using C# and protobuf-csharp-port. I'm aware of protobuf-net as an alternative port; so far I have been working with first option but I'm open for changes if it's required based on my needs.
The class to be serialized in it's simplified form is as follows:
class Entity<T> where T: IComparable<T>
{
int id;
T metaData;
}
So at compile time I have no clues about metaData. Googling I noticed that extensions are the right path to follow (as suggested on google's page and this question); hence I'm defining the Entity.proto file for class Entity as following:
message Entity
{
required int32 id = 1 [default = 0];
extensions 2 to max;
}
and I would like the user to provide his own .proto file for T without the need to access or re-compile Entity.proto. In this regard, my questions are:
Do I need to change Entity.proto ?
What should be the T.proto ?
How can I access T in my C# code ?
With that scheme, any extensions are going to be child values (not subclasses) of the non-generic Entity. That doesn't sound like generics, but ultimately storage (serialization) is often quite different to implementation (Entity<T> etc). If you can manuallyap between them: fine. But it isn't something the library will provide, AFAIK.
For completeness, in protobuf-net terms: it is perfectly fine with Entity<T> - it essentially considers each (Entity<Foo>, Entity<Bar>, etc) to be completely separate messages. Protobuf-net isn't hugely motivated by .proto schemas (although a code-gen tool is provided, for completeness) - it mainly uses runtime metadata.
I have a collection with a collection of documents. Each document has around 20 different properties with different data types (e.g. Int, Double, String).
I am searching for an efficient way or the appropriate way to add side notes to each property.
My thought (I am using C# to model the document structure) is for each property, instead of
:
public int PageRank {get; set; }
to use:
public Dictionary<int, string> PageRank {get; set;}
This means that each item in the document is a collection of both the value and the string for the side note.
The side notes will be seen at the front-end by the user.
Any better implementation?
Idan, for performance reasons, you should consider your use case from the MongoDB point of view -- not from the object oriented language point of view. The way it ends up looking in C# is an afterthought -- its the DB performance that counts. So, when querying your documents, if the side notes are mostly not needed, it will be better to place them into a separate collection (possibly) thus reducing the size of each document and enabling MongoDB to read more of them into the available memory. If the user does need to look at the side notes, you would do this with a separate query. You know your usage scenario better, so its up to you to decide how to do this, but its these kinds of design decisions that you need to concern yourself with -- and the C# code will be shaped according to your schema
I am writing a piece of software in c# .net 4.0 and am running into a wall in making sure that the code-base is extensible, re-usable and flexible in a particular area.
We have data coming into it that needs to be broken down in discrete organizational units. These units will need to be changed, sorted, deleted, and added to as the company grows.
No matter how we slice the data structure we keep running into a boat-load of conditional statements (upwards of 100 or so to start) that we are trying to avoid, allowing us to modify the OUs easily.
We are hoping to find an object-oriented method that would allow us to route the object to different workflows based on properties of that object without having to add switch statements every time.
So, for example, let's say I have an object called "Order" come into the system. This object has 'orderItems' inside of it. Each of those different kinds of 'orderItems' would need to fire a different function in the code to be handled appropriately. Each 'orderItem' has a different workflow. The conditional looks basically like this -
if(order.orderitem == 'photo')
{do this}
else if(order.orderitem == 'canvas')
{do this}
edit: Trying to clarify.
I'm not sure your question is very well defined, you need a lot more specifics here - a sample piece of data, sample piece of code, what have you tried...
No matter how we slice the data structure we keep running into a boat-load of conditional statements (upwards of 100 or so to start) that we are trying to avoid
This usually means you're trying to encode data in your code - just add a data field (or a few).
Chances are your ifs are linked to each other, it's hard to come up with 100 independent ifs - that would imply you have 100 independent branches for 100 independent data conditions. I haven't encountered such a thing in my career that really would require hard-coding 100 ifs.
Worst case scenario you can make an additional data field contain a config file or even a script of your choice. Either case - your data is incomplete if you need 100 ifs
With the update you've put in your question here's one simple approach, kind of low tech. You can do better with dependency injection and some configuration but that can get excessive too, so be careful:
public class OrderHandler{
public static Dictionary<string,OrderHandler> Handlers = new Dictionary<string,OrderHandler>(){
{"photo", new PhotoHandler()},
{"canvas", new CanvasHandler()},
};
public virtual void Handle(Order order){
var handler = handlers[order.OrderType];
handler.Handle(order);
}
}
public class PhotoHandler: OrderHandler{...}
public class CanvasHandler: OrderHandler{...}
What you could do is called - "Message Based Routing" or "Message Content Based" Routing - depending on how you implement it.
In short, instead of using conditional statements in your business logic, you should implement organizational units to look for the messages they are interested in.
For example:
Say your organization has following departments - "Plant Products", "Paper Products", "Utilities". Say there is only one place where the orders come in - Ordering (module).
here is a sample incoming message.
Party:"ABC Cop"
Department: "Plant Product"
Qty: 50
Product: "Some plan"
Publish out a message with this information. In the module that processes orders for "Plant Products" configure it such that it listens to a message that has "Department = Plant Products". This way, you push the onus on the department modules instead of on the main ordering module.
You can do this using NServiceBus, BizTalk, or any other ESB you might already have.
This is how you do in BizTalk and this is how you can do in NServiceBus
Have you considered sub-typing OrderItem?
public class PhotoOrderItem : OrderItem {}
public class CanvasOrderItem : OrderItem {}
Another option would be to use the Strategy pattern. Add an extra property to your OrderItem class definition for the OrderProcessStrategy and use a PhotoOrderStrategy/CanvasOrderStrategy to contain all of the different logic.
public class OrderItem{
public IOrderItemStrategy Strategy;
}
public interface IOrderItemStrategy{
public void Checkout();
public Control CheckoutStub{get;}
public bool PreCheckoutValidate();
}
public class PhotoOrderStrategy : IOrderItemStrategy{}
public class CanvasOrderStrategy : IOrderItemStrategy{}
Taking the specific example:
You could have some Evaluator that takes an order and iterates each line item. Instead of processing if logic raise events that carry in their event arguments the photo, canvas details.
Have a collection of objects 'Initiators' that define: 1)an handler that can process Evaluator messages, 2)a simple bool that can be set to indicate if they know what to do with something in the message, and 3)an Action or Process method which can perform or initiate the workflow. Design an interface to abstract these.
Issue the messages. Visit each Initiator, ask it if it can process the lineItem if it can tell it to do so. The processing is kicked off by the 'initiators' and they can call other workflows etc.
Name the pieces outlined above whatever best suits your domain. This should offer some flexibility. Problems may arise depending on concurrent processing requirements and workflow dependencies between the Initiators.
In general, without knowing a lot more detail, size of the project, workflows, use cases etc it is hard to comment.
I have some serverside data that I need replicating (pushed in real-time) from a server app to around 100 wpf clients. My problem is when a given Order object changes it typically only changes a 1 or 2 fields so I only want to send those changes over the wire Not the whole object – thus decreasing the wire payload, processing time etc as the whole Order object has around 50 fields.
The data is a Dictionary of Order objects keyed on OrderId. I use protobuf-net to seralise the data and send over the wire to the wpf clients.
Has anyone dealt with this patterm/problem before? Or have any ideas on who to achieve this?
Thanks a lot.
Create a simple proxy using Castle.DynamicProxy which saves the name of all properties that have been changed.
protobuf-net supports a number of patterns to aid this type of scenario, the simplest being (to share the pattern used by System.ComponentModel):
[ProtoMember(1)]
public string Foo { get;set; }
public bool ShouldSerializeFoo() { /* return true if Foo is "dirty" */ }
This assumes you have some mechanism for tracking the changes yourself (for hooking into the ShouldSerialize* method); protobuf-net doesn't do change tracking itself. If you don't currently have any change tracking, you might be able to use something from this answer: Comparing 2 objects and retrieve a list of fields with different values
I've done a lot of serialization development lately, mostly for sending objects over sockets, but I've run into an interesting question: Is it possible to send just a few of the properties from an object through a serializer?
My envisioned scenario is this: You have some sort of "state" object for each client, consisting of many properties (strings, ints, bools, etc). When your client first connects, the entire state object is serialized via an Xml or Binary serializer, and sent over the socket, to be recreated on the other side. Now both client and server have identical state objects. Your server then needs to change the state, and does so by simply setting one of the state object's property. The socket (either hooked to the state's events, or part of the state object itself) could synchronize the two states by reserializing the entire object, but it seems like a single "property change" object would do.
Obviously, this could be implemented manually. But it seems like a serializer should be able to serialize just a single property, and apply it like a patch on the other side. Does anyone know if this is possible, or would I have to write the entire thing from scratch?
With XmlSerializer (and protobuf-net, for a binary equivalent, since protobuf-net adopts most of XmlSerializer's patterns) you could do this by having a method:
public bool SouldSerializeFoo() {
return fooIsDirty;
}
public string Foo {get;set;}
for each property Foo - but you'd need to maintain the "what is dirty" manually in your own code (perhaps in the set). Lots of work; I've done a diffing serializer in the past - it was a real PITA, to be honest. I should also note that the [XmlIgnore] public bool FooSpecified {get{...} set{...}} pattern does the same thing, but for what you want, ShouldSerialize* is more appropriate.
As an addition to Marc's answer, here's the MSDN docs on the ShouldSerialize* methods