We Keep Coding

Dispatch database data to several consumer in different format

I've a big database which contains a lot of data from a big enterprise.
We would like to be able to dispatch this data to different external applications (external, meaning that are not developed by us, but only accessible in our local network).
Consumers can be of very different kinds: accounting, reporting, tech(business), website, ...
With a big variety of formats: CSV, webservice, RSS, Excel, ...
The execution of these exports can be of two different types: scheduled (like every hour), or on demand.
There is mostly two kind of exports: almost-real-time-data(meaning we want to have current data), or statistical data(meaning we are taking in account a period of time).
I've yet to find a good approach to allows those access.
I thought about Biztalk, but I don't know this product very well, and I'm not sure it can make scheduled calls and have business logic. Does anyone have enough knowledge of Biztalk to indicate to me if it can fit my needs?
If Biztalk isn't a good way, is there any libraries which can ease the development of a custom service?

Biztalk can be made to do what you want to do i.e. Extract data from your database, transform it into various formats and send it to various systems on a scheduled basis or as and when required by exposing this as a webservice/WCF Service (Not entirely out of the box, but you might need to purchase additional adapters, pipelines, etc).
But, the question here is, how database intensive is this task? If its large volumes of data, clearly Biztalk is not a favorite candidate, as Biztalk struggles with large data. Its good for routing (without transforming/inspecting) though, even if its large data files.
SSIS, on the other hand is good for data intensive tasks. If your existing databases are on SQL Server, then it fits even better for your data intensive exports/imports and transformations. But it falls short when it comes to the variety of ways you need to connect to external systems (protocols).
So, you are looking at a combination of a good ETL tool, like SSIS, as well as something good at routing like Biztalk. Neither of them clearly fit your needs on their own, in terms of scalability, volumes, connectivity, data formats, etc.

Your question can result in quite a broad implementation. You could consider using a service bus (pub/sub) along with some form of CQRS (if applicable).
My FOSS Shuttle ESB project is here: http://shuttle.codeplex.com/
It has a generic scheduler built in. You could, of course, go with any other service bus such as MassTransit, or NServiceBus.

I think you could use ASP.NET MVC API. http://www.asp.net/web-api
I find it the easiest way to export different kind of info and file formats.
It won't generate scheduled reports or files, you will need the client app or a windows service to call the app. Similar to webservices, but it can return different formats and also files.
And creating excel files, etc. you have to create them manually. Thats a bit of a turndown, but i like this approach because it can be easily hosted on IIS and all the functions your clients are going to call can be on the same place and even called from javascript, so as i see it is a bit more work for you, but it creates really easy to consume services.

By dispatch, I'm assuming you're looking for a pub/sub model. Take a hard look at NServiceBus's (NSB) pub/sub capabilities, http://nservicebus.com/docs/Samples/PublishSubscribe.aspx. Underneath the covers NSB makes heavy use of MSMQ, which has become a lot more stable over time.
If you want to venture outside of your .NET comfort zone, check out Apache Camel or Fuse's Enterprise Service Bus. Either of these tools will support what you need as well. I've used Camel in some extremely high throughput areas without any major issues.

Application Architecture For Ease of Application Customization

I'm looking for input on a direction to take for building an accounting application. The application needs to allow for high customization, sometimes entire processes will need to changed.
I want a way to make changes without re-compiling the entire application when a customer has a specific modification request. The back-end will be a SQL database of some sort. Most likely SQL Server Express for cost reasons. The front-end will be C#.
I'm thinking of an event-based system that will have events for when different types of actions, such as entries, are made. I would then have a plugin system that handles the event. I may need to have multiple processes apply in a specific order to the data before it is finally saved. It will need to trigger other processes as well.
I want to keep my base application the same, which works for most customers, but have a graceful way of loading the custom processes that other specific customers have.
I'm open to all suggestions. Even if they are thinking of completely different ways of approaching the problem. Our current in-house development talent is .NET and MS SQL Server. I'm not aware of a software pattern that may fit this situation.
Additional Info:
This isn't a completely blank slate system, it will have functionality that works for a large number of the customers. For various reasons, requirements change based on states and even at the region and town level where customization may be necessary.
I'd like to be able to plugin additional pre-compiled modules. When I started looking into possible options, I was imagining an empty handler that I could insert code into through a plugin. So say for example, a new entry is made to the general ledger that raises an event. The handler is called, but the handler's code is coming from a plugin, which may be my original process that fits 80% of the customers. If a customer wants a custom operation, I could add a plugin that completely replaces the original one or have it add an additional post processing step through another plugin after the original runs. Sort of a layering process I guess.

You could look at Managed Extensibility Framework
It provide rich composition layer features that allow you to build loosely-coupled plugin applications.
Update : sound like you need the pre-defined modules on different geographic areas and using chain of responsibility design patern might help you manage the principle of change.
Sorry no codes provided just throwing my thoughts

Windows Workflow Foundation (WF) (part of the .NET Framework) is a potential candidate for your requirements. It enables various actions, command-lets and script-lets to be composed dynamically so that you can more easily customize different workflows for different users/customers.
WF is used by Biztalk for large-scale systems integration and is hosted in-process by many other applications that require the ability to easily modify the orchestration of a number of smaller tasks and actions.
You might want to start with this tutorial on WF4.
HTH.

It's not just plugins or the way how do you technically resolve that plugin problem, use MEF (+1 #laptop) or something else, You got to put most effort in defining plugin "points" in your application, this is gone be most important eg. where you will put that empty "events" to put your code, or what parameters this events or plugins will have.
For example usable plugin would be in before save event, but you will have to have only one place in application that will save various types of business documents, so you can call plugins there and parameter would be abstract document object.
So you have to think real hard about your system architecture, to be abstract enough for various plugin points, and do that architecture completely, don't do just a part of the system and start coding on that.
I hope that you understood what I meant to say, because English is not my native language.

General questions about Biztalk

What is different between Business rules engine and Orchestration?
When working with Biztalk, does it focus more in drop and click then doing hard coding in C#?
If coding possible, what are you coding? I understand when you are coding a function for instance a pressed button display images in an application but in Biztalk it must be different.

Those two have completely different purpose.
Orchestration drives the process - it defines messaging choreography where you can coordinate message retrieval and sending from multiple systems and include some custom processing. The example can be orchestration which waits for Order message, transforms the message to another format and sends it to ERP system. Then dehydrates (waits serialized in database) until ERP system process the order to send some notification back.
Business rules engine is evaluation system. Your orchestration can need some decision making which should be controlled from outside. This is what business rule engine allows. Business user can define some complex rule sets including rules like: if Order.Price > 10.000 set Order.Price = Order.Price * 0,95. Such rules can be chained and injected to existing processes.
BizTalk is about developing biztalk project and about configuring everything correctly but it still includes programming. First what you should be aware of is XML related stuff - XSD, XPath, XSLT. Defining messages and transformation is all about these three but BizTalk templates in visual studio make this really easy (for example transformation can be done completely graphically). Next is C# and XLang (some very special variant of C# used directly for expressions in orchestrations).
BizTalk is about developing biztalk project and about configuring everything correctly. BizTalk offers a lot of out of the box features so you can use prepared shapes in orchestrations or you can use prepared functoids in maps but in the same time you can develop your own shapes, functoinds, pipeline components, adapters, etc. Also orchestration allows calling custom .NET code.
At least this was how it worked when I worked with BizTalk 2006 R2 two years ago.

Just to add to the reply from Ladislav...
I've been using BizTalk from BizTalk 2004 through to 2010. There are many BizTalk "Consultants" out there who don't have a clue about how BizTalk really works, when to use maps, when to use C# etc. You need to get some experience using it. There is a useful section on MSDN (MSDN) which has loads of useful information. Also look at EndpointTV on Channel9.
You will need to know WCF as it is important and will become more so in the future. I understand the question about sharepoint, it is quite common for people to ask you to do workflow or user interaction in sharepoint but it isn't an essential skill.
Also learn xslt, I haven't had to do much up until now as I've managed to solve problems using custom functoids or the built in functoids but my current client has some interesting schemas and I've found myself lacking in that area...

I can't create a clear picture, why and when to use RESTful services? [duplicate]

This question already has answers here:
Why do we need RESTful Web Services?
(8 answers)
Closed 9 years ago.
Why and when to use RESTful services?
I know how to create a WCF webservice. But I am not able to comprehend when to use a SOAP based service and when to use a RESTful service. I read many articles on SOAP vs REST, but still, I don't have a clear picture of why and when to use RESTful services.
What are some concrete points in order to easily decide between these services?

This is a worthy question, and one for which a short answer does no justice. Forgetting about the fact that most people may be more familiar with SOAP than REST, I think there are a few key points in this:
First and foremost, I would suggest using REST wherever it fits naturally. If your main use scenarios involve reading and updating data atoms ("resources"), REST provides a more lightweight, discoverable and straightforward approach to data access. Also, building really thin clients (mobile devices, JavaScript, even shell scripts) is often easier with REST.
For example: If your data model is all about customers and your main operations involve reading the customers and writing back changes, REST will do just fine. Using GET/POST/PUT/DELETE HTTP protocols is an excellent way to make the protocol very discoverable and easy to use, even for somebody not intimately familiar with your application.
This, however, brings us to the second point.
What if you need to offer a web API with querying capabilities? For example, a typical scenario might be "Get me the 5 newest customers". In this scenario, pure REST provides little in terms of API discoverability. Enter OData (www.odata.org), and you're rolling again; from this viewpoint, OData URI based query syntax adds a touch of well-known abstraction above the normal extremely simplistic, ID-based addressing of REST services.
But then, there are aspects which can be reasonably hard to represent in terms of REST. Point three: If you can't model it reasonably cleanly, consider SOA.
For example, if a common usage scenario involves transitioning customers between workflow stages (say, "new customer", "credit request received", "credit approved"), modeling such stages with REST may prove complex. Should different stages be represented just as an attribute value in an entity? Or perhaps, should the different stages be modeled as containers wherein the customers lie? If it's an attribute, do you always want to do a full PUT when updating it? Should you perhaps use a custom HTTP verb ("APPROVE http://mysite/customers/contoso HTTP/1.0")?
These are valid questions to which there are no universal answers. Everything can be modeled in REST, but at some point the abstraction breaks down so much that much of REST's human-facing benefits (discoverability, ease of understanding) are lost. Of course, technical benefits (like all the HTTP-level goodness) can still be reaped, but in most realities they're not really the critical argument anyway.
Fourth and finally, there are things which the SOA model simply does great. Perhaps the most important of these is transactions. While it's a pretty complex generic problem in the WS-* world as well, generic transactions are rarely needed and can often be replaced with reasonably simple, atomic operations.
For example, consider a scenario where you want to create an operation that allows the merging of two customers and all their purchases under one account. Of course, all this needs to happen or not happen; a typical transaction scenario. Modeling this in REST requires a nontrivial amount of effort. For a specialized scenario such as this, the easy SOA approach would be to create one operation (MergeCustomers) which implements the transaction internally.
For more advanced scenarios, the WS-* stack provides facilities not readily available in the REST world (including WS-Transaction, WS-Security and whatnot). While most APIs need none of this (or are better off implementing them in a more simple way), I don't think it's worth the effort to rewrite all that just to be 100% REST.
Look into the best of both worlds. For the vast majority of scenarios, it is completely acceptable to have the basic CRUD in REST and provide a few specialized operations in SOA.
Also, these APIs can be designed to act together. For example, what should a SOA-based MergeCustomers operation return? It might return a serialized copy of the merged customer, but in most cases I would opt for returning a URI of the REST resource that is the newly merged customer. This way, you would always have a single representation of the customer even if SOA were necessary for specialized scenarios.
The previous approach has the drawback that it requires client support for both REST and SOA. However, this is rarely a real problem (apart from the purely architectural perspective). The simplest clients usually have REST capabilities by the very definition of having an HTTP stack, and they rarely run the more complex operations.
Of course, your mileage may vary. The needs of your application (and its clients), local policies and backward compatibility requirements often seem to dominate these discusssions in forehand, so the REST vs. SOA discussion is rarely on a pure technical merit basis.

An eternal question! SOAP vs. REST....
SOAP is great because it's industrial-strength, the services are self-describing in a machine-readable and -interpretable way, e.g. your computer can read, understand a SOAP service and create a client for it. SOAP is great because it's methods and all the data it will ever pass around are described and defined in great detail. But SOAP is a bit heavy-weight - it takes a lot of infrastructure to support it. SOAP is also mostly method-oriented, e.g. you define your services by means of what methods can be executed on the service.
REST is much more light-weight - it uses only established HTTP protocol and procedures, so any device that has an HTTP stack (and what doesn't, these days) can access a REST service. No SOAP heavy-lifting needed. REST is also more resource-centric, e.g. you think about resources, collections of resources, and their properties and how to deal with those (so you're basically back to the core function of create, read, update, delete). REST doesn't currently have any machine-readable service description, so you're either left to try and see, or you need some documentation from the service provider, to know what resources you have at hand.
My personal bottom line: if you want to expose some collections of data, and reach (being able to access it from any device) and ease-of-use is more important than reliability and other enterprise-grade features, then REST is the way to go. If you need serious, business-to-business, well-defined, well-documented services that implement things like reliability, transaction support etc. - then you're definitely on the right track with SOAP.
But I really don't see a REST exclusive or SOAP kind of approach - they'll be great scenarios for both.

SOAP will give you a richer set of functionality and you can create strongly typed services, where REST is normally more lightweight and easier to get running.
Edit: If you look at the ws-* standards there is a lot of stuff http://www.soaspecs.com/ws.php. Development tools such as Visual Studio make access very easy to a lot of this though. Typically WS is used more for enterprise SOA type development while REST is used for public API's on web sites (at least thats what I think).

Android (mobile OS) do not have support for SOAP. RESTful services are a much better suit when mobile devices are the target.
I personally prefer RESTful services in most cases since they are more lightweight, easier to debug and integrate against (unless you use a code generator for SOAP).

One point that hasn't been mentioned is overhead. A recent REST project I worked on involved file transfer with items up to 2 GB allowed. If it had been implemented as a SOAP service, that'd be an automatic 30%+ increase in data due to encoding. The overhead with a REST service is all headers, the rest is straight data.

What am I missing about WCF?

I've been developing in MS technologies for longer than I care to remember at this stage. When .NET arrived on the scene I thought they hit the nail on the head and with each iteration and version I thought their technologies were getting stronger and stronger and looked forward to each release.
However, having had to work with WCF for the last year I must say I found the technology very difficult to work with and understand. Initially it's quite appealing but when you start getting into the guts of it, configuration is a nightmare, having to override behaviours for message sizes, number of objects contained in a messages, the complexity of the security model, disposing of proxies when faulted and finally moving back to defining interfaces in code rather than in XML.
It just does not work out of the box and I think it should. We found all of the above issues while either testing ourselves or else when our products were out on site.
I do understand the rationale behind it all, but surely they could have come up with simpler implementation mechanism.
I suppose what I'm asking is,
Am I looking at WCF the wrong way?
What strengths does it have over the
alternatives?
Under what circumstances should I
choose to use WCF?
OK Folks, Sorry about the delay in responding, work does have a nasty habit of get in the way sometimes :)
Some clarifications
My main paint point with WCF I suppose falls down into the following areas
While it does work out of the box, your left with some major surprises under the hood. As pointed out above basic things are restricted until they are overridden
Size of string than can be passed can't be over 8K
Number of objects that can be passed in a single message is restricted
Proxies not automatically recovering from failures
The amount of configuration while it's there is a good thing, but understanding it all and what to use what and under which circumstances can be difficult to understand. Especially when deploying software on site with different security requirements etc. When talking about configuration, we've had to hide lots of ours in a back-end database because security and network people on-site were trying to change things in configuration files without understanding it.
Keeping the configuration of the interfaces in code rather than moving to explicitly defined interfaces in XML, which can be published and consumed by almost anything. I know we can export the XML from the assembly, but it's full of rubbish and certain code generators choke on it.
I know the world moves on, I've moved on a number of times over the last (ahem 22 years I've been developing) and am actively using WCF, so don't get me wrong, I do understand what it's for and where it's heading.
I just think there should be simpler configuration/deployment options available, easier set-up and better management for configuration (SQL config provider maybe, rather than just the web.config/app.config files).

I use WCF all the time now and I share your pain. It seems like it was grossly over-engineered, but we are going to be stuck with it for a long, long time so I'm trying to learn it.
One thing I am certain about, XML sucks. I've had nothing but problems using XML to control it and have since switched to handling everything via code.

The concerns you listed were:
Size of string than can be passed can't be over 8K
Number of objects that can be passed in a single message is restricted
Proxies not automatically recovering from failures
The amount of configuration while it's there is a good thing, but understanding it all and what to use what and under which circumstances can be difficult to understand. Especially when deploying software on site with different security requirements etc. When talking about configuration, we've had to hide lots of ours in a back-end database because security and network people on-site were trying to change things in configuration files without understanding it.
Keeping the configuration of the interfaces in code rather than moving to explicitly defined interfaces in XML, which can be published and consumed by almost anything. I know we can export the XML from the assembly, but it's full of rubbish and certain code generators choke on it.
here's my take:
(1) addressed a valid concern that customers had with ASMX. It was too wide-open, with no way to easily control it. The 8k limit is easily lifted if you know where to look. I guess you can count that as a surprise, but it's more of a one-time thing. Once you know about it, you can lift it and be done with it forever, if you choose.
(2) is also configurable.
(3) is known, but there are boilerplate ways to work around this. The StockTrader code for example, demonstrates a proven pattern. You can re-use the code in your own app. Not sure if this is fixed in WCF for .NET 4.0. I know it was an open request.
(4) The config is a beast. This is a concern for a lot of people. The problem here is that WCF is so flexible, and config of all of that flexibility is exposed through xml files. It can be overwhelming. An approach that seems to work is to take it in small bites, as you need it.
(5) I don't understand.

I vastly prefer ASP.NET MVC and Web API over WCF. If I had to summarize WCF to a developer who was just being introduced to it, I would say, "WCF is a well-meaning attempt to replace over-engineered, Java EE style RPC development." Unfortunately, many of the decisions made require you to become an expert in configuring low level, unimportant items (message sizes, timeouts, uninteresting protocol elements, etc.) while abstracting absolutely critical pieces (URL design, parameter serialization, response serialization, etc.). The difference in productivity and aggravation between teams I know using WCF vs. Web API is night and day.
To come clean a little: I have always hated the core concept of .NET Remoting. I feel that developers need a thorough understanding of the resource structure of their application and how these resources are serialized. Furthermore, the use of the "POST" verb for simple data retrieval is worrisome in a read heavy application that needs to scale.

I'll address the rest of your issues after clarification. In the meantime, I can address your question on when you should choose to use WCF: always.
WCF is the replacement for the old ASMX technologies, including WSE. It is also the replacement for .NET Remoting. It is the only technology upon which high-level communications features in .NET will be based for the forseeable future.
For example, consider Windows Azure. It was not inevitable that the new concept of "cloud computing" would have its communications aspects covered by WCF. Yet, WCF was flexible enough to be extended to cover those cases, with very little change in code.
If you're having trouble with WCF, then you'd do well to make sure Microsoft knows about it. WCF is the present and future of web service and other service-oriented development in .NET, so they've got a very strong incentive to listen to you and resolve your pain points. Either contact them directly through Connect, or ask questions here on SO (tag with WCF, please), and a lot of people will help you.

Biggest advantage of using WCF from a programmer's point of view: separates the definition of exposed services (operations, contracts, etc.) from the protocol's specific details, unlike ASMX where you expose a class as a web service directly in the code using attributes. Using a real example of mine: we where able to easily switch the transport protocol between web services and named pipes, whatever suited better the deployment and performance needs, without changing a line of code.

WCF is intended to SOA methodologies. Work professionally using it is a nightmare. I delivered a SOA solution using WCF as tool and hell, hundreds configurations and hidden tips! My past distributed solution using old style Web Services and Remoting were more stable. I've spent days working out the solution for the error "The underlying connection was closed: An unexpected error occurred" which makes no sense to happen for one method among 4 in the same contract. I'm very disappointed. It took me back through time where .net was first introduced with lots of promises and when we got hands on, hell, log problems came up!

To address the problem of maintenance nightmare of application config, some standard like UDDI or WS-Discovery exist, WS-Discovery will be supported by WCF in .NET 4.0.
Keeping the configuration of the
interfaces in code rather than moving
to explicitly defined interfaces in
XML, which can be published and
consumed by almost anything. I know we
can export the XML from the assembley,
but it's full of rubbish and certain
code generators choke on it.
Can you be more explicit ? I think you are talking about service behavior configured in code.
You can easily code behavior extensions to configure what your are talking about in config file instead of code BUT I think that if microsoft didn't do that there is a good reason.
For example a service with this behavior :
[ServiceBehavior(InstanceContextMode=InstanceContextMode.PerCall, ConcurrencyMode=ConcurrencyMode.Single)]
The implementation knows that the instance is not shared between multiple thread so it's developed differently than :
[ServiceBehavior(InstanceContextMode=InstanceContextMode.Single, ConcurrencyMode=ConcurrencyMode.Multiple)]
In this case the service implementation should take care about concurency problems.
The implementation is coupled with the attribute ServiceBehavior, so moving this behavior in a XML file is not a good idea.
What if you can change a InstanceContextMode.PerCall service to a InstanceContextMode.Single service inside the config file ? You break the application !

Looking at how you mention XML and SQL, you are using WCF to build a web application or an actual web service (service on the Web, and not just SOAP exchange).
It helps thinking about WCF as a replacement for .NET Remoting (or DCOM, CORBA etc), which also happens to support web services as one of the transports. Interfaces declared in assemblies, behavior of proxies, certain configuration options and other aspects of the framework that look unnatural and complicated from perspective of web apps - actually do work out of the box for DCOM-style systems of distributed objects.
To answer the question: no, you are not missing anything and using WCF for web applications is complicated, because WCF is not a framework for building web applications. Probably such framework can be built on top of it, but I would hate to see WCF itself changed to move into web realm.