I've a big database which contains a lot of data from a big enterprise.
We would like to be able to dispatch this data to different external applications (external, meaning that are not developed by us, but only accessible in our local network).
Consumers can be of very different kinds: accounting, reporting, tech(business), website, ...
With a big variety of formats: CSV, webservice, RSS, Excel, ...
The execution of these exports can be of two different types: scheduled (like every hour), or on demand.
There is mostly two kind of exports: almost-real-time-data(meaning we want to have current data), or statistical data(meaning we are taking in account a period of time).
I've yet to find a good approach to allows those access.
I thought about Biztalk, but I don't know this product very well, and I'm not sure it can make scheduled calls and have business logic. Does anyone have enough knowledge of Biztalk to indicate to me if it can fit my needs?
If Biztalk isn't a good way, is there any libraries which can ease the development of a custom service?
Biztalk can be made to do what you want to do i.e. Extract data from your database, transform it into various formats and send it to various systems on a scheduled basis or as and when required by exposing this as a webservice/WCF Service (Not entirely out of the box, but you might need to purchase additional adapters, pipelines, etc).
But, the question here is, how database intensive is this task? If its large volumes of data, clearly Biztalk is not a favorite candidate, as Biztalk struggles with large data. Its good for routing (without transforming/inspecting) though, even if its large data files.
SSIS, on the other hand is good for data intensive tasks. If your existing databases are on SQL Server, then it fits even better for your data intensive exports/imports and transformations. But it falls short when it comes to the variety of ways you need to connect to external systems (protocols).
So, you are looking at a combination of a good ETL tool, like SSIS, as well as something good at routing like Biztalk. Neither of them clearly fit your needs on their own, in terms of scalability, volumes, connectivity, data formats, etc.
Your question can result in quite a broad implementation. You could consider using a service bus (pub/sub) along with some form of CQRS (if applicable).
My FOSS Shuttle ESB project is here: http://shuttle.codeplex.com/
It has a generic scheduler built in. You could, of course, go with any other service bus such as MassTransit, or NServiceBus.
I think you could use ASP.NET MVC API. http://www.asp.net/web-api
I find it the easiest way to export different kind of info and file formats.
It won't generate scheduled reports or files, you will need the client app or a windows service to call the app. Similar to webservices, but it can return different formats and also files.
And creating excel files, etc. you have to create them manually. Thats a bit of a turndown, but i like this approach because it can be easily hosted on IIS and all the functions your clients are going to call can be on the same place and even called from javascript, so as i see it is a bit more work for you, but it creates really easy to consume services.
By dispatch, I'm assuming you're looking for a pub/sub model. Take a hard look at NServiceBus's (NSB) pub/sub capabilities, http://nservicebus.com/docs/Samples/PublishSubscribe.aspx. Underneath the covers NSB makes heavy use of MSMQ, which has become a lot more stable over time.
If you want to venture outside of your .NET comfort zone, check out Apache Camel or Fuse's Enterprise Service Bus. Either of these tools will support what you need as well. I've used Camel in some extremely high throughput areas without any major issues.
Related
I am currently involved in a simple to medium complex IOT project. The main purpose of our application is gathering data from our devices and analyzing that data as well as calculating statistics.
On the server side we run a MVC application. Up until now we used Hangfire to schedule the calculations. Hangfire is an amazing tool for scheduling emails and other simple stuff, for more advanced things it's too slow. The calculations can take up a lot of time and are processor-intensive (we are trying to optimize them though), so we need to call them in a background task, a simple API call won't be enough.
I thought about splitting the application into multiple parts, the website, the core and a windows service.
The problem is, I never tried that before and I have no idea what the best practice is to achieve that kind of thing. I searched for examples and articles, but all I found were suggestions to use Hangfire and/or Quartz.NET.
Does anyone have any resources on what the best practice is to build a MVC application, a Windows service and how they could communicate (probably through a queue)? What is the best practice in such a situation?
Although there may be many different possible ways to connect a site with a windows service, I'd probably chose one of the following two, based on your statements:
Direct communication
One way of letting your site send data to your backend windows service would be to use WCF. The service would expose an endpoint. For simplicity's sake this could be a basicHttpBinding or a netTcpBinding. The choice should be made based on your specific requirements; if the data is small then basicHttp may be "sufficient".
The advantage of this approach is that there's relatively little overhead needed: You'll just have to setup the windows service (which you'll have to do anyway) and open a port for the WCF binding. The site acts as client, the service as server. There's nothing special with it, just because the client being a MVC site. You can take almost any WCF tutorial as a starting point.
Note that instead of WCF you could use another technology like .NET Remoting or even sockets just as well. Personally, I often use WCF because I'm quite used to it, but this choice is pretty opinion based.
Queued communication
If reliability and integrity is crucial for your project, then using a queue might be a good idea. Again: depending on your needs, there may come diffeent products into consideration. If you don't need much monitoring and out-of-the-box management goodies, then even a very simplistic technology like MSMQ may be sufficient.
If your demands to the aforementioned points are more relevant, then maybe you should look for something else. Just recently I got in touch with Service Bus for Windows Server (SBWS). It's the Azure Service Bus's little brother which can be used on premises locally on your windows server. The nice thing about it is, that it comes at no extra charge as it's already licensed with your windows server licence.
As with the first point: MSMQ and SBWS are just two examples. There may be a lot of other products like NServiceBus, ZeroMQ or others usable, you name it.
My company is about to undergo a large project in which our client wants a large customer portal with a cms, crm implementing. This will require interaction with data from multiple sources across our customers business, these sources include XML office backend systems, sql datbases, webservices etc.
Our proposed solution would be to write an API in c# to provide a common interface with all these systems. This would be scalable for future and concurrent projects within the company.
Our client expressed an interest in using Biztalk rather than a custom API for this integration, as they feel it is an enterprise solution that any of their suppliers could pick up and use, and it will be better supported.
We feel that the configuration work using Biztalk would be rather heavy for all their custom business rules which are required and an interface for the new application to get data to and from Biztalk would still need to be written.
Are we right to prefer a custom API solution above Biztalk? Would Biztalk be suitable as a databroker layer to provide an interface for the new Customer portal we are writing. We have not experience with using Biztalk before so any input would be appreciated.
Reading your requirements, i would say you would want to focus on the business core parts. I.e. how to use the above mentioned services together. The topic where you want to spent as little as possible is 'the plumbing' of this.
BizTalk server will take away a large part of this plumbing for you. Instead of dealing with "how to guarentee consistency if normalization goes wrong" you will be dealing with "how to normalize the data".
BizTalk is also very 'future proof' in the sense that you can always add/remove/change systems in a BizTalk enviroment without the need to 'take it down for a change'. (within limits of course and if implemented correctly).
I would advise to reevaluate the "do it yourself" aprroach and see how much effort is needed if you would go the "do it yourself" way. Take a close look at the amount of "plumbing code" versus "core compentency code". Remember, after writting it, you have to maintain/bugfix it. BizTalk is a proven technology for hosting these kind of requirements.
From above description i would say; "BizTalk is likely the better option to choose".
Hope this helps,
When looking at the interface of BizTalk there is one major truth to realise;
'There is no interface'
BizTalk does not specify specific interfaces. It allows you to set up a "named message exchange pattern" (like Request-Response, OneWay, etc).
The incoming message is "Published" into BizTalk (by what we call a 'Receive Port'+'Receive location' combination). You can have an Orchestration (piece of business logic) or a SendPort (connection to external system->out) 'subscribe' to messages. This subscription can be based on context information or content information (although the later requires the information to be uplifted from the message content to the message context).
BizTalk therefore allows you to hook in any system at any given point of time by becoming a "Publisher" or "Subscriber" to messages. This can even be done when the system is fully up and running in production.
Any BizTalk project can still utilize the full .Net API on many locations, giving you the full power to write 'anything you could in plain .Net' also inside of BizTalk.
I would like to advise one thing though;
"Please make sure at least one or two persons in your project team will get a BizTalk ramp-up/course". BizTalk is like a concealed gun; Extremely powerfull but dangerous in the wrong hands.
I need to monitor several systems going thru stress testing. I want to make a client-server .NET(C#) application that will ping the systems to check temperature, memory usage etc.
The client will provide system info (cpu-mem-hdd configuration) at start then it will undergo through several benchmark/stress tests. The server will keep track of what is been executed and also will be able to detect system crashes. I pretty much have all the code for the client (have been running on the system, using WMI)
I have no experience with .NET remoting nor WCF, but I think this is a great opportunity to learn them.
What technology would you use?
WCF is meant to unify .net remoting with a handful of other Microsoft technologies. WCF gives you a lot of flexibility to change the design of your client-server architecture simply by changing a few .net attributes.
I recommend you proceed with caution and make sure you have a good WCF reference at hand. You will spend a lot of time spinning your wheels in the mud without one.
Snmp, maybe based on http://www.snmpsharpnet.com/.
Nothing fancy, new and shiny, but a protocol that is meant to be used for exactly the stuff you describe. Would definitely be my first choice.
If that is not an option for you: Second choice for me would be WCF, because that can be used from different platforms (you are more flexible in the future).
i will choose service bus.
I want to separate modules of my program to communicate with each other. They could be on the same computer, but possibly on different ones.
I was considering 2 methods:
create a class with all details. Send it of to the communication layer. This one serializes it, sends it, the other side deserializes it back to the class and than handles it further.
Create a hashtable (key/value thing). Put all data in it. Send it of to the communicationlayer etc etc
So it boils down to hashtable vs class.
If I think 'loosely coupled', I favor hashtable. It's easy to have one module updated, include new extra params in the hastable, without updating the other side.
Then again with a class I get compile-time type checking, instead of runtime.
Has anyone tackled this previously and has suggestions about this?
Thanks!
edit:
I've awarded points to the answer which was most relevant to my original question, although it isn't the one which was upvoted the most
It sounds like you simply want to incorporate some IPC (Inter-Process Communication) into your system.
The best way of accomplishing this in .NET (3.0 onwards) is with the Windows Communication Foundation (WCF) - a generic framework developed by Microsoft for communication between programs in various different manners (transports) on a common basis.
Although I suspect you will probably want to use named pipes for the purposes of efficiency and robustness, there are a number of other transports available such as TCP and HTTP (see this MSDN article), not to mention a variety of serialisation formats from binary to XML to JSON.
One tends to hit this kind of problem in distributed systems design. It surfaces in Web Service (the WSDL defining the paramers and return types) Messaging systems where the formats of messages might be XML or some other well-defined format. The problem of controlling the coupling of client and server remains in all cases.
What happens with your hash table? Suppose your request contains "NAME" and "PHONE-NUMBER", and suddenly you realise that you need to differentiate "LANDLINE-NUMBER" and "CELL-NUMBER". If you just change the hash table entries to use new values, then your server needs changing at the same time. Suppose at this point you don't just have one client and one server, but are perhaps dealing with some kind of exchange or broker systems, many clients implemented by many teams, many servers implemented by many teams. Asking all of them to upgrade to a new message format at the same time is quite an undertaking.
Hence we tend to seek back-comptible solutions such as additive change, we preserve "PHONE-NUMBER" and add the new fields. The server now tolerates messages containg either old or new format.
Different distribution technologies have different in-built degrees of toleration for back-compatibility. When dealing with serialized classes can you deal with old and new versions? When dealing with WSDL, will the message parsers tolerate additive change.
I would follow the following though process:
1). Will you have a simple relationship between client and server, for example do you code and control both, are free to dictate their release cycles. If "no", then favour flexibility, use hash tables or XML.
2). Even if you are in control look at how easily your serialization framework supports versioning. It's likely that a strongly typed, serialized class interface will be easier to work with, providing you have a clear picture of what it's going to take to make a change to the interface.
You can use Sockets, Remoting, or WCF, eash has pros and cons.
But if the performance is not crucial you can use WCF and serialize and deserialize your classes, and for maximum performance I recommend sockets
What ever happened to the built in support for Remoting?
http://msdn.microsoft.com/en-us/library/aa185916.aspx
It works on TCP/IP or IPC if you want. Its quicker than WCF, and is pretty transparent to your code.
In our experience using WCF extensively over the last few years with various bindings we found WCF not be worth the hassle.
It is just to complicated to correctly use WCF including handling errors on channels correctly while retaining good performance (we gave up on high performance with wcf early on).
For authenticated client scenarios we switched to http rest (without wcf) and do json/protobuf payloading.
For high-speed non-authenticated scenarios (or at least non-kerberos authenticated scenarios) we are using zeromq and protobuf now.
Is there a product (ideally open source, but not necessary), that would enable a zero dependency deployment? every service bus or queue library I've been able to find has a dependency on one of the queue apps (like msmq), or a database. I would like a very lightweight solution that I can just add a reference to my application, build it, and deploy it with as little configuration as possible.
In an ideal world, the queue/service bus would run on IIS, and allow web and rich clients to talk to it.
Such a tool would be ideal for fast prototyping of large distributed systems on a local development machine.
Rhino Queues from Ayende is exactly what you are looking for, this is the blog post introducing it:
http://ayende.com/Blog/archive/2008/08/01/Rhino-Queues.aspx
I think that all of the limitations mentioned in this post have been fixed since then.
From the blog post, what rhino queues is:
XCopyable, Zero Administration, Embedded, Async queuing service
Robust in the face of networking outages
System.Transactions support
Fast
Works over HTTP
In a similar vein to ShuggyCoUk's suggestion, you could rig up a queue (or queues) using the Windows built-in ESENT database (comes already installed with Windows). There is a managed code access library (open source): http://www.codeplex.com/ManagedEsent. If you stick with writing / reading CLOBs or BLOBs, it should work just fine. If you want to be really clever, you can use NServiceBus and write (contribute?) ESENT-flavored subscription storage and transports. There are some forays into using ESENT on Ayende's blog as well (you'll have to poke around his SVN repository for the juicy bits).
If you're happy to be:
Windows specific
Limited to the local domain
Seriously limited in the message size supported
Wrap the underlying win32 calls in P/Invoke
Deal with the polling yourself
Deal with the hacks needed to allow back and forth communication
Deal with the shared config needed to keep the names in sync
Then a quick wrapper around the windows MailSlot API might be sufficient.
This simple example is a reasonable basis to start.
This article has some further information but assumes the use case is via a control (rather than a Component as it should be) as well as some poor WinForms integration so should be considered for incidental reading rather than a basis for any library.
This article is C++ but is of a higher standard (and a commenter has extended it to support the batching of larger messages into several smaller ones).
You get 424 bytes (so with .Net 212 chars) you may want to drop to ASCII to double your useful message length if you are talking text.
Note that despite its simplicity, limitations and lack of features it does provide multicast delivery, something often complex to layer on a point to point protocol yourself.
This ayende post provides and interesting comparison of three service buses. We use NServiceBus and think if it's not clear that Udi Dahan would respond to how you'd plug in non-dependent queue.
We work using MSMQ happily but there are other options and in theory it should be open to practically anything, given that you may lose some reliability and durability depending on your choice.
Why not Amazon's message service Simple Queue Service?
We moved our projects from MSMQ to ActiveMQ. its really better :)
ActiveMQ is open source queue ,based on Apache web server.
We used him in production on high frequently data workflow, where msmq have a lot of problem (we work with msmq a year)
The csharp implementation is nms
I'm currently working on an open source WCF based service bus. You can find it here: http://rockbus.codeplex.com/. It supports dynamic (#run-time) subscriptions, subcription repository (database), pluggable transports, XPath based content-based routing, transactional delivery over wcf protocols, roundrobin delivery, pluggable subscription evaluation, and more. Have a look!
Have you thought about using a service like IronMQ by http://Iron.io?
You wouldn't have any dependencies, could quickly prototype apps without setting up any queue infrastructure, and it's highly available and fast.
There is not currently a locally installable version but it's based on the upcoming OpenStack protocol so there will be.
Btw I work for Iron.
Try https://github.com/mcintyre321/PieQ - this is my attempt to write a threadsafe, persistent, zero-config, embedded work queue. It probably needs a little love, but I think it might be the kind of tool you are looking for.
I have developed an InMemory JMS library which can be used to in testing JMS applications without really connecting to JMS providers/server (Think of hsqldb). You don't have to deal with connection or protocol or anything, all you need to do is to send and receive messages.
https://github.com/Dhana-Krishnasamy/InMemoryJMS