We have multiple, decoupled systems. These systems provide public APIs which are consumed by mobile apps and other client applications.
We now need these systems to internally communicate with each other during some calls (for whatever reason). I am torn between the possible solutions.
Create secure API calls
Hosting an internal ASP.NET Web API for each system so that they can expose data and functionality to each other.
Advantage: Low coupling. As long as the API call structure doesn't change, the systems can evolve on their own.
Disadvantage: Very slow. An HTTP request is deathly slow when compared to a database query. This could severely affect the speed of our public calls.
Reference repository projects between solutions
Sharing and include the Repository (data access) codebase between multiple projects.
Advantage: Very fast. Retrieving the relevant data will be extremely fast as it would simply be a database query.
Disadvantage: Highly coupling. A rebuild and release on system A would likely mean the same for system B as they will share common code.
Neither of these solutions seem ideal.
Is this a job for WCF?
Very slow. An HTTP request is deathly slow when compared to a database query. This could severely affect the speed of our public calls
Well. no. For instance the RavenDb client API is built using HTTP. All db queries in it are executed over HTTP. You should also know that most database engines communicate with the client over the network (using sockets or named pipes).
What I'm saying is that the overhead that HTTP introduces is marginal.
The bright side with HTTP is that it enables you to cache queries at any time with little effort as there are dozens of ready-to-use caching solutions for HTTP. Same thing goes for load balancing when your user base grows.
Hence I would use HTTP for this.
We are using WCF for this purpose. Typically that is the way to handle the scenario you mentioned, I think, unless you have very specific requirements.
If you use a messaging broker in between, a (WCF)services solution can scale very well.
Weigh your priorities with the differences between WCF and Web API. See this
Related
I am currently involved in a simple to medium complex IOT project. The main purpose of our application is gathering data from our devices and analyzing that data as well as calculating statistics.
On the server side we run a MVC application. Up until now we used Hangfire to schedule the calculations. Hangfire is an amazing tool for scheduling emails and other simple stuff, for more advanced things it's too slow. The calculations can take up a lot of time and are processor-intensive (we are trying to optimize them though), so we need to call them in a background task, a simple API call won't be enough.
I thought about splitting the application into multiple parts, the website, the core and a windows service.
The problem is, I never tried that before and I have no idea what the best practice is to achieve that kind of thing. I searched for examples and articles, but all I found were suggestions to use Hangfire and/or Quartz.NET.
Does anyone have any resources on what the best practice is to build a MVC application, a Windows service and how they could communicate (probably through a queue)? What is the best practice in such a situation?
Although there may be many different possible ways to connect a site with a windows service, I'd probably chose one of the following two, based on your statements:
Direct communication
One way of letting your site send data to your backend windows service would be to use WCF. The service would expose an endpoint. For simplicity's sake this could be a basicHttpBinding or a netTcpBinding. The choice should be made based on your specific requirements; if the data is small then basicHttp may be "sufficient".
The advantage of this approach is that there's relatively little overhead needed: You'll just have to setup the windows service (which you'll have to do anyway) and open a port for the WCF binding. The site acts as client, the service as server. There's nothing special with it, just because the client being a MVC site. You can take almost any WCF tutorial as a starting point.
Note that instead of WCF you could use another technology like .NET Remoting or even sockets just as well. Personally, I often use WCF because I'm quite used to it, but this choice is pretty opinion based.
Queued communication
If reliability and integrity is crucial for your project, then using a queue might be a good idea. Again: depending on your needs, there may come diffeent products into consideration. If you don't need much monitoring and out-of-the-box management goodies, then even a very simplistic technology like MSMQ may be sufficient.
If your demands to the aforementioned points are more relevant, then maybe you should look for something else. Just recently I got in touch with Service Bus for Windows Server (SBWS). It's the Azure Service Bus's little brother which can be used on premises locally on your windows server. The nice thing about it is, that it comes at no extra charge as it's already licensed with your windows server licence.
As with the first point: MSMQ and SBWS are just two examples. There may be a lot of other products like NServiceBus, ZeroMQ or others usable, you name it.
I've a big database which contains a lot of data from a big enterprise.
We would like to be able to dispatch this data to different external applications (external, meaning that are not developed by us, but only accessible in our local network).
Consumers can be of very different kinds: accounting, reporting, tech(business), website, ...
With a big variety of formats: CSV, webservice, RSS, Excel, ...
The execution of these exports can be of two different types: scheduled (like every hour), or on demand.
There is mostly two kind of exports: almost-real-time-data(meaning we want to have current data), or statistical data(meaning we are taking in account a period of time).
I've yet to find a good approach to allows those access.
I thought about Biztalk, but I don't know this product very well, and I'm not sure it can make scheduled calls and have business logic. Does anyone have enough knowledge of Biztalk to indicate to me if it can fit my needs?
If Biztalk isn't a good way, is there any libraries which can ease the development of a custom service?
Biztalk can be made to do what you want to do i.e. Extract data from your database, transform it into various formats and send it to various systems on a scheduled basis or as and when required by exposing this as a webservice/WCF Service (Not entirely out of the box, but you might need to purchase additional adapters, pipelines, etc).
But, the question here is, how database intensive is this task? If its large volumes of data, clearly Biztalk is not a favorite candidate, as Biztalk struggles with large data. Its good for routing (without transforming/inspecting) though, even if its large data files.
SSIS, on the other hand is good for data intensive tasks. If your existing databases are on SQL Server, then it fits even better for your data intensive exports/imports and transformations. But it falls short when it comes to the variety of ways you need to connect to external systems (protocols).
So, you are looking at a combination of a good ETL tool, like SSIS, as well as something good at routing like Biztalk. Neither of them clearly fit your needs on their own, in terms of scalability, volumes, connectivity, data formats, etc.
Your question can result in quite a broad implementation. You could consider using a service bus (pub/sub) along with some form of CQRS (if applicable).
My FOSS Shuttle ESB project is here: http://shuttle.codeplex.com/
It has a generic scheduler built in. You could, of course, go with any other service bus such as MassTransit, or NServiceBus.
I think you could use ASP.NET MVC API. http://www.asp.net/web-api
I find it the easiest way to export different kind of info and file formats.
It won't generate scheduled reports or files, you will need the client app or a windows service to call the app. Similar to webservices, but it can return different formats and also files.
And creating excel files, etc. you have to create them manually. Thats a bit of a turndown, but i like this approach because it can be easily hosted on IIS and all the functions your clients are going to call can be on the same place and even called from javascript, so as i see it is a bit more work for you, but it creates really easy to consume services.
By dispatch, I'm assuming you're looking for a pub/sub model. Take a hard look at NServiceBus's (NSB) pub/sub capabilities, http://nservicebus.com/docs/Samples/PublishSubscribe.aspx. Underneath the covers NSB makes heavy use of MSMQ, which has become a lot more stable over time.
If you want to venture outside of your .NET comfort zone, check out Apache Camel or Fuse's Enterprise Service Bus. Either of these tools will support what you need as well. I've used Camel in some extremely high throughput areas without any major issues.
I am starting on a fairly basic Server/Client application (logic wise), but I am a bit confused as to what I should use for my needs. It looks like there a few options, but basically I am going to have a Master Server, and X amount of client applications (one per dedicated machine). The main purpose of this setup is so that I can basically do the following...
-Issue command to server (console app) via an ASP front end to install software on one of the remote clients.
- Server tells client to download zip package (from a various FTP site) to location and extract it to specific path.
I am not positive, but it looks like C# has Sockets and then some sort of WebClient type of deal. I am assuming Sockets would be the best route to take, and to use asynchronous (each remote client is connected in its own thread, dealing with the server individually of others).
Any information on this would be great!
Without going into too much detail for your specific requirements, I would definitely look at WCF.
It encompasses a lot of the existing remoting, client / server, web services scenarios in a very complete and secure framework.
Client Server Programming with WCF
WebClient allows you to make HTTP requests, so I don't think it's very relevant here.
There are many approaches you can take for this app.
One is of course going with WCF, which provides about a million time more options than you will need. However, WCF does have a learning curve and in particular it's hard to understand what exactly is hidden behind all the abstractions without prior experience. Furthermore, this solution is not available if you are targeting .NET 2.0.
You can also implement a simple TCP client/server model using sockets. While you can program against raw sockets, .NET also offers the convenience classes System.Net.Sockets.TcpListener for the server and System.Net.Sockets.TcpClient for the clients. This approach is much closer to the metal, but this is a tradeoff: it's much easier to understand what exactly you are doing, but you will have to implement a fair bit of functionality yourself.
This question already has answers here:
Why do we need RESTful Web Services?
(8 answers)
Closed 9 years ago.
Why and when to use RESTful services?
I know how to create a WCF webservice. But I am not able to comprehend when to use a SOAP based service and when to use a RESTful service. I read many articles on SOAP vs REST, but still, I don't have a clear picture of why and when to use RESTful services.
What are some concrete points in order to easily decide between these services?
This is a worthy question, and one for which a short answer does no justice. Forgetting about the fact that most people may be more familiar with SOAP than REST, I think there are a few key points in this:
First and foremost, I would suggest using REST wherever it fits naturally. If your main use scenarios involve reading and updating data atoms ("resources"), REST provides a more lightweight, discoverable and straightforward approach to data access. Also, building really thin clients (mobile devices, JavaScript, even shell scripts) is often easier with REST.
For example: If your data model is all about customers and your main operations involve reading the customers and writing back changes, REST will do just fine. Using GET/POST/PUT/DELETE HTTP protocols is an excellent way to make the protocol very discoverable and easy to use, even for somebody not intimately familiar with your application.
This, however, brings us to the second point.
What if you need to offer a web API with querying capabilities? For example, a typical scenario might be "Get me the 5 newest customers". In this scenario, pure REST provides little in terms of API discoverability. Enter OData (www.odata.org), and you're rolling again; from this viewpoint, OData URI based query syntax adds a touch of well-known abstraction above the normal extremely simplistic, ID-based addressing of REST services.
But then, there are aspects which can be reasonably hard to represent in terms of REST. Point three: If you can't model it reasonably cleanly, consider SOA.
For example, if a common usage scenario involves transitioning customers between workflow stages (say, "new customer", "credit request received", "credit approved"), modeling such stages with REST may prove complex. Should different stages be represented just as an attribute value in an entity? Or perhaps, should the different stages be modeled as containers wherein the customers lie? If it's an attribute, do you always want to do a full PUT when updating it? Should you perhaps use a custom HTTP verb ("APPROVE http://mysite/customers/contoso HTTP/1.0")?
These are valid questions to which there are no universal answers. Everything can be modeled in REST, but at some point the abstraction breaks down so much that much of REST's human-facing benefits (discoverability, ease of understanding) are lost. Of course, technical benefits (like all the HTTP-level goodness) can still be reaped, but in most realities they're not really the critical argument anyway.
Fourth and finally, there are things which the SOA model simply does great. Perhaps the most important of these is transactions. While it's a pretty complex generic problem in the WS-* world as well, generic transactions are rarely needed and can often be replaced with reasonably simple, atomic operations.
For example, consider a scenario where you want to create an operation that allows the merging of two customers and all their purchases under one account. Of course, all this needs to happen or not happen; a typical transaction scenario. Modeling this in REST requires a nontrivial amount of effort. For a specialized scenario such as this, the easy SOA approach would be to create one operation (MergeCustomers) which implements the transaction internally.
For more advanced scenarios, the WS-* stack provides facilities not readily available in the REST world (including WS-Transaction, WS-Security and whatnot). While most APIs need none of this (or are better off implementing them in a more simple way), I don't think it's worth the effort to rewrite all that just to be 100% REST.
Look into the best of both worlds. For the vast majority of scenarios, it is completely acceptable to have the basic CRUD in REST and provide a few specialized operations in SOA.
Also, these APIs can be designed to act together. For example, what should a SOA-based MergeCustomers operation return? It might return a serialized copy of the merged customer, but in most cases I would opt for returning a URI of the REST resource that is the newly merged customer. This way, you would always have a single representation of the customer even if SOA were necessary for specialized scenarios.
The previous approach has the drawback that it requires client support for both REST and SOA. However, this is rarely a real problem (apart from the purely architectural perspective). The simplest clients usually have REST capabilities by the very definition of having an HTTP stack, and they rarely run the more complex operations.
Of course, your mileage may vary. The needs of your application (and its clients), local policies and backward compatibility requirements often seem to dominate these discusssions in forehand, so the REST vs. SOA discussion is rarely on a pure technical merit basis.
An eternal question! SOAP vs. REST....
SOAP is great because it's industrial-strength, the services are self-describing in a machine-readable and -interpretable way, e.g. your computer can read, understand a SOAP service and create a client for it. SOAP is great because it's methods and all the data it will ever pass around are described and defined in great detail. But SOAP is a bit heavy-weight - it takes a lot of infrastructure to support it. SOAP is also mostly method-oriented, e.g. you define your services by means of what methods can be executed on the service.
REST is much more light-weight - it uses only established HTTP protocol and procedures, so any device that has an HTTP stack (and what doesn't, these days) can access a REST service. No SOAP heavy-lifting needed. REST is also more resource-centric, e.g. you think about resources, collections of resources, and their properties and how to deal with those (so you're basically back to the core function of create, read, update, delete). REST doesn't currently have any machine-readable service description, so you're either left to try and see, or you need some documentation from the service provider, to know what resources you have at hand.
My personal bottom line: if you want to expose some collections of data, and reach (being able to access it from any device) and ease-of-use is more important than reliability and other enterprise-grade features, then REST is the way to go. If you need serious, business-to-business, well-defined, well-documented services that implement things like reliability, transaction support etc. - then you're definitely on the right track with SOAP.
But I really don't see a REST exclusive or SOAP kind of approach - they'll be great scenarios for both.
SOAP will give you a richer set of functionality and you can create strongly typed services, where REST is normally more lightweight and easier to get running.
Edit: If you look at the ws-* standards there is a lot of stuff http://www.soaspecs.com/ws.php. Development tools such as Visual Studio make access very easy to a lot of this though. Typically WS is used more for enterprise SOA type development while REST is used for public API's on web sites (at least thats what I think).
Android (mobile OS) do not have support for SOAP. RESTful services are a much better suit when mobile devices are the target.
I personally prefer RESTful services in most cases since they are more lightweight, easier to debug and integrate against (unless you use a code generator for SOAP).
One point that hasn't been mentioned is overhead. A recent REST project I worked on involved file transfer with items up to 2 GB allowed. If it had been implemented as a SOAP service, that'd be an automatic 30%+ increase in data due to encoding. The overhead with a REST service is all headers, the rest is straight data.
More of a design/conceptual question.
At work the decision was made to have our data access layer be called through webservices. So our website would call the webservices for any/all data to and from the database. Both the website & the webservices will be on the same machine(so no trip across the wire), but the database is on a separate machine(so that would require a trip across the wire regardless). This is all in-house, the website, webservice, and database are all within the same company(AFAIK, the webservices won't be reused by another other party).
To the best of my knowledge: the website will open a port to the webservices, and the webservices will in turn open another port and go across the wire to the database server to get/submit the data. The trip across the wire can't be avoided, but I'm concerned about the webservices standing in the middle.
I do agree there needs to be distinct layers between the functionality(such as business layer, data access layer, etc...), but this seems overly complex to me. I'm also sensing there will be some performance problems down the line.
Seems to me it would be better to have the (DAL)assemblies referenced directly within the solution, thus negating the first port to port connection.
Any thoughts(or links) both for and against this idea would be appreciated
P.S. We're a .NET shop(migrating from vb to C# 3.5)
Edit/Update
Marked Dathan as answer, I'm still not completely sold(I'm still kind of on the fence, though leaning it may not be as bad as I feared), he provided a well thought out answer. I appreciated all the feedback.
Both designs (app to web service to db; app to db via DAL) are pretty standard. Web services are often used when interfacing with clients to standardize the semantics of data access. The web service is usually able to more accurately represent the semantics of your data model than the underlying persistence store, and thus helps the maintainability of the system by abstracting and encapsulating IO-specific concerns. Web services also serve the additional purpose of providing a public interface (though "public" may still mean internal to your company) to your data via a protocol that's commonly accessible across firewalls. When using a DAL to connect directly to the DB, it's possible to encapsulate the data IO concerns in a similar way, but ultimately your client has to have direct access to the database. By restricting IO to well-defined semantics (usually CRUD+Query), you add an additional layer of security. This isn't such a big deal for you, since you're running a web app, though - all DB access is already done from trusted code. The web service does provide an increase in robustness against SQL injection, though.
All web service justifications aside, the real questions are:
How much will it be used? The website/web service/database format does impose slightly higher overhead on the web server - if the website gets hammered, you want to consider long and hard before putting another service on the same machine. Otherwise, the added small inefficiency is probably not a big deal. On the other hand, if the site is getting hammered, you probably want to scale horizontally anyway, and you should be able to scale the web service at the same time.
How much to you gain? One of the big reasons for having a web service is to provide data accessibility to client code - particularly when multiple possible application versions need to be supported. Since your web app is the only client to use the web service, this isn't a concern - it's probably actually less effort to version the app by itself.
Are you looking to expand? You say it probably won't ever be used by any client other than the single web app, but these things have a way of gaining in size. If there's any chance your web app might grow in scope or popularity, consider the web service. By designing around a web service, you're already targeting a modular, multi-host solution, so your app will probably scale with fewer growing pains.
In case you couldn't guess, I'm a web service fan. But the above are also my honest (if somewhat biased) opinions on the subject. If you do go the web service route, be sure to make it simple - keep application logic in the app and service logic in the service, and try to draw a bright line between them when extending the two. And do design your service for efficiency and configure the hosting to keep it running as smoothly as possible.
This is a questionable design, but your shop isn't the only one using it.
Since you're using .NET 3.5 and running on the same machine, you should use WCF with the netNamedPipesBinding, which uses binary data transfer over named pipes, only on the same machine. That should mitigate the performance issue somewhat.
I like the idea because it gives you flexibility. We use a very similar approach because we can have more than 1 type of database storing our data (MSSQL or Oracle) depending on our customer install choices.
It also gives customers the ability to hook into our database if they choose not to use our front end web site. As a result we get an open API for little to no extra effort.
If speed is your most critical issue than you have to lessen your layers. However in most cases the time it takes for your web Service to process the request from the database does not add allot of time. (This is assuming you do your Web Service Layer Correctly, you can easily make it slow if you don't watch it.)