We Keep Coding

Combine .NET MVC website and Windows services

I am currently involved in a simple to medium complex IOT project. The main purpose of our application is gathering data from our devices and analyzing that data as well as calculating statistics.
On the server side we run a MVC application. Up until now we used Hangfire to schedule the calculations. Hangfire is an amazing tool for scheduling emails and other simple stuff, for more advanced things it's too slow. The calculations can take up a lot of time and are processor-intensive (we are trying to optimize them though), so we need to call them in a background task, a simple API call won't be enough.
I thought about splitting the application into multiple parts, the website, the core and a windows service.
The problem is, I never tried that before and I have no idea what the best practice is to achieve that kind of thing. I searched for examples and articles, but all I found were suggestions to use Hangfire and/or Quartz.NET.
Does anyone have any resources on what the best practice is to build a MVC application, a Windows service and how they could communicate (probably through a queue)? What is the best practice in such a situation?

Although there may be many different possible ways to connect a site with a windows service, I'd probably chose one of the following two, based on your statements:
Direct communication
One way of letting your site send data to your backend windows service would be to use WCF. The service would expose an endpoint. For simplicity's sake this could be a basicHttpBinding or a netTcpBinding. The choice should be made based on your specific requirements; if the data is small then basicHttp may be "sufficient".
The advantage of this approach is that there's relatively little overhead needed: You'll just have to setup the windows service (which you'll have to do anyway) and open a port for the WCF binding. The site acts as client, the service as server. There's nothing special with it, just because the client being a MVC site. You can take almost any WCF tutorial as a starting point.
Note that instead of WCF you could use another technology like .NET Remoting or even sockets just as well. Personally, I often use WCF because I'm quite used to it, but this choice is pretty opinion based.
Queued communication
If reliability and integrity is crucial for your project, then using a queue might be a good idea. Again: depending on your needs, there may come diffeent products into consideration. If you don't need much monitoring and out-of-the-box management goodies, then even a very simplistic technology like MSMQ may be sufficient.
If your demands to the aforementioned points are more relevant, then maybe you should look for something else. Just recently I got in touch with Service Bus for Windows Server (SBWS). It's the Azure Service Bus's little brother which can be used on premises locally on your windows server. The nice thing about it is, that it comes at no extra charge as it's already licensed with your windows server licence.
As with the first point: MSMQ and SBWS are just two examples. There may be a lot of other products like NServiceBus, ZeroMQ or others usable, you name it.

How to make the framework and the dependent applications loosely coupled?

I have a specific case and I want to know the best practice way to handle it.
I make a specific .NET framework (web application). This web application acts like a platform or framework to many other web applications through the following methodology :
We create our dependent web applications (classes for the project business, rdlc reports) in a separate solutions then build them.
After that we add references to the resulted dll in the framework.
And create set of user controls (one for each dependent web application) and put them in a folder in the framework it self.
It works fine but any modification to a specific user control or any modification to any one of the dependent web applications. We have to add the references again and publish the whole framework !!
What I want to do is make those different web applications and the framework loosely coupled. So I could publish the framework one and only one and any modifications to the user controls or the different web applications just publish the updated part rather than the whole framework .
How to refactor my code so I can do this?
The most important thing is :
Never publish the whole framework if the change in any dependent application, just publish the updated part belongs to this application .

If loose coupling is what you are after, develop your "framework(web application)" to function as a WCF web service. Your client applications will pass requests to your web services and receive standard responses in the form of predefined objects.
If you take this route, I recommend that you implement an additional step: Do not use the objects passed to your client applications directly in your client code. Instead, create versions of these web service objects local to each client application and upon receiving your web service response objects, map them to their local counterparts. I tend to implement this with a facade project in my client solution. The facade handles all calls to my various web services, and does the mapping between client and service objects automatically with each call. It is very convenient.
The reason for this is that the day that you decide to modify the objects that your web service serves, you only have to change the mapping algorithms in your client applications... the internal code of each client solution remains unchanged. Do not underestimate how much work this can save you!
Developing WCF web services is quite a large subject. If you are interested, a book that I recommend is Programming WCF Services. It offers a pretty good introduction to WCF development for those who come from a .NET background.

I totally agree with levib, but I also have some tips:
As an alternative to WCF (with its crazy configuration needs), I would recommend ServiceStack. Like WCF it lets you receive requests and return responses in the form of predefined objects, but with NO code generation and minimal configuration. It supports all kinds of response formats, such as JSON, XML, JSV and CSV. This makes it much easier to consume from f.ex. JavaScript and even mobile apps. It even has binaries for MonoTouch and Mono for Android! It is also highly testable and blazing fast!
A great tool for the mapping part of your code is AutoMapper, it lets you set up all your mappings in a single place and map from one object type to another by calling a simple method.
Check them out! :)

Decades of experience says: avoid the framework and you won't have a problem to solve.
Frameworks evolve like cancer. The road to hell is paved with good intentions, and a good portion of those good intentions are embodied in a colossal tumour of a framework all in the name of potential re-use that never really happens.
Get some experience and knowledge when it comes to OO and design, and you'll find endless solutions to your technical problem, such as facades, and mementos, and what have you, but they are not solutions to your real problem.
Another thing, if you are using MS technology, don't bother with anything beyond what .NET offers. Stick with what the MS gods offer because as soon as you digress and become committed to some inhouse framework, your days are numbered.

What are the benefits of migrating our application over to WCF as opposed to continuing to use .NET Remoting?

Alright, so I've asked several questions on StackOverflow about .NET Remoting, and there is always at least one person who just has to chime in, ".NET Remoting is deprecated, use WCF instead." I understand that it's deprecated and there is no guarantee of future support with new versions of the .NET Framework. But what are some other good reasons we would want to move over to WCF? I have seen a few mostly minor annoyances with .NET Remoting, however, this is not enough to change the minds the powers that be who believe firmly in "if it ain't broke, don't fix it". At this time, the only reason that attitude will change is if .NET Remoting is removed from a future version of the .NET Framework, so who knows how long that will be?
Does anybody have any insight as why exactly WCF is "better" than .NET Remoting, or why Remoting is inferior to WCF? What are the pros and cons of each technology? Are there additional things you can do with WCF and not with Remoting?
I mean, it would be great if I could convince them to let us migrate our software over to WCF just to allow a configurable TcpChannel timeout to be set on the client side (this seems to have been broken for a while, no matter what steps or troubleshooting I try), and when this happens, it makes our software look like absolute shite.
Thanks in advance for helping to shed some light on this.

There are plenty of reasons to ditch remoting; a few might include:
lack of transport flexibility
versioning requirements are huge pain
platform dependent (no sensible chance of cross-platform usage)
no chance of usage from the growing mobile market
lack of future development: whatever feature you want added - it won't be
however, I would disagree that WCF is the automatic replacement; WCF itself is a pretty versatile tool, but can be pretty complex, and has restrictions of its own. I haven't used it myself, but I have seen lots of praise for Service Stack, essentially with users describing it as "WCF done right", i.e. the good bits of WCF, without the pain points. However, there are plenty of other options too. One nice thing about the idea of Service Stack, though, is that it iterates pretty quickly, and if it lacks something you want you can change it.

.NET Remoting is now a legacy technology, quoted from MSDN:
This topic is specific to a legacy technology that is retained for backward compatibility with existing applications and is not recommended for new development. Distributed applications should now be developed using the Windows Communication Foundation (WCF).
And here is a performance comparison between WCF and .NET Remoting done in 2007: http://msdn.microsoft.com/en-us/library/bb310550.aspx
To summarize the results, WCF is 25%—50% faster than ASP.NET Web
Services, and approximately 25% faster than .NET Remoting.
So I guess speed is a good reason to drop .NET Remoting.

While the given reasons are probably the driving considerations there are other non-trivial reasons:
Transport independence
IDE tooling
Ease of testing
Maintainability
When you use WCF you can change transport merely by editing your config file. This can be very handy when some sanctimonious system admin won't open a port and you need to use HTTP on port 80 to get through the corporate firewall.
The WCF tooling in Visual Studio is phenomenal. The hardest part is figuring out the URL you need. After that it's just point and click for code generation. There are one or two gotchas with serialisation of collections but broadly speaking if you tell both ends to use arrays it will just plain work. If you need a collection at the destination you can always construct one around the received array, and since LINQ will happily operate on arrays you can fold this into other transformations.
I'm not sure what Stephan P means by pain points. Editing the config can be tricky but Microsoft provides an excellent GUI tool that takes all the guesswork out of it by providing a full tree of options yet generating a sparse config file.
WCF services are easy to test because they have a published interface to which you can connect a test harness. This is more a virtue of SoA in general rather than WCF in particular, but it's still desirable.
WCF makes things a great deal simpler in my code, since neither application nor service is polluted with "routing" code (to determine what ought to process the message content); it looks like simple method calls or implementations. I mostly use WCF as a wrapper for MSMQ, and the only visible consequence of the transport selection is that these method calls must all be void functions because it's a OneWay transport. But that's hardly surprising when the point was persistent queueing.
This all speaks to maintainability. Even for in-house applications, maintenance is a dominant cost, and when you're supporting your software at customer sites poor maintainability can be crippling.
Then there's interoperability with otherwise incompatible platforms. In this case I'm thinking of using HTTP/XML or HTTP/JSON to provide service to web apps written in (eg) PHP.
Going the other way isn't quite so easy but it's fairly straightforward.

I give points for WCF with respect to logging and security.
Logging
WCF has an integrated Logging mechanism that helps you log traces that become a boon during maintanence. In other technologies, developer has to do some work to achieve this but in WCF, all that we have to do is to enable trace by changing the config file and WCF starts providing traces for you.
Security
Security mechanism in WCF is fairly simple and out of box when you look from the implementors perspective but is very robust and highly secure. The best part is that for the highy used and recommended bindings, WCF provides default security which can be trusted to the core. The message security on WSHTTPbinding is an example on these lines.
.NET Framework remoting does not do authentication or encryption by default. Therefore, it is recommended that you take all necessary steps to make certain of the identity of clients or servers before interacting with them remotely
Moreover WCF is an framework to develop Service Oriented applications under microsoft platform mixing both message and rpc style of programming. Which was not in the remoting. Remoting is basically oriented to rpc only.

PHP + .Net Web services for data access - Bad Design Choice?

The company has a PHP app that is in horrible condition. They want to start making plans to redesign it in .NET, however they need to run with the current design because of various reasons that I won't get into here.
They want to make some enhancements to the current design but do it in such a way that those enhancements can be in-part reused by their .NET version when it comes along. One idea to do this was to make the data and business logic portion of the app reside as a .NET webservice that would be consumed by the PHP end.
My question is will this cause problems in PHP? Can PHP consume .NET web services quickly and efficiently? Or is this just a bad design decision?

My question is will this cause
problems in PHP? Can PHP consume .NET
web services quickly and efficiently?
Or is this just a bad design decision?
I have two thoughts here. First to answer your question directly. I don't believe it's a bad design and if the .NET services are written language agnostic then there should be little issue.
The second thought is a "hope". I hope the choice to go with a .NET framework was not due to poorly written PHP. Changing languages because of poor implementation in my opinion is where the design fails. There will be more effort converting to a new language than there would be if the company choose to re-write the PHP and the end result would be a single unified language base with built-in legacy support. But then I'm a PHP fan.

For PHP to consume .NET webservices quickly you'll need to use PHP5 native SOAP Client API, enabling cache to store WSDL locally. If you use PHP4 you can use Nusoap, but it isn't as fast as native classes.

The whole point of having a webservice is interoperability between various development platforms. For instance twitter is a rails-based website and its services are consumed by multitude of various desktop and web applications written in .NET, java, python, etc, through its RESTful web-api. Facebook is PHP and C++ based as far as I know and how many webapps consume it's services through api. SO I don't think it's a bad idea. The question is how you implement this webservice. Meaning, do you want to use it once and then get rid of it or sue it for a long time. If the second option is true - make sure you design your webservice api with that in mind. Also PHP can easily consume XML-RPC and SOAP. I used both ( provided by a Perl based service) without any problem or big hits on performance.
I think using .NET to migrate from PHP is not the smartest choice - but that is somewhat subjective opinion. In my experience it almost always ended being an overkill, badly designed, more expensive to maintain and more buggy - because of the nature of the beast.
P.S.:
I'm not a PHP fan, but I don't believe in converting to .NET for the sake of converting. Also .NET infrastructure is more expensive to maintain and much more labor intensive.

The primary advantage of a webservice is its interoperability, or ability to be consumed by others independent of language. PHP should have something that allows it to consume webservices so that shou;dn't be too big of a deal. The disadvantage will be that it may be a little slower, but this is something you'd have to test out to see how much of an impact that has on your overall solution. Generally most solutions aren't inherently right or wrong, you have to test their usefulness to your particular circumstance.

PHP can consume .NET web services fine, as long as you stay away from WSHTTP based web services and use BASIC HTTP. You can secure BASIC HTTP web services with SSL for security.
WHen you are in Visual Studio 2008, and you have an ASP.NET (could be MVC or not, doesn't matter) project open, right click on the project and select "Add New Item." You will see see something under Web, called Web Service. This will create a .asmx file and you can find tutorials on how to create these basic web services.
Another alternative is to use Windows Communication Foundation, which has a lot of helpful classes, but it can be more complicated. The default configuration for a WCF service is WSHTTP, but it is possible to make a BASIC HTTP web service with the WCF too.

Not really much of a problem at all, just investigate your options.
Web Services can output data in a wide variety of formats. SOAP/XML are the default but there is no reason why you can't do YML, Xml serialized objects, or my current favorite JSON (which makes calling into it from a browser really easy).
Look also into WCF services, I believe they're supposed to supplant the Web Service format.
Finally, if you're looking for best practices check out Service Oriented Architecture. Its a large and varied field and this is exactly the sort of things they talk about.

What am I missing about WCF?

I've been developing in MS technologies for longer than I care to remember at this stage. When .NET arrived on the scene I thought they hit the nail on the head and with each iteration and version I thought their technologies were getting stronger and stronger and looked forward to each release.
However, having had to work with WCF for the last year I must say I found the technology very difficult to work with and understand. Initially it's quite appealing but when you start getting into the guts of it, configuration is a nightmare, having to override behaviours for message sizes, number of objects contained in a messages, the complexity of the security model, disposing of proxies when faulted and finally moving back to defining interfaces in code rather than in XML.
It just does not work out of the box and I think it should. We found all of the above issues while either testing ourselves or else when our products were out on site.
I do understand the rationale behind it all, but surely they could have come up with simpler implementation mechanism.
I suppose what I'm asking is,
Am I looking at WCF the wrong way?
What strengths does it have over the
alternatives?
Under what circumstances should I
choose to use WCF?
OK Folks, Sorry about the delay in responding, work does have a nasty habit of get in the way sometimes :)
Some clarifications
My main paint point with WCF I suppose falls down into the following areas
While it does work out of the box, your left with some major surprises under the hood. As pointed out above basic things are restricted until they are overridden
Size of string than can be passed can't be over 8K
Number of objects that can be passed in a single message is restricted
Proxies not automatically recovering from failures
The amount of configuration while it's there is a good thing, but understanding it all and what to use what and under which circumstances can be difficult to understand. Especially when deploying software on site with different security requirements etc. When talking about configuration, we've had to hide lots of ours in a back-end database because security and network people on-site were trying to change things in configuration files without understanding it.
Keeping the configuration of the interfaces in code rather than moving to explicitly defined interfaces in XML, which can be published and consumed by almost anything. I know we can export the XML from the assembly, but it's full of rubbish and certain code generators choke on it.
I know the world moves on, I've moved on a number of times over the last (ahem 22 years I've been developing) and am actively using WCF, so don't get me wrong, I do understand what it's for and where it's heading.
I just think there should be simpler configuration/deployment options available, easier set-up and better management for configuration (SQL config provider maybe, rather than just the web.config/app.config files).

I use WCF all the time now and I share your pain. It seems like it was grossly over-engineered, but we are going to be stuck with it for a long, long time so I'm trying to learn it.
One thing I am certain about, XML sucks. I've had nothing but problems using XML to control it and have since switched to handling everything via code.

The concerns you listed were:
Size of string than can be passed can't be over 8K
Number of objects that can be passed in a single message is restricted
Proxies not automatically recovering from failures
The amount of configuration while it's there is a good thing, but understanding it all and what to use what and under which circumstances can be difficult to understand. Especially when deploying software on site with different security requirements etc. When talking about configuration, we've had to hide lots of ours in a back-end database because security and network people on-site were trying to change things in configuration files without understanding it.
Keeping the configuration of the interfaces in code rather than moving to explicitly defined interfaces in XML, which can be published and consumed by almost anything. I know we can export the XML from the assembly, but it's full of rubbish and certain code generators choke on it.
here's my take:
(1) addressed a valid concern that customers had with ASMX. It was too wide-open, with no way to easily control it. The 8k limit is easily lifted if you know where to look. I guess you can count that as a surprise, but it's more of a one-time thing. Once you know about it, you can lift it and be done with it forever, if you choose.
(2) is also configurable.
(3) is known, but there are boilerplate ways to work around this. The StockTrader code for example, demonstrates a proven pattern. You can re-use the code in your own app. Not sure if this is fixed in WCF for .NET 4.0. I know it was an open request.
(4) The config is a beast. This is a concern for a lot of people. The problem here is that WCF is so flexible, and config of all of that flexibility is exposed through xml files. It can be overwhelming. An approach that seems to work is to take it in small bites, as you need it.
(5) I don't understand.

I vastly prefer ASP.NET MVC and Web API over WCF. If I had to summarize WCF to a developer who was just being introduced to it, I would say, "WCF is a well-meaning attempt to replace over-engineered, Java EE style RPC development." Unfortunately, many of the decisions made require you to become an expert in configuring low level, unimportant items (message sizes, timeouts, uninteresting protocol elements, etc.) while abstracting absolutely critical pieces (URL design, parameter serialization, response serialization, etc.). The difference in productivity and aggravation between teams I know using WCF vs. Web API is night and day.
To come clean a little: I have always hated the core concept of .NET Remoting. I feel that developers need a thorough understanding of the resource structure of their application and how these resources are serialized. Furthermore, the use of the "POST" verb for simple data retrieval is worrisome in a read heavy application that needs to scale.

I'll address the rest of your issues after clarification. In the meantime, I can address your question on when you should choose to use WCF: always.
WCF is the replacement for the old ASMX technologies, including WSE. It is also the replacement for .NET Remoting. It is the only technology upon which high-level communications features in .NET will be based for the forseeable future.
For example, consider Windows Azure. It was not inevitable that the new concept of "cloud computing" would have its communications aspects covered by WCF. Yet, WCF was flexible enough to be extended to cover those cases, with very little change in code.
If you're having trouble with WCF, then you'd do well to make sure Microsoft knows about it. WCF is the present and future of web service and other service-oriented development in .NET, so they've got a very strong incentive to listen to you and resolve your pain points. Either contact them directly through Connect, or ask questions here on SO (tag with WCF, please), and a lot of people will help you.

Biggest advantage of using WCF from a programmer's point of view: separates the definition of exposed services (operations, contracts, etc.) from the protocol's specific details, unlike ASMX where you expose a class as a web service directly in the code using attributes. Using a real example of mine: we where able to easily switch the transport protocol between web services and named pipes, whatever suited better the deployment and performance needs, without changing a line of code.

WCF is intended to SOA methodologies. Work professionally using it is a nightmare. I delivered a SOA solution using WCF as tool and hell, hundreds configurations and hidden tips! My past distributed solution using old style Web Services and Remoting were more stable. I've spent days working out the solution for the error "The underlying connection was closed: An unexpected error occurred" which makes no sense to happen for one method among 4 in the same contract. I'm very disappointed. It took me back through time where .net was first introduced with lots of promises and when we got hands on, hell, log problems came up!

To address the problem of maintenance nightmare of application config, some standard like UDDI or WS-Discovery exist, WS-Discovery will be supported by WCF in .NET 4.0.
Keeping the configuration of the
interfaces in code rather than moving
to explicitly defined interfaces in
XML, which can be published and
consumed by almost anything. I know we
can export the XML from the assembley,
but it's full of rubbish and certain
code generators choke on it.
Can you be more explicit ? I think you are talking about service behavior configured in code.
You can easily code behavior extensions to configure what your are talking about in config file instead of code BUT I think that if microsoft didn't do that there is a good reason.
For example a service with this behavior :
[ServiceBehavior(InstanceContextMode=InstanceContextMode.PerCall, ConcurrencyMode=ConcurrencyMode.Single)]
The implementation knows that the instance is not shared between multiple thread so it's developed differently than :
[ServiceBehavior(InstanceContextMode=InstanceContextMode.Single, ConcurrencyMode=ConcurrencyMode.Multiple)]
In this case the service implementation should take care about concurency problems.
The implementation is coupled with the attribute ServiceBehavior, so moving this behavior in a XML file is not a good idea.
What if you can change a InstanceContextMode.PerCall service to a InstanceContextMode.Single service inside the config file ? You break the application !

Looking at how you mention XML and SQL, you are using WCF to build a web application or an actual web service (service on the Web, and not just SOAP exchange).
It helps thinking about WCF as a replacement for .NET Remoting (or DCOM, CORBA etc), which also happens to support web services as one of the transports. Interfaces declared in assemblies, behavior of proxies, certain configuration options and other aspects of the framework that look unnatural and complicated from perspective of web apps - actually do work out of the box for DCOM-style systems of distributed objects.
To answer the question: no, you are not missing anything and using WCF for web applications is complicated, because WCF is not a framework for building web applications. Probably such framework can be built on top of it, but I would hate to see WCF itself changed to move into web realm.