I have code on my server which works very well. It must crawl a few pages on remote sites to work properly. I know some users may want to abuse my site so instead of running the code which uses webclient and HttpRequest i would like it to run on client side so if it is abused the user may have his IP blacklisted instead of my server. How might i run this code client side? I am thinking silverlight may be a solution but i know nothing about it.
Yes, Silverlight is the solution that lets you run a limited subset of .NET code on client's machine. Just google for silverlight limitations to get more information about what's not available.
I don't know what is the scenario you're trying to implement, and whether you need real-time results, but I guess caching the crawl results could be a good idea?
In case you're after web scraping, you should be able to find a couple of JavaScript frameworks that for you.
I think your options here are Silverlight or somesort of desktop app
Unless maybe there is a jquery library or other client scripting language that can do same things
That's an interesting request (no pun). If you do use Silverlight then maybe instead of porting your logic to it, create a simple Proxy class in it that receives Requests from your server app and shuttles it forward for the dirty work. Same with the incoming Responses: have your Silverlight proxy send it back to the server app.
This way you have the option of running your server app through the Silverlight proxy in some instances, and on its own (with no proxy) in other scenarios. The silverlight plugin should provide a consistent API to program against no matter which browser it's running in.
If using a proxy solution in the web browser, you might even be able to skip Silverlight altogether and use JavaScript/AJAX calls. Of course this kind of thing is usually fraught with browser compatibility issues and it would be an obscure push/pull implementation for sure, but I think JavaScript can access domains and URLs and (in some cases of usage) not be restricted to the one it originated from.
If Silverlight security stands in the way you might look into other kinds of programmable (turing complete) browser plugins like Java, Flash, etc. If memory serves correct, for the Java plugin, it can only communicate over the network with the domain it originated from. This kind of security is too restrictive for your crawling needs.
Related
I want to check which web sites is open in browsers ( IE, Firefox, Chrome ) to write a program in C# which can block web site which is in list of forbidden web site. Is there some API of browsers?
The better solution can be to write a TCP/IP filter, like most firewalls do.
UPD: this topic can be relevant: How do I hook the TCP stack in Windows to sniff and modify packets?
There is no generic "browser API" that allows access to this kind of information across all browsers.
I'm pretty sure the idea of doing this by accessing the browsers is doomed from the start. It is going to be almost impossible to implement, require frequent updates, and always extremely easy to circumvent (there are dozens and dozens of browsers that your program will not know.).
The only reliable way is to filter traffic on network level. I would recommend looking into using an existing proxy server or TCP filtering program. There are several Open Source ones that I'm sure you can use as a basis to rewrite or hook into.
The easier solution is to write an http listener that logs the requests.
Fiddler2 is one of these, you can check it out. it logs all incomming and outcomming http content.
Is it realistic to use the C# .Net class HttpListener as the foundation for a production caliber web server?
The http web service I need to host contains no .aspx or static files. All http responses are dynamic and generated in c# code that is invoked via a few switch statements which inspect a restful url format.
My thinking is that IIS is really a user-mode wrapper around the Windows o/s HTTP-SYS kernel module that does all the heavy duty network handling and so is HttpListener.
I already have a basic multithreaded web server running which is excellent for development because it starts in debug mode in an instance, now I am thinking do I need the overkill of IIS for production. A low memory footprint is another attraction.
You have two serious choices here. And no, coding your own Web Server with HttpListener isn't production-grade.
1) Use IIS. It has a ton of features for security, performance, and perhaps more importantly, manageability, that you would have to reinvent yourself. Like remote administration, logging, integrated Windows Security, etc.
2) Use WCF and create a ServiceHost to host your files. Then you will have to implement your own services and find a way to manage their lifetimes. You can do it, but again, if you're talking RESTFul web calls, IIS really is the way to go.
Manually rolling your own should be avoided. IIS has changed a lot in the past 10 years. It's by no means a big monolithic server anymore. They've modularized just about everything, especially in Windows 2008, so you get a lean and fast system.
Well, as it was said - try to use IIS at first.
HttpListener is not bad at all - that's fastest managed listener server you can have now (faster than TcpListener and faster than Socket class). And it's actually the same core as with IIS. But IIS has a lot of more things.
I can't say IIS is monolith - hosting usage shown that it became worse in Win2008 in terms of stability and management. But your hand-made solution can be much worse. And also don't forget - http.sys much more customizable than HttpListener. I.e. you can't do streaming with HttpListener, but you can do that with http.sys - Question about HttpListener streaming
But if you will have enough power as a developer - you can try to write up own http.sys wrapper and that is best approach of writing own web server in Windows.
Rubbish, Roll your own. The architecture allows it. Be aware though there are some strange behaviours in the Class. Closing it down a couple of times in an NT service makes it flakey as a bag of puff pastries.
If you run it on Console, no problems whatsoever, run it async and all should be well, however, starting and stopping the darn thing. thats a different issue which I am currently struggling with as no errors are being produced from the hermetically Microsoft sealed classes.
I feel python coming on with a little dash of cherryPy
If you write it, then you'll have to maintain it. Microsoft has already written a web server - you should use it.
I would like to have a client-server application written in .NET which would do following:
server is running Linux
on the server there is SQL database (mySQL) containing document URLs
What we want:
- server side would regularly crawl all URLs and create a full text index for them
- client side would be able to perform a query into this index using GUI
The client application is written in .NET using C#. Besides of searching in documents it will be able to do a lot of other things which are not described here and which are done client-side very well.
We would like to use C# for the server side as well, but we have no experience in this area. How are things like this usually done?
Clarifying question now based on some answers:
The thing which is most unclear to me is how client-server communication is usually handled. Is client and server usually using sockets, caring about details like IP addresses, ports or NAT traversal? Or are there some common frameworks and patters, which would make this transparent, and make client-server messaging or procedure calling easy? Any examples or good starting points for this? Are there some common techniques how to handle the fact a single server is required to server multiple clients at the same time?
To use c# on Linux you will need to use Mono. This is an open source implementation of the CLR specification.
Next you need to decide on how to communicate between server and client, from the lowest level of just opening a TCP/IP socket and sending bits up and down, to .Net remoting, to WCF, to exposing webservices on the server. I do not know how compleat WCF implementation is on mono, also I think you may have issue with binary remoting between mono and MS .Net .
I would suggest RPC style WebServices offer a very good solution. WebServices also have the advantage of alowing clients from other platforms to connect easily.
EDIT
In response to the clarification of the question.
I would suggest using mono/ASP.NET/WebServices on the server, if you wish to use c# on both server and client.
One assumption I have made is that you can do a client pull model, where every message is initiated by the client. Using another approach could allow the server to push events to the client. Given the client has the ability to pole the server regularly I don't consider this much of a draw back but it may be depending on the type of application you are developing.
Mono allow execution of c# (compiled to IL) on a Linux box. Mono ASP.NET allows you to use the standard ASP.NET and integrate into Apache see http://www.mono-project.com/ASP.NET and finally WebServices allow you to communicate robustly in a strongly typed manner between you client and your server.
Using this approach negates most of the issues raised in your clarification and makes them someone else's problem.
Sockets/SSL - is taken care of by standard .Net runtime on the client and Apache on the server.
IPAddress/ports/NAT traversal - Is all taken care of. DNS look up will get the servers IP. Open socket will allow the server to respond through any firewall and NAT setup.
Multiple Clients - Apache is built to handle multiple clients processing at the same time as is ASP.NET, so you should not encounter any problems there.
As many have already mentioned there are a number of thing that you have mentioned which are going to cause you pain. I'm not going to go into those, instead I will answer your original question about communication.
The current popular choice in this kind of communication is web services. These allow you to make remote calls using the HTTP protocol, and encoding the requests and responses in XML. While this method has its critics I have found it incredibly simple to get up and running, and works fine for nearly all applications.
The .NET framework has built in support for web services which can definitely be called by your client. A brief look at the mono website indicates that it has support for web services also, so writing your server in C# and running it under mono should be fine. Googling for "C# Web Service Tutorial" shows many sites which have information about how to get started, here is a random pick from those results:
http://www.codeguru.com/Csharp/Csharp/cs_webservices/tutorials/article.php/c5477
have a look at Grasshopper:
"With Grasshopper, you can use your favorite development environment from Microsoft® to deploy applications on Java-enabled platforms such as Linux"
Or see here
The ideea is to convert your app to Java and then run it on Tomcat or JBoss.
Another approach: use the Mod_AspDotNet module for Apache, as described here.
This Basic Client/Server Chat Application in C# looks like a kind of example which might be a starting point for me. Relevant .NET classes are TcpClient and TcpListener
I have some shared server web hosting in the States (I'm from the UK), which allows me to publish PHP and .NET applications. I cannot install my own software onto the remote server, but I'd like to set up a web forwarding proxy for accessing sites that serve different content depending on what country you're from.
My C# and ASP.NET skills are OK, but my PHP is very limited. Are there any solutions that anyone would recommend for this sort of problem? The proxies I've investigated all seem to require installation on the server machine itself, whereas I'm just looking for something that's accessible from a URL.
Obviously, as the requests are coming from the UK, the headers will have to be manipulated by the proxy before forwarding them on. I was going to code my own HTTP handler in C#, but I don't want to reinvent the wheel if there's something out there already ;)
Although quite old, the Org.Mentalis.Proxy could be a good starting point for an example proxy implementation in C#. You can find it here: http://www.mentalis.org/soft/projects/proxy/
maybe this script is helping you? phproxy
I have 50+ kiosk style computers that I want to be able to get a status update, from a single computer, on demand as opposed to an interval. These computers are on a LAN in respect to the computer requesting the status.
I researched WCF however it looks like I'll need IIS installed and I would rather not install IIS on 50+ Windows XP boxes -- so I think that eliminates using a webservice unless it's possible to have a WinForm host a webservice?
I also researched using System.Net.Sockets and even got a barely functional prototype going however I feel I'm not skilled enough to make it a solid and reliable system. Given this path, I would need to learn more about socket programming and threading.
These boxes are running .NET 3.5 SP1, so I have complete flexibility in the .NET version however I'd like to stick to C#.
What is the best way to implement this? Should I just bite the bullet and learn Sockets more or does .NET have a better way of handling this?
edit:
I was going to go with a two way communication until I realized that all I needed was a one way communication.
edit 2:
I was avoiding the traditional server/client and going with an inverse because I wanted to avoid consuming too much bandwidth and wasn't sure what kind of overhead I was talking about. I was also hoping to have more control of the individual kiosks. After looking at it, I think I can still have that with WCF and connect by IP (which I wasn't aware I could connect by IP, I was thinking I would have to add 50 webservices or something).
WCF does not have to be hosted within IIS, it can be hosted within your Winform, as a console application or as windows service.
You can have each computer host its service within the winform, and write a program in your own computer to call each computer's service to get the status information.
Another way of doing it is to host one service in your own computer, and make the 50+ computers to call the service once their status were updated, you can use a database for the service to persist the status data of each node within the network. This option is easier to maintain and scalable.
P.S.
WCF aims to replace .net remoting, the alternatives can be net.tcp binding or net.pipe
Unless you have plans to scale this to several thousand clients I don't think WCF performance will even be a fringe issue. You can easily host WCF services from windows services or Winforms applications, and you'll find getting something working with WCF will be fairly simple once you get the key concepts.
I've deployed something similar with around 100-150 clients with great success.
There's plenty of resources out on the web to get you started - here's one to get you going:
http://msdn.microsoft.com/en-us/library/aa480190.aspx
Whether you use a web service or WCF on your central server, you only need to install and configure IIS on the server (and not on the 50+ clients).
What you're trying to do is a little unclear from the question, but if the clients need to call the server (to get a server status, for example), then they just call a method on the webservice running on the server.
If instead you need to have the server call the clients from time to time, then you'll need to have each client call a sign-in method on the server webservice each time the client starts up. The sign-in method would take a delegate method from the client as a parameter. The server would then call this delegate when it needed information from the client.
Setting up each client with its own web service would represent an inversion of the traditional (one server, multiple clients) client/server architecture, and as you've already noted this would be impractical.
Do not use remoting.
If you want robustness and scalability you end up ruling out everything but what are essentially stateless remote procedure calls. Since this is exactly the capability of web services, and web services are simpler and easier to build, remoting is an essentially pointless technology.
Callbacks with remote delegates are on the performance/reliability forbidden list, so if you were thinking of using remoting for that, think again.
Use web services.
I know you don't want to be polling, but I don't think you need to. Since you say all your units are on a single network segment then I suggest UDP for broadcast change notifications, essentially setting a dirty flag, and allowing the application to (re-)fetch on demand. It's still not reliable but it's easy and very fast because it's broadcast.
As others have said you don't need IIS, you can self-host. See ServiceHost class for details on how to do this.
I'd suggest using .NET Remoting. It's quite easy to implement and doesn't require anything else.
For me its is better to learn networking.. or the manual way of socket communication.. web services are mush slower because it contains metadata..
your clients and the servers can transform to multithreaded application. just imitate the request and response architecture. it is much easy to implement a network application like this..
If you just need a status update, you can use much simpler solution, such as simple tcp server/client messaging or like orrsella said, remoting. WCF is kinda overkill here.
One note though, if all your 50+ kiosk is connected via internet, then you might need use VPN or have an open port on each kiosk(which is a security risk) so that your server can retrieve status update from each kiosk.
We had a similiar situation, but the status is send to our server periodically, so we only have 1 port to protect/secure. The frequency of the update is configurable as to accomodate slower clients.
As someone who implemented something like this with over 500+ clients and growing:
Message Queing is the way to go.
We have gone from an internal developed TCP server and client to WCF polling and ended up with Message queing. It's the only guaranteed way to get data to and from clients and servers over the internet. As a bonus, many of these solutions have an extensive framework makeing it trivial to implement publish-subscribe, Send-one-way, point-to-point sending, Request-reply. Some of these are possible with WCF but it will involve crying, shouting, whimpering and long nights not to mention gallons of coffee.
A couple of important remarks:
Letting a process poll the clients instead of the other way around = Bad idea.. it is not scalable at all and you will soon be running in to trouble when the process is take too long to complete.. Not to mention having to handle all the ip addresses ( do you have access to all clients on the required ports ? What happpens when the ip changes etc..)
what we have done: The clients sends status updates to a central message queue on a regular interval ( you can easily implement live updates in the UI), it also listens on it's own queue for a GetStatusRequest message. if it receives this, it answers ( has a timeout).. this way, we can see overal status of all clients at all times and get a specific status of a specific client when needed.
Concerning bandwidth: kiosk usually show images/video etc.. 1Kb or less status messages will not be the big overhead.
I CANNOT stress enough that the current design you present will have a very intensive development cycle AND will not scale or extend well ( trust me, we have learned this lesson). Next to this, building a good client/server protocol for this type of stuff is a hard job that will be totally useless afterwards if you make a design error ( migrating a protocol is not easy)
We have built our solution ontop of ActiveMQ ( using NMS library c#) and are currently extending Simple Service Bus for our internal workings.
We only use WCF for the communication between our winforms app and the centralized service(s)