I want to check which web sites is open in browsers ( IE, Firefox, Chrome ) to write a program in C# which can block web site which is in list of forbidden web site. Is there some API of browsers?
The better solution can be to write a TCP/IP filter, like most firewalls do.
UPD: this topic can be relevant: How do I hook the TCP stack in Windows to sniff and modify packets?
There is no generic "browser API" that allows access to this kind of information across all browsers.
I'm pretty sure the idea of doing this by accessing the browsers is doomed from the start. It is going to be almost impossible to implement, require frequent updates, and always extremely easy to circumvent (there are dozens and dozens of browsers that your program will not know.).
The only reliable way is to filter traffic on network level. I would recommend looking into using an existing proxy server or TCP filtering program. There are several Open Source ones that I'm sure you can use as a basis to rewrite or hook into.
The easier solution is to write an http listener that logs the requests.
Fiddler2 is one of these, you can check it out. it logs all incomming and outcomming http content.
Related
I am thinking an HttpModules for IIS, a Pre and Post request handler and some custom module for Apache.
Please only suggest Open Source, we will need to modify the projects.
We want to run these on our high volume production systems so tools that emphasise debugging is not what we are looking for. We want the simplest capture paradigm.
We want to capture data for particular web sites, not for interfaces so I am not looking at packet sniffers.
We would prefer an in-process solution which means avoiding wire-shark, squid or other proxy based solutions.
A nice to have would be the ability to replay a stream of requests in another environment, but I am confident we can build this ourselves if we can capture the requests in a sensible manner.
Any ideas, suggestions and questions will gladly be entertained.
A proxy server that logs is best, but you've already ruled that out. Outside of that, you need a product per web server. For IIS, yes, an HttpModule is the way to go. To replay them, that's likely a carefully crafted log viewer. This is typically a feature of a proxy server. Even Fiddler has the ability to capture and replay.
In a Windows Service I implemented an HttpListener that will handle incoming HTTP Requests to a certain port, parse the query string, insert it in database and send a confirmation response. All works well and i was quite pleased with my solution. However, the clients said that they were a bit skeptical and asked if the same could have been done via a webpage. Like having an HTTPHandler listen to a certain port. Got me thinking. What would you do in my situation?
Go with the HttpListener/Windows Service or HTTPHandler/.aspx?
Thank you very much!
Is there any reason why you don't want to use a web server? We've implemented our own Http serving services because they are fairly unusual in the way they process the requests and would prove taxing on a normally configured IIS instance.
In your situation, this doesn't appear to be the case, so yes, I find myself wondering why you didn't go the webserver route either.
EDIT
Is there any other web facing part of your application? If not, I would concur that #Mr Disappoinment's reasoning is sound. You're only exposing what you need, which is considerably less attack surface than an IIS instance.
I would use something through IIS, simply because I think my clients' IT staffs would require a pretty significant argument for me to be telling them to install custom services on their servers. I don't know enough about the threading behavior of the HttpListener (does it use thread pools? max number of threads? queueing once a max has been hit?) to say for sure, but I'd imagine that your client has similar concerns.
I've some fishy application that makes HTTP requests to a website, i would like to intersect that request and send other data to the server. Is that possible in C#,java or C++?
EDIT: The application isn't mine, i just know the endpoint that it sends http requests
Fiddler might provide the functionality you need. At the very least it may enable you to see what is being sent to the web site.
in Java You can intercept request from Filter
You may want to look into using an HttpModule, whose purpose is to intercept incoming HTTP requests.
The ASP Column: HTTP Modules
Firstly are you aware of how it is connecting to the internet? For example, is it using the settings from Internet Explorer, or is it establishing a direct connection? If the latter, this may be tricky, there is no direct port forwarding as there in Linux, so you'll need some third-party tools to redirect the traffic to a server (which you can write in Java, C++ or C#, I would go for C# if you know it for pure speed of development) In that server you can intercept the request, and then create your own to actually send to the real destination.
Sounds like a cludge, but I think you're stuck with this approach due to the lack of direct port forwarding. You'll have to configure the third-party tool that you use to forward someother well known port to 80, and your server should write to this.
The program has a black list, it contains a list of sites. When the user opens the site in IE (Firefox, Opera, Chrome) he should get an error. (For example 404).
How can I do? It is advisable not writing to the file HOSTS.
Language C#.
What you are describing is a Proxy server:
http://www.squid-cache.org/
The concept behind what you're trying to do is monitoring port 80 outgoing traffic and block any requests addressed to sites/ips contained in the black list.
It's kind of complex posting for you the whole code here.
Regardless, this kind of operation is best suited to a network firewall filter than to a custom C# app that runs on the client.
I have code on my server which works very well. It must crawl a few pages on remote sites to work properly. I know some users may want to abuse my site so instead of running the code which uses webclient and HttpRequest i would like it to run on client side so if it is abused the user may have his IP blacklisted instead of my server. How might i run this code client side? I am thinking silverlight may be a solution but i know nothing about it.
Yes, Silverlight is the solution that lets you run a limited subset of .NET code on client's machine. Just google for silverlight limitations to get more information about what's not available.
I don't know what is the scenario you're trying to implement, and whether you need real-time results, but I guess caching the crawl results could be a good idea?
In case you're after web scraping, you should be able to find a couple of JavaScript frameworks that for you.
I think your options here are Silverlight or somesort of desktop app
Unless maybe there is a jquery library or other client scripting language that can do same things
That's an interesting request (no pun). If you do use Silverlight then maybe instead of porting your logic to it, create a simple Proxy class in it that receives Requests from your server app and shuttles it forward for the dirty work. Same with the incoming Responses: have your Silverlight proxy send it back to the server app.
This way you have the option of running your server app through the Silverlight proxy in some instances, and on its own (with no proxy) in other scenarios. The silverlight plugin should provide a consistent API to program against no matter which browser it's running in.
If using a proxy solution in the web browser, you might even be able to skip Silverlight altogether and use JavaScript/AJAX calls. Of course this kind of thing is usually fraught with browser compatibility issues and it would be an obscure push/pull implementation for sure, but I think JavaScript can access domains and URLs and (in some cases of usage) not be restricted to the one it originated from.
If Silverlight security stands in the way you might look into other kinds of programmable (turing complete) browser plugins like Java, Flash, etc. If memory serves correct, for the Java plugin, it can only communicate over the network with the domain it originated from. This kind of security is too restrictive for your crawling needs.