We Keep Coding

Intercept an HTTP request at browser end to alter some html content

I would like to do as follows. What would be the best way? A general answer will also be fine.
I would like to intercept an HTTP request at the client end to alter some html content. For example, I go to CNN.com and rather than an article that's displaying "Two LA Dogs marry", it should say "Ridiculous title blocked".
It should be smooth that even a secure certificate won't be disturbed.
I am using C#.
Thanks!
UPDATE: Thank you all for your answers!

You can do this with Privoxy via their filter files. Their fun filter is a good example of exactly the sort of substitutions you want to do.
To replace "Two LA Dogs Marry" with "Ridiculous Title Blocked" on cnn.com your action file would look something like this:
{ +filter{ridiculous-title-censor} }
.cnn.com
and your filter file would look like
# FILTER: ridiculous-title-censor Remove ridiculous titles
# This keeps CNN from getting too ridiculous
#
s/Two LA Dogs Marry/Ridiculous Title Blocked/ig

Local HTTP proxy is possible and most generic approach.
I.e. you can use Fiddler to see if it works for you. I supports modifying requests/responses in addition to regular watching for traffic.

Another option if you're using Firefox is to use Greasemonkey scripts. Here's an example script which changes the main headline on cnn.com
If you're not familar with the Javascript coding needed to make Greasemoney scripts, you can use the Platypus add-on to edit the page in place and automatically generate a script file.

You could setup a proxy with HTTPListener. But I would think if you wanted to do it right, you'll need a program that is more low level.
Open 2 TCP Ports (80 & 443) and actively listen for incoming connections.
Once received
Go out and make the request on behalf of the requester
Retrieve HTTP Response
Inspect and change the HTTP Response (where appropriate)
Perhaps modifying headers (where appropriate)
Forward on the response to the requester
I'd start with a simple proxy that just forwards all requests and returns back all responses. Once that is in place you can start inspecting the responses.
That is a good place to start.

Such an approach is the least efficient method of doing what you want to achieve.
If this is a client side application, the client may disable it and thus render it useless.
It is also hard to maintain and requires more complex programming to ensure that it works with SSL.
If using a browser plugin, or toolbar, it would need to be made for a specific browser.
If using a listening server to intercept the HTTP request, this provides complexity and difficulty when the content is encrypted, also unnecessary overhead.
If using a local proxy (meaning that the client's browser needs to point to a local proxy service), maybe the most effective client side method, but still have the disadvantages mentioned above (hard to maintain etc.)
I believe that what you are looking to do is completely reinventing the wheel.
The fact that you have offered a bounty begs the question that you indeed need to do this in C# and client side, but 'censoring bad things' means you need to prohibit content, and any client side method would eventually give the power to the client to remove this limitation.
Personally, I have had great success with Squid and it's content adaptation features.
This means, that the clients need to have a controlled Internet source. Meaning that, if they are all in a LAN and sharing a common Internet gateway, this is easily feasible if you have spare a server to act as a proxy.
I recommend you get a small linux box, which can have a simple Ubuntu Server Edition, then add Squid. The net is full of tutorials, but the level of implementation has become easy enough to do even without them.
I may have gone completely off-topic, but I hope I could assist.

you can come to China ^_^
censorship like this is everywhere, you don't have to implement your own.
Ok, that is a joke, the answer is you can implement browser plugins for such kind of tasks. or maybe you need to implement a ROUTING filter ( GFW like ) on the router.

Taken from here.
It can be done via a Layered Service Provider on Windows.
From Wikipedia:
"A Layered Service Provider (LSP) is a feature of the Microsoft Windows Winsock 2 Service Provider Interface (SPI). A Layered Service Provider is a DLL that uses Winsock APIs to insert itself into the TCP/IP stack. Once in the stack, a Layered Service Provider can intercept and modify inbound and outbound Internet traffic. It allows processing all the TCP/IP traffic taking place between the Internet and the applications that are accessing the Internet (such as a web browser, the email client, etc). "
AdMuncher for example is intercepting and inserting http code to avoid ads. Another suggestion is to find an open-source ad blocking program and see how they've implemented it.

Are you saying you want to intercept this for your own site?
In any case, it needs to be done in javascript or jQuery as C# is not a client-side language.

or, you can code a toolbar, or maybe a simple chrome addon, it's really easy but its not C#
you can search for libraries to monitor browsing trough proxy, like this:
http://httpproxynet.codeplex.com/
The same concept used by java in this project:
http://www.charlesproxy.com/
sounds intresting, good luck :)

A long time ago I've implemented this feature for IE using Plugable Mime Filters so after I've searched about it in c# here in stackoverflow I've founded this post that should help you start with it.
Hope this is useful for you.

Asynchronous Pluggable Protocols can be used for this type of thing. Although, as stated here INFO: Implementing HTTP-like Asynchronous Pluggable Protocols: "For various reasons, Microsoft neither supports nor recommends that you replace or wrap the default HTTP protocol."

TCP/IP vs Web Services for iPhone chat app

How to create a NATIVE chat app for the iPhone? So far we have been exploring a few options:
Creating a web service using php or other web based language and have the app connect to that. Only problem is we can't figure out how to create "push" messaging with this, where the user will not have to refresh the conversation constantly.
Hosting an application on a server such as Windows Azure which will communicate to the iPhone app using TCP/IP. This way it seems like "push" messaging could be achieved by simply sending a packet to the iPhone. However, we have never done this before and don't know if we would run into any unforeseen potholes.
Have any of you made such an app before? If so how did you go about doing it? If not, what method would you recommend?
Thank you in advance!
EDIT:
To tell you exactly what we're trying to do: we need to make an app where a user can join a chat room and send/ receive messages from that chat room. There will also be custom features like that users will have their own profiles, etc. We would also like to make this as flexible as possible, so that we can integrate it on other platforms like android and blackberry later on.
So essentially the part that I'm stuck on is the send/ receive messages from a chat room. What technology should we use server side?

Something with an open socket, like Socket.IO could work. Node.js is a good server-side framework to explore. Here's an related SO question: iPhone Objective-C socket communication with Socket.IO
EDIT:
Question has changed since posting this answer -- originally question asked about web apps. ALso, originally the question was not clear that you wanted answers about the server side more than the client side.
On the server side, I would still recommend Node.js -- sounds like you want to use C# though, which makes me wonder why you're asking again about what server side tech to use. Most languages will provide you with ways to connect a socket to a client and access a database, which are the two main requirements of the app that it sounds like you want to make. Use whatever language you're comfortable with. However, some are going to come with libraries that may come in handy for this type of communication -- Node.js and Ruby on Rails (more useful if you want to do a polling-based solution)
Look at http://code.google.com/p/cocoaasyncsocket/ for a good library for doing socket communication from the iPhone without having to delve too deep into the low-level functions.

I've done this several times. Scaling to 100K concurrent users is non-trivial. If you want an off-the-shelf system I suspect ejabberd may do what you want. although the protocol IMHO is too verbose and uses far more bandwidth than necessary.
If you want to write your own solution and have the flexibility to write your own protocol and have the maximum possible scalability in the future then use a language that allows you to distribute the application across several servers. It is easier to allow that from the get go rather than writing a single server solution then have to retroactively make it distributable.
Having written servers like this in c++, Java and Erlang I would say the easiest and most relevant tool was Erlang. It makes good use of multi core processors and with a good design it facilitates distributing across several servers. C++ was the hardest!
I have also used Java with tools like JETTY and RabbitMQ to write a highly scalable system that required using HTTP as the protocol.
Personally I prefer a custom binary protocol as it allows you to reduce bandwidth to a minimum, and avoids DOS attacks and such as the protocol is well defined and lengths are sent before the packet, where as non binary protocols need to be parsed as they come in, with no idea of how big the packets may be.

Why not try XMPP protocol first? XMPP is based on TCP/IP.
There are several OpenSource server solution, clients, and application libraries. XMPP already supports chat room like service. You can define extension easily.

Asynchronous Communication Between Two Applications

I'm writing a simple little game for my kids - it doesn't really matter what it does, though I couldn't tell you anyway, since I/they haven't quite decided yet! However, I think it will have a server component and a number of client components, and I'm looking at ways that the clients can communicate with the server.
ALL my previous experience... my entire career in fact... has involved the server element either being a database, a web server, or the two in tandem. Neither are appropriate in this case, so I'm curious as to what means I could & should use to communicate between the two.
Obviously, it would be preferable to adopt a technology or technique that I can re-use in my work, where I'm increasingly working with Windows Forms. I imagine there are 1001 different approaches I could adopt; it's a question of sorting the wheat from the chaff.
I've literally just started reading about WCF, but its unclear as yet, if this service-oriented approach is what I'm looking for.
I'm being deliberately vague about what the applications will do; I expect the client will announce their presence to the server, will feed user choices up to the server, and in return, the server will periodically update the client with what is going on in the wider game. The game will be turn-based rather than real-time... and quite low-tech really!
Suggestions? Ideally, with links to good learning resources if any are known.
Conclusion:
I actually thought there might be more viable alternatives; there is Remoting (now depracated), but the consensus says that WCF is the way to go - in my case, self-hosting looks appealing.
Thanks for the responses.

It looks like you're interested in WCF and that is a reasonable technology to use in this case.
When writing a network game the easiest approach is to use the client server approach here too. With WCF you have some different hosting possibilities, hosting in IIS or self hosting. I would go for self hosting to avoid the need for IIS on your home computers.
The service could be hosted in either a windows service or actually in one of the clients. I recommend running as a windows service. The service could very well run on one the same machine as one of the clients.
Edit:
If you want to host the server in one of the clients you could provide a menu option "start service" that starts a self hosted service on that computer (and automatically connect the client part to localhost). After the service has started you can present the computer name that you enter on the "other" computers to connect.
I would suggest to separate the service part into a separate project, then you can easily break out the service to a windows service later on if you like.
Edit2:
By the way, since WCF is driven by calls from the client you need to poll the server for changes. You can google wcf long polling or just long polling for methods to "push" messages from the server to the client.

Check out this article on creating a chat application using WCF -- http://www.codeproject.com/KB/IP/WCFWPFChatRoot.aspx
I've played around with this code in the past and the project was pretty simple to launch locally.

Here is a good book on WCF, which can help get you started. It has quite a bit of a learning curve (I still have barely scratched the surface myself), but I get the sense that it's a very powerful way to set up a service-based application. On the page I linked, check out the Examples link, which includes a bunch of code from the book, including Juval's ServiceModelEx, which has a wide variety of useful classes for working with WCF.

I'm going to go against the crowd here and suggestion NOT using WCF. WCF is a great technology for service oriented architecture, however it is largely written on the idea that clients will not stay connected to the server, but rather connect, send a message, perhaps receive a result, and disconnect.
There is a mechanism that can be used for long connected clients, but frankly, it doesn't work very well, has all kinds of issues and quirks, and is not very reliable in terms of knowing when clients have disconnected or not, or whether the clients know if they are still connected to the server. It's more of a bolted on solution that tries to shoehorn itself into the WCF model.
The other issue is one of firewalls. There can't be a firewall between the initiator and the receiver (or there must be an open port). That means you can't easily have two clients behind firewalls that want to talk to each other. You need some kind of exposed intermediate server that is open. While this problem applies to all solutions, it's easier to manage with straight TCP connections than WCF.
I'm not a huge proponent of rolling your own solution, but WCF really is overkill for simple two way communication between client apps.
I have not yet found a good open source network library for .NET. I know lots of people will chime in with various libraries, but every one i've seen is old and has various flaws. Most seem to be someone that ripped out their library from an app and tried to package it as a generic one, but leaving in all the assumptions of their app.
The problem is that recent versions of .NET have added lots of new network functionality, particularly in terms of asynchronous support. As of yet, i've not seen any libraries which implement these functions.
Anyone want to help me build a good, from scratch, network library based on .net 3.5+?

I would look at Remoting if I was you. It is fairly easy to impliment and you can definatly use it at work.
There is a tutorial on it here:
http://generally.wordpress.com/2007/05/31/a-simple-remoting-example-in-c/

monitor incoming/outgoing http traffic with .net

I want to write a small application that can:
1. monitor URLs requested via a web browser and/or
2. monitor incoming http responses on the local machine
I have been doing some Googling, but I am not finding any clear answers. I am thinking maybe System.Net.Sockets.TCPListener and messing around with it, but I am under the impression that it is either not what I"m looking for or can't handle both things.
I don't need a detailed step by step explanation. Just a small overview would be helpful (if this is even possible) such as (what classes to use, what events to subscribe to, any additional details needed to instantiate necessary objects) I can google the details.

Why not just extend Fiddler to suit your needs?

You may want to use a utility like Wireshark (graphical, but not scriptable from C#, AFAIK). Wireshare uses a library called WinPcap, which does the actual monitoring of the TCP streams on the computer. Decoding of the HTTP headers will need to be done in your application (this is what wireshark/tcpdump/windump should be able to do). You will need to use PInvoke to call the WinPcap DLL. You can find library wrappers by performing a web search for "WinPcap PInvoke".
This question sound very similar to the following:
Is there any way in .NET to programmatically listen to HTTP traffic?
EDIT: Also, Capturing HTTP requests may help

Capturing HTTP requests

Is there a way to monitor and capture all outgoing HTTP requests from a machine using C#?
I need a browser independent way of logging visited URLs.

I use fiddler ( http://www.fiddler2.com )

You may want to use existing network interfaces capturing libraries like pcap or winpcap to do so.
Rewriting all the necessary stuff by yourself would be quite time expensive.
Link to Pcap
Link to WinPcap
Edit : Just saw someone also wrote the C# bindings to winpcap : SharpPcap

Sounds like you're after some kind of "packet sniffing" utility.
Here's a couple of links to articles on the Code Project site for packet sniffers (with downloadable source code) written in C#:
Packet Capture and Analayzer
A Network Sniffer in C#
If you're just after capturing visited URL's, these utilities may be overkill, however, you'll be able to extract a URL from your HTTP packets and discard the rest, however, you may also wish to capture all packet information, in which case, these utilities will help.

You're probably going to save lots of time and effort with some kind of proxy setup. A decent local-machine solution would be Fiddler (requires Windows), or something like a Squid server for a networked solution.

I hope if Wirehark works for you.
It's free and cross-platform.
Also,
"Wireshark is the world's foremost network protocol analyzer, and is the de facto (and often de jure) standard across many industries and educational institutions"
Have look,
Wireshark-Wikipedia

Also, writing a simple http proxy for this purpose in C# is a trivial task.