I'm working on a solution to redirect http requests from my browser to another pc.
In a nutshell: Is there a more accurate way to send an incoming HttpListenerRequest as an HttpRequest to a client?
Background: The browser sends http requests to 127.0.0.1:9666 (yes, click'n'load), which should be answered by a download program.
In my case, the download program isn't running on the machine which runs the browser.
Port Forwarding tools like these don't work for me: http://www.quantumg.net/portforward.php
So I decided to write a specific click'n'load redirector in C#.
My problem is to redirect the browser's request. ATM, I listen on 127.0.0.1:9666, send some information to a client, the client redirects this to the downloader program, which answers. (long term short: this is redirected to the browser).
But using pre-defined cases to redirect only the core-information doesn't seem quite right for me. It should be possible to redirect the whole request. So I don't need to differ between various cases, and this should be more accurate.
Is there any other way to redirect this request, except from copying all headers and properties?
The Server uses an HttpListener to listen for the browser's requests.
Then, the server sends an HttpReqest to the Client.
The Client uses an HttpListener to listen for the server's requests.
The client sends an HttpRequest to the Downloader,
uses GetResponse, and sends this as a response to the server, which responds to the browser.
I've looked up some functions in the MSDN, but didn't find a good way to "copy a whole request".
So here is my problem code:
Thread.Sleep(1500);
//Read client's/JDownloader's stream, send to browser/redirector
do
{
Read = Outstream.Read(Buffer, 0, Buffer.Length);
Totalr += Read;
Instream.Write(Buffer, 0, Read);
Sent += Read;
// Bufferstr += ASCIIEncoding.ASCII.GetString(Buffer, 0, Read); debugging
} while (Read != 0);
Without the Thread.Sleep, the first read returns 171 bytes were read (just and exactly the HTTP header, buffer-length is 1024bytes).
second iteration: when executing Outstream.Read, nothing happens. No matter how long I wait. It seems, the reader is waiting for traffic to receive, but there is no traffic to receive (weird...) When the thread sleeps for 500-1500ms, the first read returns 351bytes were read (the complete http request), and then again, second iteration, nothing.
This happens when reading from the browser's or JDownloader's NetworkStream. They never return 0. A dirty method to get this working is to replace the while-argument with Read == Buffer.Length, dirty because it will fail when exactly Buffer.Length bytes are received (endless waiting again, yay).
The DataAvailable property also dosn't always seem to be right, sometimes it is set to false, when the program didn't even read something from the stream, but there were bytes to receive.(also weird...)
Any other ideas for proper receiving loops?
Nobody? short summary of my problem: neither the browser's request stream, nor JDownloaders response stream return 0. When attempting another read, the program just waits for more bytes to receive.
So I don't know a failsafe method to read the entire stream.
Just repeating until the amount of bytes read is lower than the buffer's length will result in previously mentioned endless wait for bytes, when streamlength % bufferlength == 0.
Also, it seems JDownloader needs more time to generate an answer and write it into the stream than my program attempts to read the stream. So I will only receive a part of the stream. Another read attempt will result in, hooray, an endless wait for bytes.
Is there another way to receive the whole stream without a static delay (via thread sleep)?
Can you just open a TcpListener and forward the raw bytes? That way you do not need to concern yourself with HTTP at all.
What you ask for is basically a proxy implementation - building a well working proxy is no simple task as you will have to understand and handle HTTP etc. in both directions.
I would recommend to either use an existing library for that OR some configurable proxy:
http://www.mentalis.org/soft/projects/proxy/ (with source)
http://fusion.corsis.eu/ (with source)
http://www.wingate.com/
http://www.squid-cache.org/
Related
Basically the title... I'd like to have same feedback on weather NamedPipeServerStream object successfully received a value. This is the starting code:
static void Main(string[] args){
Console.WriteLine("Client running!");
NamedPipeClientStream npc = new NamedPipeClientStream("somename");
npc.Connect();
// npc.WriteTimeout = 1000; does not work, says it is not supported for this stream
byte[] message = Encoding.UTF8.GetBytes("Message");
npc.Write(message);
int response = npc.ReadByte();
Console.WriteLine("response; "+response);
}
I've implemented a small echo message from the NamedPipeServerStream on every read. I imagine I could add some async timeout to check if npc.ReadByte(); did return a value in lets say 200ms. Similar to how TCP packets are ACKed.
Is there a better way of inspecting if namedPipeClientStream.Write() was successful?
I'd like to have same feedback on weather NamedPipeServerStream object successfully received a value
The only way to know for sure that the data you sent was received and successfully processed by the client at the remote endpoint, is for your own application protocol to include such acknowledgements.
As a general rule, you can assume that if your send operations are completing successfully, the connection remains viable and the remote endpoint is getting the data. If something happens to the connection, you'll eventually get an error while sending data.
However, this assumption only goes so far. Network I/O is buffered, usually at several levels. Any of your send operations almost certainly involve doing nothing more than placing the data in a local buffer for the network layer. The method call for the operation will return as soon as the data has been buffered, without regard for whether the remote endpoint has received it (and in fact, almost never will have by the time your call returns).
So if and when such a call throws an exception or otherwise reports an error, it's entirely possible that some of the previously sent data has also been lost in transit.
How best to address this possibility depends on what you're trying to do. But in general, you should not worry about it at all. It will typically not matter if a specific transmission has been received. As long as you can continue transmitting without error, the connection is fine, and asking for acknowledgement is just unnecessary overhead.
If you want to handle the case where an error occurs, invalidating the connection, forcing you to retry, and you want to make the broader operation resumable (e.g. you're streaming some data to the remote endpoint and want to ensure all of the data has been received, without having to resend data that has already been received), then you should build into your application protocol the ability to resume, where on reconnecting the remote endpoint reports the number of bytes it's received so far, or the most recent message ID, or whatever it is your application protocol would need to understand where it needs to start sending again.
See also this very closely-related question (arguably maybe even an actual duplicate…though it doesn't mention named pipes specifically, pretty much all network I/O will involve similar issues):
Does TcpClient write method guarantees the data are delivered to server?
There's a good answer there, as well as links to even more useful Q&A in that answer.
I have been reading about TCP packet and how they can be split up any number of times during their voyage. I took this to assume I would have to implement some kind of buffer on top of the buffer used for the actual network traffic in order to store each ReceiveAsync() until enough data is available to parse a message. BTW, I am sending length-prefixed, protobuf-serialized messages over TCP.
Then I read that the lower layers (ethernet?, IP?) will actually re-assemble packets transparently.
My question is, in C#, am I guaranteed to receive a full "message" over TCP? In other words, if I send 32 bytes, will I necessarily receive those 32 bytes in "one-go" (one call to ReceiveAsync())? Or do I have to "store" each receive until the number of bytes received is equal to the length-prefix?
Also, could I receive more than one message in a single call to ReceiveAsync()? Say one "protobuf message" is 32 bytes. I send 2 of them. Could I potentially receive 48 bytes in "one go" and then 16 in another?
I know this question shows up easily on google, but I can never tell if it's in the correct context (talking about the actual TCP protocol, or how C# will expose network traffic to the programmer).
Thanks.
TCP is a stream protocol - it transmits a stream of bytes. That's all. Absolutely no message framing / grouping is implied. In fact, you should forget that Ethernet packets or IP datagrams even exist when writing code using a TCP socket.
You may find yourself with 1 byte available, or 10,000 bytes available to read. The beauty of the (synchronous) Berkeley sockets API is that you, as an application programmer don't need to worry about this. Since you're using a length-prefixed message format (good job!) simply recv() as many bytes as you're expecting. If there are more bytes available than the application requests, the kernel will keep the rest buffered until the next call. If there are fewer bytes available than required, the thread will either block or the call will indicate that fewer bytes were received. In this case, you can simply sleep again until data is available.
The problem with async APIs is that it requires the application to track a lot more state itself. Even this Microsoft example of Asynchronous Client Sockets is far more complicated than it needs to be. With async APIs, you still control the amount of data you're requesting from the kernel, but when your async callback is fired, you then need to know the next amount of data to request.
Note that the C# async/await in 4.5 make asynchronous processing easier, as you can do so in a synchronous way. Have a look at this answer where the author comments:
Socket.ReceiveAsync is a strange one. It has nothing to do with async/await features in .net4.5. It was designed as an alternative socket API that wouldn't thrash memory as hard as BeginReceive/EndReceive, and only needs to be used in the most hardcore of server apps.
TCP is a stream-based octet protocol. So, from the application's perspective, you can only read or write bytes to the stream.
I have been reading about TCP packet and how they can be split up any number of times during their voyage.
TCP packets are a network implementation detail. They're used for efficiency (it would be very inefficient to send one byte at a time). Packet fragmentation is done at the device driver / hardware level, and is never exposed to applications. An application never knows what a "packet" is or where its boundaries are.
I took this to assume I would have to implement some kind of buffer on top of the buffer used for the actual network traffic in order to store each ReceiveAsync() until enough data is available to parse a message.
Yes. Because "message" is not a TCP concept. It's purely an application concept. Most application protocols do define a kind of "message" because it's easier to reason about.
Some application protocols, however, do not define the concept of a "message"; they treat the TCP stream as an actual stream, not a sequence of messages.
In order to support both kinds of application protocols, TCP/IP APIs have to be stream-based.
BTW, I am sending length-prefixed, protobuf-serialized messages over TCP.
That's good. Length prefixing is much easier to deal with than the alternatives, IMO.
My question is, in C#, am I guaranteed to receive a full "message" over TCP?
No.
Or do I have to "store" each receive until the number of bytes received is equal to the length-prefix? Also, could I receive more than one message in a single call to ReceiveAsync()?
Yes, and yes.
Even more fun:
You can get only part of your length prefix (assuming a multi-byte length prefix).
You can get any number of messages at once.
Your buffer can contain part of a message, or part of a message's length prefix.
The next read may not finish the current message, or even the current message's length prefix.
For more information on the details, see my TCP/IP .NET FAQ, particularly the sections on message framing and some example code for length-prefixed messages.
I strongly recommend using only asynchronous APIs in production; the synchronous alternative of having two threads per connection negatively impacts scalability.
Oh, and I also always recommend using SignalR if possible. Raw TCP/IP socket programming is always complex.
My question is, in C#, am I guaranteed to receive a full "message" over TCP?
No. You will not receive a full message. A single send does not result in a single receive. You must keep reading on the receiving side until you have received everything you need.
See the example here, it keeps the read data in a buffer and keeps checking to see if there is more data to be read:
private static void ReceiveCallback(IAsyncResult ar)
{
try
{
// Retrieve the state object and the client socket
// from the asynchronous state object.
StateObject state = (StateObject)ar.AsyncState;
Socket client = state.workSocket;
// Read data from the remote device.
int bytesRead = client.EndReceive(ar);
if (bytesRead > 0)
{
// There might be more data, so store the data received so far.
state.sb.Append(Encoding.ASCII.GetString(state.buffer, 0, bytesRead));
// Get the rest of the data.
client.BeginReceive(state.buffer, 0, StateObject.BufferSize, 0,
new AsyncCallback(ReceiveCallback), state);
}
else
{
// All the data has arrived; put it in response.
if (state.sb.Length > 1)
{
response = state.sb.ToString();
}
// Signal that all bytes have been received.
receiveDone.Set();
}
}
catch (Exception e)
{
Console.WriteLine(e.ToString());
}
}
See this MSDN article and this article for more details. The 2nd link goes into more details and it also has sample code.
Quite a simple question - I've been having some TCP packets go missing (they aren't received by the remote socket), so I'm considering resending any packets that don't get received.
However I need to know when send doesn't work in order to do this! I know Send() returns an integer containing the 'The number of bytes sent to the Socket', but even when the other computer receives nothing, this is always the length of the entire buffer, indicating that everything was (theoretically) sent. I also know there's a Socket.Connected property, but that is false even when that data is received, and sometimes true even when it isn't, so that doesn't help either.
So how do I know if Send() has worked?
Send() simply places the data in a buffer for the network adapter to process. When Send() returns, you have no guarantee that a single byte has "left" your computer, even when the socket is in blocking mode.
TCP ensures though that within a connection all data is received in the order it was sent. It never "forgets" a packet in the middle of a conversation, and automatically retransmits data when needed.
To determine whether retransmission is required, the protocol sends acknowledgement messages, but:
you can't access them from the Socket class
hosts may postpone sending this ACK message
The easiest way to ensure your message has arrived is to let the other party respond to them yourself. If you don't receive a response within a reasonable amount of time, you could treat that connection as broken.
About the whole discussion between Sriram Sakthivel and "the others" - I've had similar problems of receiving duplicate messages and missing others, but in my personal case this was caused by:
using BeginReceive() (the async receive method),
reusing the same buffer on each BeginReceive() call, and
calling BeginReceive() before processing that buffer, causing the buffer to be filled with new data before having read the old message.
The ideal way of doing this is to check if the bufferedAmount is zero after you've sent a message. The trick is, you'll have to let your application know you are in fact sending a message.
Problem
I have PHP client which sends a image file to a C# socket server. My problem is about 30% of the time the file is partially transferred and stops.
PHP END ->
$file = file_get_contents('a.bmp');
socket_write($socket,$file);
C# END ->
int l= Socket.Receive(buffer, buffer.Length, SocketFlags.None);
//create the file using a file stream
How can I always transfer the full file without intermediate states? And why does it happen?
From the documentation for Socket.Receive:
If you are using a connection-oriented Socket, the Receive method will read as much data as is available, up to the number of bytes specified by the size parameter. If the remote host shuts down the Socket connection with the Shutdown method, and all available data has been received, the Receive method will complete immediately and return zero bytes.
This means you may get less than the total amount. This is just the way sockets work.
So if if you get a partial read, you should call Socket.Receive again. You can use the overload of Socket.Receive to continue reading into the same buffer.
Here is an article that shows how "keep reading" until you get what you want:
Socket Send and Receive
If you don't know how big the data is, you must keep reading until Socket.Receive returns zero.
We are trying to get image uploading working with TinyPic that tracks the progress of the upload. We used HttpWebRequest earlier but since that doesn't support tracking progress, we decided to try low level methods such as TcpClient.
The code when executed gets "stuck" in this line:
int networkBytesRead = networkStream.Read(buffer, 0, buffer.Length);
The code hangs there for more than one minute.
Please have a look at this code for the full class:
http://paste2.org/p/331631
Any input is appreciated.
Thanks,
McoreD from ZScreen
Usually this error is caused when client and server are not synchronized, that is may be the server is listening at the same time when client is sending data. may be you should send an empty line or something else, to give server know that he should send data.HttpWebRequest know about that is why he work so good:) You can try to record traffic by tcp snipper in case of HttpWebRequest and compare it with the traffic of NetworkStream