I'm using the System.Net.HttpWebRequest class to implement a simple HTTP downloader that can be paused, canceled and even resumed after it was canceled (with the HTTP Range request header).
It's clear that HttpWebRequest.GetResponse() is when the HTTP request is actually sent to the server, and the method returns when a HTTP response is received (or a timeout occurs). However, the response body is represented with a Stream, which leaves me wonder whether the response body is actually transmitted with the response header (i.e. it's already downloaded when GetResponse() returns), or is it only downloaded on-demand, when I try to read from the response stream? Or maybe when I call the HttpWebResponse.GetResponseStream() method?
Unfortunately the msdn documentation doesn't tell, and I don't know enough about the HTTP protocol to be able to tell.
How do chunked transfers and the like behave in this case (that is, how should I handle them in my C# application)? When is actually the response data downloaded from the server?
This all depends on TCP, the underlying protocol of HTTP. The way TCP works is that data is sent in segments. Whenever a client sends a segment to the server, among the data sent is information about how much additional data is it ready to receive. This usually corresponds to some kind of buffer on the client's part. When the client receives some data, it also sends a segment to the server, acknowledging the received data.
So, assuming the client is very slow in processing the received data, the sequence of events could go like this:
Connection is established, the clients says how much data is it ready to receive.
Server sends one or more segments to the client, the total data in them at most the amount client said it is ready to receive
Client says to the server: I received the data you sent me, but don't send me anymore for now.
Client processes some of the data.
Client says to the server: You can send me x more bytes of data
What does this mean with regards to GetResponse()? When you call GetResponse(), the client sends the request, reads the HTTP header of the response (which usually fits into one segment, but it may be more) and returns. At this point, if you don't start reading the response stream (that you get by calling GetResponseStream()), some data from the server is received, but only to fill the buffer. When that is full, no more data is transmitted until you start reading the response stream.
Related
I'm using raw TCP sockets. I can send 200/404/302 without a problem.
If I serve 413 like a normal request it works just fine
"HTTP/1.1 413\r\nContent-Type: text/html; charset=utf-8;\r\nContent-Length: 7\r\n\r\nToo big"
However in the http 1/1 rfc it says I may close the connection before processing the request https://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.14
Request Entity Too Large
The server is refusing to process a request because the request entity is larger than the server is willing or able to process. The server MAY close the connection to prevent the client from continuing the request.
However when I serve the results and don't process the entire request both firefox and chrome will show a connection error. When I send the results THEN READ THE REQUEST it will work. The browsers seem to not process what I send them until I read the entire stream which defeats the point.
Or maybe I'm sending it wrong. How am I suppose to send the 413? I send the string above, flush the socket, sleep for 100 milliseconds, close the connection, sleep for another 100 milliseconds then restart.
A request and response exchange isn't considered complete until both the request data is fully received and the response fully sent. So the only way to send a valid 413 response is to read (and presumably discard) the full request body, as well as sending the 413 as soon as possible as it appears you are doing.
If you send the 413 immediately, it's up to the client to detect that early or not. I think most clients will just continue uploading the request though.
Alternatively, you may close the connection which is how you can stop the client from sending the whole message. But then, the HTTP request and response is not complete, and clients may show a connection error rather than the 413.
So: if you want to stop clients uploading data then close the connection. If you want clients to get a nice valid 413 response then consume the whole request.
I need to consume a third-party WebSocket API in .NET Core and C#; the WebSocket server is implemented using socket.io (using protocol version 0.9), and I am having a hard time understanding how socket.io works... besides that the API requires SSL.
I found out that the HTTP handshake must be initiated via a certain path, which is...
socket.io/1/?t=...
...whereby the value of the parameter t is a Unix-timestamp (in seconds). The service replies with a session-key, timeout information, and a list of supported transport protocols. Due to simplicity, this first request is made via HttpClient and does not involve any additional headers.
Next, another HTTP request is required, which should result in an HTTP 101 Switching Protocol response. I specified the following headers in accordance to the previous request...
Connection: Upgrade
Upgrade: websocket
Sec-WebSocket-Key: ...
Sec-WebSocket-Version: 13
...whereby the value of the Key-header is a Base64-encoded GUID-value that the server will use to calculate the Sec-WebSocket-Accept header value. I also precalculate the expected Sec-WebSocket-Accept header value, for validation...
I tried to make that request using HttpClient as well, but that does not seem to work... I actually don´t understand why, because I expect an HTTP response. I also tried to make the request using TcpClient by sending a manually prepared GET request over a SslStream, which accepts the remote certificate as expected. Sending data seems to work, but there´s no response data... the Read-method returns zero.
What do I miss here? Do I need to setup a listener for the WebSocket connection as well, and if yes how? I don´t want to implement a feature complete socket.io client, I´d just like to keep it as simple as possible to catch some events...
The best way of debugging these issues is to use a sniffer like wireshark or fiddler. Often connect using an IE and compare IE results with my application and modify my app so it works like the IE. Using WebClient instead of HttpClient will also work better because the WebClient does more automatically than the HttpClient.
A web connection uses the header of the client and the headers in the server webpage to negotiate a connection mode. Adding additional headers to you client will change the connection mode. Cookies are also used to select the connection mode. Cookies are the results of previous connection to the same server which shortens the negotiations and stores info from previous connection so less data has to be downloaded from server. The server remembers the cookies. Cookies have a timeout and is kept until timeout expires. The IE history in your client has a list of IP addresses and Net automatically sends the cookies associated with the server IP.
If a bad connection is made to the server the cookies is also bad so the only was of connection is to remove the cookie. Usually I go into the IE and delete cookies manually in the IE history.
To check if a response is good the server returns a status. A completed response contains a status 200 DONE. You can get status which are errors. You can also get a 100 Continue which means you need to send another request to get the rest of the webpage.
Http has 1.0 (stream mode) and 1.1 (chunk mode). Net library doesn't work with chunk. Chunk requires client to send message to get next chunk and I have not found a way in Net to send the next chunk message. So if a server responds with a 1.1 then you have to add to your client headers to use 1.0 only.
Http uses TCP as the transport layer. So in a sniffer you will see TCP and HTTP. Usually you can filter sniffer just to return Http and look at header for debugging. Occasionally TCP disconnects and then you have to look at TCP to find why the disconnect occurs.
I understand that the keep-alive message aims to prevent the underlying TCP connection from closing so that multiple requests can be sent over the same TCP connection.
However it is unclear how this affects a request that has timed out:
If multiple requests are handled over the same TCP connection, when the client times-out, how will ASP.NET know? I'm not sure how to ask this question correctly, but I suspect that HttpWorkerRequest.IsClientConnected is false if the underlying TCP connection is closed? How would I determine if the client has timed out? Surely the response can no longer be sent?
Since the client can send keep-alive messages, what effect, if any does it have on the request? I expect that it will have no effect and mechanisms such as a timeout will work as usual.
Thank you for your time and assistance.
HTTP keep-alive (which should not be confused with TCP keep alive) asks the server to not close the connection after the HTTP response is fully sent. Thus HTTP keep-alive is only relevant for the time after the HTTP response is sent and before the next HTTP request will be received.
... when the client times-out, how will ASP.NET know?
The server is trying to read data. If the client closes the connection the server will get notified (like by reading 0 bytes) and will close the connection too.
... Surely the response can no longer be sent?
To repeat myself, HTTP keep-alive is only relevant after the full response was sent. So this question is unrelated to HTTP keep-alive.
Since the client can send keep-alive messages, what effect, if any does it have on the request?
No effect on the current request. But the client must be aware that the server might close the connection anytime after the HTTP response was sent and before the server gets the new request. Thus it might happen that the server closes the connection exactly in the moment when the client is trying to send a new request. In this case idempotent requests (GET) must be retried by the client within a new connection, while other requests should fail.
I'm using HttpClient to send an asynchronous POST request to a remote web server. That remote web server responds with the Connection header set to keep-alive. Eventually, it will close the connection.
What I can't figure out how to do is keep receiving data until the connection is set to close.
For example, consider the following code:
HttpClient client = new HttpClient();
string requestUri = ...; //My GET uri
string content = ...; //url-encoded contents of the request
HttpResponseMessage response = await client.PostAsync(requestUri, new StringContent(content));
In the code above, response will have the Connection header set to keep-alive. I couldn't find any members on HttpResponseMessage that seem to give me any means of continuing to receive information after the first keep-alive is received.
I can't figure out how to continue receiving responses until the Connection closes.
I was able to find all sorts of resources that dealt with sending keep-alive requests. That's not my issue. My issue is in receiving/handling a keep-alive response from the server.
Please let me know if there is anymore information that I could provide to improve this question.
Connection: keep-alive purpose is not to enable server to send multiple responses for the same single request - this would fundamentally violate how HTTP works.
Connection: keep-alive is server's way of signaling to the client that client can submit next request using the same, already established, TCP connection, thus help avoid the costs of establishing new connection (and possibly TLS handshake as well).
Reuse of the connection is (typically) handled by underlying library, and is abstracted away such that user (as in developer) of the library need not worry about handling.
All you should be doing is keep making as many client.Post/Get/etc requests, and the HttpClient library will take care of managing the underlying TCP connection and its reuse.
There is one area in HTTP that may look counter to the above and thus confusing, but is not. The case is when server returns HTTP Status 100 Continue. In this case you will see another response come from the server for still the same original request (and by "see" I mean not from HttpClient, but rather on the wire if you were to snoop with a tool like WireShark or a proxy). What is happening in this case is if a client is making a large request, the server simply signals to the client that it is continuing to accept and read request, and for the client to continue sending that original request. Once full request is received, the server will process and respond with a final response code and message. And again, HttpClient will abstract away this interim 100 Continue response, so as developer you need not worry about it (in most if not all cases).
If all you want is receive the response for a given request you don't need to deal with keep-alive at all. Keep-alive is purely a performance optimization. The framework transparently manages connections.
All you do is send a request and receive a response. There is exactly one response per request.
One side will eventually close the connection which is not of any concern to you. You don't notice any effect of that.
I have an HTTPListener based .NET webserver, which generates data on request and outputs it. I am using StreamWriter to write to the output stream. When I write the data, not all of it gets received in Chrome, but all of it gets sent in the code. If I go to view source, it only shows part of the data that was sent.
It almost seems like some kind of timeout, or it getting too big, or something. Is there something that needs to be set in the HTTPListener or the request context?
As per the comments:
Call Flush on the stream before it is sent.