I need to consume a third-party WebSocket API in .NET Core and C#; the WebSocket server is implemented using socket.io (using protocol version 0.9), and I am having a hard time understanding how socket.io works... besides that the API requires SSL.
I found out that the HTTP handshake must be initiated via a certain path, which is...
socket.io/1/?t=...
...whereby the value of the parameter t is a Unix-timestamp (in seconds). The service replies with a session-key, timeout information, and a list of supported transport protocols. Due to simplicity, this first request is made via HttpClient and does not involve any additional headers.
Next, another HTTP request is required, which should result in an HTTP 101 Switching Protocol response. I specified the following headers in accordance to the previous request...
Connection: Upgrade
Upgrade: websocket
Sec-WebSocket-Key: ...
Sec-WebSocket-Version: 13
...whereby the value of the Key-header is a Base64-encoded GUID-value that the server will use to calculate the Sec-WebSocket-Accept header value. I also precalculate the expected Sec-WebSocket-Accept header value, for validation...
I tried to make that request using HttpClient as well, but that does not seem to work... I actually don´t understand why, because I expect an HTTP response. I also tried to make the request using TcpClient by sending a manually prepared GET request over a SslStream, which accepts the remote certificate as expected. Sending data seems to work, but there´s no response data... the Read-method returns zero.
What do I miss here? Do I need to setup a listener for the WebSocket connection as well, and if yes how? I don´t want to implement a feature complete socket.io client, I´d just like to keep it as simple as possible to catch some events...
The best way of debugging these issues is to use a sniffer like wireshark or fiddler. Often connect using an IE and compare IE results with my application and modify my app so it works like the IE. Using WebClient instead of HttpClient will also work better because the WebClient does more automatically than the HttpClient.
A web connection uses the header of the client and the headers in the server webpage to negotiate a connection mode. Adding additional headers to you client will change the connection mode. Cookies are also used to select the connection mode. Cookies are the results of previous connection to the same server which shortens the negotiations and stores info from previous connection so less data has to be downloaded from server. The server remembers the cookies. Cookies have a timeout and is kept until timeout expires. The IE history in your client has a list of IP addresses and Net automatically sends the cookies associated with the server IP.
If a bad connection is made to the server the cookies is also bad so the only was of connection is to remove the cookie. Usually I go into the IE and delete cookies manually in the IE history.
To check if a response is good the server returns a status. A completed response contains a status 200 DONE. You can get status which are errors. You can also get a 100 Continue which means you need to send another request to get the rest of the webpage.
Http has 1.0 (stream mode) and 1.1 (chunk mode). Net library doesn't work with chunk. Chunk requires client to send message to get next chunk and I have not found a way in Net to send the next chunk message. So if a server responds with a 1.1 then you have to add to your client headers to use 1.0 only.
Http uses TCP as the transport layer. So in a sniffer you will see TCP and HTTP. Usually you can filter sniffer just to return Http and look at header for debugging. Occasionally TCP disconnects and then you have to look at TCP to find why the disconnect occurs.
Related
I'm using raw TCP sockets. I can send 200/404/302 without a problem.
If I serve 413 like a normal request it works just fine
"HTTP/1.1 413\r\nContent-Type: text/html; charset=utf-8;\r\nContent-Length: 7\r\n\r\nToo big"
However in the http 1/1 rfc it says I may close the connection before processing the request https://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.14
Request Entity Too Large
The server is refusing to process a request because the request entity is larger than the server is willing or able to process. The server MAY close the connection to prevent the client from continuing the request.
However when I serve the results and don't process the entire request both firefox and chrome will show a connection error. When I send the results THEN READ THE REQUEST it will work. The browsers seem to not process what I send them until I read the entire stream which defeats the point.
Or maybe I'm sending it wrong. How am I suppose to send the 413? I send the string above, flush the socket, sleep for 100 milliseconds, close the connection, sleep for another 100 milliseconds then restart.
A request and response exchange isn't considered complete until both the request data is fully received and the response fully sent. So the only way to send a valid 413 response is to read (and presumably discard) the full request body, as well as sending the 413 as soon as possible as it appears you are doing.
If you send the 413 immediately, it's up to the client to detect that early or not. I think most clients will just continue uploading the request though.
Alternatively, you may close the connection which is how you can stop the client from sending the whole message. But then, the HTTP request and response is not complete, and clients may show a connection error rather than the 413.
So: if you want to stop clients uploading data then close the connection. If you want clients to get a nice valid 413 response then consume the whole request.
The response from the server is not a valid HTTP response. This problem occurs when the .NET Framework detects that the server response does not comply with HTTP 1.1 RFC. This problem may occur when the response contains incorrect headers or incorrect header delimiters.RFC 2616 defines HTTP 1.1 and the valid format for the response from the server. For more information, see RFC 2616 - Hypertext Transfer Protocol -- HTTP/1.1 at Internet Engineering Task Force (IETF) website.
Get a network trace of the transaction and examine the headers in the response.
If your application requires the server response without parsing (this could be a security issue), set useUnsafeHeaderParsing to true in the configuration file. See Element
It seems like you just forgentting to clear the buffer before you receive new message.
2 quick options for you:
-add '\0' in the end of each of your messages on your server side. This will prevent gliches like that from happening.
-use Array.Clear(buffer,0,buffer.length) to clear buffer before receiving on your client side.
Currently I am working on a custom HTTP publisher for the Peach Fuzzing framework.
In order to determine if the server responed in an unusual way I need to examine the decrypted incoming / outgoing packages with Wireshark (or the PcapMonitor that is included in the Peach Framework) - to do that I need the SSL keys that are being generated by the C# HttpWebRequest (similar to the content of the SSLKEYLOGFILE) since I am trying to fuzz an SSL protected RESTful webservice.
If it is not possible to get the keys or if it is very difficult - is there any other way to see the raw HTTP request / response?
Thanks!
Just found a way to get the decrypted packages:
Create a loopback interface that can be monitored with Wireshark
Create a reverse proxy (I used nginx) and redirect all traffic from "localhost" to the target URL (https://)
Tell your application not to send the requests to target (https://) but to localhost (http://localhost/)
Start listening on the loopback interface and start your application
Hope this is helpful for somebody in a similar situation
I'm using HttpClient to send an asynchronous POST request to a remote web server. That remote web server responds with the Connection header set to keep-alive. Eventually, it will close the connection.
What I can't figure out how to do is keep receiving data until the connection is set to close.
For example, consider the following code:
HttpClient client = new HttpClient();
string requestUri = ...; //My GET uri
string content = ...; //url-encoded contents of the request
HttpResponseMessage response = await client.PostAsync(requestUri, new StringContent(content));
In the code above, response will have the Connection header set to keep-alive. I couldn't find any members on HttpResponseMessage that seem to give me any means of continuing to receive information after the first keep-alive is received.
I can't figure out how to continue receiving responses until the Connection closes.
I was able to find all sorts of resources that dealt with sending keep-alive requests. That's not my issue. My issue is in receiving/handling a keep-alive response from the server.
Please let me know if there is anymore information that I could provide to improve this question.
Connection: keep-alive purpose is not to enable server to send multiple responses for the same single request - this would fundamentally violate how HTTP works.
Connection: keep-alive is server's way of signaling to the client that client can submit next request using the same, already established, TCP connection, thus help avoid the costs of establishing new connection (and possibly TLS handshake as well).
Reuse of the connection is (typically) handled by underlying library, and is abstracted away such that user (as in developer) of the library need not worry about handling.
All you should be doing is keep making as many client.Post/Get/etc requests, and the HttpClient library will take care of managing the underlying TCP connection and its reuse.
There is one area in HTTP that may look counter to the above and thus confusing, but is not. The case is when server returns HTTP Status 100 Continue. In this case you will see another response come from the server for still the same original request (and by "see" I mean not from HttpClient, but rather on the wire if you were to snoop with a tool like WireShark or a proxy). What is happening in this case is if a client is making a large request, the server simply signals to the client that it is continuing to accept and read request, and for the client to continue sending that original request. Once full request is received, the server will process and respond with a final response code and message. And again, HttpClient will abstract away this interim 100 Continue response, so as developer you need not worry about it (in most if not all cases).
If all you want is receive the response for a given request you don't need to deal with keep-alive at all. Keep-alive is purely a performance optimization. The framework transparently manages connections.
All you do is send a request and receive a response. There is exactly one response per request.
One side will eventually close the connection which is not of any concern to you. You don't notice any effect of that.
I have built a web proxy from scratch (using Socket and NetworkStream classes). I am now trying to implement SSL support for it so that it can handle HTTPS requests and responses. I have a good idea of what I need to do (using SslStream) but I don't know how to determine if the request I get from the client is SSL or not.
I have searched for hours on this subject and have been unable to find a suitable solution.
After I do this:
TcpListener pServer = new TcpListener(localIP, port);
pServer.Start(256);
Socket a_socket = pServer.AcceptSocket();
How do I know if I need to read the information using SslStream or NetworkStream?
Client will send you a CONNECT method request after this point you need to just redirect the traffic.
Sample Connect :
CONNECT www.google.com:443 HTTP/1.1
After seeing this just switch to data redirect mode. You can not intercept or read the data so you don't need to worry about SSLStream anyway, you won't touch it.
However if you want to MITM (man in the middle) then you need to switch to SSL otherwise just redirect whatever comes to the target URL and port, that's it.
Obviously client browser will popup with an SSL certificate exception if you intercept the request.
You need to add support for the CONNECT command.
http://www.codeproject.com/KB/IP/akashhttpproxy.aspx
This is why proxy clients use one proxy for HTTP and different one for HTTPS. You can't know what type of connection you're going to receive.