We Keep Coding

C# HTTP Server - Respond post request without processing post data

I am currently working on a local http server written in C#. At this point I am not yet processing post data, but only distinguish between get and post request. In both cases, of course, a 200 should be answered. While testing the server, I noticed that if I send an empty post request, it is answered by the server with a 200 and an html page just like a get request without any problems. However, if there are images attached to the post request, as in my example, the connection to the server fails immediately.
I handle a client connection as follows. I know it's not ideal to store the received bytes in a string, but for testing purposes I haven't seen any problems with it.
private void HandleClient(TcpClient client)
{
Byte[] bytes;
String requestData = "";
NetworkStream ns = client.GetStream();
if (client.ReceiveBufferSize > 0)
{
bytes = new byte[client.ReceiveBufferSize];
ns.Read(bytes, 0, client.ReceiveBufferSize);
requestData = Encoding.ASCII.GetString(bytes);
}
// Get Request out of message
Request request = Request.GetRequest(requestData);
// Create Response
Response response = Response.From(request);
response.Post(client.GetStream());
}
And here is the method I use to determine what type of request it is.
public static Request GetRequest(String request)
{
//return if request is null
if(String.IsNullOrEmpty(request))
{
return null;
}
//Split Request to get tokens - split by spaces
String[] tokens = request.Split(' ');
String type = tokens[0];
String url = tokens[1];
String host = tokens[4];
return new Request(type, url, host);
}
Surely it must be possible to read only the headers from a get as well as post request and then still give a 200 response. Is there a rule of behavior for an http server on how it should handle post-request data?

The answer to my question was quite simple in the end. The input stream of a request must be read completely before the server can respond to the request. In my case it was so, that I only read the header of the request to know if it is a Post or Get request, therefore the server could not respond to the request in case of an attached image, because the input stream was not read completely.

Strategy and how to create Request and Response pipes in .net core 3.0 middleware

I am developing a .net core middle-ware (api) and thinking to use pipes with following flow, Can someone tell me is this is a good approach and comply best practices or should i use different strategy.
Request comes to api
Authorization pipe validates the request.
Request pipe logs the request into db.
Request goes to api and perform action and return a result.
Response pipe gets the response and logs into db and return the result to client.
I know that we can read stream only time (point 3) but i figured this out already and after reading i have attach it to request stream again.
So, confusion is where to write the response? In api? or in separate pipe.
If i do it in separate pipe then i am handling my response two time (one is creating response in api, second is reading response in separate pipe) which is a performance hit.
Can i pass the data from point number 4 to 5 like from api to my pipe and from there that response should added into response stream and if it is correct then how can i pass the data from api to pipe?

Yes, response stream can only be read once. You can use the MemoryStream to load the response , reference article :
First, read the request and format it into a string.
Next, create a dummy MemoryStream to load the new response into.
Then, wait for the server to return a response.
Finally, copy the dummy MemoryStream (containing the actual response) into the original stream, which gets returned to the client.
Code sample :
public class RequestResponseLoggingMiddleware
{
private readonly RequestDelegate _next;
public RequestResponseLoggingMiddleware(RequestDelegate next)
{
_next = next;
}
public async Task Invoke(HttpContext context)
{
//First, get the incoming request
var request = await FormatRequest(context.Request);
//Copy a pointer to the original response body stream
var originalBodyStream = context.Response.Body;
//Create a new memory stream...
using (var responseBody = new MemoryStream())
{
//...and use that for the temporary response body
context.Response.Body = responseBody;
//Continue down the Middleware pipeline, eventually returning to this class
await _next(context);
//Format the response from the server
var response = await FormatResponse(context.Response);
//TODO: Save log to chosen datastore
//Copy the contents of the new memory stream (which contains the response) to the original stream, which is then returned to the client.
await responseBody.CopyToAsync(originalBodyStream);
}
}
private async Task<string> FormatRequest(HttpRequest request)
{
var body = request.Body;
//This line allows us to set the reader for the request back at the beginning of its stream.
request.EnableRewind();
//We now need to read the request stream. First, we create a new byte[] with the same length as the request stream...
var buffer = new byte[Convert.ToInt32(request.ContentLength)];
//...Then we copy the entire request stream into the new buffer.
await request.Body.ReadAsync(buffer, 0, buffer.Length);
//We convert the byte[] into a string using UTF8 encoding...
var bodyAsText = Encoding.UTF8.GetString(buffer);
//..and finally, assign the read body back to the request body, which is allowed because of EnableRewind()
request.Body = body;
return $"{request.Scheme} {request.Host}{request.Path} {request.QueryString} {bodyAsText}";
}
private async Task<string> FormatResponse(HttpResponse response)
{
//We need to read the response stream from the beginning...
response.Body.Seek(0, SeekOrigin.Begin);
//...and copy it into a string
string text = await new StreamReader(response.Body).ReadToEndAsync();
//We need to reset the reader for the response so that the client can read it.
response.Body.Seek(0, SeekOrigin.Begin);
//Return the string for the response, including the status code (e.g. 200, 404, 401, etc.)
return $"{response.StatusCode}: {text}";
}
}
And register the middleware :
app.UseMiddleware<RequestResponseLoggingMiddleware>();

Why the (httpwebresponse) ResponseStream stops reading data from internet radio?

I am using this code to get the data from an icecast radio, but the ResponseStream stops reading data at 64K recieved. Can you help me with this?
HttpWebRequest request = (HttpWebRequest) WebRequest.Create("http://icecast6.play.cz/radio1-128.mp3");
request.AllowReadStreamBuffering = false;
request.Method = "GET";
request.BeginGetResponse(new AsyncCallback(GetShoutAsync), request);
void GetShoutAsync(IAsyncResult res)
{
HttpWebRequest request = (HttpWebRequest) res.AsyncState;
HttpWebResponse response = (HttpWebResponse) request.EndGetResponse(res);
Stream r = response.GetResponseStream();
byte[] data = new byte[4096];
int read;
while ((read = r.Read(data, 0, data.Length)) > 0)
{
Debug.WriteLine(data[0]);
}
}

I don’t see any obvious problems in your code. Apart from not using async-await which greatly simplifies the kind of asyncronous code you’re developing :-)
What do you mean “the ResponseStream stops reading”?
If the connection is dropped, then my #1 idea — server does that. Use wireshark to confirm, and then use wireshark to compare the request’s HTTP header with e.g. Winamp that starts playing that stream. I’m sure you’ll find some important differences.
If however it merely pauses, it’s normal.
Upon connect, streaming servers typically send you some initial amount of data, and then they only send you their data in real-time. So, after you’ve received that initial buffer, you’ll only get the data # the rate of your stream, i.e. 16 kbytes/sec for your 128 kbit/sec radio.
BTW, some clients send “Initial-Burst” HTTP header with the request, but I was unable to find the documentation about that header. When I worked on my radio for WP7, I basically replicated the behavior of some other, iOS app.

Finally I write this code to solve the issue, it is completely necessary to use the namespace : Windows.Web.Http, And it is like..
Uri url = new Uri("http://icecast6.play.cz/radio1-128.mp3");
HttpResponseMessage response = await httpClient.GetAsync(
url,
HttpCompletionOption.ResponseHeadersRead);
IInputStream inputStream = await response.Content.ReadAsInputStreamAsync();
try
{
ulong totalBytesRead =
IBuffer buffer = new Windows.Storage.Streams.Buffer(100000);
while (buffer.Length > 0);
{
uffer = await inputStream.ReadAsync(
buffer,
buffer.Capacity,
InputStreamOptions.Partial);
//
// Some stuff here...
totalBytesRead += buffer.Length;
Debug.WriteLine(buffer.Length + " " + totalBytesRead);
}
Debug.WriteLine(totalBytesRead);
I hope you guys enjoy it.

HttpWebRequest BeginGetRequestStream callback never called

In my Xamarin application I use HttpWebRequest class to send POST messages to the server (I use it because it is available out-of-the box in PCL libraries).
Here is some request preparation code:
request.BeginGetRequestStream (asyncResult => {
Mvx.Trace ("BeginGetRequestStream callback");
request = (HttpWebRequest)asyncResult.AsyncState;
Stream postStream = request.EndGetRequestStream (asyncResult);
string postData = jsonConverter.SerializeObject (objectToSend);
Mvx.Trace ("Posting following JSON: {0}", postData);
byte[] byteArray = Encoding.UTF8.GetBytes (postData);
postStream.Write (byteArray, 0, byteArray.Length);
MakeRequest (request, timeoutMilliseconds, successAction, errorAction);
}, request);
When I start application and execute this code for the first and the second time everything works fine. But when this is executed for the 3rd time (exactly!) the callback is not called and line "BeginGetRequestStream callback" is never printed to log. Is it a bug in class implementation or maybe I do something incorrectly?
If it is not possible to make this working in Xamarin please suggest reliable and convenient class for sending Http GET and POST request with timeout.
Also created related, more general question: Sending Http requests from Xamarin Portable Class Library

My solution to send and receive messages JSON in Xamarin PCL:
public async Task<string> SendMessageJSON(string message, string url)
{
HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create(new Uri(url));
request.ContentType = "application/json";
request.Method = "POST";
// Send data to server
IAsyncResult resultRequest = request.BeginGetRequestStream(null, null);
resultRequest.AsyncWaitHandle.WaitOne(30000); // 30 seconds for timeout
Stream streamInput = request.EndGetRequestStream(resultRequest);
byte[] byteArray = Encoding.UTF8.GetBytes(message);
await streamInput.WriteAsync(byteArray, 0, byteArray.Length);
await streamInput.FlushAsync();
// Receive data from server
IAsyncResult resultResponse = request.BeginGetResponse(null, null);
resultResponse.AsyncWaitHandle.WaitOne(30000); // 30 seconds for timeout
HttpWebResponse response = (HttpWebResponse)request.EndGetResponse(resultResponse);
Stream streamResponse = response.GetResponseStream();
StreamReader streamRead = new StreamReader(streamResponse);
string result = await streamRead.ReadToEndAsync();
await streamResponse.FlushAsync();
return result;
}

Finally solved this by switching to Profile 78 and HttpClient, which works well in all cases.

HttpWebRequest is slow with chunked data

I'm using HttpWebRequest to connect to my in-house built HTTP server. My problem is that it is a lot slower than connecting to the server via for instance PostMan (https://chrome.google.com/webstore/detail/postman-rest-client/fdmmgilgnpjigdojojpjoooidkmcomcm?hl=en), which is probably using the built-in functions in Chrome to request data.
The server is built using this example on MSDN (http://msdn.microsoft.com/en-us/library/dxkwh6zw.aspx) and uses a buffer size of 64. The request is a HTTP request with some data in the body.
When connecting via PostMan, the request is split into a bunch of chunks and BeginRecieve() is called multiple times, each time receiving 64B and taking about 2 milliseconds. Except the last one, which receives less than 64B.
But when connecting with my client using HttpWebRequest, the first BeginRecieve() callback receives 64B and takes about 1 ms, the following receives only 47B and takes almost 200 ms, and finally the third receives about 58B and takes 2ms.
What is up with the second BeginRecieve? I note that the connection is established as soon as I start to write data to the HttpWebRequest input stream, but the data reception does not start until I call GetResponse().
Here is my HttpWebRequest code:
var request = (HttpWebRequest)WebRequest.Create(url);
request.Method = verb;
request.Timeout = timeout;
request.Proxy = null;
request.KeepAlive = false;
request.Headers.Add("Content-Encoding", "UTF-8");
System.Net.ServicePointManager.Expect100Continue = false;
request.ServicePoint.Expect100Continue = false;
if ((verb == "POST" || verb == "PUT") && !String.IsNullOrEmpty(data))
{
var dataBytes = Encoding.UTF8.GetBytes(data);
try
{
var dataStream = request.GetRequestStream();
dataStream.Write(dataBytes, 0, dataBytes.Length);
dataStream.Close();
}
catch (Exception ex)
{
throw;
}
}
WebResponse response = null;
try
{
response = request.GetResponse();
}
catch (Exception ex)
{
throw;
}
var responseReader = new StreamReader(rStream, Encoding.UTF8);
var responseStr = responseReader.ReadToEnd();
responseReader.Close();
response.Close();
What am I doing wrong? Why is it behaving so much differently than a HTTP request from a web browser? This is effectively adding 200ms of lag to my application.

This looks like a typical case of the Nagle algorithm clashing with TCP Delayed Acknowledgement. In your case you are sending a small Http Request (~170 bytes according to your numbers). This is likely less than the MSS (Maximum Segment Size) meaning that the Nagle Algorithm will kick in. The server is probably delaying the ACK resulting in a delay of up to 500 ms. See links for details.
You can disable Nagle via ServicePointManager.UseNagleAlgorithm = false (before issuing the first request), see MSDN.
Also see Nagle’s Algorithm is Not Friendly towards Small Requests for a detailed discussion including a Wireshark analysis.
Note: In your answer you are running into the same situation when you do write-write-read. When you switch to write-read you overcome this problem. However I do not believe you can instruct the HttpWebRequest (or HttpClient for that matter) to send small requests as a single TCP write operation. That would probably be a good optimization in some cases. Althought it may lead to some additional array copying, affecting performance negatively.

200ms is the typical latency of the Nagle algorithm. This gives rise to the suspicion that the server or the client is using Nagling. You say you are using a sample from MSDN as the server... Well there you go. Use a proper server or disable Nagling.
Assuming that the built-in HttpWebRequest class has an unnecessary 200ms latency is very unlikely. Look elsewhere. Look at your code to find the problem.

It seems like HttpWebRequest is just really slow.
Funny thing: I implemented my own HTTP client using Sockets, and I found a clue to why HttpWebRequest is so slow. If I encoded my ASCII headers into its own byte array and sent them on the stream, followed by the byte array encoded from my data, my Sockets-based HTTP client behaved exactly like HttpWebRequest: first it fills one buffer with data (part of the header), then it uses another buffer partially (the rest of the header), waits 200 ms and then sends the rest of the data.
The code:
TcpClient client = new TcpClient(server, port);
NetworkStream stream = client.GetStream();
// Send this out
stream.Write(headerData, 0, headerData.Length);
stream.Write(bodyData, 0, bodyData.Length);
stream.Flush();
The solution was of course to append the two byte arrays before sending them out on the stream. My application is now behaving as espected.
The code with a single stream write:
TcpClient client = new TcpClient(server, port);
NetworkStream stream = client.GetStream();
var totalData = new byte[headerBytes.Length + bodyData.Length];
Array.Copy(headerBytes,totalData,headerBytes.Length);
Array.Copy(bodyData,0,totalData,headerBytes.Length,bodyData.Length);
// Send this out
stream.Write(totalData, 0, totalData.Length);
stream.Flush();
And HttpWebRequest seems to send the header before I write to the request stream, so it might be implemented somewhat like my first code sample. Does this make sense at all?
Hope this is helpful for anyone with the same problem!

Try this: you need to dispose of your IDisposables:
var request = (HttpWebRequest)WebRequest.Create(url);
request.Method = verb;
request.Timeout = timeout;
request.Proxy = null;
request.KeepAlive = false;
request.Headers.Add("Content-Encoding", "UTF-8");
System.Net.ServicePointManager.Expect100Continue = false;
request.ServicePoint.Expect100Continue = false;
if ((verb == "POST" || verb == "PUT") && !String.IsNullOrEmpty(data))
{
var dataBytes = Encoding.UTF8.GetBytes(data);
using (var dataStream = request.GetRequestStream())
{
dataStream.Write(dataBytes, 0, dataBytes.Length);
}
}
string responseStr;
using (var response = request.GetResponse())
{
using (var responseReader = new StreamReader(rStream, Encoding.UTF8))
{
responseStr = responseReader.ReadToEnd();
}
}