We Keep Coding

Httpwebrequest(client writemode) to TCPclient(Server Readmode) Approach

I am working on sending data across the network with the traditional server-client model.
Here Server starts the Tcplistener in a particular address and port. In this case, it is the local host.
The client makes use of WebRequest class in .net and takes the request stream and starts writing data into the request stream.
Let me walk through the Server code class:
TcpListener tcpListener = new TcpListener(IPAddress.Parse("127.0.0.1"),9309);
tcpListener.Start()//Start the listener and wait for client to connect.
while(true)
{
TcpClient newclient = tcpListener.AcceptTcpClient();
if(newclient.Connected())
{
break;
}
}
while(true)
{
ReadData(newclient);
}
public void ReadData(TcpClient newclient)
{
byte[] buffer = newbyte[50];
Stream ns = newclient.GetStream();
ns.Read(buffer, 0, buffer.Length);
Console.WriteLine( Encoding.UTF8.GetString(buffer));
}
//End of Server class.
Now let's see the Client Code class:-
WebRequest Request = HttpWebRequest.Create(http://127.0.0.1:9309/DataChannel);
Request.Method = "POST";
//Below method registers to Server's AcceptTcpClient and tcpclient is assigned.
Stream NetworkStream = ModifyCollimationRequest.GetRequestStream();
int DataWritten = 0;
while(true)
{
string Dname = "\r\nPosting server with Data as {0}\r\n";
byte[] dbytes = Encoding.UTF8.GetBytes(string.Format(Dname, ++DataWritten));
ns.WriteAsync(dbytes, 0, dbytes.Length);
ns.FlushAsync();
}
//End of Client code. Once the connection is established, the client keeps writing into the stream till the buffer size of >65000 is reached without issue but the problem is with the Server.
In Server,
Stream ns = newclient.GetStream(); -> This line under ReadData() method of the server executes,
but the the next line the code where Read() is used -> the code does not throw exception nor reaches next line. It just exits while debugging or times out. Someone it feels like, I am not able to fetch the stream or stream is empty. But the client keeps writing without any issue.
Can sometimes try this out and help me with what I am missing. Ultimately, I should be able to read the data available in a stream in any case but not sure why. Please add in your suggestions?

At the end of the client code, we need to call the below method for the data to
start actual streaming
ResponseStream = (HttpWebResponse)Request.GetResponse();
This is because there seems to be a bug in this .NET API where the buffered data is sent only when GetResponse() stream is called. This fixed the issue for me.
Below StackOverflow URL helped me learn more on this problem
HttpWebRequest.GetRequestStream returned stream does not send data immediately in .NET Core

HttpListener in C# didn't get all of the queries from Google Actions Server, how to solve it?

I'm creating a smart home server and I want to support Google Smart Home actions. The server app is written in C# using HttpListener and I'm using Mono 5.18 (5.20 version have problems with httpcfg) to run it on Debian 10 server. The server app is working correctly but 1 out of 5 queries isn't received on program. Tcpdump is showing some type of traffic but app doesn't get any.
I tried reinstalling Debian 2 times, using different mono version, changing port, running it on Windows 10, disabling the part of the code that it provides MQTT and MySQL support, disabling firewall and nothing happen. The main problem is that Google Server isn't sending any packets after only one fail and I must disconnect and reconnect my devices in Google Home App.
There is my code with HttpListner:
static HttpListener listener;
//...
static void Main(string[] args)
{
//...
HttpServiceMain();
}
//...
private static void HttpServiceMain()
{
listener = new HttpListener();
listener.Prefixes.Add("https://*:2030/");
listener.Start();
while (true)
{
ProcessRequest();
}
}
static void ProcessRequest()
{
var result = listener.BeginGetContext(ListenerCallback, listener);
var startNew = Stopwatch.StartNew();
result.AsyncWaitHandle.WaitOne();
startNew.Stop();
Console.WriteLine("Elapsed miliseconds: " + startNew.ElapsedMilliseconds);
}
static void ListenerCallback(IAsyncResult ar)
{
Console.WriteLine("Listening...");
HttpListenerContext context = listener.EndGetContext(ar);
HttpListenerRequest request = context.Request;
string documentContents;
using (Stream receiveStream = request.InputStream)
{
using (StreamReader readStream = new StreamReader(receiveStream, Encoding.UTF8))
{
documentContents = readStream.ReadToEnd();
}
}
string responseString = "{}";
//Creating response and exporting it to 'responseString'
byte[] buffer = System.Text.Encoding.UTF8.GetBytes(responseString);
HttpListenerResponse httpResponse = context.Response;
httpResponse.StatusCode = 200;
httpResponse.StatusDescription = "OK";
httpResponse.ContentLength64 = buffer.Length;
System.IO.Stream output = httpResponse.OutputStream;
output.Write(buffer, 0, buffer.Length);
httpResponse.Close();
}
That I said, everything is working fine in 4 out of 5 times, but after some requests server didn't get query and I must reconnect Google Home App with my service. It is bug in HttpListener, in my code or in Google Server? Have you any ideas?

The fact that you didn’t get all the results while implementing the QUERY intent means that there is some problem while communicating with the server. You can try to troubleshoot following the Troubleshooting Guide, to see if we are sending requests to your server.
When you don’t get requests as you expected, the other culprit might also be the OAuth implementation, as you found out. If Google does not get valid Access Tokens for users, it might not send all the requests to your server.

Asynchronous Base64 downloadstream to file

I'm trying to develop a .NET app which will communicate with a .NET Core Server app (which I did not develop). The main goal is to download a file. Since the client app will have a WPF gui, the whole download should happen asynchronously.
From reading the server app's API I know, that the repsonse to my request is a Base64-encoded String containing the contents of a file.
What I wanted to do, is to async send a request, take its response-stream, async read from that stream to a charArray, base64-decode, async write to file (see code below).
But Convert.FromBase64CharArray most often fails with an exception
invalid length for a base-64 char array or string
Occasionally this one succeeds but the download ends prematurely (downloadedLength < totalLength)
It seems, as if the connection was closed too early, but I'm not entirly sure whether that's true.
What I tried to resolve this issue so far:
Using streamReader.ReadToEndAsync(), decode the complete string, async write: worked, but to download 115MB around 600MB RAM were used
Make the block of { read, decode, write } async instead of async read, decode, async write: no improvement
No async at all: fails sometimes, but not as often as the async version
Use FromBase64Transform::TransformBlock instead of Convert.FromBase64CharArray: didn't finish the download in reasonable time, since inputBlockSize is set fix to 1Byte (Downloading about 115MB)
Communicating through an SSH tunnel to omit the Apache Server: Download didn't even start
Having client and server running on the same machine: seemed to work fine
Some specs:
Client: Windows 7 x64, .NET 4.6.1
Server: Ubuntu 16.04, Apache 2.4, .NET Core 2.1.4
And finally: the Code
The function that requests the file:
private async Task<WebResponse> DoGetRequestAsync(string requestString)
{
var completeRequestUrl = $"{_options.StoreUrl}/api/{requestString}";
try
{
RequestStarted?.Invoke(true);
var request = (HttpWebRequest)WebRequest.Create(completeRequestUrl);
request.ContentType = "text/plain";
request.Method = "GET";
var response = await request.GetResponseAsync();
RequestFinished?.Invoke(true);
return response;
}
catch (Exception e)
{
Console.WriteLine($"ERROR: {e.Message}");
}
return null;
}
The function that handles the reponse:
public async Task<string> DownloadPackage(string vendor, string package)
{
// declaring some vars
using (var response = await DoGetRequestAsync(requestString))
{
var totalLength = response.ContentLength;
var downloadedLength = 0;
var charBuffer = new char[4 * 1024];
try
{
using (var stream = response.GetResponseStream())
{
if (stream != null)
{
using (var reader = new StreamReader(stream))
using (var fStream = File.Create(filename))
{
while (!reader.EndOfStream)
{
var readBytes = await reader.ReadAsync(charBuffer, 0, charBuffer.Length);
var decoded = Convert.FromBase64CharArray(charBuffer, 0, readBytes);
await fStream.WriteAsync(decoded, 0, decoded.Length);
downloadedLength += readBytes;
DownloadProgress?.Invoke((float)downloadedLength / totalLength * 100.0f);
}
}
}
}
if (downloadedLength < totalLength)
{
throw new Exception($"Download failed due to a network error. Downloaded {downloadedLength} Bytes.");
}
// some follow-up stuff
return filename;
}
catch (Exception e)
{
Console.WriteLine("Error!");
Console.WriteLine(e.Message);
throw;
}
}
}
Any ideas what could cause the error?
EDIT:
Ok, I tried to implement the solution Fildor proposed. Since I do not delete the decoded contents of the secondary buffer, more memory is needed now to perform the download. But I could omit the StreamReader and read from the Stream directly. This lead to another exception:
Unable to read data from the transport connection: The connection was closed
No matter whether synchronously or asnychronously. Seems to be a proof for my first suspicion. But still I don't know how to solve this problem.

C# Client/Server returning answer from server to client

I've created a simple client/server program which takes an input from the client and returns a answer to the input by looking in the text file to see if there is an answer associated with the input.
The issue I'm having is that I get the response on the server side but I don't know how to send it back to the client (it just returns the input on the client side).
The second issue is that it will execute once, as in, it will only take in one input. I tried adding a loop but couldn't get it to work.
Server.cs
using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using System.Net.Sockets;
using System.Net;
namespace server
{
class server
{
const string SERVER_IP = "127.0.0.1";
static void Main(string[] args)
{
//Create Dictionary
Dictionary<string, string> dict = new Dictionary<string, string>();
//---Read Text File containing commands ---
StreamReader sr = new StreamReader(#"C:\Users\Desktop\potato.txt");
string line;
//Splits the text into commands:responses
while ((line = sr.ReadLine()) != null)
{
string[] arr = line.Split(';');
dict.Add(arr[0], arr[1]);
}
//Print dictionary TESTING FUNCTION
foreach (KeyValuePair<string, string> kvp in dict)
{
Console.WriteLine("Command = {0} Response = {1}", kvp.Key, kvp.Value);
}
//---Input the port number for clients to conect---
Console.Write("Input port" + System.Environment.NewLine);
int PORT_NO = int.Parse(Console.ReadLine());
//---listen at the specified IP and port no.---
IPAddress localAdd = IPAddress.Parse(SERVER_IP);
TcpListener listener = new TcpListener(localAdd, PORT_NO);
Console.WriteLine("Listening for Commands");
listener.Start();
//---incoming client connected---
TcpClient client = listener.AcceptTcpClient();
//---get the incoming data through a network stream---
NetworkStream nwStream = client.GetStream();
byte[] buffer = new byte[client.ReceiveBufferSize];
//---read incoming stream---
int bytesRead = nwStream.Read(buffer, 0, client.ReceiveBufferSize);
//---convert the command data received into a string---
string dataReceived = Encoding.ASCII.GetString(buffer, 0, bytesRead);
Console.WriteLine("Received Command : " + dataReceived);
//---Search Command and send a response
string Response;
if (dict.TryGetValue(dataReceived, out Response))
{
Console.WriteLine(Response);
}
//---write back the response to the client---
Console.WriteLine("Sending Response : " + Response);
nwStream.Write(buffer, 0, bytesRead);
Console.ReadLine();
}
}
}

You need to convert Response to a byte[] just as you do in the client sending your request (i.e. bytesToSend). E.g.:
Console.WriteLine("Sending Response : " + Response);
byte[] bytesToSend = ASCIIEncoding.ASCII.GetBytes(Response);
nwStream.Write(bytesToSend, 0, bytesToSend.Length);
Console.ReadLine();
That said, you have made the classic mistake every person does who tries to write TCP code without first reading references about TCP and sockets: you mistakenly believe that when you read from a socket, you will always receive in that single operation every byte that was read. So even with the above fix (which does address the issue on the server side), it is possible you will see partial responses on the client side (not as likely when testing locally, but much more likely if you move to running on the Internet or across LANs, and especially as the message size increases).
For low-volume network interaction, you may want to consider wrapping your NetworkStream with StreamReader and StreamWriter objects, and use ReadLine() and WriteLine() to receive and send data (i.e. use line-breaks as the delimiter for the data).
As for dealing with multiple requests, given the code you have presented here, the simplest approach is to add a loop around the server code after the listener.Start() method. I.e. containing all the code after that statement, starting with the call to listener.AcceptTcpClient() and going to the last statement in the method. However, again this is only appropriate for low-volume network code. If you anticipate clients will need your server to handle multiple requests and especially if in quick succession, what you really want is for the client to maintain the connection, i.e. only connect once and then have it send multiple requests on that same connection.
Similarly, if you want to be able to handle multiple clients at once, you cannot run the server in a single thread. Instead, at the very least you'll need to use the thread-blocking logic you're using now, where you have a new thread created for each client. Better, would be to use the non-blocking API (e.g. NetworkStream.BeginRead(), StreamReader.ReadLineAsync(), etc. … there are many asynchronous options to choose from), and let .NET deal with the threading for you.
Doing it that way will require significant changes to the server code. You really should look carefully at various samples on MSDN and Stack Overflow to see how this sort of thing is done. I also strongly recommend you read through the Winsock Programmer's FAQ. It is not specifically about .NET at all, but does cover all of the key details you'll need to know in order to effectively and correctly use the .NET API.

Minimize latency when downloading through asp.net web api and IIS

I have a web api, hosted on IIS, that returns a 4MB memory buffer through the StreamContent class.
public class RestPerfController : ApiController
{
byte[] data = new byte[4 * 1024 * 1024]; // 4MB
public HttpResponseMessage Get()
{
return new HttpResponseMessage(HttpStatusCode.OK)
{
Content = new StreamContent(new MemoryStream(data))
};
}
}
I also have .net client running on another machine that GETs the data 128 times in a loop and calculates average latency.
static void Main(string[] args)
{
Stopwatch timer = new Stopwatch();
byte[] buffer = new byte[FourMB];
for (int i = 0; i < 128; ++i)
{
// Create the request
var request = WebRequest.Create("https://<IpAdddress>/RestPerfController/") as HttpWebRequest;
request.Method = "GET";
request.ContentLength = 0;
// Start the timer
timer.Restart();
// Download the response
WebResponse response = request.GetResponse();
var responseStream = response.GetResponseStream();
long bytesRead = 0;
do
{
bytesRead = responseStream.Read(buffer, 0, FourMB);
}
while (bytesRead > 0);
Console.WriteLine(timer.ElapsedMilliseconds);
}
}
The client and server are connected through a 10Gbps LAN.
Using default settings, the client sees an average latency of 90ms.
Then I changed the server code to use PushStreamContent instead of StreamContent
return new HttpResponseMessage(HttpStatusCode.OK)
{
Content = //new StreamContent(new MemoryStream(data))
new PushStreamContent(async (s, hc, tc) =>
{
await s.WriteAsync(data, 0, data.Length);
s.Close();
},
"application/octet-stream")
};
This caused the average latency on the client to drop from 90ms to 50ms
Why is PushStreamContent almost twice as fast as StreamContent?
Is there a way of reducing the latency even further on the client? 50ms too seems pretty high for a 4MB transfer on a 10 Gigabit LAN.
EDIT: When I used http instead of https, the latency dropped from 50ms to 18ms. So it appears a large part of the latency was coming from the use of https.
Next, I did another experiment using ntttcp
Server: ntttcp.exe -r -m 1,*,<ipaddress> -rb 2M -a 2 -t 15
Client: ntttcp.exe -s -m 1,*,<ipaddress> -l 4M -a 2 -t 15
This showed an average latency of 11.4ms for 4MB transfers. This I believe is the fastest I can get from tcp.
Since I am constrained to use https, I am interested in knowing if there are ways to bring down the 50ms latency.

Did you try to work with less buffer than 4Mb? I think it´s too large and may cause some system bottleneck. Remember that, at this rate, some VIRTUAL/PAGE operations may occurr if RAM is not available. Try something like 32kb-256Kb.
The problem may be not in the LAN itself but in the Windows to manage data at this rate.
The PushStreamContent forces the system to transmit the buffer, stoping some other activities - a kind of HIGH PRIORITY at streams. The problem is about som error than can be occurr of the Stream is not well aligned/complete (the data itself).
Another problem is related to network checks are performed internally by StreamContent and not performed by PushStreamContent. As the name says, you´re forcing the communication (a kind of transmit anyway order).