Long story short I am trying to send a string via TcpClient using StreamWriter.
Without changing any other code except swapping out these samples. They produce different results.
In code sample 1 the StreamReader picks up that it has DataAvailable and the message is received.
In code sample 2 it does not have DataAvailable so no message is received. I need to keep my underlying stream open hence needing to use the constructor of StreamWrite in sample 2.
Sample 1 - Write Method
public void SendMessage(string message)
{
message = "TestMessage";
//WORKING - Sample 1
using (var sw = new StreamWriter(stream))
{
sw.Write(message);
sw.Flush();
}
}
Sample 2 - Write Method
public void SendMessage(string message)
{
message = "TestMessage";
//NOT WORKING - Sample 2
var encoding = new UTF8Encoding(false, true);
using (var sw = new StreamWriter(stream, encoding, 1024, true))
{
sw.Write(message);
sw.Flush();
}
}
Read Method
public string ReadMessage()
{
if (!stream.DataAvailable)
return null;
//I have also tried
//if(sr.Peek() == 0)
// return null;
string message = sr.ReadToEnd();
return message;
}
NOTE: If I put both samples together with the working one last I get the message received "TestMessageTestMessage" so it is definitely writing to stream however it is not setting DataAvailable to true?
Any Idea's why?
The problem is your ReadToEnd() command which blocks indefinitely on a NetworkStream which has no end until closed. I tested your code and I went past the DataAvailable query and blocked on the ReadToEnd() command.
Your method that uses a constructor that allows the BaseStream to stay open means that you never have an end to your stream. When the working method closes the stream the ReadMessage method returns with everything in the stream.
The solution: Do not attempt to read to the end. Read in blocks while the data is available or introduce a terminating character and read to that character.
From MSDN:
ReadToEnd assumes that the stream knows when it has reached an end. For interactive protocols in which the server sends data only when you ask for it and does not close the connection, ReadToEnd might block indefinitely because it does not reach an end, and should be avoided.
Related
I am working with two C# stream APIs, one of which is a data source and the other of which is a data sink.
Neither API actually exposes a stream object; both expect you to pass a stream into them and they handle writing/reading from the stream.
Is there a way to link these APIs together such that the output of the source is streamed into the sink without having to buffer the entire source in a MemoryStream? This is a very RAM-sensitive application.
Here's an example that uses the MemoryStream approach that I'm trying to avoid, since it buffers the entire stream in RAM before writing it out to S3:
using (var buffer = new MemoryStream())
using (var transferUtil = new TransferUtility(s3client))
{
// This destructor finishes the file and transferUtil closes
// the stream, so we need this weird using nesting to keep everyone happy.
using (var parquetWriter = new ParquetWriter(schema, buffer))
using (var rowGroupWriter = parquetWriter.CreateRowGroup())
{
rowGroupWriter.WriteColumn(...);
...
}
transferUtil.Upload(buffer, _bucketName, _key.Replace(".gz", "") + ".parquet");
}
You are looking for a stream that can be passed to both the data source and sink and that can 'transfer' the data between the two asynchronously. There are a number of possible solutions and I might have considered a producer-consumer pattern around a BlockingCollection.
Recently, the addition of the System.IO.Pipelines, Span and Memory types have really focused on high performance IO and I think it would be a good fit here. The Pipe class with it's associated Reader and Writer, can automatically handle the flow control, back pressure and IO between themselves whilst utilising all the new Span and Memory related types.
I have uploaded a Gist at PipeStream that will give you a custom stream with an internal Pipe implementation that you can pass to both your API classes. Whatever is written to the WriteAsync (or Write) method will be made available to the ReadAsync (or Read) method without requiring any further byte[] or MemoryStream allocations
In your case you would simply substite the MemoryStream for this new class and it should work out of the box. I haven't got a full S3 test working but reading directly from the Parquet stream and dumping it to the console window shows that it works asynchronously.
// Create some very badly 'mocked' data
var idColumn = new DataColumn(
new DataField<int>("id"),
Enumerable.Range(0, 10000).Select(i => i).ToArray());
var cityColumn = new DataColumn(
new DataField<string>("city"),
Enumerable.Range(0, 10000).Select(i => i % 2 == 0 ? "London" : "Grimsby").ToArray());
var schema = new Schema(idColumn.Field, cityColumn.Field);
using (var pipeStream = new PipeStream())
{
var buffer = new byte[4096];
int read = 0;
var readTask = Task.Run(async () =>
{
//transferUtil.Upload(readStream, "bucketName", "key"); // Execute this in a Task / Thread
while ((read = await pipeStream.ReadAsync(buffer, 0, buffer.Length)) > 0)
{
var incoming = Encoding.ASCII.GetString(buffer, 0, read);
Console.WriteLine(incoming);
// await Task.Delay(5000); uncomment this to simulate very slow consumer
}
});
using (var parquetWriter = new ParquetWriter(schema, pipeStream)) // This destructor finishes the file and transferUtil closes the stream, so we need this weird using nesting to keep everyone happy.
using (var rowGroupWriter = parquetWriter.CreateRowGroup())
{
rowGroupWriter.WriteColumn(idColumn); // Step through both these statements to see data read before the parquetWriter completes
rowGroupWriter.WriteColumn(cityColumn);
}
}
The implementation is not completely finished but I think it shows a nice approach. In the console 'readTask' you can un-comment the Task.Delay to simulate a slow read (transferUtil) and you should see the pipe automatically throttles the write task.
You need to be using C# 7.2 or later (VS 2017 -> Project Properties -> Build -> Advanced -> Language Version) for one of the Span extension methods but it should be compatible with any .Net Framework. You may need the Nuget Package
The stream is readable and writable (obviously!) but not seekable which should work for you in this scenario but wouldn't work reading from the Parquet SDK which requires seekable streams.
Hope it helps
Using System.IO.Pipelines it would look something like this:
var pipe = new System.IO.Pipelines.Pipe();
using (var buffer = pipe.Writer.AsStream())
using (var transferUtil = new TransferUtility(s3client))
{
// we can start the consumer first because it will just block
// on the stream until data is available
Task consumer = transferUtil.UploadAsync(pipe.Reader.AsStream(), _bucketName, _key.Replace(".gz", "") + ".parquet");
// start a task to produce data
Task producer = WriteParquetAsync(buffer, ..);
// start pumping data; we can wait here because the producer will
// necessarily finish before the consumer does
await producer;
// this is key; disposing of the buffer early here causes the consumer stream
// to terminate, else it will just hang waiting on the stream to finish.
// see the documentation for Writer.AsStream(bool leaveOpen = false)
buffer.Dispose();
// wait the upload to finish
await consumer;
}
I'm having some trouble with a simple TCP Read/Write application where I need to write a command to a device/host. Normally I can do this using a stream.Write() command however with this particular device, it seems to send an initial welcome message back (PJLINK 0) before any command can be sent to it. I can send the commands fine using PuTTY but when using C# I think my connection is closing before I can get my command through.
So my question would be how can I adjust my code below to receive that welcome message and then send my command back (I don't need to read a response) without the TcpClient closing the connection early?
Any help would be greatly appreciated.
using (tcpClientA = new TcpClient())
{
int portA = 4352;
if (!tcpClientA.BeginConnect("10.0.2.201", portA, null, null).AsyncWaitHandle.WaitOne(TimeSpan.FromSeconds(1.0)))
{
throw new Exception("Failed to connect.");
}
while (tcpClientA.Connected)
{
using (streamA = tcpClientA.GetStream())
{
if (type == "raw")
{
// Buffer to store the response bytes.
byte[] writeBufferC = Encoding.ASCII.GetBytes("%1 INPT 32$0D"); //Command I need to send
byte[] readBufferC = new byte[tcpClientA.ReceiveBufferSize];
string fullServerReply = null;
using (var writer = new MemoryStream())
{
do
{
int numberOfBytesRead = streamA.Read(readBufferC, 0, readBufferC.Length);
if (numberOfBytesRead <= 0)
{
break;
}
writer.Write(writeBufferC, 0, writeBufferC.Length);
} while (streamA.DataAvailable);
fullServerReply = Encoding.UTF8.GetString(writer.ToArray());
Console.WriteLine(fullServerReply.Trim());
}
}
}
}
}
Update 1
Removed the BeginConnect and Async methods.
using (tcpClientA = new TcpClient())
{
int portA = 4352;
tcpClientA.Connect("10.0.2.201", portA);
while (tcpClientA.Connected)
{
using (streamA = tcpClientA.GetStream())
{
if (type == "raw")
{
byte[] readBufferC = new byte[tcpClientA.ReceiveBufferSize];
byte[] writeBufferC = Encoding.ASCII.GetBytes("%1 INPT 31$0D"); //Command I need to send
string fullServerReply = null;
using (var writer = new MemoryStream())
{
do
{
streamA.Read(readBufferC, 0, readBufferC.Length); //First read
writer.Write(writeBufferC, 0, writeBufferC.Length); //Send command
} while (streamA.DataAvailable);
fullServerReply = Encoding.UTF8.GetString(readBufferC.ToArray());
Console.WriteLine(fullServerReply.Trim());
tcpClientA.Close();
}
}
}
}
}
DataAvailable does not tell you how much data will be sent in the future by the remote side. It's use is almost always a bug. Here, it causes you to randomly exit the loop early.
Read, until you have all the bytes you expect or until the stream is being closed.
Is this a line-based protocol? Instantiate a StreamReader and draw entire lines from the stream.
while (tcpClientA.Connected) accomplishes nothing. Even if it returns true, the connection could be lost 1 nanosecond later. Your code has to deal with that anyway. It should be while (true). This is not a bug, it just shows weak TCP understanding so I point it out.
Remove all usages of ReceiveBufferSize. This value means nothing of significance. Instead, use a fixed buffer size. I find that 4096 works well with not very high throughput connections.
numberOfBytesRead <= 0 should be ==0. Again, not a bug but you don't seem to understand exactly what the API does. This is dangerous.
In the updated code you're not using the return value of streamA.Read which is a bug. You have tried to fix that bug by trimming off the resulting \0 chars. That's just treating the symptoms and is not a true fix.
You need a socket tutorial. This carnage comes because you are not relying on best practices. Socket reading loops are actually rather simple if done right. This code is a collection of what can go wrong.
I am using Pcap.Net take Pcap File and transmit all it's packet through my machine Network Adapter.
So in order to do that i am using the code example Sending packets using Send Buffer:
class Program
{
static void Main(string[] args)
{
string file = #"C:\file_1.pcap";
string file2 = #"C:\file_2.pcap";
// Retrieve the device list from the local machine
IList<LivePacketDevice> allDevices = LivePacketDevice.AllLocalMachine;
// Take the selected adapter
PacketDevice selectedOutputDevice = allDevices[1];
SendPackets(selectedOutputDevice, file);
SendPackets(selectedOutputDevice, file2);
}
static void SendPackets(PacketDevice selectedOutputDevice, string file)
{
// Retrieve the length of the capture file
long capLength = new FileInfo(file).Length;
// Chek if the timestamps must be respected
bool isSync = false;
// Open the capture file
OfflinePacketDevice selectedInputDevice = new OfflinePacketDevice(file);
using (PacketCommunicator inputCommunicator = selectedInputDevice.Open(65536, PacketDeviceOpenAttributes.Promiscuous, 1000))
{
using (PacketCommunicator outputCommunicator = selectedOutputDevice.Open(100, PacketDeviceOpenAttributes.Promiscuous, 1000))
{
// Allocate a send buffer
using (PacketSendBuffer sendBuffer = new PacketSendBuffer((uint)capLength))
{
// Fill the buffer with the packets from the file
Packet packet;
while (inputCommunicator.ReceivePacket(out packet) == PacketCommunicatorReceiveResult.Ok)
{
//outputCommunicator.SendPacket(packet);
sendBuffer.Enqueue(packet);
}
// Transmit the queue
outputCommunicator.Transmit(sendBuffer, isSync);
inputCommunicator.Dispose();
}
outputCommunicator.Dispose();
}
//inputCommunicator.Dispose();
}
}
}
In order to send packet Pcap.Net offers 2 ways:
Send buffer.
Send each packet using SendPacket().
Now after finish to send my 2 files (like in my example) i want to use the Dispose() to free resources.
when using the first option all works fine and this finish to handle my 2 Pcap files.
When using the second option SendPacket() (currently in my code example this is as a comments) after the first file finish my application is closing and not reach to the second file.
I try it also in Console Application and in WPF and in both cases same result.
With UI (WPF) my application GUI just close without any error.
Any suggestions what could cause this ?
When you use the using keyword it means you implicitly call Dispose() at the end of the scope.
If you call Dispose() explicitly as well, it means you call Dispose() twice on the same instance, which is likely to crash your program.
I have a piece of code that reads a JSON stream from a server on the public internet. I am trying to make the connection a little more robust by catching the exception and trying to restart it on a given interval but I haven't been able to figure out how to restart it.
My stream code is as follows
TcpClient connection = new TcpClient(hostname, port);
NetworkStream stream = connection.GetStream();
thread = new Thread(ProcessStream);
thread.Start(stream);
My ProcessStream method is
private void ProcessStream(object stream)
{
Stream source = (NetworkStream)stream;
byte[] line;
int count;
const int capacity = 300;
ReadState readState;
while ((readState = ReadStreamLine(source, out line, out count, capacity)) != ReadState.EOF && _stopFeed == false)
{
if (readState != ReadState.Error && count > 4)
{
byte[] line1 = new byte[count];
Array.Copy(line, line1, count);
Process(line1); // return ignored in stream mode
}
else
{
ReadFail(line, count);
}
}
}
and my ReadStream function takes the stream s, does an s.ReadByte and then catches the exception when the network connection is broken. It is here that I am not sure how to try and restart the stream on a timed basis. It does not restart automatically when the network is restored.
That is not possible. It is like you calling your friend on the phone and he hangs up in the middle of a conversation. No matter how long you wait, you'll never hear from him again. All you can do is hang-up the phone and dial the number again. Unless the server supports restartable downloads (use HttpWebRequest.AddRange), you'll have to download the json again from the beginning.
If this happens a lot, so it can't be explained by the server going offline or getting overloaded, do keep in mind that the server might well be doing this on purpose. Usually because you exceeded some kind of quota. Talk to the server owner, they typically have a paid plan to allow you to use more resources.
From what I can tell, you instantiate your TcpClient before you start your method. So, in order to restart your stream, you need to re-instantiate or re-initialize your connection stream before trying again.
try
{
// Do something
}
catch (Exception ex)
{
// Caught your exception, might be ideal to log it too
// Have a count, if count is less than goal
// Call your method again
if (count < 5)
{
// re-initialize or re-instantiate connection
TcpClient connection = new TcpClient(host, port);
NetworkStream stream = connection.GetStream();
ProcessStream(stream);
}
}
I hope this helps.
You coculd at first add your stream to a static list of running streams and after finishing reading remove it from there.
Remember to use locking!
Then in the NetworkGone-Catch you can copy your list to a "todoAfterNetworkIsUpAgain"-List and start a timer that checks for network and after your network is up again restarts reading the streams again.
This Might look a bit tuff but its not the case.
Use Recursion and threading in a better way and your problem might get resolved
For Recursion
http://www.dotnetperls.com/recursion
For Threading
Take a look to msdn documentation or take consepts from albahari
I have following code:
public static async Task<string> ReadLineAsync(this Stream stream, Encoding encoding)
{
byte[] byteArray = null;
using (MemoryStream ms = new MemoryStream())
{
int bytesRead= 0;
do
{
byte[] buf = new byte[1024];
try
{
bytesRead = await stream.ReadAsync(buf, 0, 1024);
await ms.WriteAsync(buf, 0, bytesRead);
}
catch (Exception e)
{
Console.WriteLine(e.Message + e.StackTrace);
}
} while (stream.CanRead && bytesRead> 0);
byteArray = ms.ToArray();
return encoding.GetString(ms.ToArray());
}
I am trying to read Stream to write into MemoryStream asynchronously, but the Do...while loop is failing to break. I mean it's an infinite loop. How to solve this?
First, in an exceptional situation, your loop would continue indefinitely. You shouldn't catch and ignore exceptions.
Secondly, if the stream doesn't actually end, then bytesRead would never be zero. I suspect this is the case because the name of the method (ReadLineAsync) doesn't imply to me that it will read until the end of the stream.
P.S. CanRead does not ever change for a specific stream. It's whether it makes semantic sense for a stream to do a read operation, not whether it can read right now.
You have your loop condition set to run as long as CanRead is true and bytesRead is greater then 0. CanRead will always be true if your file is readable. This means as long as you start reading your bytes will always be greater than zero. You need to have a maximum number of bytes to be read as well as a minimum or set some other control to break out.
So, you are taking a stream from IMAP and this method is for converting that steam into text?
Why not construct a SteamReader round the stream and call either it's ReadToEndAsync or just ReadToEnd? I doubt the need for making this an Async operation, if the stream is something like an e-mail then it is unlikely to be so big that a user will notice the UI blocking while it reads.
If, as one of your comments suggests, this isn't a UI app at all then it is probably even less of an issue.
If my assumptions are wrong then could I ask you to update your question with some more information about how this function is being used. The more information you can tell us, the better our answers can be.
EDIT:
I just noticed that your method is called ReadLineAsync, although I can't see anywhere in the code that you are looking for a line ending. If your intention is to read a line of text then the SteamReader also provides ReadLine and ReadLineAsync.
I took your method and modified it just a tad by shortening the read buffer size and adding some debug statements
public static async Task<string> ReadLineAsync(this Stream stream, Encoding encoding)
{
const int count = 2;
byte[] byteArray = Enumerable.Empty<byte>().ToArray();
using (MemoryStream ms = new MemoryStream())
{
int bytesRead = 0;
do
{
byte[] buf = new byte[count];
try
{
bytesRead = await stream.ReadAsync(buf, 0, count);
await ms.WriteAsync(buf, 0, bytesRead);
Console.WriteLine("{0:ffffff}:{1}:{2}",DateTime.Now, stream.CanRead, bytesRead);
}
catch (Exception e)
{
Console.WriteLine(e.Message + e.StackTrace);
}
} while (stream.CanRead && bytesRead > 0);
byteArray = ms.ToArray();
return encoding.GetString(byteArray);
}
}
but basically it worked as expected with the following call:
private static void Main(string[] args)
{
FileStream stream = File.OpenRead(#"C:\in.txt");
Encoding encoding = Encoding.GetEncoding(1252);
Task<string> result = stream.ReadLineAsync(encoding);
result.ContinueWith(o =>
{
Console.Write(o.Result);
stream.Dispose();
});
Console.WriteLine("Press ENTER to continue...");
Console.ReadLine();
}
so I'm wondering could it be something with your input file? Mine was (encoded in Windows-1252 in Notepad++)
one
two
three
and my output was
Press ENTER to continue...
869993:True:2
875993:True:2
875993:True:2
875993:True:2
875993:True:2
875993:True:2
875993:True:2
875993:True:1
875993:True:0
one
two
three
note how the "Press ENTER to continue..." was printed first as expected because the main method was invoked asynchronously, and CanRead is always true because it means the file is readable. Its the state of how the file was opened, not the state meaning that the cursor is at the EOF.
From my POV, looks like your code is trying to do the following:
read an entire stream as a sequence of 1024-octet chunks,
concatenate all those chunks into a MemoryStream (which uses a byte array as its backing store),
convert the MemoryStream to a string using the specified encoding
return that string to the caller.
This seems...complicated to me. Maybe I'm missing something, but to use async and await, you've got to be using VS2012 and .Net 4.5, or VS2010. .Net 4.0 and the Async CTP, right? If so, why wouldn't you simply use a StreamReader and its StreamReader.ReadToEndAsync() method?
public static async Task<string> MyReadLineAsync(this Stream stream, Encoding encoding)
{
using ( StreamReader reader = new StreamReader( stream , encoding ) )
{
return await reader.ReadToEndAsync() ;
}
}
The overlapping i/o idea is nice, but the time required to write to a memory stream is, to say the least, not enough to make one whit of difference with respect to the time required to peform actual I/O (presumably your input stream is doing disk or network i/o).