Related
I'm working with a 3rd party, command-line tool called "sam-ba" v3.5 (available here for free). It's a C++ / QML command line tool that interfaces with a hardware module to read/write data. Output from commands, in most cases, is sent to Standard Error.
I have a C# / .NET application that creates a Process object to execute the sam-ba tool and run commands. Executing the commands works as expected. What doesn't always work is the redirect of the Standard Error output. In some commands, part or all of the output is not received by the C# application. For example, here is the execution of a command using the sam-ba tool directly at the Windows 10 command line:
C:\Temp\Stuff\sam-ba_3.5>sam-ba -p serial:COM5 -d sama5d3 -m version
Error: Cannot open invalid port 'COM5'
Cannot open invalid port 'COM5'
Here is some simple code from a C# application to create a Process object to execute the sam-ba tool with the same command:
Process p = new Process
{
StartInfo = new ProcessStartInfo("sam-ba.exe", "-p serial:COM5 -d sama5d3 -m version")
{
RedirectStandardOutput = true,
RedirectStandardError = true,
UseShellExecute = false,
CreateNoWindow = true
}
};
p.Start();
string output = p.StandardOutput.ReadToEnd();
string error = p.StandardError.ReadToEnd();
p.WaitForExit();
Console.WriteLine("Standard Out: " + output);
Console.WriteLine("Standard Error: " + error);
The Output of the C# application:
Standard Out:
Standard Error: Cannot open invalid port 'COM5'
In this simple example, only 1 of the output lines is redirected to Standard Error while the other is not. I've tried many different commands and results are mixed. Sometimes I get everything, sometimes partial output, sometimes no output.
Now ... here's the real issue. The following is a python script (v3.8) that does exactly what the C# application is doing:
import subprocess
import sys
result = subprocess.run("sam-ba.exe -p serial:COM5 -d sama5d3 -m version", capture_output=True, text=True)
print("stdout:", result.stdout)
print("stderr:", result.stderr)
This script always returns the correct output to standard error. BUT ... when I run this script from the C# app to create a chain of C# -> python -> sam-ba, I get the same issue of output missing from the stream.
This has led me to 2 conclusions:
Something in that sam-ba tool is different about the way it is outputting its text. Format, content, ... something. There's an inconsistency going on somewhere in that code
Something is different about the environment created by the C# Process object when executing external applications that doesn't happen when the external application is run directly. Otherwise, why would the python script get all the output when run directly, but not when run through the C# Process object?
It is #2 that has brought me here. I'm looking for any insight on how to diagnose this. Something I'm doing wrong, settings I can try within the Process object, thoughts on how data can go into a stream and not come out on redirect, or if anyone has ever seen something like this before and how they resolved it.
UPDATE
Got a hold of the sam-ba tool's source code. The output the C# app is not capturing is coming from the QML files. They are using this 'print()' method that I can't really find any details on. The output the C# app can capture is being delivered back to the C++ side via signals and then sent to standard error. This feeds back into my conclusion #1 where they have inconsistencies in their code.
Still, this potentially means there is a conflict between C# and QT/QML, which would explain why the Python script gets the QML output, but the C# app does not.
The following uses ShellExecute instead of CreateProcess when running
process. When using ShellExecute one can't re-direct StandardOutput and/or StandardError for Process. To work around this, both StandardOutput and StandardError are re-directed to a temp file, and then the data is read from the temp file--which seems to result in the same output that one sees when running from a cmd window.
Note: In the following code it's necessary to use %windir%\system32\cmd.exe (ex: C:\Windows\system32\cmd.exe) with the /c option. See the usage section below.
Add using statement: using System.Diagnostics;
Then try the following:
public string RunProcess(string fqExePath, string arguments, bool runAsAdministrator = false)
{
string result = string.Empty;
string tempFilename = System.IO.Path.Combine(System.IO.Path.GetTempPath(), "tempSam-ba.txt");
string tempArguments = arguments;
if (String.IsNullOrEmpty(fqExePath))
{
Debug.WriteLine("fqExePath not specified");
return "Error: fqExePath not specified";
}
//redirect both StandardOutput and StandardError to a temp file
if (!arguments.Contains("2>&1"))
{
tempArguments += String.Format(" {0} {1} {2}", #"1>", tempFilename, #"2>&1");
}
//create new instance
ProcessStartInfo startInfo = new ProcessStartInfo(fqExePath, tempArguments);
if (runAsAdministrator)
{
startInfo.Verb = "runas"; //elevates permissions
}//if
//set environment variables
//pStartInfo.EnvironmentVariables["SomeVar"] = "someValue";
startInfo.RedirectStandardError = false;
startInfo.RedirectStandardOutput = false;
startInfo.RedirectStandardInput = false;
startInfo.UseShellExecute = true; //use ShellExecute instead of CreateProcess
startInfo.CreateNoWindow = false;
startInfo.WindowStyle = ProcessWindowStyle.Hidden;
startInfo.ErrorDialog = false;
startInfo.WorkingDirectory = System.IO.Path.GetDirectoryName(fqExePath);
using (Process p = Process.Start(startInfo))
{
//start
p.Start();
//waits until the process is finished before continuing
p.WaitForExit();
}
//read output from temp file
//file may still be in use, so try to read it.
//if it is still in use, sleep and try again
if (System.IO.File.Exists(tempFilename))
{
string errMsg = string.Empty;
int count = 0;
do
{
//re-initialize
errMsg = string.Empty;
try
{
result = System.IO.File.ReadAllText(tempFilename);
Debug.WriteLine(result);
}
catch(System.IO.IOException ex)
{
errMsg = ex.Message;
}
catch (Exception ex)
{
errMsg = ex.Message;
}
System.Threading.Thread.Sleep(125);
count += 1; //increment
} while (!String.IsNullOrEmpty(errMsg) && count < 10);
//delete temp file
System.IO.File.Delete(tempFilename);
}
return result;
}
Usage:
RunProcess(#"C:\Windows\system32\cmd.exe", #"/c C:\Temp\sam-ba_3.5\sam-ba.exe -p serial:COM5 -d sama5d3 -m version");
Note: /c C:\Temp\sam-ba_3.5\sam-ba.exe -p serial:COM5 -d sama5d3 -m version is the value of the process "Argument" property.
Update:
Option 2:
Here's a solution that uses named pipes. Process is used to redirect the output to a named pipe instead of a file. One creates a named pipe "server" which listens for a connection from a client.Then System.Diagnostics.Process is used to run the desired command and redirect the output to the named pipe server. The "server" reads the output, and then raises event "DataReceived" which will return the data to any subscribers.
The named pipe server code is from here, however I've modified it. I've added numerous events--which can be subscribed to. I've also added the ability for the server to shut itself down after it's finished reading the data by setting "ShutdownWhenOperationComplete" to "true".
Create a class named: HelperNamedPipeServer.cs
HelperNamedPipeServer.cs
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using System.IO.Pipes;
using System.IO;
using System.Diagnostics;
using System.Threading;
using System.Security.Principal;
namespace ProcessTest
{
public class HelperNamedPipeServer : IDisposable
{
//delegates
public delegate void EventHandlerClientConnected(object sender, bool e);
public delegate void EventHandlerDataReceived(object sender, string data);
public delegate void EventHandlerOperationCompleted(object sender, bool e);
public delegate void EventHandlerMessageComplete(object sender, bool e);
public delegate void EventHandlerReadComplete(object sender, bool e);
public delegate void EventHandlerServerShutdown(object sender, bool e);
public delegate void EventHandlerServerStarted(object sender, bool e);
//event that subscribers can subscribe to
public event EventHandlerClientConnected ClientConnected;
public event EventHandlerDataReceived DataReceived;
public event EventHandlerMessageComplete MessageReadComplete;
public event EventHandlerOperationCompleted OperationCompleted;
public event EventHandlerReadComplete ReadComplete;
public event EventHandlerServerShutdown ServerShutdown;
public event EventHandlerServerStarted ServerStarted;
public bool IsClientConnected
{
get
{
if (_pipeServer == null)
{
return false;
}
else
{
return _pipeServer.IsConnected;
}
}
}
public string PipeName { get; set; } = string.Empty;
public bool ShutdownWhenOperationComplete { get; set; } = false;
//private int _bufferSize = 4096;
private int _bufferSize = 65535;
//private volatile NamedPipeServerStream _pipeServer = null;
private NamedPipeServerStream _pipeServer = null;
public HelperNamedPipeServer()
{
PipeName = "sam-ba-pipe";
}
public HelperNamedPipeServer(string pipeName)
{
PipeName = pipeName;
}
private NamedPipeServerStream CreateNamedPipeServerStream(string pipeName)
{
//named pipe with security
//SecurityIdentifier sid = new SecurityIdentifier(WellKnownSidType.BuiltinAdministratorsSid, null); //member of Administrators group
//SecurityIdentifier sid = new SecurityIdentifier(WellKnownSidType.WorldSid, null); //everyone
//SecurityIdentifier sid = new SecurityIdentifier(WellKnownSidType.BuiltinUsersSid, null); //member of Users group
//PipeAccessRule rule = new PipeAccessRule(sid, PipeAccessRights.ReadWrite, System.Security.AccessControl.AccessControlType.Allow);
//PipeSecurity pSec = new PipeSecurity();
//pSec.AddAccessRule(rule);
//named pipe - with specified security
//return new NamedPipeServerStream(PipeName, PipeDirection.InOut, NamedPipeServerStream.MaxAllowedServerInstances, PipeTransmissionMode.Byte, PipeOptions.Asynchronous, _bufferSize, _bufferSize, pSec);
//named pipe - access for everyone
//return new System.IO.Pipes.NamedPipeServerStream(pipeName, PipeDirection.InOut, NamedPipeServerStream.MaxAllowedServerInstances, PipeTransmissionMode.Message, PipeOptions.Asynchronous);
return new System.IO.Pipes.NamedPipeServerStream(pipeName, PipeDirection.InOut, NamedPipeServerStream.MaxAllowedServerInstances, PipeTransmissionMode.Byte, PipeOptions.Asynchronous);
}
public void Dispose()
{
Shutdown();
}
private void OnClientConnected()
{
LogMsg("OnClientConnected");
//raise event
if (ClientConnected != null)
ClientConnected(this, true);
}
private void OnDataReceived(string data)
{
LogMsg("OnClientConnected");
//raise event
if (DataReceived != null && !String.IsNullOrEmpty(data))
{
if (DataReceived != null)
DataReceived(this, data);
}
}
private void OnMessageReadComplete()
{
LogMsg("OnMessageReadComplete");
//raise event
if (MessageReadComplete != null)
MessageReadComplete(this, true);
}
private void OnOperationCompleted()
{
LogMsg("OnOperationCompleted");
//raise event
if (OperationCompleted != null)
OperationCompleted(this, true);
}
private void OnReadComplete()
{
LogMsg("OnReadComplete");
//raise event
if (ReadComplete != null)
ReadComplete(this, true);
}
private void OnServerShutdown()
{
LogMsg("OnServerShutdown");
//raise event
if (ServerShutdown != null)
ServerShutdown(this, true);
}
private void OnServerStarted()
{
LogMsg("OnServerStarted");
//raise event
if (ServerStarted != null)
ServerStarted(this, true);
}
private async void DoConnectionLoop(IAsyncResult result)
{ //wait for connection, then process the data
if (!result.IsCompleted) return;
if (_pipeServer == null) return;
//IOException = pipe is broken
//ObjectDisposedException = cannot access closed pipe
//OperationCanceledException - read was canceled
//accept client connection
try
{
//client connected - stop waiting for connection
_pipeServer.EndWaitForConnection(result);
OnClientConnected(); //raise event
}
catch (IOException) { RebuildNamedPipe(); return; }
catch (ObjectDisposedException) { RebuildNamedPipe(); return; }
catch (OperationCanceledException) { RebuildNamedPipe(); return; }
while (IsClientConnected)
{
if (_pipeServer == null) break;
try
{
// read from client
string clientMessage = await ReadClientMessageAsync(_pipeServer);
OnDataReceived(clientMessage); //raise event
}
catch (IOException) { RebuildNamedPipe(); return; }
catch (ObjectDisposedException) { RebuildNamedPipe(); return; }
catch (OperationCanceledException) { RebuildNamedPipe(); return; }
}
//raise event
OnOperationCompleted();
if (!ShutdownWhenOperationComplete)
{
//client disconnected. start listening for clients again
if (_pipeServer != null)
RebuildNamedPipe();
}
else
{
Shutdown();
}
}
private void LogMsg(string msg)
{
//ToDo: log message
string output = String.Format("{0} - {1}", DateTime.Now.ToString("yyyy/MM/dd HH:mm:ss"), msg);
//ToDo: uncomment this line, if desired
//Debug.WriteLine(output);
}
private void RebuildNamedPipe()
{
Shutdown();
_pipeServer = CreateNamedPipeServerStream(PipeName);
_pipeServer.BeginWaitForConnection(DoConnectionLoop, null);
}
private async Task<string> ReadClientMessageAsync(NamedPipeServerStream stream)
{
byte[] buffer = null;
string clientMsg = string.Empty;
StringBuilder sb = new StringBuilder();
int msgIndex = 0;
int read = 0;
LogMsg("Reading message...");
if (stream.ReadMode == PipeTransmissionMode.Byte)
{
LogMsg("PipeTransmissionMode.Byte");
//byte mode ignores message boundaries
do
{
//create instance
buffer = new byte[_bufferSize];
read = await stream.ReadAsync(buffer, 0, buffer.Length);
if (read > 0)
{
clientMsg = Encoding.UTF8.GetString(buffer, 0, read);
//string clientMsg = Encoding.Default.GetString(buffer, 0, read);
//remove newline
//clientMsg = System.Text.RegularExpressions.Regex.Replace(clientString, #"\r\n|\t|\n|\r|", "");
//LogMsg("clientMsg [" + msgIndex + "]: " + clientMsg);
sb.Append(clientMsg);
msgIndex += 1; //increment
}
} while (read > 0);
//raise event
OnReadComplete();
OnMessageReadComplete();
}
else if (stream.ReadMode == PipeTransmissionMode.Message)
{
LogMsg("PipeTransmissionMode.Message");
do
{
do
{
//create instance
buffer = new byte[_bufferSize];
read = await stream.ReadAsync(buffer, 0, buffer.Length);
if (read > 0)
{
clientMsg = Encoding.UTF8.GetString(buffer, 0, read);
//string clientMsg = Encoding.Default.GetString(buffer, 0, read);
//remove newline
//clientMsg = System.Text.RegularExpressions.Regex.Replace(clientString, #"\r\n|\t|\n|\r|", "");
//LogMsg("clientMsg [" + msgIndex + "]: " + clientMsg);
sb.Append(clientMsg);
msgIndex += 1; //increment
}
} while (!stream.IsMessageComplete);
//raise event
OnMessageReadComplete();
} while (read > 0);
//raise event
OnReadComplete();
LogMsg("message completed");
}
return sb.ToString();
}
private void Shutdown()
{
LogMsg("Shutting down named pipe server");
if (_pipeServer != null)
{
try { _pipeServer.Close(); } catch { }
try { _pipeServer.Dispose(); } catch { }
_pipeServer = null;
}
}
public void StartServer(object obj = null)
{
LogMsg("Info: Starting named pipe server...");
_pipeServer = CreateNamedPipeServerStream(PipeName);
_pipeServer.BeginWaitForConnection(DoConnectionLoop, null);
}
public void StopServer()
{
Shutdown();
OnServerShutdown(); //raise event
LogMsg("Info: Server shutdown.");
}
}
}
Next, I've created a "Helper" class that contains the code to start the named pipe server, run the command using Process, and return the data. There are three ways to get the data. It's returned by the method, one can subscribe to the "DataReceived" event, or once the method completes, the data will be in property "Data".
Helper.cs
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using System.Diagnostics;
using System.Runtime.InteropServices;
using System.IO.Pipes;
using System.IO;
using System.Threading;
namespace ProcessTest
{
public class Helper : IDisposable
{
public delegate void EventHandlerDataReceived(object sender, string data);
//event that subscribers can subscribe to
public event EventHandlerDataReceived DataReceived;
private StringBuilder _sbData = new StringBuilder();
private HelperNamedPipeServer _helperNamedPipeServer = null;
private bool _namedPipeServerOperationComplete = false;
public string Data { get; private set; } = string.Empty;
public Helper()
{
}
private void OnDataReceived(string data)
{
if (!String.IsNullOrEmpty(data) && DataReceived != null)
{
DataReceived(this, data);
//Debug.Write("Data: " + data);
}
}
public void Dispose()
{
ShutdownNamedPipeServer();
}
public async Task<string> RunSambaNamedPipesAsync(string fqExePath, string arguments, string pipeName = "sam-ba-pipe", string serverName = ".", bool runAsAdministrator = false)
{
string result = string.Empty;
string tempArguments = arguments;
//re-initialize
_namedPipeServerOperationComplete = false;
_sbData = new StringBuilder();
Data = string.Empty;
if (String.IsNullOrEmpty(fqExePath))
{
Debug.WriteLine("fqExePath not specified");
return "fqExePath not specified";
}
//create new instance
_helperNamedPipeServer = new HelperNamedPipeServer(pipeName);
_helperNamedPipeServer.ShutdownWhenOperationComplete = true;
//subscribe to events
_helperNamedPipeServer.DataReceived += HelperNamedPipeServer_DataReceived;
_helperNamedPipeServer.OperationCompleted += HelperNamedPipeServer_OperationCompleted;
//start named pipe server on it's own thread
Thread t = new Thread(_helperNamedPipeServer.StartServer);
t.Start();
//get pipe name to use with Process
//this is where output from the process
//will be redirected to
string fqNamedPipe = string.Empty;
if (String.IsNullOrEmpty(serverName))
{
fqNamedPipe = String.Format(#"\\{0}\pipe\{1}", serverName, pipeName);
}
else
{
fqNamedPipe = String.Format(#"\\{0}\pipe\{1}", ".", pipeName);
}
//redirect both StandardOutput and StandardError to named pipe
if (!arguments.Contains("2>&1"))
{
tempArguments += String.Format(" {0} {1} {2}", #"1>", fqNamedPipe, #"2>&1");
}
//run Process
RunProcess(fqExePath, tempArguments, runAsAdministrator);
while (!_namedPipeServerOperationComplete)
{
await Task.Delay(125);
}
//set value
Data = _sbData.ToString();
return Data;
}
public void RunProcess(string fqExePath, string arguments, bool runAsAdministrator = false)
{
if (String.IsNullOrEmpty(fqExePath))
{
Debug.WriteLine("fqExePath not specified");
throw new Exception( "Error: fqExePath not specified");
}
//create new instance
ProcessStartInfo startInfo = new ProcessStartInfo(fqExePath, arguments);
if (runAsAdministrator)
{
startInfo.Verb = "runas"; //elevates permissions
}//if
//set environment variables
//pStartInfo.EnvironmentVariables["SomeVar"] = "someValue";
startInfo.RedirectStandardError = false;
startInfo.RedirectStandardOutput = false;
startInfo.RedirectStandardInput = false;
startInfo.UseShellExecute = true; //use ShellExecute instead of CreateProcess
startInfo.WindowStyle = ProcessWindowStyle.Hidden;
startInfo.ErrorDialog = false;
startInfo.WorkingDirectory = System.IO.Path.GetDirectoryName(fqExePath);
using (Process p = Process.Start(startInfo))
{
//start
p.Start();
//waits until the process is finished before continuing
p.WaitForExit();
}
}
private void HelperNamedPipeServer_OperationCompleted(object sender, bool e)
{
//Debug.WriteLine("Info: Named pipe server - Operation completed.");
//set value
Data = _sbData.ToString();
//set value
_namedPipeServerOperationComplete = true;
}
private void HelperNamedPipeServer_DataReceived(object sender, string data)
{
Debug.WriteLine("Info: Data received from named pipe server.");
if (!String.IsNullOrEmpty(data))
{
//append
_sbData.Append(data.TrimEnd('\0'));
//send data to subscribers
OnDataReceived(data);
}
}
private void ShutdownNamedPipeServer()
{
Debug.WriteLine("Info: ShutdownNamedPipeServer");
try
{
if (_helperNamedPipeServer != null)
{
//unsubscribe from events
_helperNamedPipeServer.DataReceived -= HelperNamedPipeServer_DataReceived;
_helperNamedPipeServer.OperationCompleted -= HelperNamedPipeServer_OperationCompleted;
_helperNamedPipeServer.Dispose();
_helperNamedPipeServer = null;
}
}
catch (Exception ex)
{
}
}
}
}
Usage:
private async void btnRunUsingNamedPipes_Click(object sender, EventArgs e)
{
//Button name: btnRunUsingNamedPipes
using (Helper helper = new Helper())
{
//subscribe to event
helper.DataReceived += Helper_DataReceived;
var result = await helper.RunSambaNamedPipesAsync(#"C:\Windows\system32\cmd.exe", #"/c C:\Temp\sam-ba_3.5\sam-ba.exe -p serial:COM5 -d sama5d3 -m version");
Debug.WriteLine("Result: " + result);
//unsubscribe from event
helper.DataReceived -= Helper_DataReceived;
}
}
private void Helper_DataReceived(object sender, string data)
{
//System.Diagnostics.Debug.WriteLine(data);
//RichTextBox name: richTextBoxOutput
if (richTextBoxOutput.InvokeRequired)
{
richTextBoxOutput.Invoke((MethodInvoker)delegate
{
richTextBoxOutput.Text = data;
richTextBoxOutput.Refresh();
});
}
}
Resources:
When do we need to set ProcessStartInfo.UseShellExecute to True?
Redirecting error messages from Command Prompt: STDERR/STDOUT
PipeTransmissionMode.Message: How do .NET named pipes distinguish between messages?
WellKnownSidType Enum
I've created a service, which "locks" the desktop background, so you can't change it. In general, the service compares the current wallpaper with the one I want to be the new wallpaper. If it's not the same, it gets overwritten. This is my OnTimer()-Method, which gets executed every 2 seconds:
private void OnTimer(object sender, ElapsedEventArgs e)
{
if (!File.Exists("C:/Program Files/image.jpg"))
{
Assembly myAssembly = Assembly.GetExecutingAssembly();
Stream s = myAssembly.GetManifestResourceStream("MsService.Gandalf.jpg"); ;
byte[] b;
using (BinaryReader br = new BinaryReader(s))
{
b = br.ReadBytes((int)s.Length);
}
while (true)
{
try
{
File.WriteAllBytes("C:/Program Files/image.jpg", b);
break;
}
catch (Exception)
{
//stuff
}
}
}
//{path} is %Appdata%
if (!FileEquals($"{path}\\Microsoft\\Windows\\Themes\\TranscodedWallpaper", "C:/Program Files/image.jpg"))
{
byte[] file = File.ReadAllBytes("C:/Program Files/image.jpg");
while (true)
{
try
{
File.WriteAllBytes($"{path}\\Microsoft\\Windows\\Themes\\TranscodedWallpaper", file);
break;
}
catch (Exception)
{
//stuff
}
}
}
}
That's the FileEquals Method:
static bool FileEquals(string path1, string path2)
{
byte[] file1 = File.ReadAllBytes(path1);
byte[] file2 = File.ReadAllBytes(path2);
if (file1.Length == file2.Length)
{
for (int i = 0; i < file1.Length; i++)
{
if (file1[i] != file2[i])
{
return false;
}
}
return true;
}
return false;
}
When I start the service, It only changes the wallpaper if I change it actively, even if it already is another one that the one I want it to be, and only once. The service outputs no error when debugging, and the service doesn't crash. Also when I change the wallpaper, the breakpoints in the if(!File.Equals(...)) gets triggered. Virus scanner isn't alerting anything too. Why doesn't it work anyway?
I am using this code to monitor creation of files in certain folder:
_watcher = new RecoveringFileSystemWatcher(SourceFolder, "*.xml");
_watcher.Created += (_, e) =>
{
ProcessFile(e.Name);
};
RecoveringFileSystemWatcher is a fileSystemWatcher wrapper. It's constructor is:
public RecoveringFileSystemWatcher (string path, string filter)
{
_containedFSW = new FileSystemWatcher(path, filter);
}
The process works as expected but for some files, randomly, an exception is thrown telling that the file is used by another process.
This is the method that is launched upon file creation:
var nfo = new FileInfo(filePath);
if (nfo.Exists)
{
var archivoXml = nfo.Name;
string archivo = String.Empty;
try
{
string content = Task.Run(async () => await GetFileContent(filePath)).Result;
if (String.IsNullOrEmpty(content))
return false;
XmlDocument xml = new XmlDocument();
xml.LoadXml(content);
//The rest of the method
}
}
the method GetFileContent is this:
private async Task<string> GetFileContent(string filePath)
{
string content = String.Empty;
try
{
Console.Write("ONE - "); InfoLog.Save($"ONE {filePath}");
using (StreamReader sr = new StreamReader(filePath))
{
Console.Write("TWO - "); InfoLog.Save($"TWO {filePath}");
content = await sr.ReadToEndAsync().ConfigureAwait(false);
Console.Write($"THREE {(sr.BaseStream == null ? "Closed" : "Opened")} - "); InfoLog.Save($"THREE {(sr.BaseStream == null ? "Closed" : "Opened")} {filePath}");
sr.Close();
Console.WriteLine($"FOUR {(sr.BaseStream == null ? "Closed" : "Opened")}"); InfoLog.Save($"FOUR {(sr.BaseStream == null ? "Closed" : "Opened")} {filePath}");
}
}
catch (Exception ex)
{
InfoLog.Save($"XML file could be read -> {filePath}. See error log.");
ErrorLog.Save(ex);
}
return content;
}
Look at the log information I am writing to debug the process.
I got one case with a file called 1112186.xml.... this is recorded in the log:
18/12/2018 19:12:10 ONE D:\GestorDocumental\Origen\1112186.xml
18/12/2018 19:12:10 XML file could not be read -> D:\GestorDocumental\Origen\1112186.xml. See error log.
As you see, the exception is thrown at the "using" instruction.
If I see the full log, I can see that file, 1112186.xml, is never used before, so the only chance is that FSW keeps the file opened. I don't know why, but it seems this is happening.
It is clear also that this process is locking the file, because when I exit the console application and then run again, the file can be processed.
Any help about this, please?
thanks
Jaime
I usually use this method to check if file is locked. I got it from one of the link in stackoverflow.
public static bool IsFileClosed(string filepath)
{
bool fileClosed = false;
int retries = 20;
const int delay = 400; // set a delay period = retries*delay milliseconds
if (!File.Exists(filepath))
return false;
do
{
try
{
// Attempts to open then close the file in RW mode, denying other users to place any locks.
FileStream fs = File.Open(filepath, FileMode.Open, FileAccess.ReadWrite, FileShare.None);
fs.Close();
fileClosed = true; // success
}
catch (IOException) { }
retries--;
if (!fileClosed)
Thread.Sleep(delay);
}
while (!fileClosed && retries > 0);
return fileClosed;
}
This is a new class called FileTimerWatcher (it will have logger injected):
public FileTimerWatcher(ILogger logger) : base(logger)
{
if (timer == null)
{
// Create a timer with a 1.5 second interval.
// monitor the files after 1.5 seconds.
timer = new Timer(delay);
// Hook up the event handler for the Elapsed event.
timer.Elapsed += new ElapsedEventHandler(ProcessFolder);
timer.AutoReset = true;
timer.Enabled = true;
}
}
private void ProcessFolder(object sender, ElapsedEventArgs e)
{
var LastChecked = DateTime.Now;
string[] files = System.IO.Directory.GetFiles(SourceDirectory, somefilter, System.IO.SearchOption.TopDirectoryOnly);
foreach (string file in files)
{
ProcessFile(file); // process file here
}
}
I'm trying to make a simple application that communicates with a console program that operates in a similar way to the Windows classical cmd.exe.
The execution sequence is expected to be as follows:
Setup a background worker and run it to read the output.
Run the process and wait for a short while.
Feed a command to the Process.StandardInput
Wait for a few seconds
Exit the background thread and kill the process
But what's going on is not as expected because:
Not all the output of the cmd.exe is read. Even when excluding the step which writes to the StandardInput
The string was not passed to the command from the StandardInput even though it has AutoFlush = true.
I also tried Process.StandardInput.Flush() which does nothing
I also tried to pass it a string padded with spaces to a length larger than 4096 which is the buffer size with no results.
Nothing happens!! Why?
I tried this on dot net 4.5.2 and 4.7.1
Similar questions exist here, here, and here but none of the answers work. Others are implemented in another language. i.e. Java, Delphi etc
This is a simplified version of my code:
BackgroundWorker _outputWorker;
Process _process;
StreamWriter _inputWriter;
TextReader _outputReader;
void Main()
{
_outputWorker = new BackgroundWorker { WorkerSupportsCancellation = true };
_outputWorker.DoWork += OnOutputWorkerDoWork;
_outputWorker.RunWorkerAsync();
_process = new Process
{
EnableRaisingEvents = true,
StartInfo = new ProcessStartInfo
{
FileName = "cmd.exe",
Arguments = string.Empty,
UseShellExecute = false,
CreateNoWindow = true,
WindowStyle = ProcessWindowStyle.Hidden,
WorkingDirectory = Directory.GetCurrentDirectory(),
StandardOutputEncoding = Encoding.UTF8,
StandardErrorEncoding = Encoding.UTF8,
RedirectStandardInput = true,
RedirectStandardOutput = true,
RedirectStandardError = true
}
};
Console.WriteLine("Starting...");
if (!_process.Start()) return;
_inputWriter = _process.StandardInput;
_inputWriter.AutoFlush = true; // does nothing
_outputReader = TextReader.Synchronized(_process.StandardOutput);
// You can exclude this step too and still not get the expected output
Thread.Sleep(500);
_inputWriter.WriteLine("dir");
_inputWriter.Flush(); // does nothing, private field carpos = 0
_inputWriter.BaseStream.Flush(); // does nothing, private field carpos = 5 which is equal to length of "dir" command + 2 characters (NewLine \r\n)
//_inputWriter.WriteLine("dir".PadLeft(4096)); // does nothing
// also closing the stream does nothing and does something that I can't afford which is closing the exe
// _inputWriter.Close();
//
_process.WaitForExit(5000);
_outputWorker.CancelAsync();
_process.Kill();
Console.WriteLine("Done");
}
void OnOutput(string data)
{
// never mind thread safety for now. It's just a single line static call
Console.WriteLine(data);
}
void OnOutputWorkerDoWork(object sender, DoWorkEventArgs e)
{
const int BUFFER_SIZE = 4096;
StringBuilder builder = new StringBuilder(BUFFER_SIZE);
while (!_outputWorker.CancellationPending)
{
/*
* It'll keep on running untill it's canceled to reduce thread costs
* because the program will run different executables sequentially which
* all are similar to cmd.exe.
*/
try
{
// Simplified version without locking
if (_outputReader == null) continue;
TextReader reader = _outputReader;
if (reader.Peek() < 1) continue;
char[] buffer = new char[BUFFER_SIZE];
do
{
int count = reader.Read(buffer, 0, buffer.Length);
if (count > 0) builder.Append(buffer, 0, count);
}
while (reader.Peek() > 0);
}
catch (Exception ex)
{
// handle exception in debug mode
Console.WriteLine(ex.Message); // no exception generated!
continue;
}
if (builder.Length == 0) continue;
OnOutput(builder.ToString());
builder.Length = 0;
}
if (!IsWaitable(_process)) return;
try
{
if (_outputReader == null) return;
TextReader reader = _outputReader;
if (reader.Peek() < 1) return;
char[] buffer = new char[BUFFER_SIZE];
do
{
int count = reader.Read(buffer, 0, buffer.Length);
if (count > 0) builder.Append(buffer, 0, count);
}
while (reader.Peek() > 0);
}
catch (Exception ex)
{
// handle exception in debug mode
Console.WriteLine(ex.Message); // no exception generated!
return;
}
if (builder.Length > 0) OnOutput(builder.ToString());
}
bool IsWaitable(Process thisValue)
{
return thisValue != null && !thisValue.HasExited;
}
I can't use Process.BeginOutputReadLine because it waits for a NewLine to be present in the stream which does not happen for the last line of output. See this and this for details
I get
Microsoft Windows [Version xxxxx]
(c) copyright line
And the program does not display the command-line prompt unless more output comes from the process that contains a NewLine
The points of interest are:
1. Why does this code not read all the output as it should to the end?
After trying many things, it seems as if the buffer does not contain any more text to read and the stream seems to be missing some data from the original output of the executable. What is weird is that happens randomly. Sometimes I get the second line and sometimes not. In any case, I never got the output that should result from feeding internal commands to the cmd process.
2. The StandardInput (StreamWriter) actually does flush the buffer (or it thinks it does) and sets its charpos to zero while the BaseStream still has its charpos field = [length of buffer]. Why is this happening?
Any insights into what could be the problem or a workaround would be greatly appreciated.
Well, I got it to work eventually after 2 days of frustration. A very simple solution:
Process _process;
StreamWriter _inputWriter;
void Main()
{
_process = new Process
{
EnableRaisingEvents = true,
StartInfo = new ProcessStartInfo
{
FileName = "cmd.exe",
Arguments = string.Empty,
UseShellExecute = false,
CreateNoWindow = true,
WindowStyle = ProcessWindowStyle.Hidden,
WorkingDirectory = Directory.GetCurrentDirectory(),
StandardOutputEncoding = Encoding.UTF8,
StandardErrorEncoding = Encoding.UTF8,
RedirectStandardInput = true,
RedirectStandardOutput = true,
RedirectStandardError = true
}
};
_process.OutputDataReceived += (s, e) => // instead of using a background worker
{
if (e.Data == null) return;
Console.WriteLine(e.Data);
};
Console.WriteLine("Starting...");
if (!_process.Start()) return;
_process.BeginOutputReadLine(); // <- using BeginOutputReadLine
_inputWriter = _process.StandardInput;
_inputWriter.AutoFlush = true;
_inputWriter.WriteLine(); // <- my little trick here
// using LINQPad, replace it with Console.ReadLine();
string input = Util.ReadLine<string> ("Enter command:");
if (!string.IsNullOrEmpty(input)) _inputWriter.WriteLine(input);
_process.WaitForExit(5000);
_process.Kill();
Console.WriteLine("Done");
}
void OnOutput(string data)
{
Console.WriteLine(data);
}
Forgot to mention that I have to wait for user input otherwise it would work normally using Process.BeginOutputReadLine() without resorting to use a BackgroundWorker.
Anyway, I hope nobody goes through the 2 days user/dev experience and waste 2 days of their lives over this.
Edit:
Unfortunately, the previous solution is flawed.
The problem is caused by the internal class AsyncStreamReader.
Source code can be found here
After modifying its code to skip splitting its buffer's content into lines and skip queueing strings, it's working as expected and it's even faster!
Here is the modified version:
//
// Copyright (c) Microsoft Corporation. All rights reserved.
//
// ==--==
/*============================================================
**
** Class: AsyncStreamReader
**
** Purpose: For reading text from streams using a particular
** encoding in an asychronous manner used by the process class
**
**
===========================================================*/
using System;
using System.Diagnostics;
using System.IO;
using System.Text;
using System.Threading;
namespace System.Diagnostics
{
/// <summary>
/// http://www.dotnetframework.org/default.aspx/DotNET/DotNET/8#0/untmp/whidbey/REDBITS/ndp/fx/src/Services/Monitoring/system/Diagnosticts/AsyncStreamReader#cs/1/AsyncStreamReader#cs
/// </summary>
public sealed class AsyncStreamReader : DisposableBase, IDisposable
{
internal const int DEFAULT_BUFFER_SIZE = 4096; // Byte buffer size
private const int MIN_BUFFER_SIZE = 128;
private Decoder _decoder;
private byte[] _byteBuffer;
private char[] _charBuffer;
// Record the number of valid bytes in the byteBuffer, for a few checks.
// This is the maximum number of chars we can get from one call to
// ReadBuffer. Used so ReadBuffer can tell when to copy data into
// a user's char[] directly, instead of our internal char[].
private int _maxCharsPerBuffer;
// Store a backpointer to the process class, to check for user callbacks
private Process _process;
private StringBuilder _sb;
// Delegate to call user function.
private Action<string> _userCallBack;
// Internal Cancel operation
private bool _cancelOperation;
private ManualResetEvent _eofEvent;
public AsyncStreamReader(Process process, Stream stream, Action<string> callback, Encoding encoding)
: this(process, stream, callback, encoding, DEFAULT_BUFFER_SIZE)
{
}
// Creates a new AsyncStreamReader for the given stream. The
// character encoding is set by encoding and the buffer size,
// in number of 16-bit characters, is set by bufferSize.
public AsyncStreamReader(Process process, Stream stream, Action<string> callback, Encoding encoding, int bufferSize)
{
Debug.Assert(process != null && stream != null && encoding != null && callback != null, "Invalid arguments!");
Debug.Assert(stream.CanRead, "Stream must be readable!");
Debug.Assert(bufferSize > 0, "Invalid buffer size!");
Init(process, stream, callback, encoding, bufferSize);
}
private void Init(Process process, Stream stream, Action<string> callback, Encoding encoding, int bufferSize)
{
_process = process;
BaseStream = stream;
CurrentEncoding = encoding;
_userCallBack = callback;
_decoder = encoding.GetDecoder();
if (bufferSize < MIN_BUFFER_SIZE) bufferSize = MIN_BUFFER_SIZE;
_byteBuffer = new byte[bufferSize];
_maxCharsPerBuffer = encoding.GetMaxCharCount(bufferSize);
_charBuffer = new char[_maxCharsPerBuffer];
_sb = new StringBuilder(_charBuffer.Length);
_cancelOperation = false;
_eofEvent = new ManualResetEvent(false);
}
public void Close()
{
Dispose(true);
}
void IDisposable.Dispose()
{
Dispose(true);
}
protected override void Dispose(bool disposing)
{
if (disposing)
{
if (BaseStream != null)
{
BaseStream.Close();
BaseStream = null;
}
}
if (BaseStream != null)
{
BaseStream = null;
CurrentEncoding = null;
_decoder = null;
_byteBuffer = null;
_charBuffer = null;
}
if (_eofEvent != null)
{
_eofEvent.Close();
_eofEvent = null;
}
}
public Encoding CurrentEncoding { get; private set; }
public Stream BaseStream { get; private set; }
// User calls BeginRead to start the asynchronous read
public void BeginRead()
{
_cancelOperation = false;
BaseStream.BeginRead(_byteBuffer, 0, _byteBuffer.Length, ReadBuffer, null);
}
public void CancelOperation()
{
_cancelOperation = true;
}
// This is the async callback function. Only one thread could/should call this.
private void ReadBuffer(IAsyncResult ar)
{
if (_cancelOperation) return;
int byteLen;
try
{
byteLen = BaseStream.EndRead(ar);
}
catch (IOException)
{
// We should ideally consume errors from operations getting cancelled
// so that we don't crash the unsuspecting parent with an unhandled exc.
// This seems to come in 2 forms of exceptions (depending on platform and scenario),
// namely OperationCanceledException and IOException (for errorcode that we don't
// map explicitly).
byteLen = 0; // Treat this as EOF
}
catch (OperationCanceledException)
{
// We should consume any OperationCanceledException from child read here
// so that we don't crash the parent with an unhandled exc
byteLen = 0; // Treat this as EOF
}
if (byteLen == 0)
{
// We're at EOF, we won't call this function again from here on.
_eofEvent.Set();
}
else
{
int charLen = _decoder.GetChars(_byteBuffer, 0, byteLen, _charBuffer, 0);
if (charLen > 0)
{
_sb.Length = 0;
_sb.Append(_charBuffer, 0, charLen);
_userCallBack(_sb.ToString());
}
BaseStream.BeginRead(_byteBuffer, 0, _byteBuffer.Length, ReadBuffer, null);
}
}
// Wait until we hit EOF. This is called from Process.WaitForExit
// We will lose some information if we don't do this.
public void WaitUtilEof()
{
if (_eofEvent != null)
{
_eofEvent.WaitOne();
_eofEvent.Close();
_eofEvent = null;
}
}
}
}
Usage:
Process _process;
StreamWriter _inputWriter;
AsyncStreamReader _output;
void Main()
{
_process = new Process
{
EnableRaisingEvents = true,
StartInfo = new ProcessStartInfo
{
FileName = "cmd.exe",
Arguments = string.Empty,
UseShellExecute = false,
CreateNoWindow = true,
WindowStyle = ProcessWindowStyle.Hidden,
WorkingDirectory = Directory.GetCurrentDirectory(),
StandardOutputEncoding = Encoding.UTF8,
StandardErrorEncoding = Encoding.UTF8,
RedirectStandardInput = true,
RedirectStandardOutput = true,
RedirectStandardError = true
}
};
Console.WriteLine("Starting...");
if (!_process.Start()) return;
BeginRead();
_inputWriter = _process.StandardInput;
_inputWriter.AutoFlush = true;
Thread.Sleep(500);
string input = Util.ReadLine<string>("Type a command:");
if (!string.IsNullOrEmpty(input)) _inputWriter.WriteLine(input);
_process.WaitForExit(5000);
CancelRead();
_process.Kill();
Console.WriteLine("Done");
}
void OnOutput(string data)
{
Console.Write(data);
}
void BeginRead()
{
if (_output == null) _output = new AsyncStreamReader(_process, _process.StandardOutput.BaseStream, OnOutput, _process.StandardOutput.CurrentEncoding);
_output.BeginRead();
}
void CancelRead()
{
_output.CancelOperation();
}
In my program i write data to files if i am not able to connect to database and I have a separate thread that has timer which checks for connection availability after every 25 seconds and if it can connect it transfers the data from files to the main db and deletes the file.
the problem is i never stop this timer,can this lead to memory leaks?
if i just run my program and monitor the task manager i can see the memory usage increasing continuously if i disable the timer and then run my application then the memory is stable
public BackgroundWorker()
{
_backgroundWorkerThread = new Thread(new ThreadStart(ThreadEntryPoint));
_timer = new SWF.Timer();
_timer.Tick += new EventHandler(_timer_Tick);
_timer.Interval = 25 * 1000;
_timer.Enabled = true;
}
void _timer_Tick(object sender, EventArgs e)
{
bool lanAvailabe = NetworkInterface.GetIsNetworkAvailable();
if (lanAvailabe)
{
if (!GetListOfFiles())
{
return;
}
}
else
return;
}
implementation of GetListofFiles()
private bool GetListOfFiles()
{
string sourceDirectory = pathOfXmlFiles;
if (!Directory.Exists(sourceDirectory))
{
return false;
}
var xmlFiles = Directory.GetFiles(sourceDirectory, "*.xml");
if (!xmlFiles.Any())
{
return false;
}
foreach (var item in xmlFiles)
{
ReadXmlFile(item);
}
foreach (var item in xmlFiles)
{
if (_writtenToDb)
{
File.Delete(item);
}
}
return true;
}
method that reads xml files
private void ReadXmlFile(string filename)
{
string[] patientInfo = new string[15];
using (StreamReader sr = new StreamReader(filename, Encoding.Default))
{
String line;
line = sr.ReadToEnd();
if (line.IndexOf("<ID>") > 0)
{
patientInfo[0] = GetTagValue(line, "<ID>", "</ID>");
}
if (line.IndexOf("<PatientID>") > 0)
{
patientInfo[1] = GetTagValue(line, "<PatientID>", "</PatientID>");
}
if (line.IndexOf("<PatientName>") > 0)
{
patientInfo[2] = GetTagValue(line, "<PatientName>", "</PatientName>");
}
if (line.IndexOf("<Room>") > 0)
{
patientInfo[3] = GetTagValue(line, "<Room>", "</Room>");
}
}
WriteToDb(patientInfo);
}
if i just run my program and monitor the task manager i can see the memory usage increasing continuously
Get a profiler. Task Manager is not a right tool. Can't tell what is going on. It doesn't mean that you have a leak. Maybe just GC doesn't run cause there is enough space etc.