Language: C#
I have to pass a huge string array (built dynamically) as an argument to run an exe.
I am thinking of acheiving it by the below 2 ways. But I am not feeling confident.
I can create it as one string delimited by spaces. I can invoke the exe through Process.Start.
Hence the running child process considers the space and holds as a string array.
However I am unsure about the string array limitation. Suppose if my string array count exceeds more than 10,000
I can create it as one string delimited by a special symbol, which never fall in data. I can invoke the exe with the string. The running child process considers it as one single string, where i can split it with the same delimiter to get the string array back. However, here i am unsure about the command size. Will that do, if the command line string length is large
Can anyone help me in letting me know the parameter size limitations
It depends on the OS:
See Command prompt (Cmd. exe) command-line string limitation on the Microsoft Support site.
On computers running Microsoft Windows XP or later, the maximum length of the string that you can use at the command prompt is 8191 characters. On computers running Microsoft Windows 2000 or Windows NT 4.0, the maximum length of the string that you can use at the command prompt is 2047 characters.
(emphasis mine)
In regards to the size of a string array - if you have many millions of strings in a string array - you are mostly limited by the amount of memory available.
If you are passing 10,000 arguments to a program, you should be putting those arguments in a file and reading the file from disk.
Although a bad idea, Process.start with useshellexecute=false would invoke createprocess() which allows for 32767 characters in the command line (although this is also the maximum size for the entire environment block)
You could store the arguments in a text file and pass that text file as the argument. Your application can then parse the text file to analyse the arguments.
It is not really good practice to use command line arguments for huge arrays. Put your arguments in a configuration file instead, and just pass the filename as a command line argument.
The OS limit varies with the Windows version. It could be about 2k or 8k:
http://support.microsoft.com/kb/830473
You may want to consider creating a parameter file and passing the file as the parameter.
I found this:
For OS: maximum command line lenght is 32767 characters (this limit is from unicode string structure), command prompt maximum lenght is 8192 characters (this limit is from cmd.exe). You may also check:
http://support.microsoft.com/kb/830473
Hope this help.
The limitation in Command prompt (Cmd. exe) is 8191. Command prompt (Cmd. exe) command-line string limitation
The limitation in c# code using Process is 32768 chars in win10.
Tested using Process.start()
If starting the child process directly from the parent process is acceptable (UseShellExecute= false), then you could redirect the StandardInput of the child process and pass arbitrary size of data throw it. Here is an example passing an array of 100000 strings and other stuff, serializing them in binary format.
static void Main(string[] args)
{
if (args.Length == 0)
{
var exeFilePath = Assembly.GetExecutingAssembly().Location;
var psi = new ProcessStartInfo(exeFilePath, "CHILD");
psi.UseShellExecute = false;
psi.RedirectStandardInput = true;
Console.WriteLine("Parent - Starting child process");
var childProcess = Process.Start(psi);
var bf = new BinaryFormatter();
object[] data = Enumerable.Range(1, 100000)
.Select(i => (object)$"String-{i}")
.Append(13)
.Append(DateTime.Now)
.Append(new DataTable("Customers"))
.ToArray();
Console.WriteLine("Parent - Sending data");
bf.Serialize(childProcess.StandardInput.BaseStream, data);
Console.WriteLine("Parent - WaitForExit");
childProcess.WaitForExit();
Console.WriteLine("Parent - Closing");
}
else
{
Console.WriteLine("Child - Started");
var bf = new BinaryFormatter();
Console.WriteLine("Child - Reading data");
var data = (object[])bf.Deserialize(Console.OpenStandardInput());
Console.WriteLine($"Child - Data.Length: {data.Length}");
Console.WriteLine("Child - Closing");
}
}
Output:
Parent - Starting child process
Child - Started
Child - Reading data
Parent - Sending data
Parent - WaitForExit
Child - Data.Length: 100003
Child - Closing
Parent - Closing
This example executes in 6 sec in my machine.
Related
The docs for Console.ReadLine say:
By default, the method reads input from a 256-character input buffer. Because this includes the Environment.NewLine character(s), the method can read lines that contain up to 254 characters. To read longer lines, call the OpenStandardInput(Int32) method.
However, I can read more than that many characters just fine. Running a simple program like:
string s = Console.ReadLine();
Console.WriteLine(s);
Console.WriteLine(s.Length);
Console.ReadKey();
with an input like:
aaaaaaaaaabbbbbbbbbbccccccccccddddddddddeeeeeeeeeeffffffffffgggggggggghhhhhhhhhhiiiiiiiiiijjjjjjjjjjaaaaaaaaaabbbbbbbbbbccccccccccddddddddddeeeeeeeeeeffffffffffgggggggggghhhhhhhhhhiiiiiiiiiijjjjjjjjjjaaaaaaaaaabbbbbbbbbbccccccccccddddddddddeeeeeeeeeeffffffffffgggggggggghhhhhhhhhhiiiiiiiiiijjjjjjjjjjaaaaaaaaaabbbbbbbbbbccccccccccddddddddddeeeeeeeeeeffffffffffgggggggggghhhhhhhhhhiiiiiiiiiijjjjjjjjjjaaaaaaaaaabbbbbbbbbbccccccccccddddddddddeeeeeeeeeeffffffffffgggggggggghhhhhhhhhhiiiiiiiiiijjjjjjjjjj
yields the same input back with length 500.
Runnable example here.
Are the docs outdated then, or are the words "by default" key here?
An update:
Jeremy found a limit of 4096 defined in the source code.
I have verified that a .NET Core app will only read the first 4094 characters from stdin (excluding newlines).
In my case, I actually have a .NET Core 3.1 process that starts a .NET Framework 4.6 process, redirecting its StandardOut and StandardIn. I have verified that the .NET Framework process can successfully read 1 billion characters via Console.ReadLine() where the .NET Core 3.1 process sends the Framework process stuff via fwProcess.StandardInput.WriteLine(Serialize(<some stuff>));
This also works when the .NET Framework process is replaced with a .NET Core process.
So it seems like the 256-character limit doesn't apply when redirecting stdout/stdin, but if someone can dig up the definitive proof/docs explaining this, I would appreciate it. If there is still a limit (excluding the OutOfMemory case), but it's 1.1 billion characters, I would like to know. I'd also like to know if this is platform dependent (I'm on Windows 10).
If it helps, this is the code that I'm running.
ConsoleApp1:
ProcessStartInfo processInfo = new ProcessStartInfo {
CreateNoWindow = true,
FileName = "ConsoleApp2.exe",
UseShellExecute = false,
RedirectStandardOutput = true,
RedirectStandardInput = true
};
StringBuilder s = new StringBuilder();
var proc = Process.Start(processInfo);
int n = 1_000_000_000;
for (int i = 0; i < n; i++)
s.Append("a");
proc.StandardInput.WriteLine(s.ToString());
string s = proc.StandardOutput.ReadLine();
Console.WriteLine(s.Length == n); // logs True
Console.ReadKey();
ConsoleApp2:
string s = Console.ReadLine();
Console.WriteLine(s);
Paraphrasing from here:
The default cmd console mode is "ENABLE_LINE_INPUT", which means that
when code issues a ::ReadFile call against stdin, ::ReadFile doesn't
return to the caller until it encounters a carriage return. But the
call to ReadFile only has a limited size buffer that was passed to it.
Which means that cmd looks at the buffer size provided to it, and
determines based on that how many characters long the line can be...
if the line were longer, it wouldn't be able to store all the data
into the buffer.
The default buffer size used when opening Console.In is now 4096 (source)
The documentation is open source, and available here if you would like to submit an issue or pull request.
When redirecting StandardOut/StandardIn, this limit does not apply. From here:
The limit is how much memory you pass in.
This question already has an answer here:
Reading from a process, StreamReader.Peek() not working as expected
(1 answer)
Closed 6 years ago.
I am having trouble setting up a c# application that creates and interacts with a python process1. A simplistic example is given below.
EDIT: Further research on SO unveiled that my question is a possible duplicate. A potentially related known bug in the .NET Framework is duscussed here and here. It seems that back in 2014 the only easy workaround is indeed to ask the child process to write something in both stdOut and stdErr. But I'd like to know if this assumption is correct and wonder if there hasn't been a fix since 2014?
I have to fulfill the following boundary conditions:
I am not able to close the python process after handing over a script or a command, but I have to keep the process alive. Edit: For that reason I can not make use of the Process.WaitForExit() Method
As the std's remain open all the time, I believe I can't check for EndOfStream, as that would require to read to the end of the stream, which does not exist.
Furthermore, my application has to wait for the response of the python process, therefore the asynchronous option using BeginOutputReadLine() with OnOutputDataReceived seems not appropriate to me.
As the commands that will be sent to python are arbitrary user input, pythons result might be either in stdOut or stdErr ("4+7" results in "11" stored in stdOut; "4+a" results in "name 'a' is not defined" in stdErr)
What I do is to:
set up a python process in interactive mode (Argument "-i")
enable redirect of StdIn, Out, and Err
start the process
get StreamReaders and Writers for the Std's
After that, I want to initially check the StdOut and StdErr. I know that python writes the following piece of information to the StdErr
Python 2.7.11 (v2.7.11:6d1b6a68f775, Dec 5 2015, 20:32:19) [MSC
v.1500 32 bit (Intel)] on win32
and I am able to get this line by using errorReader.Peek() and reading character-based from the errorReader 2.
However, the situation with another process might be totally different. Even with Python, I run into the following problem: when I want to initially read from the outputReader, there is nothing contained in it and outputReader.Peek() seems to run into a deadlock. As mentioned above, the same holds for outputReader.EndOfStream or outputReader.ReadToEnd(). So how do I know if the stdOut can be used at all without causing a deadlock?
Code:
// create the python process StartupInfo object
ProcessStartInfo _tempProcessStartInfo = new ProcessStartInfo(#"C:\TMP\Python27\python.exe");
// ProcessStartInfo _tempProcessStartInfo = new ProcessStartInfo(PathToPython + "python.exe");
// python uses "-i" to run in interactive mode
_tempProcessStartInfo.Arguments = "-i";
// Only start the python process, but don't show a (console) window
_tempProcessStartInfo.WindowStyle = ProcessWindowStyle.Minimized;
_tempProcessStartInfo.CreateNoWindow = true;
// Enable the redirection of python process std's
_tempProcessStartInfo.UseShellExecute = false;
_tempProcessStartInfo.RedirectStandardOutput = true;
_tempProcessStartInfo.RedirectStandardInput = true;
_tempProcessStartInfo.RedirectStandardError = true;
// Create the python process object and apply the startupInfos from above
Process _tempProcess = new Process();
_tempProcess.StartInfo = _tempProcessStartInfo;
// Start the process
bool _hasStarted = _tempProcess.Start();
//// ASynch reading seems not appropriate to me:
// _tempProcess.BeginOutputReadLine();
// _tempProcess.BeginErrorReadLine();
// Create StreamReaders and Writers for the Std's
StreamReader outputReader = _tempProcess.StandardOutput;
StreamReader errorReader = _tempProcess.StandardError;
StreamWriter commandWriter = _tempProcess.StandardInput;
// Create StringBuilder that collects results and ErrorMessages
StringBuilder tmp = new StringBuilder("");
// Create temp variable that is used to peek into streams. C# uses -1 to indicate that there is no more byte to read
int currentPeek = -1;
// Get Initial Error Message. In this specific case, this is the python version
tmp.AppendLine("INITIAL ERROR MESSAGE:");
currentPeek = errorReader.Peek();
while (currentPeek >= 0)
{
char text = (char)errorReader.Read();
tmp.Append(text);
currentPeek = errorReader.Peek();
}
// Get initial output Message. In this specific case, this is EMPTY, which seems to cause this problem, as ...
tmp.AppendLine("INITIAL STDOUT MESSAGE:");
//// ... the following command CREATES a well defined output, and afterwards everything works fine (?) but ...
//commandWriter.WriteLine(#"print 'Hello World'");
//// ... without the the above command, neither
//bool isEndOfStream = outputReader.EndOfStream;
//// ... nor
// currentPeek = outputReader.Peek();
//// ... nor
// tmp.AppendLine(outputReader.ReadLine());
//// ... nor
//tmp.AppendLine(outputReader.ReadToEnd());
//// ... works
// Therefore, the following command creates a deadlock
currentPeek = outputReader.Peek();
while (currentPeek >= 0)
{
char text = (char)outputReader.Read();
tmp.Append(text);
currentPeek = errorReader.Peek();
}
_currentPythonProcess = _tempProcess;
return true;
1 An easy fix to this very specific problem is to send a valid command to the process first, for example simply "4", which returns a "4" as well... However, I want to understand how process streams, pipes and the corresponing readers and writers work and how I can use them in C#. Who knows what future brings, maybe I run into buffer problems when pythons response is 2^n+1 bytes long...
2 I know that I can also read line-based. However, the Peek() prevents me from reporting problems that are related to truncated lines.
If you can wait for the process to end and then read the buffers, you might be able to use Process.WaitForExit. There is also another method you can check, Process.WaitForInputIdle, but it depends on the process having a message loop, which I don't think a Python script gets when executing.
Normally shell automation InvokeVerb only accepts a string command, based on the documentation here: https://msdn.microsoft.com/en-us/library/ms723189(v=vs.85).aspx
However I noticed a cmd line EXE utility that accepts a number instead
http://www.technosys.net/products/utils/pintotaskbar
This is interesting because in Windows 10 "&Pin to taskbar" is removed from shell automation, despite being visible in the UI. ( https://connect.microsoft.com/PowerShell/feedback/details/1609288/pin-to-taskbar-no-longer-working-in-windows-10 )
However with this EXE passing the "number" 5386 as the command works.
I am interested to know what methods can be used to achieve this in either PowerShell/VBScript/.NET language or Win32 C/C++.
I understand this is almost certainly unsupported and likely to break any time as these numbers have changed between OS releases.
An example using the string version in PowerShell
$filepath = "C:\windows\system32\notepad.exe"
$path = Split-Path $FilePath
$shell= New-Object -com "Shell.Application"
$folder=$shell.Namespace($path)
$item = $folder.Parsename((split-path $FilePath -leaf))
$item.InvokeVerb("&Pin to taskbar")
The underlying shell context menu extension system actually works with numeric IDs natively; string verbs are an optional layer on top. However as you observe, command IDs are internal to a context menu extension and are not guaranteed to stay the same between versions (or even between invocations) - the whole point of verbs is that they allow commands to be invoked programatically irrespective of their ID.
In C/C++ you can invoke a command by its ID by casting the numeric ID as a string pointer (the rationale being that since all valid string pointers are higher than 65535, it's "safe" to pass a number as a string pointer as long as it fits into 16 bits, since the receiving API is able to correctly determine whether it is a "real string" or not).
Note that this is different from printing the number as a string, which is not correct. The MAKEINTRESOURCE macro exists for exactly this purpose.
E.g., a command with ID 1234 would be invoked using MAKEINTRESOURCE(1234) as the verb.
I ran into the same problem these days, in a Windows Script Host based script.
Invoking the verb with the localized string didn't work as it used to.
However you can still get the item's "Verbs" collection and then enumerate them until you get the verb you want to invoke. Then you can invoke it using it's "DoIt" method, without the need to pass the verbs name in.
In a WSH JScript it looks like this:
/* workaround for 'item.InvokeVerb("&Delete")' which no longer works */
var verbs = item.Verbs(); // item == return of folder.ParseName(..)
for (var x = 0; x < verbs.Count; x++)
{
var verb = verbs.Item(x);
if (verb.Name == "&Delete") //replace with "&Pin to taskbar"
{
try
{
verb.DoIt();
WSH.Echo ("Deleted:", item.Name);
}
catch(e)
{
WSH.Echo ("Error Deleting:", item.Name + "\r\n\t", e.description, e.number);
}
break;
}
}
}
Likely it also works somehow in PowerShell
ffr:
https://learn.microsoft.com/en-us/windows/win32/shell/folderitem-verbs
https://learn.microsoft.com/en-us/windows/win32/shell/folderitemverb
I am sending email body to the console application as parameter, the thing is I only see first 4 characters <div on the console application what happens to the other part? Can I send html email text as parameter to console application? Also is there any way to return a string[] array from console app?
My so far code below:
TestingConsoleApp to check the send n receive:
static void Main(string[] args)
{
string emailBody = System.IO.File.ReadAllText(#"C:\Users\ehsankayani\Desktop\email1Html.txt");
CallProcess(emailBody);
}
static void CallProcess(string body)
{
string path = #"F:\Scrappers\emailParser_app\emailParser_app\bin\Debug\emailParser_app.exe";
Process.Start(path, body);
}
Main console app:
static void Main(string[] args)
{
Console.WriteLine("EMAIL BODY = ");
string[] dataToReturn = new string[8];
//string emailBody = System.IO.File.ReadAllText(#"C:\Users\ehsankayani\Desktop\email1Html.txt");
string emailBody = args[0];
Console.WriteLine(emailBody);
Console.WriteLine(emailBody.Length);
Console.ReadLine();
var doc = new HtmlAgilityPack.HtmlDocument();
doc.LoadHtml(emailBody);
}
Any suggestions?
You are passing body as the parameter set in CallProcess. So if body were eg
xxxx yyyy ....
then args[0] would be just xxxx. You'll need to put "" around the text and escape in "'s in the text too.
A far better solution though would be to set up your Process to redirect stdin and to write the body to the process' stdin. This will avoid issues with whitespace and quotes. Take a look at http://msdn.microsoft.com/en-us/library/system.diagnostics.processstartinfo.redirectstandardinput.aspx for details on doing this.
A simply HttpUtility.EncodeHtml(Emailbody) while sending and HttpUtility.DecodeHtml(emailBody) on receiving does the job.
Since you ask for suggestions: For complex and/or large data, I'd rather pass only the information where to find the data to the process being called. Same can be done for returned data
This has several positive effects, the more complex and the larger these data are: You don't have to make sure the data are matching the command line restrictions, you don't have to allocate lots of memory to encode/decode the data in both processes, and you don't need to implement the logic to do that. Instead, write the data into a file, a shared memory region (MMF) or the like and pass that address.
Nearly the only downside is that you have to think about who's responsible for cleaning up.
Another possible approach could involve interprocess communication, but I think that's a bit overkill here.
I am currently trying to develop a program which takes the output of an existing program (written in C) and uses it as input (in C#). The problem I am having is that the existing program prints data in redundant format but it dynamically changes. An example could be a random name generator and I need to make a program that logs all of the random names as they appear.
Could I just pipe this and the output will be grabbed as it comes? The C program is run from a CLI.
You could redirect the output streams from the Process object to get direct access to it. You make a call and it will invoke an event of your choice when output is received. One of my questions may provide some explanation on how to do this - C# Shell - IO redirection:
processObject.StartInfo.RedirectStandardOutput = true;
processObject.OutputDataReceived += new DataReceivedEventHandler(processObject_OutputDataReceived);
/* ... */
processObject.Start();
processObject.BeginOutputReadLine();
And then later:
public void processObject_OutputDataReceived(object sender, DataReceivedEventArgs e) {
ProcessNewData(e.Data);
}
Note that this includes the trailing newline.
Or you can just pipe it and use Console.ReadLine to read it in. On the command line, you would execute:
cprogram | csharp_program
Unfortunately, you can't do this directly with the Process object - just use the method above. If you do choose to go this route, you can use:
string input = "";
int currentChar = 0;
while ( (currentChar = Console.Read()) > -1 ) {
input += Convert.ToChar(currentChar);
}
Which will read from the input until EOF.
If the C# program is also command-line based, running this command from the command-line will chain them together:
my_c_program.exe | my_csharp_program.exe
The C# program will receive the output of the C program, via its standard input stream (Console.Read, etc.)