my curl process is downloading, then overwriting the files - c#

I'm using Process to start a curl binary and download some urls on windows. I'm finding that one url (poss more) is downloaded and written to disk, then instantly overwritten with a filesize of 0, then written again. Why is this? Is it linked to my use of -O three times? (I'm grabbing three urls) It's the only way I've found to write the files to disk as remote-name-all doesn't appear to do anything. I'm using a curl binary version 7.53.1, which according to the docs will have remote-name-all
My Process start Info:
_p.StartInfo = new ProcessStartInfo
{
FileName = _config.PathToCurlBinary,
Arguments = $"{urls} {args}", // urls are space separated
CreateNoWindow = true,
UseShellExecute = false, // should this be true?,
WorkingDirectory = _config.OutputDirectory
};
and my curl arguments:
var args = $"--silent --compressed --connect-timeout 100 --user-agent \"{_config.UserAgent}\" --location" +
$" -O -O -O --retry 25 --retry-connrefused --stderr error.log" +
$" --cookie-jar \"{cookieFileName}\"";

Related

How can I programmatically turn off or on 'Windows Features'

When I try to update Windows features; When I update UseShellExecute to "true"; "The Process object must have the UseShellExecute property set to false in order to redirect IO streams." I get an error. When I set it to False; Unable to update. How can I do it ? Do you have any other suggestions?
static void InstallIISSetupFeature()
{
var featureNames = new List<string>() {
"IIS-WebServerRole",
"IIS-WebServer",
"IIS-CommonHttpFeatures",
"IIS-HttpErrors",
"IIS-HttpRedirect",
"IIS-ApplicationDevelopment",
"IIS-Security",
"IIS-RequestFiltering",
"IIS-NetFxExtensibility",
"IIS-NetFxExtensibility45",
"IIS-HealthAndDiagnostics",
"IIS-HttpLogging",
"IIS-LoggingLibraries",
"IIS-RequestMonitor",
"IIS-HttpTracing",
"IIS-URLAuthorization",
"IIS-IPSecurity",
"IIS-Performance",
"IIS-HttpCompressionDynamic",
"IIS-WebServerManagementTools",
"IIS-ManagementScriptingTools",
"IIS-IIS6ManagementCompatibility",
"IIS-Metabase",
"IIS-HostableWebCore","IIS-StaticContent",
"IIS-DefaultDocument",
"IIS-DirectoryBrowsing",
"IIS-WebDAV",
"IIS-WebSockets",
"IIS-ApplicationInit",
"IIS-ASPNET",
"IIS-ASPNET45",
"IIS-ASP",
"IIS-CGI",
"IIS-ISAPIExtensions",
"IIS-ISAPIFilter",
"IIS-ServerSideIncludes",
"IIS-CustomLogging",
"IIS-BasicAuthentication",
"IIS-HttpCompressionStatic",
"IIS-ManagementConsole",
"IIS-ManagementService",
"IIS-WMICompatibility",
"IIS-LegacyScripts",
"IIS-LegacySnapIn",
"IIS-FTPServer",
"IIS-FTPSvc",
"IIS-FTPExtensibility",
"IIS-CertProvider",
"IIS-WindowsAuthentication",
"IIS-DigestAuthentication",
"IIS-ClientCertificateMappingAuthentication",
"IIS-IISCertificateMappingAuthentication",
"IIS-ODBCLogging",
"NetFx4-AdvSrvs",
"NetFx4Extended-ASPNET45",
"NetFx3",
"WAS-WindowsActivationService",
"WCF-HTTP-Activation",
"WCF-HTTP-Activation45",
"WCF-MSMQ-Activation45",
"WCF-NonHTTP-Activation",
"WCF-Pipe-Activation45",
"WCF-TCP-Activation45",
"WCF-TCP-PortSharing45",
"WCF-Services45",
};
ManagementObjectSearcher obj = new ManagementObjectSearcher("select * from Win32_OperatingSystem");
foreach (ManagementObject wmi in obj.Get())
{
string Name = wmi.GetPropertyValue("Caption").ToString();
Name = Regex.Replace(Name.ToString(), "[^A-Za-z0-9 ]", "");
if (Name.Contains("Server 2008 R2") || Name.Contains("Windows 7"))
{
featureNames.Add("IIS-ASPNET");
featureNames.Add("IIS-NetFxExtensibility");
featureNames.Add("WCF-HTTP-Activation");
featureNames.Add("WCF-MSMQ-Activation");
featureNames.Add("WCF-Pipe-Activation");
featureNames.Add("WCF-TCP-Activation");
featureNames.Add("WCF-TCP-Activation");
}
string Version = (string)wmi["Version"];
string Architecture = (string)wmi["OSArchitecture"];
}
foreach (var featureName in featureNames)
{
Run(string.Format("dism/online/Enable-Feature:{0}", featureName));
}
}
static void Run(string arguments)
{
try
{
string systemPath = Path.Combine(Environment.ExpandEnvironmentVariables("%windir%"), "system32");
var dism = new Process();
dism.StartInfo.WorkingDirectory = systemPath;
dism.StartInfo.Arguments = arguments;
dism.StartInfo.FileName = "dism.exe";
dism.StartInfo.Verb = "runas";
dism.StartInfo.UseShellExecute = true;
dism.StartInfo.RedirectStandardOutput = true;
dism.Start();
var result = dism.StandardOutput.ReadToEnd();
dism.WaitForExit();
}
catch (Exception ex)
{
}
}`
I tried to update the feature with dism.exe and cmd.exe, when it gave an authorization error, I used the Verb property
`
Since the use of .Verb = "RunAs" requires .UseShellExecute = true, and since the latter cannot be combined with RedirectStandardOutput = true, you cannot directly capture the elevated process' output in memory.
It seems that the system itself, by security-minded design, prevents a non-elevated process from directly capturing an elevated process' output.
The workaround is to launch the target executable (dism.exe, in your case) indirectly, via a shell, and then use the latter's redirection feature (>) to capture the target executable's output (invariably) in a file, as shown below.
string systemPath = Path.Combine(Environment.ExpandEnvironmentVariables("%windir%"), "system32");
// Create a temp. file to capture the elevated process' output in.
string tempOutFile = Path.GetTempFileName();
var dism = new Process();
dism.StartInfo.WorkingDirectory = systemPath;
// Use cmd.exe as the executable, and pass it a command line via /c
dism.StartInfo.FileName = "cmd.exe" ;
// Use a ">" redirection to capture the elevated process' output.
// Use "2> ..." to also capture *stderr* output.
// Append "2>&1" to capture *both* stdout and stderr in the file targeted with ">"
dism.StartInfo.Arguments =
String.Format(
"/c {0} {1} > \"{2}\"",
"dism.exe", arguments, tempOutFile
);
dism.StartInfo.Verb = "RunAs";
dism.StartInfo.UseShellExecute = true;
dism.Start();
dism.WaitForExit();
// Read the temp. file in which the output was captured...
var result = File.ReadAllText(tempOutFile);
// ... and delete it.
File.Delete(tempOutFile);
First, you can use WindowsPrincipal::IsInRole() to check if you're running elevated.
See Microsoft Learn for details.
Second, this may be one of those cases where using native PS is easier than the cmdlet approach (admittedly, still not great).
If the script is supposed to run on clients as well as server operating systems: use Get-WmiObject or Get-CimInstance to get a reference to what you're running on. ActiveDirectory also has that information (in operatingSystem attribute).
For servers use Get-WindowsFeature in ServerManager module.
For clients use Get-WindowsOptionalFeature with switch -Online in DISM module which, if you indeed need to support OSes older than 6.3.xxxx, can be copied over from a machine that has it and added to $Env:Path before C:\Windows and C:\Windows\System32.
For either platform just pass the list of features to configure.
If in a (binary) cmdlet you have to call external tools then the advantage of them is mostly gone. It may be possible to access Windows CBS using a managed API to avoid this but even then the script based approach gets more results faster, especially since you can just just put together a quick wrapper around dism.exe .

Process.WaitForExitAsync stucks infinitely while calling ffmpeg on Windows

I am working with ffmpeg via C#'s Process class.
I have a script that runs ffmpeg to generate thumbnails from video. Initially it was called manually from command line - .\ffmpeg.exe -i .\input.mp4 -ss 00:00:01.000 -vframes:v 1 output.png, it starts ffmpeg instance, outputs some warnings/errors during the execution:
[image2 # 000001e51095ec80] The specified filename 'output.png' does not contain an image sequence pattern or a pattern is invalid.
[image2 # 000001e51095ec80] Use a pattern such as %03d for an image sequence or use the -update option (with -frames:v 1 if needed) to write a single image.
frame= 1 fps=0.0 q=-0.0 Lsize=N/A time=00:00:00.00 bitrate=N/A speed= 0x
video:73kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown
but anyway exits the process and correctly generates thumbnail image at output.png.
I want to execute it from C#.
Let's see the code:
var ffmpegProcess = new Process
{
StartInfo = new ProcessStartInfo
{
FileName = _config.FfmpegExecutablePath,
Arguments = CreateArgumentsForFfmpegProcessToRun(videoTempFilePath, thumbnailTempFilePath),
RedirectStandardError = true,
RedirectStandardInput = true,
RedirectStandardOutput = true,
UseShellExecute = false,
CreateNoWindow = true
},
EnableRaisingEvents = true
};
ffmpegProcess.Start();
await ffmpegProcess.WaitForExitAsync();
Method CreateArgumentsForFfmpegProcessToRun returns exactly the same arguments as in the script above -i .\input.mp4 -ss 00:00:01.000 -vframes:v 1 output.png.
However, when I run this code, it stucks/blocked at line ffmpegProcess.WaitForExitAsync() infinitely and no output written to output path.
If I omit WaitForExitAsync call and just go to the next line, then it doesn't stuck and writes the output as expected and finish the process with -1 exit code.
I am trying to figure out why block/stuck happens and what is the best way to resolve this situation? As far I know, WaitForExitAsync should return as process ends, no matter how process ends - 0 or another exit code, am I right?
Update #1:
Community advised to search if somewhere up the stack I am blocking my code. I wrote xunit-test and it still stucks.
[Theory]
[InlineData("assets/input.mp4", "assets/ffmpeg.exe")]
public async Task CreateThumbnailFromVideo(string videoFilePath, string ffmpegExePath)
{
var config = new VideoThumbnailServiceConfig
{
FfmpegExecutablePath = ffmpegExePath,
ThumbnailImageExtension = ".png"
};
var sut = new VideoThumbnailService(config);
using var fileStream = File.OpenRead(videoFilePath);
await sut.CreateThumbnailFromVideo(fileStream);
}
Inside sut.CreateThumbnailFromVideom I call process start method and awaits WaitForExitAsync().

Running console command from inside MVC controller not getting output

We have an MVC web app that allows downloading dynamically generated PDF reports. I am trying to allow viewing the report in the browser, and because of browser compatibility issues, we can't use a JS PDF viewer, so am working on a controller action that generated the PDF using existing code, then converts it to HTML using a third party program and returns the HTML.
The third party program, pdf2htmlEX, is used via a command line interface, but when I try to invoke the program to convert the PDF to HTML nothing happens. I do not receive an error, but no HTML file is generated.
I first tried just a single line to start the conversion Process.Start("commands here"), but when that didn't work I tried a more advanced way to start the process and allow capturing the StdOut found on this answer: How To: Execute command line in C#, get STD OUT results, but I don't seem to be getting any output either. I am not familiar with invoking command line programs using c#, so I am not sure if I am making a simple mistake. My current controller action looks like this:
public ActionResult GetReportPdfAsHtml(int userId, ReportType reportType, int page = 1)
{
// get pdf
var pdfService = new PdfServiceClient();
var getPdfResponse = pdfService.GetPdfForUser(new GetPdfForUserRequest {
UserId = userId,
ReportType = reportType,
BaseUri = Request.Url.Host
});
pdfService.Close();
// save pdf to temp location
var folderRoot = Server.MapPath("~");
var location = Path.Combine(folderRoot, "pdfTemp");
var outputDir = $"{location}\\output";
var fileName = $"{userId}_{reportType}";
Directory.CreateDirectory(outputDir);
var file = $"{location}\\{fileName}.pdf";
//IOFile is alias of system.IO.File to avoid collision with the 'File' Method already on the controller
IOFile.WriteAllBytes(file, getPdfResponse.Pdf);
//********************************************************************
//***** Works fine up to here, PDF is successfully generated and saved
//********************************************************************
// Convert pdf above to html
var arguments = $"{file} --dest-dir {outputDir} -f {page} -l {page}";
// Start the child process.
var p = new Process {
StartInfo = {
UseShellExecute = false,
RedirectStandardOutput = true,
FileName = Server.MapPath("~\\pdf2htmlEX.exe"),
Arguments = arguments
}
};
p.Start();
// Read the output stream first and then wait.
var output = p.StandardOutput.ReadToEnd();
p.WaitForExit();
// Function continues and returns fine, but MVC then errors because the
// file isn't created so the path below doesn't exist
return File($"{outputDir}\\{fileName}.html", "text/html");
}
Update: I have tried running the command in a cmd console and it works fine. However when I try and run it via the Process.Start() method, i get following output from Pdf2htmlEX:
>temporary dir: [Redacted]\pdfTemp\temp/pdf2htmlEX-a46244
>Preprocessing: 0/1
>Preprocessing: 1/1
>Add new temporary file: [Redacted]\pdfTemp\temp/pdf2htmlEX-a46244/__css
>Add new temporary file: [Redacted]\pdfTemp\temp/pdf2htmlEX-a46244/__outline
>Add new temporary file: [Redacted]\pdfTemp\temp/pdf2htmlEX-a46244/__pages
>Working: 0/1
>Install font 1: (14 0) SUBSET+LatoLightItalic
>Add new temporary file: [Redacted]\pdfTemp\temp/pdf2htmlEX-a46244/f1.ttf
>Embed font: [Redacted]\pdfTemp\temp/pdf2htmlEX-a46244/f1.ttf 1
>Add new temporary file: [Redacted]\pdfTemp\temp/pdf2htmlEX-a46244/__raw_font_1.ttf
>em size: 2000
>Add new temporary file: [Redacted]\pdfTemp\temp/pdf2htmlEX-a46244/f1.map
>Missing space width in font 1: set to 0.5
>space width: 0.5
>Add new temporary file: [Redacted]\pdfTemp\temp/pdf2htmlEX-a46244/__tmp_font1.ttf
>Add new temporary file: [Redacted]\pdfTemp\temp/pdf2htmlEX-a46244/__tmp_font2.ttf
>Internal Error: Attempt to output 2147483647 into a 16-bit field. It will be truncated and the file may not be useful.
>Internal Error: File Offset wrong for ttf table (name-data), -1 expected 150
>Save Failed
>Cannot save font to [Redacted]\pdfTemp\temp/pdf2htmlEX-a46244/__tmp_font1.ttf

Docusign Retrieve via ProcessBuilder

I'm attempting to build a java process to execute the docusign retrieve product via the command line. I've written the process to execute based on a given property file.
buildRoot = isWindowsOs() ? "C:" + "\\Program Files (x86)\\DocuSign, Inc\\Retrieve 3.2" : "\\Program Files (x86)\\DocuSign, Inc\\Retrieve 3.2" ;
String[] command = new String [2];
command[0] = "\""+buildRoot+ "\\" + docuSignAppName+"\"";
logger.info(command[0].toString());
//ADDED FOR EXPLANATION - "C:\Program Files (x86)\DocuSign, Inc\Retrieve 3.2\DocuSignRetrieve.exe"
command[1] = arguments;
logger.info(command[1].toString());
ProcessBuilder processBuilder = new ProcessBuilder(command);
logger.info("ProcessBuilder starting directory" +processBuilder.directory());
processBuilder.redirectErrorStream(true);
p = processBuilder.start();
InputStream is = p.getInputStream();
InputStreamReader isr = new InputStreamReader(is);
stdout = new BufferedReader(isr);
Once I pass in the built out string of parameters the executed code looks like the sample provided but always results in the error back to the screen "Missing "accountid" parameter".
The parameter list looks like the following.
/endpoint "Demo"
/userid "REMOVED"
/password "REMOVED"
/accountid "REMOVED"
/span "-1"
/spanfilter "Completed"
/statusfilter "Completed"
/fieldlog "LIST OF FIELDS"
/nstyle "EnvelopeID"
/save "MergedPdfWithoutCert"
/dir "D:\DocuSignStore"
/includeheaders "true"
Any help or assistance would be appreciated.
The solution was found in a StackOverflow discussion regarding common issues with the ProcessBuilder.
My problem was that I expected by changing the putting in the full path, that I could run the executable. For the reason I'm not sure right now, that wasn't working as expected. The solution was to run the CMD command which exists on the %PATH% on any windows OS.
String[] command = new String [2];
command[0] = "\""+buildRoot+ "\\" + docuSignAppName+"\"";
logger.info(command[0].toString());
//ADDED FOR EXPLANATION - "C:\Program Files (x86)\DocuSign, Inc\Retrieve 3.2\DocuSignRetrieve.exe"
command[1] = arguments;
logger.info(command[1].toString());
//This starts a new command prompt
ProcessBuilder processBuilder = new ProcessBuilder("cmd","/c","DocusignRetreive.exe);
//This sets the directory to run the command prompt from
File newLoc = new File("C:/Program Files (x86)/DocuSign, Inc/Retrieve 3.2");
processBuilder.directory(newLoc);
logger.info("ProcessBuilder starting directory" +processBuilder.directory());
processBuilder.redirectErrorStream(true);
/*When the process builder starts the prompt looks like
*C:\Program Files (x86)\DocuSign, Inc\Retrieve 3.2
*Now DocusignRetrieve.exe is an executable in the directory to be run
*/
p = processBuilder.start();

Getting a poster frame(thumbnail) with ffmpeg

I am trying to get a poster frame from a video file, using ffmpeg.
I have been following this tutorial and come up with the following code(which is taken/adapted from the link I gave):
public bool GetVideoThumbnail(string path, string saveThumbnailTo, int seconds)
{
string parameters = string.Format("-i {0} {1} -vcodec mjpeg -ss {2} -vframes 1 -an -f rawvideo", path, saveThumbnailTo, seconds);
if (File.Exists(saveThumbnailTo))
{
return true;
}
else
{
using (Process process = Process.Start(pathToConvertor, parameters))
{
process.WaitForExit();
}
return File.Exists(saveThumbnailTo);
}
}
At the moment this code is successfully creating a file in the correct destination (saveThumbnailTo) only the picture is completely black. I have tried changing the seconds value in the code to ensure that I am not just getting a blank picture from the start of the video. The path refers to where my video is stored, by the way.
I am currently calling the above code like so:
GetVideoThumbnail(videoPath, folderPath + "/poster.jpg", 100)
..and then passing it out to my view to display the picture. I just wonder whether ".jpg" is the extension I should be giving to this file as I am not entirely sure?
Edit: When I run the same command from the command line I get the following errors:
Incompatible pixel format 'yuv420p' for codec 'mjpeg', auto-selecting
format 'yuvj420p'
which appears in yellow, and
[image2 # 02S96AE0] Could not get frame filename number 2 from pattern
'poster.jpg' an_interleaved_write_frame(): Invalid argument
which appears in red.
Could anyone help me with getting this working properly as I am completely unfamiliar with the ffmpeg command line and not sure what I am doing wrong. I have tried removing the vcodec parameter and get the same error message.
Try this:
public bool GetVideoThumbnail(string path, string saveThumbnailTo, int seconds)
{
string parameters = string.Format("-ss {0} -i {1} -f image2 -vframes 1 -y {2}", seconds, path, saveThumbnailTo);
var processInfo = new ProcessStartInfo();
processInfo.FileName = pathToConvertor;
processInfo.Arguments = parameters;
processInfo.CreateNoWindow = true;
processInfo.UseShellExecute = false;
File.Delete(saveThumbnailTo);
using(var process = new Process())
{
process.StartInfo = processInfo;
process.Start();
process.WaitForExit();
}
return File.Exists(saveThumbnailTo);
}
Short explanation:
f image2 : output is image
vframes 1 : take one frame from the input
y : overwrite output file
"processInfo.CreateNoWindow = true" : do not show the ffmpeg window
Try several times with different values for the "seconds" parameter.
Also, make sure the "pathToConvertor" is correct.
This worked for me, with recent build of ffmpeg.exe on a Windows machine.
Let me know how it goes.

Categories