We Keep Coding

File Enumeration performance between .net core and powershell

I currently need to scan a folder within a c# application for files that match a pattern such as RecordId*.*
The folder currently has several hundred thousand files in it and we cannot change that, at least at the moment.
Doing this:
var folderPath = #"\\server\folder\";
var options = new EnumerationOptions()
{
BufferSize = 64000,
RecurseSubdirectories = false,
MatchType = MatchType.Simple,
MatchCasing = MatchCasing.CaseInsensitive
};
var searchPattern = $"{request.RecordId}*.*";
var isMediaFileInFolder = _fileSystem.Directory
.EnumerateFiles(folderPath, searchPattern, options);
Would take about 10 minutes to return the list of found files.
However, we can run this within PowerShell ISE and it runs in less than a second.
$Directory = "\\server\folder\"
$Filter = "REC1234*.*"
Get-ChildItem -Path $Directory -Filter $Filter
I tried running the powershell within c# through something like this:
public static List<string> PSFiles(string path, string pattern)
{
try
{
ICollection<PSObject> collection = null;
List<string> returnStrings = new List<string>();
script = "Get-ChildItem -Path " + path + " -Filter" + pattern;
using (PowerShell powerShell = PowerShell.Create())
{
powerShell.AddScript(script);
collection = powerShell.Invoke();
}
foreach (PSObject obj in collection)
{
if (!String.IsNullOrEmpty(obj.ToString()))
{
foreach(var i in Regex.Split(obj.ImmediateBaseObject.ToString().Trim(), "\r\n"))
{
returnStrings.Add(i.Trim());
}
}
}
return returnStrings;
}
catch (Exception e)
{
Console.WriteLine(e);
throw;
}
}
And it took longer to run than the enumerate files attempt with nearly all of the time spent on the powerShell.Invoice(); step.
After testing in Powershell Core, that is also taking over 10 minutes. Further, using the Directory.EnumerateFiles() on .NET took less than a second.
So the question is, is there a way to mirror the performance of file enumeration within .NET when running on .NET Core?

Parse log files for relevant data from multiple lines

C#, Winforms:
I have a log file I need to parse. This file contains transactions requests from a program, but the program writes the transaction across multiple lines.
I need to get the ID# and if the request was processed or denied for whatever reason. The problem is that these requests are on multiple lines. My only saving grace is that they contain the same time stamp from the logger. The (##) is not usable since it is a temporary placeholder, thus (19) may repeat multiple times throughout the log.
I was thinking of scanning for a PR_Request, substringing the ID# and the time stamp, but I dont know how to make a streamreader move down to the next 4 lines and write it out to be one single line in a file.
Examples:
06/10/16 08:09:33.031 (1) PR_Request: IID=caa23b14,
06/10/16 08:09:33.031 (1) PR_Mon: IID=caa23b14,
06/10/16 08:09:33.031 (1) RESUME|BEGIN
06/10/16 08:09:33.031 (1) RESUME_TRIG|SC-TI
06/10/16 08:19:04.384 (19) PR_Request: IID=90dg01b,
06/10/16 08:19:04.384 (19) PR_Mon: IID=90dg01b,
06/10/16 08:19:04.384 (19) RESUME|DENIED: Access not granted.
I need output to be in a single line for a file. That way, I can just parse it with another program and feed the data into a database.
06/10/16 08:09:33.031 PR_Request: IID=caa23b14 | RESUME | BEGIN | RESUME_TRIG | SC-TI
06/10/16 08:19:04.384 PR_Request: IID=90dg01b | RESUME | DENIED: Access not granted.
EDIT:
Okay I think I have a base code here. It works, kind of. It takes such a long time because I had to open another file streamer when it found a match to PR_Request, then scan the file again with the same fullstamp (date + process number). It will then look for RESUME|BEGIN or RESUME|DENIED and then write out that it succeeded or failed.
Is there any way to perhaps speed this up by getting the streamreader line where it originally found the PR_Request, have it start on another line, count maybe to 5 more lines, then stop it? This would help speed up the program considerably.
string inputfolder = inputloctxt.Text;
string outputfolder = saveloctxt.Text;
string outputfile = #"ParsedFile.txt";
try
{
string[] readfromdir = Directory.GetFiles(outputfolder);
foreach (string readnow in readfromdir)
{
using (StreamReader fileread = new StreamReader(readnow))
{
string fileisreading;
while ((fileisreading = fileread.ReadLine()) != null)
{
if (fileisreading.Contains("PR_Request"))
{
string resumed = null;
string fullstamp = fileisreading.Substring(1, 26);
string datestamp = fileisreading.Substring(1, 21);
string requesttype = fileisreading.Substring(27, 22);
string iidnum = fileisreading.Substring(53, 8);
using (StreamReader grabnext01 = new StreamReader(readnow))
{
string grabnow01;
while ((grabnow01 = grabnext01.ReadLine()) != null)
{
if (grabnow01.Contains(fullstamp))
{
if (grabnow01.Contains("RESUME|BEGIN"))
{
resumed = "TRUE";
break;
}
else if (grabnow01.Contains("RESUME|DENIED"))
{
resumed = "FALSE";
break;
}
}
}
}
File.AppendAllText(outputfolder + outputfile,
datestamp + " " + requesttype + " " + iidnum + " " + resumed + Environment.NewLine);
resumed = null;
}
}
}
}
}

This sounds like you need to use Regular Expressions. There is a namespace System.Text.RegularExpressions you can use and reference the capture groups that I made for you in the example.
Use these sites for reference:
https://regex101.com/
https://msdn.microsoft.com/en-us/library/bs2twtah(v=vs.110).aspx
I started off the Regex for you, it is not pretty but it should get the job done.
(?:\d{2}\/\d{2}\/\d{2}\s\d{2}\:\d{2}\:\d{2}\.\d{3}\s\(\d+\)\s)(PR_Request: IID=[^,\n]+)(?:\,\n\d{2}\/\d{2}\/\d{2}\s\d{2}\:\d{2}\:\d{2}\.\d{3}\s\(\d+\)\sPR_Mon: IID=[^,\n]*\,\n\d{2}\/\d{2}\/\d{2}\s\d{2}\:\d{2}\:\d{2}\.\d{3}\s\(\d+\)\s)((RESUME|BEGIN|\||DENIED: Access not granted.)*)(?:\n\d{2}\/\d{2}\/\d{2}\s\d{2}\:\d{2}\:\d{2}\.\d{3}\s\(\d+\)\s)*((RESUME_TRIG|SC\-TI|\|)*)

FFMpegConverter. ConvertLiveMedia method doesn't work

I need to convert files from AWS cloud, which mounted like a local drive by using TntDrive.
I tried to use FFMpegConverter.ConvertMedia method, but it works slow. In another question someone explained to me next:
"It looks like you need to use the FFMpegConverter.ConvertLiveMedia method instead of the FFMpegConverter.ConvertMedia method to achieve this. Overall you will still find that you will be probably I/O bound rather than CPU bound."
But next code didn't work for me:
static void Main(string[] args)
{
string input_path =
#"D:\WAV\ALBUM1\UNDER_ALBUM1\APOV01_10 POV 8_MAIN.WAV";
string dest_path = #"D:\result.mp3";
using (FileStream inputStream = new FileStream(input_path, FileMode.Open))
{
var converter = new FFMpegConverter();
var result = converter.ConvertLiveMedia(
inputStream, "WAV",
dest_path, "MP3",
new ConvertSettings
{
AudioSampleRate = 44100,
CustomOutputArgs = " -b:a 192k "
}
);
Console.ReadLine();
}
}
What's wrong with my code ?

var ffMpeg = new FFMpegConverter();
ffMpeg.ConvertMedia(input_path, null, dest_path, "mp3", new ConvertSettings() {
CustomOutputArgs = " -b:a 192k -ar 44100 -y -f mp3 -movflags faststart"
});

You have to start the conversion I think...
result.Start();

Xamarin C# android and PARSE - Downloading a parseFile inside a parseObject

I previously made a post asking how to send a .3gpp audio file up to the parse cloud here:
Xamarin C# Android - converting .3gpp audio to bytes & sending to parseObject
I have managed to do this successfully, on parse's data manager, I can click the file's link and play the sound sent from my android device successfully.
Here's the code for uploading the data to the cloud:
async Task sendToCloud(string filename)
{
ParseClient.Initialize ("Censored Key", "Censored Key");
string LoadPath = System.Environment.GetFolderPath(System.Environment.SpecialFolder.ApplicationData);
string savetheFile = sName + ".3gpp";
string tempUserName;
LoadPath += savetheFile;
Console.WriteLine ("loadPath: " + LoadPath);
try
{
byte[] data = File.ReadAllBytes(LoadPath);
ParseFile file = new ParseFile(savetheFile, data);
await file.SaveAsync();
var auidoParseObject = new ParseObject("AudioWithData");
//Console.WriteLine(ParseUser.getUserName());
if (ParseUser.CurrentUser != null)
{
tempUserName = ParseUser.CurrentUser.Username.ToString();
}
else{
tempUserName = "Anonymous";
}
//tempUserName = ParseUser.CurrentUser.Username.ToString();
Console.WriteLine("PARSE USERNAME: " + tempUserName);
auidoParseObject["userName"] = tempUserName;
auidoParseObject["userName"] = tempUserName;
auidoParseObject["file"] = file;
await auidoParseObject.SaveAsync();
}
catch (Exception e)
{
Console.WriteLine("Failed to await audio object! {0}" + e);
}
}
So as you can see, I'm sending a ParseObject called "AudioWithData".
This object contains two children:
-The username of the user who uploaded the file (string)
-The parseFile called "file" (which has the following two children)
---SaveTheFile (A string containing the name of the audio file, input by the user, with the .3gpp extension added on the end, for example "myAudioFile.3gpp"
---data (this contains the bytes of the audio file)
I need to be able to download the file onto my android device, and play it through a mediaplayer object.
I've checked over the documentation on the parse website, but I haven't managed to do this:
(excuse my pseudo querying syntax here)
SELECT (audio files) FROM (the parseObject) WHERE (the username = current user)
I then, eventually, want to place all of these files into a listview, and when the user clicks the file, it plays the audio.
I've tried the following but I don't really know what I'm doing with it...
async Task RetrieveSound(string filename)
{
ParseClient.Initialize ("Censored key", "Censored key");
Console.WriteLine ("Hit RetrieveSound, filename = " + filename);
string username;
var auidoParseObject = new ParseObject("AudioWithData");
if (ParseUser.CurrentUser != null) {
username = ParseUser.CurrentUser.Username.ToString ();
} else {
username = "Anonymous";
}
string cloudFileName;
Console.WriteLine ("username set to: " + username);
var HoldThefile = auidoParseObject.Get<ParseFile>("audio");
//fgher
var query = from audioParseObject in ParseObject.GetQuery("userName")
where audioParseObject.Get<String>("userName") == username
select file;
IEnumerable<ParseFile> results = await query.FindAsync();
Console.WriteLine ("passed the query");
//wfojh
byte[] data = await new HttpClient().GetByteArrayAsync(results.Url);
Console.WriteLine ("putting in player...");
_player.SetDataSourceAsync (data);
_player.Prepare;
_player.Start ();
}
Any help would be GREATLY APPRECIATED! Even a point in the right direction would be great!
Thanks!
EDIT--
I'm actually getting a query error on the following lines
(I can't post images because of my reputation - I lost access to my main stackOverflow account :/ )
Links to images here:
first error: http://i.stack.imgur.com/PZBJr.png
second error: http://i.stack.imgur.com/UkHvX.png
Any ideas? The parse documentation is vague about this.

this line will return a collection of results
IEnumerable<ParseFile> results = await query.FindAsync();
you either need to iterate through them with foreach, or just pick the first one
// for testing, just pick the first one
if (results.Count > 0) {
var result = results[0];
byte[] data = await new HttpClient().GetByteArrayAsync(result.Url);
File.WriteAllBytes(some_path_to_a_temp_file, data);
// at this point, you can just initialize your player with the audio file path and play as normal
}

C# Problem Reading Console Output to string

i want to launch ffmpeg from my app and retrive all console output that ffmpeg produces. Thing seems obvious, i followed many forum threads/articles like this one but i have problem, though i follow all information included there I seem to end up in dead end.
String that should contain output from ffmpeg is always empty. I've tried to see where is the problem so i made simple c# console application that only lists all execution parameters that are passed to ffmpeg, just to check if problem is caused by ffmpeg itself. In that case everything work as expected.
I also did preview console window of my app. When i launch ffmpeg i see all the output in console but the function that should recieve that output for further processing reports that string was empty. When my param-listing app is launched the only thing I see is the expected report from function that gets output.
So my question is what to do to get ffmpeg output as i intended at first place.
Thanks in advance
MTH

This is a long shot, but have you tried redirecting StandardError too?

Here is a part of my ffmpeg wrapper class, in particular showing how to collect the output and errors from ffmpeg.
I have put the Process in the GetVideoDuration() function just so you can see everything in the one place.
Setup:
My ffmpeg is on the desktop, ffPath is used to point to it.
namespace ChildTools.Tools
{
public class FFMpegWrapper
{
//path to ffmpeg (I HATE!!! MS special folders)
string ffPath = System.Environment.GetFolderPath(Environment.SpecialFolder.Desktop) + "\\ffmpeg.exe";
//outputLines receives each line of output, only if they are not zero length
List<string> outputLines = new List<string>();
//In GetVideoDuration I only want the one line of output and in text form.
//To get the whole output just remove the filter I use (my search for 'Duration') and either return the List<>
//Or joint the strings from List<> (you could have used StringBuilder, but I find a List<> handier.
public string GetVideoDuration(FileInfo fi)
{
outputLines.Clear();
//I only use the information flag in this function
string strCommand = string.Concat(" -i \"", fi.FullName, "\"");
//Point ffPath to my ffmpeg
string ffPath = System.Environment.GetFolderPath(Environment.SpecialFolder.Desktop) + "\\ffmpeg.exe";
Process processFfmpeg = new Process();
processFfmpeg.StartInfo.Arguments = strCommand;
processFfmpeg.StartInfo.FileName = ffPath;
//I have to say that I struggled for a while with the order that I setup the process.
//But this order below I know to work
processFfmpeg.StartInfo.UseShellExecute = false;
processFfmpeg.StartInfo.RedirectStandardOutput = true;
processFfmpeg.StartInfo.RedirectStandardError = true;
processFfmpeg.StartInfo.CreateNoWindow = true;
processFfmpeg.ErrorDataReceived += processFfmpeg_OutData;
processFfmpeg.OutputDataReceived += processFfmpeg_OutData;
processFfmpeg.EnableRaisingEvents = true;
processFfmpeg.Start();
processFfmpeg.BeginOutputReadLine();
processFfmpeg.BeginErrorReadLine();
processFfmpeg.WaitForExit();
//I filter the lines because I only want 'Duration' this time
string oStr = "";
foreach (string str in outputLines)
{
if (str.Contains("Duration"))
{
oStr = str;
}
}
//return a single string with the duration line
return oStr;
}
private void processFfmpeg_OutData(object sender, DataReceivedEventArgs e)
{
//The data we want is in e.Data, you must be careful of null strings
string strMessage = e.Data;
if outputLines != null && strMessage != null && strMessage.Length > 0)
{
outputLines.Add(string.Concat( strMessage,"\n"));
//Try a Console output here to see all of the output. Particularly
//useful when you are examining the packets and working out timeframes
//Console.WriteLine(strMessage);
}
}
}
}