C#, Winforms:
I have a log file I need to parse. This file contains transactions requests from a program, but the program writes the transaction across multiple lines.
I need to get the ID# and if the request was processed or denied for whatever reason. The problem is that these requests are on multiple lines. My only saving grace is that they contain the same time stamp from the logger. The (##) is not usable since it is a temporary placeholder, thus (19) may repeat multiple times throughout the log.
I was thinking of scanning for a PR_Request, substringing the ID# and the time stamp, but I dont know how to make a streamreader move down to the next 4 lines and write it out to be one single line in a file.
Examples:
06/10/16 08:09:33.031 (1) PR_Request: IID=caa23b14,
06/10/16 08:09:33.031 (1) PR_Mon: IID=caa23b14,
06/10/16 08:09:33.031 (1) RESUME|BEGIN
06/10/16 08:09:33.031 (1) RESUME_TRIG|SC-TI
06/10/16 08:19:04.384 (19) PR_Request: IID=90dg01b,
06/10/16 08:19:04.384 (19) PR_Mon: IID=90dg01b,
06/10/16 08:19:04.384 (19) RESUME|DENIED: Access not granted.
I need output to be in a single line for a file. That way, I can just parse it with another program and feed the data into a database.
06/10/16 08:09:33.031 PR_Request: IID=caa23b14 | RESUME | BEGIN | RESUME_TRIG | SC-TI
06/10/16 08:19:04.384 PR_Request: IID=90dg01b | RESUME | DENIED: Access not granted.
EDIT:
Okay I think I have a base code here. It works, kind of. It takes such a long time because I had to open another file streamer when it found a match to PR_Request, then scan the file again with the same fullstamp (date + process number). It will then look for RESUME|BEGIN or RESUME|DENIED and then write out that it succeeded or failed.
Is there any way to perhaps speed this up by getting the streamreader line where it originally found the PR_Request, have it start on another line, count maybe to 5 more lines, then stop it? This would help speed up the program considerably.
string inputfolder = inputloctxt.Text;
string outputfolder = saveloctxt.Text;
string outputfile = #"ParsedFile.txt";
try
{
string[] readfromdir = Directory.GetFiles(outputfolder);
foreach (string readnow in readfromdir)
{
using (StreamReader fileread = new StreamReader(readnow))
{
string fileisreading;
while ((fileisreading = fileread.ReadLine()) != null)
{
if (fileisreading.Contains("PR_Request"))
{
string resumed = null;
string fullstamp = fileisreading.Substring(1, 26);
string datestamp = fileisreading.Substring(1, 21);
string requesttype = fileisreading.Substring(27, 22);
string iidnum = fileisreading.Substring(53, 8);
using (StreamReader grabnext01 = new StreamReader(readnow))
{
string grabnow01;
while ((grabnow01 = grabnext01.ReadLine()) != null)
{
if (grabnow01.Contains(fullstamp))
{
if (grabnow01.Contains("RESUME|BEGIN"))
{
resumed = "TRUE";
break;
}
else if (grabnow01.Contains("RESUME|DENIED"))
{
resumed = "FALSE";
break;
}
}
}
}
File.AppendAllText(outputfolder + outputfile,
datestamp + " " + requesttype + " " + iidnum + " " + resumed + Environment.NewLine);
resumed = null;
}
}
}
}
}
This sounds like you need to use Regular Expressions. There is a namespace System.Text.RegularExpressions you can use and reference the capture groups that I made for you in the example.
Use these sites for reference:
https://regex101.com/
https://msdn.microsoft.com/en-us/library/bs2twtah(v=vs.110).aspx
I started off the Regex for you, it is not pretty but it should get the job done.
(?:\d{2}\/\d{2}\/\d{2}\s\d{2}\:\d{2}\:\d{2}\.\d{3}\s\(\d+\)\s)(PR_Request: IID=[^,\n]+)(?:\,\n\d{2}\/\d{2}\/\d{2}\s\d{2}\:\d{2}\:\d{2}\.\d{3}\s\(\d+\)\sPR_Mon: IID=[^,\n]*\,\n\d{2}\/\d{2}\/\d{2}\s\d{2}\:\d{2}\:\d{2}\.\d{3}\s\(\d+\)\s)((RESUME|BEGIN|\||DENIED: Access not granted.)*)(?:\n\d{2}\/\d{2}\/\d{2}\s\d{2}\:\d{2}\:\d{2}\.\d{3}\s\(\d+\)\s)*((RESUME_TRIG|SC\-TI|\|)*)
Related
I have been working on converting a GUI script from another language to C# in VS2017 for a customer. With help from the folks here I am 95% of the way there, but have run into a couple of snags; just not sure I am doing things in the best way. I'm including just the relevant portions of code below, please let me know if I am not providing enough:
The majority of the code is centered on the wpf form, which collects data for low level technicians to batch deploy a number of Virtual Machines into the VMware environment. This number could easily range into the dozens or even a hundred VMs at once. The information for each VM is specified in the form, then collected in a listview. Once the listview is fully populated it is exported to a csv. Up to this point everything works just fine.
I've next been working on actually launching the powershell/powerCLI script (also working) and capturing output. The log file is opened with a specific reader application the customer uses, which updates in real time, and the captured output is fed to the log. It is important for the technicians to see the output from the code line by line so they can react if there is an issue.
I started with something like this as a test:
string sPSScript = "C:\\Users\\Admin\\Desktop\\TestC#.ps1";
string logFile = "C:\\Users\\Admin\\Desktop\\My.log";
string logReader = "C:\\Users\\Admin\\Documents\\CMTrace.exe";
string standard_output;
System.Diagnostics.Process PSScript = new System.Diagnostics.Process();
PSScript.StartInfo.FileName =
Environment.GetFolderPath(Environment.SpecialFolder.SystemX86) +
"\\WindowsPowerShell\\v1.0\\powershell.exe";
PSScript.StartInfo.Arguments = "-command . '" + sPSScript + "' " +
vCenter.Text;
PSScript.StartInfo.RedirectStandardOutput = true;
PSScript.StartInfo.UseShellExecute = false;
PSScript.Start();
System.Diagnostics.Process LogFile = new System.Diagnostics.Process();
LogFile.StartInfo.FileName = logReader;
LogFile.StartInfo.Arguments = logFile;
LogFile.Start(); while ((standard_output =
PSScript.StandardOutput.ReadLine()) != null)
{
if (standard_output != "")
{
using (StreamWriter file = new StreamWriter(logFile, append: true))
{
file.WriteLine(standard_output);
}
}
}
While this writes to the log file in real time as expected, it creates 100 instances of the logReader application. I understand why, since I am declaring a new StreamWriter object through every pass, but am unsure how better to go about this.
I tried creating the file outside the loop, like this:
StreamWriter file = new StreamWriter(logFile, append: true) { };
System.Diagnostics.Process LogFile = new System.Diagnostics.Process();
LogFile.StartInfo.FileName = logReader;
LogFile.StartInfo.Arguments = logFile;
System.Diagnostics.Process PSScript = new System.Diagnostics.Process();
PSScript.StartInfo.FileName = Environment.GetFolderPath(Environment.SpecialFolder.SystemX86) + "\\WindowsPowerShell\\v1.0\\powershell.exe";
PSScript.StartInfo.Arguments = "-command . '" + sPSScript + "' " + vCenter.Text;
PSScript.StartInfo.RedirectStandardOutput = true;
PSScript.StartInfo.UseShellExecute = false;
LogFile.Start();
PSScript.Start();
System.Threading.Thread.Sleep(1500);
while ((standard_output = PSScript.StandardOutput.ReadLine()) != null)
{
if (standard_output != "")
{
file.WriteLine(standard_output);
}
}
It doesn't create multiple instances, but it also does not update the log file in real time as the previous code does. It only updates once the script runs, and then only partially. The script produces ~1000 lines of output, and I consistently see only about 840 written to the log file.
I thought about doing something like this:
FileStream logFS;
logFS = new FileStream(logFile, FileMode.Append);
but it appears the only options available to me to write to the file are expecting a byte array.
I am sure that I am missing something stupid simple in this, but would appreciate any suggestions on the easiest way to create the log file, open it in the reader, and then update it with the standard output from the powershell script.
why did the previous code writes in real time?
because you are wrapping it with using. And at the end of using block its gonna call dispose which calls .Flush to write to disk
Your second code block calls WriteLine but never called Flush so it writes to the disk whenever the buffer is full. Just add a .Flush call after WriteLine and you will have real time logging
hope this question makes sense: Is there a way I can download files from a drive without reiterating the service account every time? So for example, I have a program that allows me to backup my Gapps organization drives. The program currently works like this:
Logs in to each account with the service I created in the Google developer console
Checks the token as a source for changed files from the last time the backup ran to the current execution
IF the token is different, executes a file list request and records the fileId and actual fileName for the files changed, then puts both values into a temporary text document as 2 columns ("fileId,fileName"). Here is what I'm using for the file resource list
Console.WriteLine("Changes detected. Making notes while we go through these.");
if (File.Exists(savelocation + ".deltalog.tok"))
File.Delete(savelocation + ".deltalog.tok");
using (StreamWriter deltalog = new StreamWriter(savelocation + ".deltalog.tok", true))
{
while (pageToken != null)
{
counter1++;
var request = CreateService.BuildService(user).Changes.List(pageToken);
//StreamWriter deltalog = new StreamWriter(savelocation + ".deltalog.tok", true);
request.Fields = "*";
request.Spaces = "drive";
var changes = request.Execute();
foreach (var change in changes.Changes)
{
try
{
string updatedfile = change.File.Name;
//string updatedfile = CreateService.BuildService(user).Files.Get(change.FileId).Execute().Name;
// Record the changed file
Console.WriteLine(user + ": New or changed file found: " + updatedfile + "\n");
logFile.WriteLine(user + ": New or changed file found: " + change.FileId + " --- " + updatedfile);
deltalog.Write(change.FileId + "," + updatedfile+"\n");
deltalog.Flush();
}
Start exporting the files out as documents to my server for backup. To do this, I log into the Gapps domain with my service account 3 different times for one file for each user. Right now, I'm reading the file created from step 3 and splitting the values so that I have the fileId and Filename on hand. The code looks like this:
FilesResource.ListRequest listRequest = CreateService.BuildService(user).Files.List();
listRequest.PageSize = 1000;
listRequest.Fields = "nextPageToken, files(id, name)";
string[] deltafiles = File.ReadAllLines(savelocation + ".deltalog.tok");
IList<Google.Apis.Drive.v3.Data.File> files = listRequest.Execute()
.Files;
Console.WriteLine("\nFiles to backup:\n");
if (deltafiles == null)
{
return;
}
else
{
foreach (var file in deltafiles)
{
try
{
// Our file is a CSV. Column 1 = file ID, Column 2 = File name
var values = file.Split(',');
string fileId = values[0];
string fileName = values[1];
fileName = fileName.Replace('\\', '_').Replace('/', '_').Replace(':', '_').Replace('!', '_').Replace('\'', '_').Replace('*', '_');
Console.WriteLine("Filename: " + values[1]);
logFile.WriteLine("ID: " + values[0] + " - Filename: " + values[1]);
var requestfileid = CreateService.BuildService(user).Files.Get(fileId);
var getfile = CreateService.BuildService(user).Files.Get(fileId).Execute();
var request = CreateService.BuildService(user).Files.Export(fileId, getfile.MimeType);
and so forth.
If I try to change the requestfileid to values[0] (which would be the fileId for that file in the loop), then the MediaDownloader doesn't work because it's no longer part of the Files.Get constructor.
Is there anything I can do, or overlooking, so that the service account only has to log in once to do everything it needs per account?
Hope that gibberish makes sense.
I've been using https://developers.google.com/drive/v3/web/quickstart/dotnet and the API documentation as my source for information, and I got everything working the way I want it to, Except for having to log in as the service multiple times for One file. Any help or point in the right direction would sure be appreciated. Thank you!
I have this question,
My boss wants a program which you can enter a path, Console.ReadLine(directory);
This is irrelevant, I got this part working. In fact, the whole code/program is working as it should be.
The point of the path is that the program scans all the files in the given directory/subdirectories for the last write time. He want to pay minimum effort in this. So the plan is to use Windows to start this program once every 24 hours.
Only problem with this "minimum effort" part is that you have to enter the path EVERYTIME when it's started. So it actually doesn't go automaticly.
The question is: is there a way to avoid this?
For example Thread.Sleep(); and when it's done sleeping goto a label right under the Console.ReadLine(directory);?
So not once a day, but sleeping for 24 hours and for 1 whole minute working?
If it's any help, here's the code:
using System.IO;
using System.Security.Permissions;
namespace CheckWithinTime
{
class Program
{
public static void Main(string[] args)
{
Console.WriteLine("Which folder do you wish to scan?");
string path = Console.ReadLine();
//copies everything from the console to a .txt file
FileStream filestream = new FileStream(#"C:\Logs\Log.txt", FileMode.Create);
var streamwriter = new StreamWriter(filestream);
streamwriter.AutoFlush = true;
Console.SetOut(streamwriter);
Console.SetError(streamwriter);
//this is the path which you type in a the beginning of the program
string[] files = Directory.GetFiles(path, "*.*", System.IO.SearchOption.AllDirectories);
List<string> updatedFiles = new List<string>();
DateTime from = DateTime.Now.AddDays(-1);
DateTime to = DateTime.Now;
foreach (string name in files)
{
FileInfo file = new FileInfo(name);
string fullname = file.FullName;
//checks if the last writed time occured less then 24 hours ago, if it's not it will not be loggeed
if (file.LastWriteTime >= from && file.LastWriteTime <= to)
{
updatedFiles.Add(name);
Console.WriteLine(file.FullName + " ; " + "last changed at >> " + " ; " + file.LastWriteTime.ToString());
//Console.WriteLine("File created at >> " + file.CreationTime.ToString());
//Console.WriteLine("File last opened at >> " + file.LastAccessTime.ToString());
Console.WriteLine();
}
}
streamwriter.Close();
filestream.Close();
//The mail class basicly sends an e-mail to the server with the log file, but this is irrelevant
Mail();
}
}
}
It used to be just a simple file system watcher. After that it was like it is now, but without the Console.ReadLine() part. And now he wants to give a path.
If anyone can tell me a way to avoid the Console.ReadLine(), but only use call it when you need it. I would appreciate it!
Sorry in advance for my big texts.
The best way to do this would be to either create an XML file or use a notepad file and have windows run a task manager.
You can set up a program to run every so often in windows task manager and all you need to do is save the path to a file and have your C# console application read that file and get the path, the path will always be store and the program will run all the time.
We do this at work; a program has been running for 4 months doing through a bunch of paths and we don't edit it anymore.
Xml file/config is overkill for one setting. Pass it in to the command line string[] args:
string path;
if(args.Length > 0)
{
path = args[0];
Console.WriteLine("Using path: " + path);
}
else
{
Console.WriteLine("Which folder do you wish to scan?");
path = Console.ReadLine();
}
Then use task scheduler as already suggested, but pass the path as a command line argument. You can also then launch it with different paths without your app supporting multiple paths.
Wondering how to best deal with a problem I am having with xsltransform. Long story short, everything works in my test environment, but it crashes when I run it on the server due to the filenames it tries to deal with, which are output from another program, over which I have no control.
For example. "4Copy (2) of Fed_Around_Six__TFVC020-12.mov.xml" a simple # would solve this, but it's actually running on a service, and this service gets all files of that type in the directory and processes them one by one.
string[] filepaths = Directory.GetFiles(path, Filetype);
I keep the file name variable in:
FileInfo f = new FileInfo(filepaths[i]);
But the method I use for the transform:
myXslTransform = new XslCompiledTransform();
myXslTransform.Transform(filename,OutputFileName);
Only accepts (String, String) and thus when it sees "4Copy (2) of Fed_Around_Six__TFVC020-12.mov.xml" it has a heart attack and cuts it off.
I was thinking save the original name, rename, remove whitespace, transform, and rename back. But I think there is a smarter way to handle it out there, just not sure where to look. Is there a way of telling C# to handle a variable as a literal? Or a different transform method that accepts these weird filenames with very bad naming conventions?
Any insight that helps would be great!
The error & exception message I recieve from the Eventvwr is
Cannot Translate
\\9g031\Export\4Copy (2) of Fed_Around_Six__TFVC020-12.mov.xml
OutputName = \\9g031\Export\done\4Copy (2) of Fed_Around_Six__TFVC020-12.mov.xml
XSL LOC = C:\CXS.xsl
System.IO.IOException: The specified path is invalid.
private void PreformTranslation(FileInfo FileName, String OutputFileName , String result)
{
try
{
XslCompiledTransform myXslTransform;
myXslTransform = new XslCompiledTransform();
myXslTransform.Load(XSLname);
EventLog.WriteEntry(FileName.ToString(), OutputFileName);
myXslTransform.Transform(FileName.Name,OutputFileName);
EventLog.WriteEntry("TranslationComplete");
if (File.Exists(path + result))
{
MoveVideoFiles(path + result, outputPath + result);
}
// Rename(OutputFileName, FileName, Out);
}
catch (Exception e)
{
EventLog.WriteEntry("Cannot Translate " + FileName + " OutputName = " + OutputFileName + " \r\n"+
"XSL LOC = " + XSLname + "\r\n" + e);
}
}
The default directory when running a service is something like "windows/system32" and this isn't the directory of the executable.
This is probably the reason the XML file isn't found.
i want to launch ffmpeg from my app and retrive all console output that ffmpeg produces. Thing seems obvious, i followed many forum threads/articles like this one but i have problem, though i follow all information included there I seem to end up in dead end.
String that should contain output from ffmpeg is always empty. I've tried to see where is the problem so i made simple c# console application that only lists all execution parameters that are passed to ffmpeg, just to check if problem is caused by ffmpeg itself. In that case everything work as expected.
I also did preview console window of my app. When i launch ffmpeg i see all the output in console but the function that should recieve that output for further processing reports that string was empty. When my param-listing app is launched the only thing I see is the expected report from function that gets output.
So my question is what to do to get ffmpeg output as i intended at first place.
Thanks in advance
MTH
This is a long shot, but have you tried redirecting StandardError too?
Here is a part of my ffmpeg wrapper class, in particular showing how to collect the output and errors from ffmpeg.
I have put the Process in the GetVideoDuration() function just so you can see everything in the one place.
Setup:
My ffmpeg is on the desktop, ffPath is used to point to it.
namespace ChildTools.Tools
{
public class FFMpegWrapper
{
//path to ffmpeg (I HATE!!! MS special folders)
string ffPath = System.Environment.GetFolderPath(Environment.SpecialFolder.Desktop) + "\\ffmpeg.exe";
//outputLines receives each line of output, only if they are not zero length
List<string> outputLines = new List<string>();
//In GetVideoDuration I only want the one line of output and in text form.
//To get the whole output just remove the filter I use (my search for 'Duration') and either return the List<>
//Or joint the strings from List<> (you could have used StringBuilder, but I find a List<> handier.
public string GetVideoDuration(FileInfo fi)
{
outputLines.Clear();
//I only use the information flag in this function
string strCommand = string.Concat(" -i \"", fi.FullName, "\"");
//Point ffPath to my ffmpeg
string ffPath = System.Environment.GetFolderPath(Environment.SpecialFolder.Desktop) + "\\ffmpeg.exe";
Process processFfmpeg = new Process();
processFfmpeg.StartInfo.Arguments = strCommand;
processFfmpeg.StartInfo.FileName = ffPath;
//I have to say that I struggled for a while with the order that I setup the process.
//But this order below I know to work
processFfmpeg.StartInfo.UseShellExecute = false;
processFfmpeg.StartInfo.RedirectStandardOutput = true;
processFfmpeg.StartInfo.RedirectStandardError = true;
processFfmpeg.StartInfo.CreateNoWindow = true;
processFfmpeg.ErrorDataReceived += processFfmpeg_OutData;
processFfmpeg.OutputDataReceived += processFfmpeg_OutData;
processFfmpeg.EnableRaisingEvents = true;
processFfmpeg.Start();
processFfmpeg.BeginOutputReadLine();
processFfmpeg.BeginErrorReadLine();
processFfmpeg.WaitForExit();
//I filter the lines because I only want 'Duration' this time
string oStr = "";
foreach (string str in outputLines)
{
if (str.Contains("Duration"))
{
oStr = str;
}
}
//return a single string with the duration line
return oStr;
}
private void processFfmpeg_OutData(object sender, DataReceivedEventArgs e)
{
//The data we want is in e.Data, you must be careful of null strings
string strMessage = e.Data;
if outputLines != null && strMessage != null && strMessage.Length > 0)
{
outputLines.Add(string.Concat( strMessage,"\n"));
//Try a Console output here to see all of the output. Particularly
//useful when you are examining the packets and working out timeframes
//Console.WriteLine(strMessage);
}
}
}
}