We Keep Coding

How can I log low-level OS file transactions in C#?

Since File/Process Monitor falls short in terms of filtering and unnecessary duplication when logging, I want to recreate what that program does and log all Windows file operations live in realtime.
I want to record various attributes such as the time, process name, source path, destination path, operation, result, and detail, just like Process Monitor does.
How can I get C# to extract this information from the OS?
EDIT: As zett42 pointed out, the FileSystemWatcher won't quite work as for example, file events created from processes themselves won't be intercepted. For instance, none of these transactions show up, even though I added the events: Changed, Created, Renamed, and Deleted to the FileSystemWatcher and set the EnableRaisingEvents flag to true.
EDIT 2: Using SimonMourier's suggestion of the Microsoft.Diagnostics.Tracing.TraceEvent nuget package, I managed to knock up the code below.
This section is put into a background worker:
Console.CancelKeyPress += (sender, e) => session.Stop();
session.EnableKernelProvider(KernelTraceEventParser.Keywords.All);
session.Source.Kernel.FileIOWrite += Kernel_FileIOWrite;
session.Source.Process();
And then the FileIOWrite event created runs the following when called (automatically):
private void Kernel_FileIOWrite(Microsoft.Diagnostics.Tracing.Parsers.Kernel.FileIOReadWriteTraceData obj)
{
string filename = obj.FileName;
string processpath = "";
if (obj.ProcessID == 0) processpath = "System Idle Process";
else if (obj.ProcessID == 4) processpath = "System";
else
{
try { processpath = "ID: " + obj.ProcessID + ": "+ Process.GetProcessById(obj.ProcessID).MainModule.FileName; }
catch { processpath = "error ID: "+ obj.ProcessID; }
}
richTextBox1.AppendText(filename + " ............."+ processpath +"\n");
}
Unfortunately, FileIOReadWriteTraceData.FileName is not picking up things Procmon picks up such as (for example), very common Chrome stuff such as writing to C:\Users\Dan\AppData\Local\Google\Chrome\User Data\Default\Cookies-journal or C:\Users\Dan\AppData\Local\Google\Chrome\User Data\Default\Current Session

You can never capture all the things that Process Monitor captures in C#. One of the reasons that ProcMon is so good at capturing all of those things is because ProcMon contains a driver that is loaded in kernel mode and hooks itself and listens for those events. If you want to replicate the process monitor, you will have to write your own driver to capture all the things that you want to. A windows driver cannot be written in C# and you will have to write the driver in C/C++.
The other option is to get Process Monitor to log everything to file and filter the events yourself.

Did you try to use WMI? The ManagementEventWatcher could provide all information you need, even though it's not that handy as the FileSytemWatcher.
I'm not sure it will work and unfortunately cannot try it myself at the moment, but this is the point where I would start.
Take a look at the anwer of RRUZ in this thread, which does exactly what you want. You will just need to find out if WMI provides all required information.

SshStream.Expect() breaks code when SshStream is no longer valid

Im trying to do automation over SSH but the server is faulty and does not implement exec channel properly so I ended up doing a workaround using CreateShellStream(). I can expect that upon running the program a connection wouldnt be available and disconnections are a thing. My solution:
while(!_cancellationToken.IsCancellationRequested))
{
ShellStream stream = null;
while(stream == null)
{
stream = await GetSshStream();
}
while(stream.CanWrite && stream.CanRead)
{
stream.WriteLine(command);
//this breaks everything if stream is not valid
var rep = stream.Expect(new Regex(#"[$>]"));
var delimiters = new[] { " ", "\r\n", "\t" };
var values = rep.Split(delimiters, StringSplitOptions.RemoveEmptyEntries);
DealWithValues(values);
}
}
This works and waits for connection and once connected starts coms. The problem arises with that stream.CanWrite && stream.CanRead is not enough to detect that stream is healthy and once connection is lost and stream becomes invalid and used with Expect(); everything breaks. Jumps out of all loops, goes through try {} catch{} and even makes Visual Studio debugger steping break down and continue the program in another thread (multi threaded program). Is there a way to stop this from happening and throwing execution back to first while? I could possibly create a new stream every time I need access to the server but since Im polling parameters about once a second I wouldnt want to have the overhead of reconnecting each time.

I misundertood that thread dissapearing ment code execution stopped. Instead thread sleeps waiting for input. Expect has a timeout overload and I should of been using it.

What is the correct way to update a GUI in a multi-threaded Winforms app?

I have a C# multi-threaded app with a bunch of worker threads in different pools all calling this function to update a textbox in a WinForms app.
It has a very subtle bug (or maybe not if you know what you are doing) that results in an Out-Of-Memory exception over a long period of time (I think because the stack never gets to unroll, but I'm not sure).
What is the correct way to update a textbox from any thread in my app without encountering this issue? The root problem is Application.DoEvents, but I don't know how to force a GUI update without it (if I take it out, the GUI never updates).
Ideally the solution would be a function that I can call from any thread that just does the right thing.
private void WriteStatus(string s)
{
if (textBoxStatus.InvokeRequired)
{
textBoxStatus.Invoke(new Action<string>(WriteStatus), new object[] { s });
}
else
{
StringBuilder sb = new StringBuilder(s + Environment.NewLine);
sb.Append(textBoxStatus.Text.Substring(0, textBoxStatus.Text.Length > 40000 ? 40000 : textBoxStatus.Text.Length));
textBoxStatus.Text = sb.ToString();
// don't call DoEvents more than once a second - this prevents stack over flow from frequent updates
if (DateTime.Now - lastGUIUpdate > TimeSpan.FromSeconds(1))
{
Application.DoEvents();
lastGUIUpdate = DateTime.Now;
}
}
}
I've looked at other SO solutions for this problem, but they seem to ignore the Application.DoEvents step. How are they forcing their GUI to update in this circumstance? In my app commenting this line out fixes my memory problems, but my textbox never updates.

Createhardlink in managed code

I'm developing an application that creates hundreds of thousands of hardlinks (this is the core feature of the application).
I use the parallel programming features that are now available in dotNET 4.0. This works really well. See example snippits below.
Either:
Parallel.For(from, until, delegate(int i)
{
j += 1;
fileIndex = Convert.ToInt32(Math.Round(j * 0.001) + 1);
//determine the hardlink files; we have to have an unique name for the hardlink for each individual hardlink
fileName = fiArray[fileIndex].Name; //Path.GetFileNameWithoutExtension(textBoxFile.Text);
destinationFileName = Path.Combine(textBoxDestination.Text, string.Concat(fileName, "_", i.ToString(), ".txt"));
fr.CreateHardLink(destinationFileName, fiArray[fileIndex].FullName);
});
Or:
//loop that does the actual work
for (int i = 0; i < nudThreads.Value; i++)
{
//determine the work package per task
from = 0 + until + 1;
until = (i * (Convert.ToInt32(HardLinks / ThreadNo))) + 1;
var compute = Task.Factory.StartNew(() =>
{
token.ThrowIfCancellationRequested(); //uit boek
return Work(from, until, false);//todo: counter moet nog worden meegenomen
}, tokenSource.Token);
tasks.Add(compute);
var displayResults = compute.ContinueWith(resultTask => UpdateControls(),
CancellationToken.None,
TaskContinuationOptions.OnlyOnRanToCompletion,
ui);
CheckedListBoxFiles.Items.Add(DateTime.Now.ToString() + " : Created the hardlinks for: " + displayResults + " files.");
Application.DoEvents();
var displayCancelledTasks = compute.ContinueWith(resultTask => UpdateControls(),
CancellationToken.None,
TaskContinuationOptions.OnlyOnCanceled, ui);
CheckedListBoxFiles.Items.Add(DateTime.Now.ToString() + " : Cancelled a task at: " + displayCancelledTasks + " files.");
Application.DoEvents();
}
The question i have is this: CreateHardlink is part of Kernel32.dll and hence runs in UNMANAGED code. What i know about the parallel ctp is that parallel tasks have to run in managed code.
Is there a managed alternative for createhardlink? Does anyone know how to create a hardlink in managed code and does anyone have any thougths about the parallel programming and using unmanaged code?

There's little point in trying to create hard links in a parallel fashion. This is not a CPU bound operation, it is I/O bound. I would not expect any performance benefit from this approach when compared against the naive serial approach.
Your question about managed and unmanaged code relating to hardlink creation is interesting. You must remember that any I/O access from managed code will at some point call into unmanaged code. The OS is not managed and the only way to create a hardlink is to go through the OS. I think you need to be more precise about exactly what this restriction of the CTP to managed code really means.

If you want to be able to cancel hardlink creation, but don't want multithreading, what I would do is implement a producer/consumer queue - one or more threads add "create a hard link at this path" work items to the queue, and a single thread grabs work items from the queue and creates hard links for them.
This gives you some flexibility over cancelling - you can abort the worker thread if you want to stop all hardlink creation, or if you want to cancel just one item you can find it in the queue and remove it.
Side note from looking at the code you posted - calling Application.DoEvents() is pure, unmitigated evil. Every time you call it, a baby seal clubs itself to death. A much better way to do it is for your UI to queue a work item on the thread pool, then use Dispatcher.Invoke (assuming this is WPF - if it's Winforms I know there's an equivalent but I don't know what it is) to update the UI asynchronously.

Using unmanaged OS calls in parallel is no more a problem than doing it from managed code. Simple as that. :)
Managed code simply means it runs within a managed context (so the CLR can keep track of things). Execution wise it doesn't matter if it is managed or unmanaged.
If I am not remembering it wrong any kernel32-call is atomic, meaning it is thread safe.

Port scanning using threadpool

I am trying to run a small app that scans ports and checks to see if they are open using and practicing with threadpools. The console window will ask a number and scans ports from 1 to X and will display each port whether they are open or closed. My problem is that as it goes through each port, it sometimes stops prematurely. It doesn't stop at just one number either, its pretty random. For example it I specify 200. The console will scroll through each port then stops at 110. Next time I run it, it stops at 80.
Code
Left out some of the things, assume all variables are declared where they should. First part is in Main.
static void Main(string[] args)
{
string portNum;
int convertedNum;
Console.WriteLine("Scanning ports 1-X");
portNum = Console.ReadLine();
convertedNum = Convert.ToInt32(portNum);
try
{
for (int i = 1; i <= convertedNum; i++)
{
ThreadPool.QueueUserWorkItem(scanPort, i);
Thread.Sleep(100);
}
}
catch (Exception e)
{
Console.WriteLine("exception " + e);
}
}
static void scanPort(object o)
{
TcpClient scanner = new TcpClient();
try
{
scanner.Connect("127.0.0.1",(int)o);
Console.WriteLine("Port {0} open", o);
}
catch
{
Console.WriteLine("Port {0} closed",o);
}
}
}

If this is the entire code, then the error is probably caused by you just falling through to the end of main() without waiting for all your thread pool threads to finish. The ThreadPool threads are all aborted once your main thread exits after falling through main().
Try removing the Thread.Sleep(100) (it is not needed, this is the wrong way, you'd never know for how long to sleep for and it partially defeats the purpose of using a ThreadPool in the first place) and you will probably not even check a single port!
Instead you could have each of your worker threads set an event and use WaitAll in main for all events to finish. See http://msdn.microsoft.com/en-us/library/3dasc8as.aspx for an example.
Edit:
Thinking this through, the solution referenced at the link above is probably less than ideal for you as well (it might involve having to allocate an array of 65000 events, this would be excessive). In .net 4 you could use a CountdownEvent like this:
Sorry, I gotta run, but check this example http://msdn.microsoft.com/en-us/library/system.threading.countdownevent.aspx and let us know when you have further questions, I'm sure someone can and will elaborate or suggest a better solution and a solution more suitable for .net3

What OS? Don't forget, different versions of XP have tcp connection limits, while you may also be triggering anti DDOS protection as well.
Also, your logic is flawed. Just because TcpClient.Connect excepted, doesn't mean the port is closed. You should be capturing and displaying that exception's details as I imagine it will offer you greater insight into why your code is stopping. Keep in mind, its possible to throw a SocketException or SecurityException as well.

Concerning the threading part, you could consider using the Task Parallel Library (TPL) instead of directly accessing the ThreadPool.
IMHO it offers a more simple use and a more intuitive/readable syntax.