We Keep Coding

Detect Duplicate Items in DataFlow

I've been building out a service that processes files using a Queue<string> object to manage the items.
public partial class BasicQueueService : ServiceBase
{
private readonly EventWaitHandle completeHandle =
new EventWaitHandle(false, EventResetMode.ManualReset, "ThreadCompleters");
public BasicQueueService()
{
QueueManager = new Queue<string>();
}
public bool Stopping { get; set; }
private Queue<string> QueueManager { get; }
protected override void OnStart(string[] args)
{
Stopping = false;
ProcessFiles();
}
protected override void OnStop()
{
Stopping = true;
}
private void ProcessFiles()
{
while (!Stopping)
{
var count = QueueManager.Count;
for (var i = 0; i < count; i++)
{
//Check the Stopping Variable again.
if (Stopping) break;
var fileName = QueueManager.Dequeue();
if (string.IsNullOrWhiteSpace(fileName) || !File.Exists(fileName))
continue;
Console.WriteLine($"Processing {fileName}");
Task.Run(() =>
{
DoWork(fileName);
})
.ContinueWith(ThreadComplete);
}
if (Stopping) continue;
Console.WriteLine("Waiting for thread to finish, or 1 minute.");
completeHandle.WaitOne(new TimeSpan(0, 0, 15));
completeHandle.Reset();
}
}
partial void DoWork(string fileName);
private void ThreadComplete(Task task)
{
completeHandle.Set();
}
public void AddToQueue(string file)
{
//Called by FileWatcher/Manual classes, not included for brevity.
lock (QueueManager)
{
if (QueueManager.Contains(file)) return;
QueueManager.Enqueue(file);
}
}
}
Whilst researching how to limit the number of threads on this (I've tried a manual class with an incrementing int, but there's an issue where it doesn't decrement properly in my code), I came across TPL DataFlow, which seems like its a better fit for what I'm trying to achieve - specifically, it allows me to let the framework handle threading/queueing, etc.
This is now my service:
public partial class BasicDataFlowService : ServiceBase
{
private readonly ActionBlock<string> workerBlock;
public BasicDataFlowService()
{
workerBlock = new ActionBlock<string>(file => DoWork(file), new ExecutionDataflowBlockOptions()
{
MaxDegreeOfParallelism = 32
});
}
public bool Stopping { get; set; }
protected override void OnStart(string[] args)
{
Stopping = false;
}
protected override void OnStop()
{
Stopping = true;
}
partial void DoWork(string fileName);
private void AddToDataFlow(string file)
{
workerBlock.Post(file);
}
}
This works well. However, I want to ensure that a file is only ever added to the TPL DataFlow once. With the Queue, I can check that using .Contains(). Is there a mechanism that I can use for TPL DataFlow?

Your solution with Queue works only if file goes into your service twice in a small period of time. If it came again in, say, few hours, queue will not contain it, as you Dequeue it from there.
If this solution is expected, then you may use a MemoryCache to store file paths being already handled, like this:
using System.Runtime.Caching;
private static object _lock = new object();
private void AddToDataFlow(string file)
{
lock (_lock)
{
if (MemoryCache.Default.Contains(file))
{
return;
}
// no matter what to put into the cache
MemoryCache.Default[file] = true;
// we can now exit the lock
}
workerBlock.Post(file);
}
However, if your application must run for a long time (which service is intended to do), you'll eventually run out of memory. In that case you probably need to store your file paths in database or something, so even after restarting the service your code will restore the state.

You can check it inside of DoWork.
You have to save in Hash already works items and check current filename doesn't exist in hash.

Why is my `StandardOleMarshalObject` COM Object called from multiple threads?

I have a COM object which is implemented in C#, and inherits from StandardOleMarshalObject to disable the NTA default behavior. For some reason, when I make a call to a server that makes a reentrant call to the client, the callback ends up on a different thread.
How do I ensure that all calls are made on the main thread?
[ComVisible(true)]
[InterfaceType(ComInterfaceType.InterfaceIsDual)]
public interface IComChat
{
void WriteLine(string text);
void Subscribe(IComChat callback);
}
public class ComChatServer : StandardOleMarshalObject, IComChat
{
private List<IComChat> Clients = new List<IComChat>();
public void WriteLine(string text)
{
foreach (var client in Clients)
{
// this makes a reentrant callback into the calling client
client.WriteLine(text);
}
}
public void Subscribe(IComChat client) => Clients.Add(client);
}
public class ComChatClient : StandardOleMarshalObject, IComChat
{
private IComChat Server;
private Thread MainThread;
public ComChatClient()
{
this.MainThread = Thread.CurrentThread;
this.Server = /* get server by some means */;
this.Server.Subscribe(this);
}
void IComChat.WriteLine(string text)
{
// this throws as the call ends up on a different thread
Contract.Assert(Thread.CurrentThread == MainThread);
Console.WriteLine(text);
}
void IComChat.Subscribe(IComChat callback) => throw new NotSupportedException();
public void WriteLine(string text) => Server.WriteLine(text);
}
public static class Program
{
public static void Main(string[] args)
{
var client = new ComChatClient();
Application.Run(new ChatWindow(client));
}
}

StandardOleMarshalObject only keeps things on the main thread if you create the object on a STA thread. Mark your entrypoint with [STAThread] to set your main thread as being single-threaded:
public static class Program
{
[STAThread]
public static void Main(string[] args)
{
var client = new ComChatClient();
Application.Run(new ChatWindow(client));
}
}

How to check that a thread is complete?

I am having a lot of trouble with this. Consider this example:
public class Test {
Thread t;
public Test() {
t = new Thread(ThreadFunction);
}
public void Start() {
t.Start();
}
private void ThreadFunction() {
Thread.Sleep(5000);
Console.WriteLine("Function Complete");
}
}
public static class Main {
public Main() {
Test test = new Test();
test.Start();
// sleep longer than my worker so it finishes
Thread.Sleep(10000);
// a place to place a breakpoint
bool breakPointHere = true;
}
}
Now, I see the output of the console.log, but when I inspect Test's thread object, I see that IsAlive is still true, and ThreadStatus = TheadStatus.Running. Why is this? I wish to detect that the thread is truly complete, but I am confused as to how it can still be considered running if ThreadFunction() completes?
EDIT 2:
I finally tracked down the cause, Updating the code, and am going to answer my own question
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading;
using System.Threading.Tasks;
namespace ConsoleApplication1 {
public abstract class Worker {
protected bool shutdown;
protected Thread t;
private bool _isStopped = true;
public bool IsStopped {
get {
return t.ThreadState == ThreadState.Stopped;
}
}
private bool _isPaused = false;
public bool IsPaused {
get {
return _isPaused;
}
}
private string stringRepresentation;
public Worker() {
t = new Thread(ThreadFunction);
stringRepresentation = "Thread id:" + t.ManagedThreadId;
t.Name = stringRepresentation;
}
public void Start() {
OnBeforeThreadStart();
t.Start();
}
public void ScheduleStop() {
shutdown = true;
}
public void SchedulePause() {
OnPauseRequest();
_isPaused = true;
}
public void Unpause() {
_isPaused = false;
}
public void ForceStop() {
t.Abort();
}
/// <summary>
/// The main thread loop.
/// </summary>
private void ThreadFunction() {
OnThreadStart();
while (!shutdown) {
if (!IsPaused) {
if (!OnLoop()) {
break;
}
}
Thread.Sleep(1000);
}
OnShutdown();
}
public abstract void OnBeforeThreadStart();
public abstract void OnThreadStart();
public abstract bool OnLoop();
public abstract void OnShutdown();
public abstract void OnPauseRequest();
public override string ToString() {
return stringRepresentation;
}
}
public class Test : Worker {
public override void OnBeforeThreadStart() {
Log.WriteLine(this + ": Thread about to be started...");
}
public override void OnThreadStart() {
Log.WriteLine(this + ": Thread Started!");
}
public override bool OnLoop() {
Log.WriteLine(this + ": I am doing the things...");
return true;
}
public override void OnShutdown() {
Log.WriteLine(this + ": Shutting down!");
}
public override void OnPauseRequest() {
}
}
public static class Log {
public delegate void LogDelegate(string text, string eventTime, Severity severity);
public static event LogDelegate OnWriteLine;
private static Queue<string> _pendingFileWrites = new Queue<string>();
public enum Severity {
Info,
Warning,
Error
}
public static void WriteLine(object line, Severity severity = Severity.Info) {
string eventTime = DateTime.Now.ToString("yyyy-MM-dd HH:mm:ss");
string formatted = "[" + eventTime + "]: " + line;
Console.WriteLine(formatted);
lock (_pendingFileWrites) {
_pendingFileWrites.Enqueue(formatted);
}
if (OnWriteLine != null) {
// this is the offending line:
OnWriteLine.Invoke((string)line, eventTime, severity);
}
}
public static void WriteToFile(string path) {
lock(_pendingFileWrites) {
StreamWriter sw = File.AppendText(path);
while(_pendingFileWrites.Count > 0) {
sw.WriteLine(
_pendingFileWrites.Dequeue()
);
}
sw.Close();
}
}
}
class Program {
static void Main(string[] args) {
List<Test> tests = new List<Test>();
for(int i = 0; i < 10; i++) {
Test test = new Test();
test.Start();
tests.Add(test);
}
// sleep a little bit so they do the things
Thread.Sleep(10000);
foreach (Test test in tests) {
test.ScheduleStop();
}
bool allStopped;
do {
allStopped = true;
foreach (Test test in tests) {
if (!test.IsStopped) {
allStopped = false;
break;
}
}
} while (!allStopped);
Console.WriteLine("Done!");
// a place to place a breakpoint
bool breakPointHere = true;
}
}
}

I think your original testing that lead you to believe .IsAlive would be true had some flaw in it, I tweaked your program in your question to the following to make it compile and to be able to see which thread it created.
public class Program
{
public class Test
{
Thread t;
public Test()
{
t = new Thread(ThreadFunction);
t.Name = "TestThread";
}
public void Start()
{
t.Start();
}
private void ThreadFunction()
{
Thread.Sleep(5000);
Console.WriteLine("Function Complete");
}
}
public static void Main()
{
Test test = new Test();
test.Start();
// sleep longer than my worker so it finishes
Thread.Sleep(10000);
// a place to place a breakpoint
bool breakPointHere = true;
}
}
here is a screenshot of the running threads from inside ThreadFunction
Here is a screenshot from the end of the program
Notice that there is no "TestThread" thread.
Here is a screenshot from the locals window
IsAlive is false.

Do you really need to sleep to wait for your thread to finish?
If you don't, a better and more robust solution would be using Thread.Join()
public static class Main {
public Main() {
Test test = new Test();
test.Start();
test.Join(); // Waits for test to complete
bool breakPointHere = true;
}
}

So it turns out that my issue was that my logging method was calling a UI thread function like so:
private void LogToForm(object line, string eventTime, Log.Severity severity) {
if (dataGridView_LogInfo.InvokeRequired) {
dataGridView_LogInfo.Invoke (
new Action<object, string, Log.Severity>(LogtoFormCallback),
new object[] { line, eventTime, severity }
);
} else {
LogtoFormCallback(line, eventTime, severity);
}
}
At the Invoke() line, the thread would hang forever. The solution was to replace it with BeginInvoke() instead.
EDIT: Also, my example was/is quite poor for this. I thought I didn't understand threads at a fundamental level, and that my examples would have been enough. Hopefully someone googles this though and has this same cause, and can try this solution.

Static class / Object? How to dispose

i'm really struggeling with OOP. I would like to start a process in my additional class. The process is a shell and I need to access this shell from severel forms and classes to write the commands and to receive the output. I use events to get the data. Here is my class for the process.
My class for the
public class ADBShell
{
public static string output = String.Empty;
public static Process adbshell = new Process();
public void Start_ADBShell()
{
if (adbshell != null && !adbshell.HasExited)
return;
adbshell = new Process();
adbshell.StartInfo.UseShellExecute = false;
adbshell.StartInfo.FileName = #"D:\adb\adb.exe";
adbshell.StartInfo.Arguments = "shell";
adbshell.StartInfo.RedirectStandardOutput = true;
adbshell.StartInfo.RedirectStandardInput = true;
//adb.StartInfo.RedirectStandardError = true;
adbshell.EnableRaisingEvents = true;
adbshell.StartInfo.CreateNoWindow = true;
//adb.ErrorDataReceived += new DataReceivedEventHandler(adb_ErrorDataReceived);
adbshell.OutputDataReceived += new DataReceivedEventHandler(adbshell_OutputDataReceived);
try { var started = adbshell.Start(); }
catch (Exception ex)
{
Console.WriteLine(ex.Message + Environment.NewLine + ex.StackTrace);
}
//adb.BeginErrorReadLine();
adbshell.BeginOutputReadLine();
}
void adbshell_OutputDataReceived(object sender, DataReceivedEventArgs e)
{
output += (e.Data) + Environment.NewLine;
}
public void press_touch(string x, string y)
{
adbshell.StandardInput.WriteLine("input tap " + String.Format("{0} {1}", x, y));
Debug.WriteLine("pressed");
}
}
My Form class looks like
public partial class Form1 : Form
{
private bool _record;
private bool _selecting;
private Rectangle _selection;
//---------------------------------------------------------------------
public Form1()
{
InitializeComponent();
}
//---------------------------------------------------------------------
private void Form1_Load(object sender, System.EventArgs e)
{
ADBShell adbshell = new ADBShell();
adbshell.Start_ADBShell();
}
Everytime I have to make a new object in my methods, but i dont want to create everytime a new object. I would like make one time the object and access everytime to the same object. I do not want to make servel processes. I need only proccess and send and receive everytime the data to this process.
Do I have to make a static class?
How I can dispose and close my process after I'm quit my Form Class?

1: You do not want a static class. You want a SINGLETON - that is a class that has only one instance. This is normally accessed using a static property. At the easiest way this works like this:
public class A () {
private A () {}
public static A Instance {get; } = new A();
}
Access is via:
A.Instance
2: You do not. Processes do not get disposed. You exit the last thread that is not a background thread then the process ends. Otherwise you kill it, if that has to be done "In force" from the outside.

Move the ADBShell intialization in constructor of Form class. So this object will live till Form is not exited and to release resources by process make sure you call Process.close() in ADBShell class (Either in destructor or implement a IDisposable)
public partial class Form1 : Form
{
private bool _record;
private bool _selecting;
ADBShell adbshell;
private Rectangle _selection;
//---------------------------------------------------------------------
public Form1()
{
InitializeComponent();
adbshell = new ADBShell();
}
//---------------------------------------------------------------------
private void Form1_Load(object sender, System.EventArgs e)
{
adbshell.Start_ADBShell();
}
Dipose Process like this by adding Destructor
~ADBShell()
{
process.Close();
}
or implement Dispose method of IDisposable
Class ABDShell : IDisposable
{
...
...
public void Dispose()
{
Dispose(true);
GC.SuppressFinalize(this);
}
protected virtual void Dispose(bool disposing)
{
if (disposing)
{
process.Close();
}
}
}
Updated singleton class
sealed class ADBShell
{
public static string output = String.Empty;
private ABDShell _instance;
private Process _processInstance;
// Note: constructor is 'private'
private ADBShell()
{
}
public Process ProcessInstance
{
if(_processInstance==null)
_processInstance = new Process();
get _processInstance ;
}
public static ADBShell Instance
{
get
{
if (_instance == null)
{
_instance = new ABDShell();
}
return _instance;
}
}
}
Now from your Form just do this
Process process = ABDShell.Instance.ProcessInstance;

// Sealed class makes sure it is not inherited. If inheritance required, go to Abstract Pattern.
class ADBShell
{
//public static property used to expose Singleton instance.
public static ADBShell Instance;
// private constructor
private ADBShell() { }
public static ADBShell getInstance()
{
if (Instance == null)
{
Instance = new Process;
}
}
}
Update
Thank you with your helps I solved my problems and now the ADB runs much faster instead of start everytime a new process.
public class ADBShell
{
private static ADBShell instance;
//private List<Employee> employeeList = null;
private Process shell = null;
private StreamWriter myWriter = null;
private static readonly object syncRoot = new object();
private ADBShell()
{
if (shell == null)
{
shell = new Process();
shell.StartInfo.FileName = (#"D:\ADB\ADB.exe");
shell.StartInfo.Arguments = "shell";
shell.StartInfo.RedirectStandardInput = true;
shell.StartInfo.UseShellExecute = false;
shell.StartInfo.WindowStyle = ProcessWindowStyle.Hidden;
shell.StartInfo.RedirectStandardOutput = true;
shell.StartInfo.CreateNoWindow = true;
shell.EnableRaisingEvents = true;
shell.OutputDataReceived += (sender, a) => Console.WriteLine(a.Data);
shell.Start();
myWriter = shell.StandardInput;
shell.BeginOutputReadLine();
}
}
public static ADBShell Instance()
{
if (instance == null)
{
lock (syncRoot)
{
if (instance == null)
{
instance = new ADBShell();
}
}
}
return instance;
}
public void tap(int x, int y)
{
myWriter.WriteLine("input tap {0} {1}", x.ToString(), y.ToString());
Thread.Sleep(10);
}
public void tap(string x, string y)
{
myWriter.WriteLine("input tap {0} {1}", x, y);
Thread.Sleep(10);
}
public void exit()
{
myWriter.WriteLine("exit");
}
public void Close()
{
myWriter.WriteLine("exit");
shell.WaitForExit();
if (!shell.HasExited)
{
shell.Kill();
}
shell.Close();
shell.Dispose();
myWriter.Close();
myWriter.Dispose();
}
}

Joining a thread started with StartNew()

When using the StartNew() method to kick off a process on a new thread, I need to figure out how to make another call into this object in that same thread (I assume this would be some sort of Join operation?).
The following example is dumbed down to illustrate the meat of what I am trying to do. I am well aware it is severely lacking in basic concurrency considerations. But I didn't want to cloud the code with all of that logic, so please forgive me on that.
The following console app shows what I am trying to accomplish. Assume on the StartNew() call a new thread with ID 9976 is created and the method invoked there. I would like the subsequent call to ProcessImmediate() in the file system watcher change event handler to be made on thread 9976 as well. As it stands, the call would share the same thread that is used for the file system watcher change event.
Can this be done, and if so, how?
namespace ConsoleApplication1
{
class Program
{
static void Main(string[] args)
{
var runner = new Runner();
runner.Run();
Console.ReadKey();
}
}
public class Runner
{
private Activity _activity = null;
private FileSystemWatcher _fileSystemWatcher;
public void Run()
{
_activity = new Activity();
// start activity on a new thread
Task.Factory.StartNew(() => _activity.Go());
_fileSystemWatcher = new FileSystemWatcher();
_fileSystemWatcher.Filter = "*.watcher";
_fileSystemWatcher.Path = "c:\temp";
_fileSystemWatcher.Changed += FileSystemWatcher_Changed;
_fileSystemWatcher.EnableRaisingEvents = true;
}
private void FileSystemWatcher_Changed(object sender, FileSystemEventArgs e)
{
// WANT TO CALL THIS FOR ACTIVITY RUNNING ON PREVIOUSLY CALLED THREAD
_activity.ProcessImmediate();
}
}
public class Activity
{
public void Go()
{
while (!Stop)
{
// for purposes of this example, magically assume that ProcessImmediate has not been called when this is called
DoSomethingInteresting();
System.Threading.Thread.Sleep(2000);
}
}
protected virtual void DoSomethingInteresting() { }
public void ProcessImmediate()
{
// for purposes of this example, assume that Go is magically in its sleep state when ProcessImmediate is called
DoSomethingInteresting();
}
public bool Stop { get; set; }
}
}
* UPDATE *
Thanks for the excellent responses. I took Mike's suggestion and implemented it for my console app. Below is the full working code which also includes the use of a cancellation token. I post this in case someone else might find it useful.
namespace ConsoleApplication1
{
class Program
{
static void Main(string[] args)
{
var runner = new Runner();
runner.Run();
Console.ReadKey();
runner.Stop();
Console.ReadKey();
}
}
public class Runner
{
private Activity _activity = null;
private FileSystemWatcher _fileSystemWatcher;
private CancellationTokenSource _cts = new CancellationTokenSource();
public void Stop() { _cts.Cancel(); }
public void Run()
{
_activity = new Activity();
// start activity on a new thread
var task = new Task(() => _activity.Go(_cts.Token), _cts.Token, TaskCreationOptions.LongRunning);
task.Start();
_fileSystemWatcher = new FileSystemWatcher();
_fileSystemWatcher.Filter = "*.watcher";
_fileSystemWatcher.Path = "C:\\Temp\\FileSystemWatcherPath";
_fileSystemWatcher.Changed += FileSystemWatcher_Changed;
_fileSystemWatcher.EnableRaisingEvents = true;
}
private void FileSystemWatcher_Changed(object sender, FileSystemEventArgs e)
{
// WANT TO CALL THIS FOR ACTIVITY RUNNING ON PREVIOUSLY CALLED THREAD
_activity.ProcessImmediate();
}
}
public class Activity : IDisposable
{
private AutoResetEvent _processing = new AutoResetEvent(false);
public void Go(CancellationToken ct)
{
Thread.CurrentThread.Name = "Go";
while (!ct.IsCancellationRequested)
{
// for purposes of this example, magically assume that ProcessImmediate has not been called when this is called
DoSomethingInteresting();
_processing.WaitOne(5000);
}
Console.WriteLine("Exiting");
}
protected virtual void DoSomethingInteresting()
{
Console.WriteLine(string.Format("Doing Something Interesting on thread {0}", Thread.CurrentThread.ManagedThreadId));
}
public void ProcessImmediate()
{
// for purposes of this example, assume that Go is magically in its sleep state when ProcessImmediate is called
_processing.Set();
}
public void Dispose()
{
if (_processing != null)
{
_processing.Dispose();
_processing = null;
}
}
}
}

First, you should use TaskCreationOptions.LongRunning if you are creating a task that will not complete quickly. Second, use an AutoResetEvent to signal the waiting thread to wake up. Note that below ProcessImmediate will return before DoSomethingInteresting has completed running on the other thread. Example:
using System.Threading;
public class Activity : IDisposable
{
private AutoResetEvent _processing = new AutoResetEvent(false);
public void Go()
{
while (!Stop)
{
// for purposes of this example, magically assume that ProcessImmediate has not been called when this is called
DoSomethingInteresting();
_processing.WaitOne(2000);
}
}
protected virtual void DoSomethingInteresting() { }
public void ProcessImmediate()
{
_processing.Set();
}
public bool Stop { get; set; }
public void Dispose()
{
if (_processing != null)
{
_processing.Dispose();
_processing = null;
}
}
}

User mike has given a better solution, which will be appropriate when you like to call the same method immediately. If you want to call a different methods immediately I'll expand mike's answer to achieve that.
using System.Threading;
public class Activity : IDisposable
{
private AutoResetEvent _processing = new AutoResetEvent(false);
private ConcurrentQueue<Action> actionsToProcess = new ConcurrentQueue<Action>();
public void Go()
{
while (!Stop)
{
// for purposes of this example, magically assume that ProcessImmediate has not been called when this is called
DoSomethingInteresting();
_processing.WaitOne(2000);
while(!actionsToProcess.IsEmpty)
{
Action action;
if(actionsToProcess.TryDeque(out action))
action();
}
}
}
protected virtual void DoSomethingInteresting() { }
public void ProcessImmediate(Action action)
{
actionsToProcess.Enqueue(action);
_processing.Set();
}
public bool Stop { get; set; }
public void Dispose()
{
if (_processing != null)
{
_processing.Dispose();
_processing = null;
}
}
}

To execute different methods on the same thread you can use a message loop that dispatches incoming requests. A simple option would be to use the event loop scheduler of the Reactive Extensions and to "recursively" schedule your Go() function - if in the mean time a different operation is scheduled it would be processed before the next Go() operation.
Here is a sample:
class Loop
: IDisposable
{
IScheduler scheduler = new EventLoopScheduler();
MultipleAssignmentDisposable stopper = new MultipleAssignmentDisposable();
public Loop()
{
Next();
}
void Next()
{
if (!stopper.IsDisposed)
stopper.Disposable = scheduler.Schedule(Handler);
}
void Handler()
{
Thread.Sleep(1000);
Console.WriteLine("Handler: {0}", Thread.CurrentThread.ManagedThreadId);
Next();
}
public void Notify()
{
scheduler.Schedule(() =>
{
Console.WriteLine("Notify: {0}", Thread.CurrentThread.ManagedThreadId);
});
}
public void Dispose()
{
stopper.Dispose();
}
}
static void Main(string[] args)
{
using (var l = new Loop())
{
Console.WriteLine("Press 'q' to quit.");
while (Console.ReadKey().Key != ConsoleKey.Q)
l.Notify();
}
}