I am using the below code to look up for any new process started. This function is running in the thread.
I need to log in the process name started. For which I use two arraylist. On one arraylist
i store all the process names before it starts the thread and the other arraylist, I fill
the current process inside the thread and compare two arraylist to find the new process.
The problem now is, since the string to write to the log file is in for each loop, I see
repeated process names in the log file. I want it to be logged only once. How can I solve this?
class ProcessMonitor
{
public static ArrayList ExistingProcess = new ArrayList();
public static void Monitor()
{
existingProcesses = GetExistingProcess();
while (true)
{
ArrayList currentProcesses = new ArrayList();
currentProcesses = GetCurrentProcess();
ArrayList NewApps = new ArrayList(GetCurrentProcess());
foreach (var p in ExistingProcess)
{
NewApps.Remove(p);
}
string str = "";
foreach (string NewApp in NewApps)
{
str += "Launched ProcessName/ID : " + NewApp + "/" + System.Diagnostics.Process.GetProcessesByName(NewApp)[0].Id.ToString() + Environment.NewLine;
}
if(str!="")
{
Log.Info(str);
}
}
}
public static ArrayList GetExistingProcess()
{
Process[] processlist = Process.GetProcesses();
foreach (Process Proc in processlist)
{
ExistingProcess.Add(Proc.ProcessName);
}
return ExistingProcess;
}
public static ArrayList GetCurrentProcess()
{
ArrayList CurrentProcesses = new ArrayList();
Process[] processlist = Process.GetProcesses();
foreach (Process Proc in processlist)
{
CurrentProcesses.Add(Proc.ProcessName);
}
return CurrentProcesses;
}
}
Iterating processes is very expensive on Windows. There's a better way to do it with WMI, Win32_ProcessStartTrace class. It also automatically solves your problem since it will only tell you about new processes getting started. And doesn't need a thread.
You'll find the code you need in this answer.
I'm not exactly sure what you are doing here, but the first two lines and the last line from the excerpt below basically do the same thing, only the last line is more costly (since your creating a second array list from the one returned by GetCurrentProcess:
ArrayList currentProcesses = new ArrayList();
currentProcesses = GetCurrentProcess();
ArrayList NewApps = new ArrayList(GetCurrentProcess());
Second, you never seem to use the currentProcess variable as far as I can tell...so its 100% waste. Third, why is it a problem if there are duplicates of process names? The same process can be started more than once, more than one instance of a process may be running concurrently, a process may run, be stopped, then be run again, etc. It is not necessarily "incorrect" for a process name to be listed twice.
(UPDATE: One reason why you may be getting "duplicates" in your log is that you get existingProcesses only once. Every time through the loop (which, btw, will happen at maximum speed continuously), you will be getting the list of processes again, and comparing them with the original existingProcesses, so the same processes listed in the previous loop...if they are still running, will be listed again. I've updated my code example to demonstrate how to solve this problem.)
You seem to have some fundamental code errors, and possibly flaws in your expectations. I would revisit your code in general, eliminate useless code (like the first two lines above), and generally streamline your code. (Hint: ArrayList is a REALLY bad choice...I would use IEnumerable<T>, which does not require ANY conversion or population from a raw Array). If I were to duplicate the code above with more efficient code:
public static void Monitor()
{
var existingProcesses = Process.GetProcesses();
bool doProcessing = true;
while (doProcessing)
{
var currentProcesses = Process.GetProcesses();
var newProcesses = currentProcesses.Except(existingProcesses);
int capacity = newProcesses.Count() * 60;
var builder = new StringBuilder(capacity);
foreach (var newProcess in newProcesses)
{
builder.Append("Launched ProcessName/ID : ");
builder.Append(newProcess.ProcessName);
builder.Append("/");
builder.Append(newProcess.Id);
builder.AppendLine();
}
string newProcessLogEntry = builder.ToString();
if(!String.IsNullOrEmpty(newProcessLogEntry))
{
Log.Info(newProcessLogEntry);
}
existingProcesses = currentProcesses; // Update existing processes, so you don't reprocess previously processed running apps and get "duplicate log entries"
if (requestToStopMonitoring) // do something to kill this loop gracefully at some point
{
doProcessing = false;
continue;
}
Thread.Sleep(5000); // Wait about 5 seconds before iterating again
}
}
Related
I am executing/processing very big files in multi threaded mode in a console app.
When I don't update/write to the console from threads, for testing the whole process take about 1 minute.
But when I try to update/write to console from threads to show the progress, the process stuck and it never finishes (waited several minutes even hours). And also console text/window does not updated as it should.
Update-1: As requested by few kind responder, i added minimal code that can reproduce the same error/problem
Here is the code from the thread function/method:
using System;
using System.Collections;
using System.Collections.Concurrent;
using System.Collections.Generic;
using System.IO;
using System.Threading;
using System.Threading.Tasks;
namespace Large_Text_To_Small_Text
{
class Program
{
static string sAppPath;
static ArrayList objThreadList;
private struct ThreadFileInfo
{
public string sBaseDir, sRFile;
public int iCurFile, iTFile;
public bool bIncludesExtension;
}
static void Main(string[] args)
{
string sFileDir;
DateTime dtStart;
Console.Clear();
sAppPath = Path.GetDirectoryName(System.Reflection.Assembly.GetExecutingAssembly().Location);
sFileDir = #"d:\Test";
dtStart = DateTime.Now;
///process in multi threaded mode
List<string> lFiles;
lFiles = new List<string>();
lFiles.AddRange(Directory.GetFiles(sFileDir, "*.*", SearchOption.AllDirectories));
if (Directory.Exists(sFileDir + "-Processed") == true)
{
Directory.Delete(sFileDir + "-Processed", true);
}
Directory.CreateDirectory(sFileDir + "-Processed");
sPrepareThreading();
for (int iFLoop = 0; iFLoop < lFiles.Count; iFLoop++)
{
//Console.WriteLine(string.Format("{0}/{1}", (iFLoop + 1), lFiles.Count));
sThreadProcessFile(sFileDir + "-Processed", lFiles[iFLoop], (iFLoop + 1), lFiles.Count, Convert.ToBoolean(args[3]));
}
sFinishThreading();
Console.WriteLine(DateTime.Now.Subtract(dtStart).ToString());
Console.ReadKey();
return;
}
private static void sProcSO(object oThreadInfo)
{
var inputLines = new BlockingCollection<string>();
long lACounter, lCCounter;
ThreadFileInfo oProcInfo;
lACounter = 0;
lCCounter = 0;
oProcInfo = (ThreadFileInfo)oThreadInfo;
var readLines = Task.Factory.StartNew(() =>
{
foreach (var line in File.ReadLines(oProcInfo.sRFile))
{
inputLines.Add(line);
lACounter++;
}
inputLines.CompleteAdding();
});
var processLines = Task.Factory.StartNew(() =>
{
Parallel.ForEach(inputLines.GetConsumingEnumerable(), line =>
{
lCCounter++;
/*
some process goes here
*/
/*If i Comment out these lines program get stuck!*/
//Console.SetCursorPosition(0, oProcInfo.iCurFile);
//Console.Write(oProcInfo.iCurFile + " = " + lCCounter.ToString());
});
});
Task.WaitAll(readLines, processLines);
}
private static void sPrepareThreading()
{
objThreadList = new ArrayList();
for (var iTLoop = 0; iTLoop < 5; iTLoop++)
{
objThreadList.Add(null);
}
}
private static void sThreadProcessFile(string sBaseDir, string sRFile, int iCurFile, int iTFile, bool bIncludesExtension)
{
Boolean bMatched;
Thread oCurThread;
ThreadFileInfo oProcInfo;
Salma_RecheckThread:
bMatched = false;
for (int iTLoop = 0; iTLoop < 5; iTLoop++)
{
if (objThreadList[iTLoop] == null || ((System.Threading.Thread)(objThreadList[iTLoop])).IsAlive == false)
{
oProcInfo = new ThreadFileInfo()
{
sBaseDir = sBaseDir,
sRFile = sRFile,
iCurFile = iCurFile,
iTFile = iTFile,
bIncludesExtension = bIncludesExtension
};
oCurThread = new Thread(sProcSO);
oCurThread.IsBackground = true;
oCurThread.Start(oProcInfo);
objThreadList[iTLoop] = oCurThread;
bMatched = true;
break;
}
}
if (bMatched == false)
{
System.Threading.Thread.Sleep(250);
goto Salma_RecheckThread;
}
}
private static void sFinishThreading()
{
Boolean bRunning;
Salma_RecheckThread:
bRunning = false;
for (int iTLoop = 0; iTLoop < 5; iTLoop++)
{
if (objThreadList[iTLoop] != null && ((System.Threading.Thread)(objThreadList[iTLoop])).IsAlive == true)
{
bRunning = true;
}
}
if (bRunning == true)
{
System.Threading.Thread.Sleep(250);
goto Salma_RecheckThread;
}
}
}
}
And here is the screenshot, if I try to update console window:
You see? Nor the line number (oProcInfo.iCurFile) or the whole line is correct!
It should be like this:
1 = xxxxx
2 = xxxxx
3 = xxxxx
4 = xxxxx
5 = xxxxx
Update-1: To test just change the sFileDir to any folder that has some big text file or if you like you can download some big text files from following link:
https://wetransfer.com/downloads/8aecfe05bb44e35582fc338f623ad43b20210602005845/bcdbb5
Am I missing any function/method to update console text from threads?
I can't reproduce it. In my tests the process always runs to completion, without getting stuck. The output is all over the place though, because the two lines below are not synchronized:
Console.SetCursorPosition(0, oProcInfo.iCurFile);
Console.Write(oProcInfo.iCurFile + " = " + lCCounter.ToString());
Each thread of the many threads involved in the computation invokes these two statements concurrently with the other threads. This makes it possible for one thread to preempt another, and move the cursor before the first thread has the chance to write in the console. To solve this problem you must add proper synchronization, and the easiest way to do it is to use the lock statement:
class Program
{
static object _locker = new object();
And in the sProcSO method:
lock (_locker)
{
Console.SetCursorPosition(0, oProcInfo.iCurFile);
Console.Write(oProcInfo.iCurFile + " = " + lCCounter.ToString());
}
If you want to know more about thread synchronization, I recommend this online resource: Threading in C# - Part 2: Basic Synchronization
If you would like to hear my opinion about the code in the question, and you don't mind receiving criticism, my opinion is that honestly the code is so much riddled with problems that the best course of action would be to throw it away and restart from scratch. Use of archaic data structures (ArrayList???), liberal use of casting from object to specific types, liberal use of the goto statement, use of hungarian notation in public type members, all make the code difficult to follow, and easy for bugs to creep in. I found particularly problematic that each file is processed concurrently with all other files using a dedicated thread, and then each dedicated thread uses a ThreadPool thread (Task.Factory.StartNew) to starts a parallel loop (Parallel.ForEach) with unconfigured MaxDegreeOfParallelism. This setup ensures that the ThreadPool will be saturated so badly, that there is no hope that the availability of threads will ever match the demand. Most probably it will also result to a highly inefficient use of the storage device, especially if the hardware is a classic hard disk.
Your freezing problem may not be C# or code related
on the top left of your console window, on the icon .. right click
select Properties
remove the option of Quick Edit Mode and Insert Mode
you can google that feature, but essentially manifests in the problem you describe above
The formatting problem on the other hand does seem to be, here you need to create a class that serializes writes to the console window from a singe thread. a consumer/producer pattern would work (you could use a BlockingCollection to implement this quite easily)
I have a folder with many CSV files in it, which are around 3MB each in size.
example of content of one CSV:
afkla890sdfa9f8sadfkljsdfjas98sdf098,-1dskjdl4kjff;
afkla890sdfa9f8sadfkljsdfjas98sdf099,-1kskjd11kjsj;
afkla890sdfa9f8sadfkljsdfjas98sdf100,-1asfjdl1kjgf;
etc...
Now I have a Console app written in C#, that searches each CSV file for a certain string.
And those strings to search for are in a txt file.
example of search txt file:
-1gnmjdl5dghs
-17kn3mskjfj4
-1plo3nds3ddd
then I call the method to search each search string in all files in given folder:
private static object _lockObject = new object();
public static IEnumerable<string> SearchContentListInFiles(string searchFolder, List<string> searchList)
{
var result = new List<string>();
var files = Directory.EnumerateFiles(searchFolder);
Parallel.ForEach(files, (file) =>
{
var fileContent = File.ReadLines(file);
if (fileContent.Any(x => searchList.Any(y => x.ToLower().Contains(y))))
{
lock (_lockObject)
{
foreach (string searchFound in fileContent.Where(x => searchList.Any(y => x.ToLower().Contains(y))))
{
result.Add(searchFound);
}
}
}
});
return result;
}
Question now is, can I anyhow improve performance of this operation?
I have around 100GB of files to search trough.
It takes aproximatly 1 hour to search all ~30.000 files with around 25 search strings, on a SSD disk and a good i7 CPU.
Would it make a difference to have larger CSV files or smaller CSV? I just want this search to be as fast as possible.
UPDATE
I have tried every suggestion that you wrote, and this is now what best performed for me (Removing ToLower from the LINQ yielded best performance boost. Search time from 1hour is now 16minutes!):
public static IEnumerable<string> SearchContentListInFiles(string searchFolder, HashSet<string> searchList)
{
var result = new BlockingCollection<string>();
var files = Directory.EnumerateFiles(searchFolder);
Parallel.ForEach(files, (file) =>
{
var fileContent = File.ReadLines(file); //.Select(x => x.ToLower());
if (fileContent.Any(x => searchList.Any(y => x.Contains(y))))
{
foreach (string searchFound in fileContent.Where(x => searchList.Any(y => x.Contains(y))))
{
result.Add(searchFound);
}
}
});
return result;
}
Probably something like Lucene could be a performance boost: why don't you index your data so you can search it easily?
Take a look at Lucene .NET
You'll avoid searching data sequentially. In addition, you can model many indexes based on the same data to be able to get to certain results at the light speed.
Try to:
Do .ToLower one time for a line instead of do .ToLower for each element in searchList.
Do one scan of file instead of two pass any and where. Get the list and then add with lock if any found. In your sample you waste time for two pass and block all threads when search and add.
If you know position where to look for (in your sample you know) you can scan from position, not in all string
Use producer consumer pattern for example use: BlockingCollection<T>, so no need to use lock
If you need to strictly search in field, build HashSet of searchList and do searchHash.Contains(fieldValue) this will increase process dramatically
So here a sample (not tested):
using(var searcher = new FilesSearcher(
searchFolder: "path",
searchList: toLookFor))
{
searcher.SearchContentListInFiles();
}
here is the searcher:
public class FilesSearcher : IDisposable
{
private readonly BlockingCollection<string[]> filesInMemory;
private readonly string searchFolder;
private readonly string[] searchList;
public FilesSearcher(string searchFolder, string[] searchList)
{
// reader thread stores lines here
this.filesInMemory = new BlockingCollection<string[]>(
// limit count of files stored in memory, so if processing threads not so fast, reader will take a break and wait
boundedCapacity: 100);
this.searchFolder = searchFolder;
this.searchList = searchList;
}
public IEnumerable<string> SearchContentListInFiles()
{
// start read,
// we not need many threads here, probably 1 thread by 1 storage device is the optimum
var filesReaderTask = Task.Factory.StartNew(ReadFiles, TaskCreationOptions.LongRunning);
// at least one proccessing thread, because reader thread is IO bound
var taskCount = Math.Max(1, Environment.ProcessorCount - 1);
// start search threads
var tasks = Enumerable
.Range(0, taskCount)
.Select(x => Task<string[]>.Factory.StartNew(Search, TaskCreationOptions.LongRunning))
.ToArray();
// await for results
Task.WaitAll(tasks);
// combine results
return tasks
.SelectMany(t => t.Result)
.ToArray();
}
private string[] Search()
{
// if you always get unique results use list
var results = new List<string>();
//var results = new HashSet<string>();
foreach (var content in this.filesInMemory.GetConsumingEnumerable())
{
// one pass by a file
var currentFileMatches = content
.Where(sourceLine =>
{
// to lower one time for a line, and we don't need to make lowerd copy of file
var lower = sourceLine.ToLower();
return this.searchList.Any(sourceLine.Contains);
});
// store current file matches
foreach (var currentMatch in currentFileMatches)
{
results.Add(currentMatch);
}
}
return results.ToArray();
}
private void ReadFiles()
{
var files = Directory.EnumerateFiles(this.searchFolder);
try
{
foreach (var file in files)
{
var fileContent = File.ReadLines(file);
// add file, or wait if filesInMemory are full
this.filesInMemory.Add(fileContent.ToArray());
}
}
finally
{
this.filesInMemory.CompleteAdding();
}
}
public void Dispose()
{
if (filesInMemory != null)
filesInMemory.Dispose();
}
}
This operation is first and foremost disk bound. Disk bound operations do not benefit from Multithreading. Indeed all you will do is swamp the Disk controler with a ton of conflictign requests at the same time, that a feature like NCQ has to striahgten out again.
If you had loaded all the files into memory first, your operation would be Memory Bound. And memory bound operations do not benefit from Multithreading either (usually; it goes into details of CPU and memory architecture here).
While a certain amount of Multitasking is mandatory in Programming, true Multithreading only helps with CPU bound operations. Nothing in there looks remotely CPU bound. So multithreading taht search (one thread per file) will not make it faster. And indeed likely make it slower due to all the Thread switching and synchronization overhead.
There are several (tons) of resources for finding parent/child processes from a PID. Too many to even list. I am finding that they are all incredibly slow.
Let's say I'm trying to implement a simple version of Process Explorer. I can enumerate all the processes easily and quickly with Process.GetProcesses(). However, getting the parent/child relationships takes forever.
I have tried using the NtQueryInformationProcess method, and found that this takes ~.1 seconds PER QUERY, that is, to get one parent from one process. To get children you basically have to build the whole tree and run this on every running process.
I then tried using ManagementObject to query for a process's parent and found it to be even slower at ~.12 seconds.
I tried going the other way and using ManagementObject to query the children directly rather than querying parents and building a tree. Querying for the children took between .25 and .5 seconds PER QUERY.
With any of these methods, that means populating a model of the current process tree takes 6-15 full seconds. That just seems crazy to me, like an order of magnitude higher than I would have guestimated. I can open process explorer and the parent/child relationships in the whole tree are just right there, immediatly.
Is there something wrong with my computer to make it this slow? Is this just a thing that for some reason takes way longer to discover than you'd think? How can process explorer do it so fast?
So, I came up with this which is a significant improvement. Since it's apparently WMI which is dog slow, I reduced it down to a single query for all processes and their parents, and then (with code) fill in the children so they are easily retrievable.
Notes on this code: it's a static class, sue me. it's called ProcessTree, but the data is not stored as a tree, it just models the tree, again sue me. also, there might be bugs, this is a proof of concept.
This gets the speed down to a third of a second, instead of 5-15 seconds. Since this is still expensive, it caches data for the interval ispecified in 'timeout'. This cache is not made to work with threads.
public static class ProcessTree
{
private static Dictionary<int, ProcessTreeNode> processes;
private static DateTime timeFilled = DateTime.MinValue;
private static TimeSpan timeout = TimeSpan.FromSeconds(3);
static ProcessTree()
{
processes = new Dictionary<int, ProcessTreeNode>();
}
public static List<int> GetChildPids(int pid)
{
if (DateTime.Now > timeFilled + timeout) FillTree();
return processes[pid].Children;
}
public static int GetParentPid(int pid)
{
if (DateTime.Now > timeFilled + timeout) FillTree();
return processes[pid].Parent;
}
private static void FillTree()
{
processes.Clear();
var results = new List<Process>();
string queryText = string.Format("select processid, parentprocessid from win32_process");
using (var searcher = new ManagementObjectSearcher(queryText))
{
foreach (var obj in searcher.Get())
{
object pidObj = obj.Properties["processid"].Value;
object parentPidObj = obj.Properties["parentprocessid"].Value;
if (pidObj != null)
{
int pid = Convert.ToInt32(pidObj);
if (!processes.ContainsKey(pid)) processes.Add(pid, new ProcessTreeNode(pid));
ProcessTreeNode currentNode = processes[pid];
if (parentPidObj != null)
{
currentNode.Parent = Convert.ToInt32(parentPidObj);
}
}
}
}
foreach (ProcessTreeNode node in processes.Values)
{
if (node.Parent != 0 && processes.ContainsKey(node.Parent)) processes[node.Parent].Children.Add(node.Pid);
}
timeFilled = DateTime.Now;
}
}
public class ProcessTreeNode
{
public List<int> Children { get; set; }
public int Pid { get; private set; }
public int Parent { get; set; }
public ProcessTreeNode(int pid)
{
Pid = pid;
Children = new List<int>();
}
}
I will leave the question not-answered in case someone has a better solution.
You could p/invoke Process32First, Process32Next, CreateToolhelp32Snapshot. The PROCESSENTRY32 structures returned have a parent process id, which you could use to establish parent relationship. These API's are blazingly fast.
I have the following code that starts robocopy as a Process. I also need to do database queries to determine which directories I need to copy each time robocopy is called so I used ProcessStartInfo to control the arguments passed.
internal class Program
{
private static void Main(string[] args)
{
using (var context = new MyDbContext())
{
IEnumerable<ProcessStartInfo> processInfos = GetProcessInfos(context, args[0]);
foreach (ProcessStartInfo processInfo in processInfos)
{
// How can I reuse robocopy Process instances and
// how can I dispose of them properly?
Process.Start(processInfo);
}
}
}
private static IEnumerable<ProcessStartInfo> GetProcessInfos(MyDbContext context,
string directory)
{
const string defaultRobocopyFormatString = "{0} {1} /mir /tee /fft /r:3 /w:10 /xd *Temp*";
var directoryInfo = new DirectoryInfo(directory);
return from dir in directoryInfo.GetDirectories()
from myEntity in context.MyEntities
where dir.Name == myEntity.Name
select new ProcessStartInfo("robocopy",
string.Format(defaultRobocopyFormatString,
Path.Combine("C:\Test", dir.Name),
Path.Combine("C:\Test_bak", dir.Name)));
}
}
How can I reuse Process instances returned by the static Process.Start(ProcessStartInfo) inside the foreach loop and how can I Dispose of them properly?
You cannot re-use a Process object. The Process class behaves like all of the other .NET classes that wrap an operating system object. Like Socket, Bitmap, Mutex, FileStream, etcetera. They are tiny little cookies that are very cheap to bake and take very little space on the GC heap. They track the lifetime of the underlying OS object carefully, once the object is dead, the .NET wrapper object is no longer useful either.
The Process class signals that the cookie was eaten with its Exited event and HasExited property. It has some post-bite properties that are useful, ExitCode and ExitTime.
But that's where it ends, if you want to create another process then you have to bake another cookie. Simple to do with the new keyword or the Start() factory function. Don't try to optimize it, there's no point and it can't work. Re-using ProcessStartInfo is fine, it is not a wrapper class.
You don't really need to reuse the Process class - that's just a wrapper for the underlying process. And when processes end, they're gone, completely - that's the main point of having a process in the first place.
Instead, it seems like you really want to just make sure that only one of those robocopy processes runs at a time, which is pretty easy:
using (var context = new MyDbContext())
{
IEnumerable<ProcessStartInfo> processInfos = GetProcessInfos(context, args[0]);
foreach (ProcessStartInfo processInfo in processInfos)
{
using (var process = Process.Start(processInfo))
{
// Blocks until the process ends
process.WaitForExit();
}
// When the `using` block is left, `process.Dispose()` is called.
}
}
I'm doing what amounts to a glorified mail merge and then file conversion to PDF... Based on .Net 4.5 I see a couple ways I can do the threading. The one using a thread safe queue seems interesting (Plan A), but I can see a potential problem. What do you think? I'll try to keep it short, but put in what is needed.
This works on the assumption that it will take far more time to do the database processing than the PDF conversion.
In both cases, the database processing for each file is done in its own thread/task, but PDF conversion could be done in many single threads/tasks (Plan B) or it can be done in a single long running thread (Plan A). It is that PDF conversion I am wondering about. It is all in a try/catch statement, but that thread must not fail or all fails (Plan A). Do you think that is a good idea? Any suggestions would be appreciated.
/* A class to process a file: */
public class c_FileToConvert
{
public string InFileName { get; set; }
public int FileProcessingState { get; set; }
public string ErrorMessage { get; set; }
public List<string> listData = null;
c_FileToConvert(string inFileName)
{
InFileName = inFileName;
FileProcessingState = 0;
ErrorMessage = ""; // yah, yah, yah - String.Empty
listData = new List<string>();
}
public void doDbProcessing()
{
// get the data from database and put strings in this.listData
DAL.getDataForFile(this.InFileName, this.ErrorMessage); // static function
if(this.ErrorMessage != "")
this.FileProcessingState = -1; //fatal error
else // Open file and append strings to it
{
foreach(string s in this.listData}
...
FileProcessingState = 1; // enum DB_WORK_COMPLETE ...
}
}
public void doPDFProcessing()
{
PDFConverter cPDFConverter = new PDFConverter();
cPDFConverter.convertToPDF(InFileName, InFileName + ".PDF");
FileProcessingState = 2; // enum PDF_WORK_COMPLETE ...
}
}
/*** These only for Plan A ***/
public ConcurrentQueue<c_FileToConvert> ConncurrentQueueFiles = new ConcurrentQueue<c_FileToConvert>();
public bool bProcessPDFs;
public void doProcessing() // This is the main thread of the Windows Service
{
List<c_FileToConvert> listcFileToConvert = new List<c_FileToConvert>();
/*** Only for Plan A ***/
bProcessPDFs = true;
Task task1 = new Task(new Action(startProcessingPDFs)); // Start it and forget it
task1.Start();
while(1 == 1)
{
List<string> listFileNamesToProcess = new List<string>();
DAL.getFileNamesToProcessFromDb(listFileNamesToProcess);
foreach(string s in listFileNamesToProcess)
{
c_FileToConvert cFileToConvert = new c_FileToConvert(s);
listcFileToConvert.Add(cFileToConvert);
}
foreach(c_FileToConvert c in listcFileToConvert)
if(c.FileProcessingState == 0)
Thread t = new Thread(new ParameterizedThreadStart(c.doDbProcessing));
/** This is Plan A - throw it on single long running PDF processing thread **/
foreach(c_FileToConvert c in listcFileToConvert)
if(c.FileProcessingState == 1)
ConncurrentQueueFiles.Enqueue(c);
/*** This is Plan B - traditional thread for each file conversion ***/
foreach(c_FileToConvert c in listcFileToConvert)
if(c.FileProcessingState == 1)
Thread t = new Thread(new ParameterizedThreadStart(c.doPDFProcessing));
int iCount = 0;
for(int iCount = 0; iCount < c_FileToConvert.Count; iCount++;)
{
if((c.FileProcessingState == -1) || (c.FileProcessingState == 2))
{
DAL.updateProcessingState(c.FileProcessingState)
listcFileToConvert.RemoveAt(iCount);
}
}
sleep(1000);
}
}
public void startProcessingPDFs() /*** Only for Plan A ***/
{
while (bProcessPDFs == true)
{
if (ConncurrentQueueFiles.IsEmpty == false)
{
try
{
c_FileToConvert cFileToConvert = null;
if (ConncurrentQueueFiles.TryDequeue(out cFileToConvert) == true)
cFileToConvert.doPDFProcessing();
}
catch(Exception e)
{
cFileToConvert.FileProcessingState = -1;
cFileToConvert.ErrorMessage = e.message;
}
}
}
}
Plan A seems like a nice solution, but what if the Task fails somehow? Yes, the PDF conversion can be done with individual threads, but I want to reserve them for the database processing.
This was written in a text editor as the simplest code I could, so there may be something, but I think I got the idea across.
How many files are you working with? 10? 100,000? If the number is very large, using 1 thread to run the DB queries for each file is not a good idea.
Threads are a very low-level control flow construct, and I advise you try to avoid a lot of messy and detailed thread spawning, joining, synchronizing, etc. etc. in your application code. Keep it stupidly simple if you can.
How about this: put the data you need for each file in a thread-safe queue. Create another thread-safe queue for results. Spawn some number of threads which repeatedly pull items from the input queue, run the queries, convert to PDF, then push the output into the output queue. The threads should share absolutely nothing but the input and output queues.
You can pick any number of worker threads which you like, or experiment to see what a good number is. Don't create 1 thread for each file -- just pick a number which allows for good CPU and disk utilization.
OR, if your language/libraries have a parallel map operator, use that. It will save you a lot of messing around.