We Keep Coding

Compare time between a log file and an image file

I have 1000's of images and more than 100,000 lines of log files. I need to check if there exists an image that is associated with each unix time in log file. To do this, I first read thru all the images and stored time information in an array. Then I read thru all the lines of the log file, split each information (lat, long, time) and stored them in an array. Finally, I am taking one time element at a time and checking if it matches with image time array. If no match is found, I get the time from log, get lat and long from the same array and write it to a text file. But the overall process takes very long time. I am looking into efficiencies on how to make this process faster.
var fileList = Directory.GetFiles(imageLocation, "*.jpg");
//Array that will store all the time information obtained from image property
double[] imgTimeInfo = new double[fileList.Length];
int imgTimeCounter =0;
foreach (var fileName in fileList)
{
x++;
string fileNameShort = fileName.Substring(fileName.LastIndexOf('\\') + 1);
richTextBox1.AppendText("Getting time information from image " + x + " of " + fileList.Length + " : " + fileNameShort + Environment.NewLine);
richTextBox1.Refresh();
using (var fs = File.OpenRead(fileName))
{
//create an instance of a bitmap image
var image = new Bitmap(fs);
//get the date/time image property of the image
PropertyItem property = image.GetPropertyItem(36867);
System.Text.Encoding encoding = new System.Text.ASCIIEncoding();
string valueFrmProperty = encoding.GetString(property.Value);
//Format the value obtained to convert it into unix equivalent for comparison
string valueCorrected = valueFrmProperty.Split(' ')[0].Replace(":", "/") + " " + valueFrmProperty.Split(' ')[1];
var unixTime = ConvertToUnixTimeStamp(DateTime.Parse(valueCorrected));
imgTimeInfo[imgTimeCounter] = unixTime;
imgTimeCounter++;
//It is very important to dispose the image resource before trying to read the property of another image. image.dispose frees the resources or else we get
//outofmemoryexception.
image.Dispose();
}
}
MessageBox.Show("Images done.");
richTextBox1.AppendText("Fetching time information from log files..."+Environment.NewLine);
richTextBox1.Refresh();
int counter4Time = contentBathy.Length / 6;
//assign counter for lat,long and time
int timeCounter = 3;
for (int i = 0; i < counter4Time; i++)
{
richTextBox1.AppendText("Searching time match with image files..." + Environment.NewLine);
richTextBox1.Refresh();
double timeValue = Int32.Parse(contentBathy[timeCounter]);
//Looks for values that is +- 3 seconds different in the image file.
if (Array.Exists(imgTimeInfo, a => a == timeValue || a == timeValue + 1 || a == timeValue + 2|| a == timeValue+3
||a == timeValue-1|| a== timeValue-2||a == timeValue-3))
{
File.AppendAllText(#"c:\temp\matched.txt", "Lat : " + contentBathy[timeCounter - 3] + " Log : " + contentBathy[timeCounter - 2] + Environment.NewLine);
richTextBox1.AppendText("Image with same time information found. Looking for another match."+ Environment.NewLine);
}
else
{
//richTextBox1.AppendText("Time did not match...Writing GPX cordinates..." + Environment.NewLine);
//richTextBox1.Refresh();
File.AppendAllText(gpxLocation, "Lat : " + contentBathy[timeCounter - 3] + " Log : " + contentBathy[timeCounter - 2] + Environment.NewLine);
}
if(timeCounter < contentBathy.Length-3)
timeCounter += 6;
}
}

Bizarre numerical errors

I am implementing a simple two-state particle filter. If you don't know what a particle filter is, that's fine - the short version is that I need to compute weighted means with weights between 0 and 1, and values between 0 and 1. Each particle has a value and a weight.
C# is giving me absolutely bizarre numerical problems though.
In trying to debug this, this is what my code looks like:
ConcurrentBag<Particle> particles; //this is inputted as an argument to my function
double mean = 0.0;
double totalWeight = 0.0;
foreach (Particle p in particles)
{
mean += p.Value * p.Weight;
totalWeight += p.Weight;
if (p.Value > 1.01 || p.Weight > 1.01)
{
Console.WriteLine("Value " + p.Value);
Console.WriteLine("Weight " + p.Weight);
Console.WriteLine("wtf");
}
}
if (totalWeight == 0.0)
{
//in this case, everything has miniscule weight, so let's just return 0.0 to avoid this precision corner case.
return new Bernoulli(0.0);
}
double oldMean = mean;
mean /= totalWeight;
return mean;
That if statement with the "wtf" is there for debug purposes, and it's being triggered. But, the print out is:
Value 1.0
Weight 0.01
This if statement shouldn't be true at all! What is happening?
Edit: A little update on debugging. This is my current entire function:
public override IDistribution createDistribution(ConcurrentBag<Particle> particles)
{
if (particles.Count == 0)
{
throw new Exception("Cannot create Distribution from empty particle collection");
}
if (!particles.ToArray()[0].GetType().IsAssignableFrom(typeof(BinaryParticle)))
{
throw new Exception("Cannot create Bernoulli Distribution from non-Binary Particle");
}
decimal mean = 0.0m;
decimal totalWeight = 0.0m;
foreach (Particle p in particles)
{
mean += (decimal)(p.Value * p.Weight);
totalWeight += (decimal)p.Weight;
if ((p.Weight > 1.01))
{
{
Console.WriteLine("Value " + p.Value);
Console.WriteLine("Weight " + p.Weight);
Console.WriteLine("Value " + p.Value.ToString("0.0000000"));
Console.WriteLine("wtf");
}
}
if (totalWeight == 0.0m)
{
//in this case, everything has miniscule weight, so let's just return 0.0 to avoid this precision corner case.
return new Bernoulli(0.0);
}
decimal oldMean = mean;
mean /= totalWeight;
try
{
return new Bernoulli((double)mean);
}
catch (Exception e)
{
decimal testMean = 0.0m;
decimal testTotalWeight = 0.0m;
Console.WriteLine(e);
foreach (Particle p in particles)
{
testMean += (decimal)(p.Value * p.Weight);
testTotalWeight += (decimal)p.Weight;
Console.WriteLine("weight is " + p.Weight);
Console.WriteLine("value is " + p.Value);
Console.WriteLine("Total mean is " + testMean);
Console.WriteLine("Total weight is " + testTotalWeight);
}
Console.WriteLine(testMean / testTotalWeight);
throw new Exception();
}
}
"mean" is giving a different value than is being printed in the writeline in the catch block. I have no idea why. Also, bizarrely, it is weight > 1.01 that is the true condition, when weight is 0.01.

Okay, you guys are going to be mad, so let me start off by saying I'm sorry :-)
The problem was in fact a race condition, and had to do with a misunderstanding on my part as to how locks in C# work. I was locking on an object whose instance could change in different methods, in which the particle bag was changing. Replacing that with a dedicated lock object fixed my problems.
Sorry ^_^;;

How to get cpu temperature? each core separately

I'm using this code to get cpu temperature, but I'm getting 'not supported' instead of the temperature.
public static string getCpuTemperature()
{
try
{
ManagementObjectSearcher searcher =
new ManagementObjectSearcher("root\\WMI",
"SELECT * FROM MSAcpi_ThermalZoneTemperature");
//Win32_TemperatureProbe
foreach (ManagementObject queryObj in searcher.Get())
{
double temp = Convert.ToDouble(queryObj["CurrentTemperature"].ToString());
double temp_critical = Convert.ToDouble(queryObj["CriticalTripPoint"].ToString());
double temp_cel = (temp / 10 - 273.15);
double temp_critical_cel = temp_critical / 10 - 273.15;
return temp_cel.ToString() + " _ " + temp_critical_cel.ToString();
}
}
catch (ManagementException e)
{
//MessageBox.Show("An error occurred while querying for WMI data: " + e.Message);
return e.Message.ToString();
}
return "";
}
I've googled the problem and have seen many answers for this including,
- the motherboard doesn't support this feature
- run VS with administration right
etc...
but none of them are true, because programs like OpenHardwareMonitor and SpeedFan show the temperature of cpu and gpu, Hdd temperature, cpu fan speed, and all other informations.
I want to know how do they do that? how is that I get 'not supported' message and these programs don't.

This is not a complete answer but hopefully it helps.
After perusing the code at https://code.google.com/p/open-hardware-monitor/source/browse/trunk/, I can't fully understand this code without downloading it all and investigating further.
The magic seems to happen here,
public override void Update() {
base.Update();
for (int i = 0; i < coreTemperatures.Length; i++) {
uint eax, edx;
// if reading is valid
if (Ring0.RdmsrTx(IA32_THERM_STATUS_MSR, out eax, out edx,
1UL << cpuid[i][0].Thread) && (eax & 0x80000000) != 0)
{
// get the dist from tjMax from bits 22:16
float deltaT = ((eax & 0x007F0000) >> 16);
float tjMax = coreTemperatures[i].Parameters[0].Value;
float tSlope = coreTemperatures[i].Parameters[1].Value;
coreTemperatures[i].Value = tjMax - tSlope * deltaT;
} else {
coreTemperatures[i].Value = null;
}
}
...
This code extracts the temperature data from the result of Ring0.RdmsrTx.
I believe Ring0 is a C implementation of a ring buffer, the code of which is in the repository here. This reads the Model Specific Register data from the CPU driver.
There is more detail in this question.

Easiest way would be probably to find a tool that can output the information you need in machine-readable way and then process that output. SpeedFan logs temperature to logs, you could just read the latest reading from the logs.
I realize this might not be an ideal solution, but it is the only universal. Querying CPU temperature in a Windows system is not an easy task.

I'm probably very late to answer this, but just in case someone stumbles upon this in the future, here's the way I did it:
public string getCPUTemp()
{
UpdateVisitor updateVisitor = new UpdateVisitor();
Computer computer = new Computer();
computer.Open();
computer.CPUEnabled = true;
computer.Accept(updateVisitor);
string res = "";
for (int i = 0; i < computer.Hardware.Length; i++)
{
if (computer.Hardware[i].HardwareType == HardwareType.CPU)
{
for (int j = 0; j < computer.Hardware[i].Sensors.Length; j++)
{
if (computer.Hardware[i].Sensors[j].SensorType == SensorType.Temperature) res = String.Concat(res, (computer.Hardware[i].Sensors[j].Name + ": " + computer.Hardware[i].Sensors[j].Value.ToString() + "ºC" + "\r"));
if (computer.Hardware[i].Sensors[j].Value.ToString() == "") { res = ""; return res; }
}
}
}
It worked perfectly with me (even though it didn't work for the GPU part). You just have to download OpenHardwareMonitor (https://openhardwaremonitor.org/), then add the reference to the OpenHardwareMonitorLib.dll which is in the \openhardwaremonitor\Bin\Debug folder, then add "using OpenHardwareMonitor.Hardware;" at the top.
Hope it can still help someone even if not OP!

create one time reference for various sample rates per file

My program reads in multiple files that contain time/value pairs sampled at different rates. I'm trying to use the file with the highest sample rate as the time scale for all sampled rates and output one master file with the unique time values from the highest sample rate file.
Each file contains time/values pairs like:
1,58
1.5,90
2,154
2.5,34
Here is my code so far:
public void ReadAndWrite(string[] fileNames)
{
var stopwatch = Stopwatch.StartNew();
List<StreamReader> readers = fileNames.Select(f => new StreamReader(f)).ToList();
try
{
using (StreamWriter writer = new StreamWriter(tbxOutputFile.Text))
{
string line = null;
// For each measurement in max measurements
for (int measNum = 0; measNum < numOfRows; measNum++)
{
// For each file's reader
for (int i = 0; i < readers.Count; i++)
{
// If line contains something, then add it to line
if ((line = readers[i].ReadLine()) != null)
{
// Process line and then write it to file
line = ProcessLine(line);
writer.Write(line);
}
else
{
writer.Write("");
}
// If it's not the last column, add delimiter
if (i < readers.Count - 1)
writer.Write(",");
}
writer.WriteLine();
// Update labels
int val = ((measNum + 1) * 100) / numOfRows;
string newText = Convert.ToString("Progress: " + val + "% " + " " + "Measurement #: " + (measNum + 1)
+ " out of " + numOfRows); // running on worker thread
this.Invoke((MethodInvoker)delegate
{
// runs on UI thread
lblStatus.Text = newText;
progressBar1.Value = val;
});
}
}
}
catch (Exception)
{
throw;
}
finally
{
foreach (var reader in readers)
{
reader.Close();
}
}
MessageBox.Show("File successfully created! " + '\n' + "Elapsed time: " +
(stopwatch.ElapsedMilliseconds/1000) + " seconds", "Processing Complete");
}
I came up with the pseudo code below (currentTime is the time from each file and uniqueTime is from an array that reads in each time from the highest sampled file):
// if time value from individual file is same as uniqueTime
if currentTime == uniqueTime
{
valueToWrite = curr_value // write the current value
}
else // currentTime is not same as uniqueTime
{
valueToWrite = prev_value // write the previous value
}
timeToWrite = uniqueTime // always write the uniqueTime
What is the best way to execute this pseudo code to make a unique time reference for all the various sample rates? Sorry if my question is confusing, I can elaborate more if need be.

To be clear about this, you do not want the values at the specific time they occurred, but you want to display one value for each source at each timepoint that the highest sampled source has?
That should be pretty straightforward. In pseudocode:
foreach (ValuePair val in highSampleRateValues) {
var aggregatedTimePointData;
aggregatedTimePointData.Add(val.Time, val.Value);
foreach (ValuePair val2 in lowSampleRateValues) {
var value = DetermineLatestEntryBackwardFrom(val.Time);
aggregatedTimePointData.Add(value);
}
}
This way, the sample rate of the higher density sampled signal serves as a clock, but you will have inaccuracies since the values from the other sources are only close, but not exactly on the timepoint of their recording. If you want those inaccuracies reduced, choose a higher sample rate and do the same thing. You can get as close to the actual timepoints as you want.

Background operation and blocking main form

Here is my scenario:
On a form I have list of direcotories, Button and control to display multiline text.
In a loop I try to find all files in each directory and delete them.
When file is deleted i want to add text to multiline control.
My problem is that when text is added I can not do anything else. Form is blocked and if I try do do anytching it just stops responding.
Files are deleted using BackgroundWorker
private void backgroundWorker1_DoWork(object sender, DoWorkEventArgs e)
{
//this is datatable with directories and other info
MainDataset.CZYSZCZENIEDataTable CZYSZCZENIE = e.Argument as MainDataset.CZYSZCZENIEDataTable;
CzyscPliki(CZYSZCZENIE, ReportProgress);
}
private void CzyscPliki(MainDataset.CZYSZCZENIEDataTable CZYSZCZENIE, ReportProgressDel del)
{
DirectoryInfo dir = null;
FileInfo[] files = null;
bool subfolder = false;
string katalog = "";
string maska = "";
string[] maski = null;
long total=0;
string dirS;
string fileS;
long fileLen;
//foreach directory to delete
foreach (DataRow r in CZYSZCZENIE.Rows)
{
//CanRead - check if row is not deleted or detached
//r["CZYSC"].AsBool() - check if directory should be cleared
if (r.CanRead() && r["CZYSC"].AsBool())
{
subfolder = r["PODKATALOGI"].AsBool();
katalog = r["KATALOG"].AsString().TrimEnd('\\');
maska = r["MASKA"].AsString();
if (maska.IsEmpty())
maska = "*";
maski = maska.Split(';');
dir = new DirectoryInfo(katalog);
if (dir.Exists)
{
foreach (string s in maski)
{
files = dir.GetFiles(s, (subfolder ? SearchOption.AllDirectories : SearchOption.TopDirectoryOnly));
dir.GetFiles();
foreach (FileInfo f in files)
{
dirS = f.Directory.FullName;
fileS = f.Name;
fileLen = f.Length;
try
{
f.Delete();
total += fileLen;
if (del != null)
//here is problem: del - delegate to report state
//when it is called it blocks form
del(dirS, fileS, fileLen, total);
}
catch (Exception ex)
{ }
}
}
}
}
}
}
//this is the delegate that appends text in multiline control
//memoEdit1 is the control
//ceReportProgress.Checked - check if report should be added
private void ReportProgress(string directory, string file, long size, long totalSize)
{
if (memoEdit1.InvokeRequired)
{
memoEdit1.BeginInvoke(new Action<string, string, long, long>(ReportProgress), directory, file, size, totalSize);
}
else
{
if (ceReportProgress.Checked)
{
if (file.IsEmpty())
memoEdit1.AppendText("\r\nCzyszczenie katalogu " + directory);
else
{
memoEdit1.AppendText(file);
if (size > 0)
{
if (size > 1048576)
{
decimal d = size / 1048576;
d = decimal.Round(d, 2);
memoEdit1.AppendText("\tWielkość : " + d.AsString() + " megabajtów", false);
}
else if (size > 1024)
{
decimal d = (decimal)size / (decimal)1024;
d = decimal.Round(d, 2);
memoEdit1.AppendText("\tWielkość : " + d.AsString() + " kilobajtów", false);
}
else
memoEdit1.AppendText("\tWielkość : " + size.AsString() + " bajtów", false);
}
if (totalSize > 0)
{
if (totalSize > 1073741824)
{
decimal d = (decimal)totalSize / (decimal)1073741824;
d = decimal.Round(d, 2);
memoEdit1.AppendText("Zwolniono dotychczas : " + d.AsString() + " gigabajtów");
}
else if (totalSize > 1048576)
{
decimal d = (decimal)totalSize / (decimal)1048576;
d = decimal.Round(d, 2);
memoEdit1.AppendText("Zwolniono dotychczas : " + d.AsString() + " megabajtów");
}
else if (totalSize > 1024)
{
decimal d = (decimal)totalSize / (decimal)1024;
d = decimal.Round(d, 2);
memoEdit1.AppendText("Zwolniono dotychczas : " + d.AsString() + " kilobajtów");
}
else
memoEdit1.AppendText("Zwolniono dotychczas : " + totalSize.AsString() + " bajtów");
}
}
//scroll to the end of control
memoEdit1.ScrollToEnd();
}
}
}
How can I improve this to make it not blocking the form?

You are calling ReportProgress too often. Do it more than about 1000 times per second and the UI thread gets flooded with requests that it cannot keep up with. It won't get around to doing its normal duties, which include painting the controls and responding to the mouse and keyboard. It looks frozen. This gets worse when the UI update code gets more expensive, updating text in a TextBox when there's already a lot of text in it can get quite slow.
The diagnostic is still seeing the UI frozen for a while after the BGW stops running, working on emptying the backlog in the invoke request queue, then suddenly jumping back alive when the queue is finally emptied.
You need to throttle the rate at which you call BeginInvoke(). It never makes more sense to call it any more frequently than once every 50 milliseconds, a human cannot perceive the difference beyond that. Collect the info in a List<> so you can BeginInvoke() a lot less frequently. That's still no complete guarantee if your worker can produce results faster than the UI thread could ever keep up with. In which case slowing down the worker would be a fix. Easy by using Invoke instead of BeginInvoke.

If this worker is running asynchronously, then you can have a form which responds to you.
Besides, problems:
You are running the loop in another function - it makes the operation non-reponsive.
You are not even checking if user wants to cancel (just a point i wanted to make) - Handle DoWorkEventArgs's Cancel property inside the foreach loop.
Move the function CzyscPliki's code in the backgroundWorker1_DoWork (it's anyway too tiny).
EDIT:
If you don't want to move the code into DoWork event handler, then better use Thread for more control. I'm not an expert on it but you will find plenty of code on how to implement so.