We Keep Coding

With LibGit2Sharp, what is the best way to force specific processing for some files in the merge operation?

We have an authoring tool that stores different kind of textual datas in a Git Repository.
For most of the files we use, the way LibGit2Sharp handles the merge of the files is OK. Basically, when the changes are simple enough, the merge is done automatically. Otherwise, we get a list of conflicts that we use to manage by calling a manual merge editor (KDiff3 by default).
But we have some specific files that are sometimes corrupted by the automatic processing. For instance, if we have a file with 100 lines, in branch A, the line 50 is moved to the 5th position, and in branch B this same line 50 is moved to line 95. In that case the automatic merge will remove the line 50 and add it both on position 5 and 95. And in our application it is not acceptable: here we have to ask the user which position is valid (when we can't figure it out automatically).
So what would be the best solution to handle a specific processing for that kind of file, either by forcing the usage of some manual merge tool, or do some specific automatic processing ?
Here is an extract of the code that handles Git merge with LibGit2Sharp:
CheckoutNotifyFlags flags = CheckoutNotifyFlags.Dirty |
CheckoutNotifyFlags.Conflict |
CheckoutNotifyFlags.Updated;
MergeOptions mo = new MergeOptions() {
CheckoutNotifyFlags = flags,
OnCheckoutNotify = CheckNotifyHandler,
FileConflictStrategy = CheckoutFileConflictStrategy.Diff3,
OnCheckoutProgress = CheckoutProgressHandler,
CommitOnSuccess = true,
};
var mergeResult = Repository.Merge(commonBranch, Signature, mo);
bool NoConflict = Repository.Index.Conflicts.Count() == 0;
if (!NoConflict)
{
if (!CheckMergeConfig()) return false;
var conflicts = Repository.Index.Conflicts;
var indexParConflict = conflicts.FirstOrDefault(conf => IsIndexParConflict(conf));
if (ProcessIndexParConflict(indexParConflict))
{
foreach (var conflict in conflicts)
{
if (!ProcessConflict(conflict))
{
Reset(); // Reset all changes
return false;
}
}
}
}
We already have written a method that, provided the 3 input files used for the merge, says if the result file is valid or not.
If no better way to do it, we could analyse the result files at the end of the merge operation, and give a chance to the user to change it before we commit the merge results. But that would increase the overall complexity, and doesn't seem to be very clean way to do it.
Thanks for any input on that topic.

get the position of a string in a text file based on the line number in c#

I have an input text file that comes from a third party and i wrote a c# program to process it and get the results. I have the results and I need to update the same file with the results. The third party updates their DB based on this output file. I need to get the position of the string to update the file.
Ex: The input file looks this way:
Company Name: <some name> ID: <some ID>
----------------------------------------------------
Transaction_ID:0000001233 Name:John Amount:40:00 Output_Code:
-----------------------------------------------------------------------
Transaction_ID:0000001234 Name:Doe Amount:40:00 Output_Code:
------------------------------------------------------------------------
Please note: transaction_ID is unique in each row.
The Output file should be:
Company Name: <some name> ID: <some ID>
----------------------------------------------------
Transaction_ID:0000001233 Name:John Amount:40:00 Output_Code:01
-----------------------------------------------------------------------
Transaction_ID:0000001234 Name:Doe Amount:40:00 Output_Code:02
---------------------------------------------------------------------------
The codes 01 and 02 are the results of the c# program and have to be updated in the response file.
I have the code find out the position of "Transaction_ID:0000001233" and "Output_Code:". I am able to update the first row. But I am not able to get the position of the "Output_Code:" for the second row. How do I identify the string based on the line number?
I cannot rewrite the whole response file as it has other unwanted columns.
The best option here would be to update the existing file.
long positionreturnCode1 = FileOps.Seek(filePath, "Output_Code:");
//gets the position of Output_Code in the first row.
byte[] bytesToInsert = System.Text.Encoding.ASCII.GetBytes("01");
FileOps.InsertBytes(bytesToInsert, newPath, positionreturnCode1);
// the above code inserts "01" in the correct position. ie:first row
long positiontransId2 = FileOps.Seek(filePath, "Transaction_ID:0000001234");
long positionreturnCode2 = FileOps.Seek(filePath, "Output_Code:");
// still gets the first row's value
long pos = positionreturnCode2 - positiontransId2;
byte[] bytesToInsert = System.Text.Encoding.ASCII.GetBytes("02");
FileOps.InsertBytes(bytesToInsert, newPath, pos);
// this inserts in a completely different position.
I know the logic is wrong. But I am trying to get the position of output code value in the second row.

Don't try to "edit" the existing file. There is too much room for error.
Rather, assuming that the file format will not change, parse the file into data, then rewrite the file completely. An example, in pseudo-code below:
public struct Entry
{
public string TransactionID;
public string Name;
public string Amount;
public string Output_Code;
}
Iterate through the file and create a list of Entry instances, one for each file line, and populate the data of each Entry instance with the contents of the line. It looks like you can split the text line using white spaces as a delimiter and then further split each entry using ':' as a delimiter.
Then, for each entry, you set the Output_Code during your processing phase.
foreach(Entry entry in entrylist)
entry.Output_Code = MyProcessingOfTheEntryFunction(entry);
Finally iterate through your list of entries and rewrite the entire file using the data in your Entry list. (Making sure to correctly write the header and any line spacers, etc..)
OpenFile();
WriteFileHeader();
foreach(Entry entry in entrylist)
{
WriteLineSpacer();
WriteEntryData(entry);
}
CloseFile();

To start with, I'll isolate the part that takes a transaction and returns a code, since I don't know what that is, and it's not relevant. (I'd do the same thing even if I did know.)
public class Transaction
{
public Transaction(string transactionId, string name, decimal amount)
{
TransactionId = transactionId;
Name = name;
Amount = amount;
}
public string TransactionId { get; }
public string Name { get; }
public decimal Amount { get; }
}
public interface ITransactionProcessor
{
// returns an output code
string ProcessTransaction(Transaction transaction);
}
Now we can write something that processes a set of strings, which could be lines from a file. That's something to think about. You get the strings from a file, but would this work any different if they didn't come from a file? Probably not. Besides, manipulating the contents of a file is harder. Manipulating strings is easier. So instead of "solving" the harder problem we're just converting it into an easier problem.
For each string it's going to do the following:
Read a transaction, including whatever fields it needs, from the string.
Process the transaction and get an output code.
Add the output code to the end of the string.
Again, I'm leaving out the part that I don't know. For now it's in a private method, but it could be described as a separate interface.
public class StringCollectionTransactionProcessor // Horrible name, sorry.
{
private readonly ITransactionProcessor _transactionProcessor;
public StringCollectionTransactionProcessor(ITransactionProcessor transactionProcessor)
{
_transactionProcessor = transactionProcessor;
}
public IEnumerable<string> ProcessTransactions(IEnumerable<string> inputs)
{
foreach (var input in inputs)
{
var transaction = ParseTransaction(input);
var outputCode = _transactionProcessor.ProcessTransaction(transaction);
var outputLine = $"{input} {outputCode}";
yield return outputLine;
}
}
private Transaction ParseTransaction(string input)
{
// Get the transaction ID and whatever values you need from the string.
}
}
The result is an IEnumerable<string> where each string is the original input, unmodified except for the output code appended that the end. If there were any extra columns in there that weren't related to your processing, that's okay. They're still there.
There are likely other factors to consider, like exception handling, but this is a starting point. It gets simpler if we completely isolate different steps from each other so that we only have to think about one thing at a time.
As you can see, I've still left things out. For example, where do the strings come from? Do they come from a file? Where do the results go? Another file? Now it's much easier to see how to add those details. They seemed like they were the most important, but now we've rearranged this so that they're the least important.
It's easy to write code that reads a file into a collection of strings.
var inputs = file.ReadLines(path);
When you're done and you have a collection of strings, it's easy to write them to a file.
File.WriteAllLines(path, linesToWrite);
We wouldn't add those details into the above classes. If we do, we've restricted those classes to only working with files, which is unnecessary. Instead we just write a new class which reads the lines, gets a collection of strings, passes it to the other class to get processed, gets back a result, and writes it to a file.
This is an iterative process that allows us to write the parts we understand and leave the parts we haven't figured out for later. That keeps us moving forward solving one problem at a time instead of getting stuck trying to solve a few at once.
A side effect is that the code is easier to understand. It lends itself to writing methods with just a few lines. Each is easy to read. It's also much easier to write unit tests.
In response to some comments:
If the output code doesn't go at the end of the line - it's somewhere in the middle, you can still update it:
var line = line.Replace("Output_Code:", "Output_Code:" + outputCode);
That's messy. If the line is delimited, you could split it, find the element that contains Output_Code, and completely replace it. That way you don't get weird results if for some reason there's already an output code.
If the step of processing a transaction includes updating a database record, that's fine. That can all be within ITransactionProcessor.ProcessTransaction.
If you want an even safer system you could break the whole thing down into two steps. First process all of the transactions, including your database updates, but don't update the file at all.
After you're done processing all of the transactions, go back through the file and update it. You could do that by looking up the output code for each transaction in the database. Or, processing transactions could return a Dictionary<string, string> containing the transaction ids and output codes. When you're done with all the processing, go through the file a second time. For each transaction ID, see if there's an output code. If there is, update that line.

The additions here are send in a position based on where your main program has already updated and keep that moving forward ahead the length of what you also added.
I believe if I am reading the code there and in your example correctly this should make you scoot along through the file.
This function is within the utils that you linked in your comment.
public static long Seek(string file, long position, string searchString)
{
//open filestream to perform a seek
using (System.IO.FileStream fs =
System.IO.File.OpenRead(file))
{
fs.Position = position;
return Seek(fs, searchString);
}
}

Aspose.Words adding paragraphs via DocumentBuilder does not update Document info

I'm trying to write a extension method for Aspose's DocumentBuilder class that allows you to check if inserting a number of paragraphs into a document will cause a page break or not, I hoped this would be rather simple, but it turns out otherwise:
public static bool WillPageBreakAfter(this DocumentBuilder builder, int numParagraphs)
{
// Get current number of pages
int pageCountBefore = builder.Document.PageCount;
for (int i = 0; i < numParagraphs; i++)
{
builder.InsertParagraph();
}
// Get the number of pages after adding those paragraphs
int pageCountAfter = builder.Document.PageCount;
// Delete the paragraphs, we don't need them anymore
...
if (pageCountBefore != pageCountAfter)
{
return true;
}
else
{
return false;
}
}
MY problem is, that inserting paragraphs does not seem to update the builder.Document.PageCount property. Even plugging in something crazy like 5000 paragraphs does seem to modify that property. I've also tried InsertBreak() (including using BreakType.PageBreak) and Writeln() but those don't work either.
What's going on here? Is there anyway I can achieve the desired result?
UPDATE
It seems that absolutely nothing done on the DocumentBuilder parameter actually happens on the DocumentBuilder that is calling the method. In other words:
If I modify the for loop to do something like builder.InsertParagraph(i.ToString()); and then remove the code that deletes the paragraphs afterwords. I can call:
myBuilder.WillPageBreakAfter(10);
And expect to see 0-9 written to the document when it is saved, however it is not. None of the Writeln()s in the extension methods seem to do anything at all.
UPDATE 2
It appears for what ever reasons, I cannot write anything with the DocumentBuilder after accessing the page count. So calling something like Writeln() before the int pageCountBefore = builder.Document.PageCount; line works, but trying to write after that line simply does nothing.

The Document.PageCount invokes page layout. You are modifying the document after using this property. Note that when you modify the document after using this property, Aspose.Words will not update the page layout automatically. In this case you should call Document.UpdatePageLayout method.
I work with Aspose as Developer Evangelist.

And it seems I've figured it out.
From the Aspose docs:
// This invokes page layout which builds the document in memory so note that with large documents this
// property can take time. After invoking this property, any rendering operation e.g rendering to PDF or image
// will be instantaneous.
int pageCount = doc.PageCount;
The most important line here:
This invokes page layout
By "invokes page layout", they mean it calls UpdatePageLayout(), for which the docs contain this note:
However, if you modify the document after rendering and then attempt to render it again - Aspose.Words will not update the page layout automatically. In this case you should call UpdatePageLayout() before rendering again.
So basically, given my original code, I have to call UpdatePageLayout() after my Writeln()s in order to get the updated page count.
// Get current number of pages
int pageCountBefore = builder.Document.PageCount;
for (int i = 0; i < numParagraphs; i++)
{
builder.InsertParagraph();
}
// Update the page layout.
builder.Document.UpdatePageLatout();
// Get the number of pages after adding those paragraphs
int pageCountAfter = builder.Document.PageCount;

What exactly would be a better way of writing a tool that changes the values in an ini file?

So for fun, I decided to write a tool that loads a game's ini file(Outlast) and then lets you change the values for certain things.
The way I did this was, in the original ini, I found the line number the variable was on, and then removed the variable name in the code leaving only the value it held, and it wrote that to the textbox.
I am horrible at explaining things, so to clarify, in this particular ini file this is what line 42 is:
NormalWalkSpeed=200,
And in the code, I did this to write that number to a textbox:
WalkSpeedTxtBox.Text = GameLocation.DefaultGameIniLines[42].Replace("NormalWalkSpeed=", "");
And to change it, this is the button's code:
GameLocation.DefaultGameIniLines[42] = "NormalWalkSpeed=" + walkspeed;
System.IO.File.WriteAllLines(GameLocation.DefaultGameIniLocation, GameLocation.DefaultGameIniLines);
MessageBox.Show("Success! Walk speed has been set to " + WalkSpeedTxtBox.Text);
System.Media.SystemSounds.Beep.Play();
So the problem right off the bat with this is, obviously, if the ini file is not exactly the same as it was before, as in the lines change, this will no longer work and I'd have to rewrite everything.
What exactly would be a better way of doing what I want to do?

INI files are structured configuration files, which allow you to find a value given its Group and Key. Treating it as a sequential collection of lines will miss that point - and treating each line as simply a bunch of text means more work for you.
Assuming you want to write this code yourself rather than using an existing INI reading library, the best thing to do is to run over the INI file, converting it into a Dictionary of Dictionaries. A Dictionary is the data structure that makes most sense, since it allows you to access a collection of Keys by their Group name, and a value by its Key name.
Something like this:
Dictionary<string, Dictionary<string,string>> Groups = new...
string currentGroupName;
foreach (string line in File.ReadAllLines(iniFile))
{
if (lineStartsWith("[") && line.EndsWith("]"))
{
currentGroupName = line.Substring(1, line.Length-2);
Groups.Add(currentGroupName, new Dictionary<string,string>());
}
else if (line.Contains("="))
{
string[] keyValue = line.Split("=");
Groups[currentGroupName].Add(keyValue[0], keyValue[1]);
}
}
This, roughly, will allow you to iterate on all the groups and their keys, getting the values. You can then construct a list of Strings back from it to save.

you could try to find that string in file and if found you can fetch its value
using (var reader = new StreamReader(file))
{
while (!reader.EndOfStream)
{
var line = reader.ReadLine();
if (line.Contains("NormalWalkSpeed="))
{
//get its value
}
}
}

Fast way to check existance of a file in a directory with filename

I have a physical Directory structure as :
Root directory (X) -> many subdirectory in side root (1,2,3,4..) -> In each sub dir many files present.
Photos(Root)
----
123456789(Child One)
----
1234567891_w.jpg (Child two)
1234567891_w1.jpg(Child two)
1234567891_w2.jpg(Child two)
1234567892_w.jpg (Child two)
1234567892_w1.jpg(Child two)
1234567892_w2.jpg(Child two)
1234567893_w.jpg(Child two)
1234567893_w1.jpg(Child two)
1234567893_w2.jpg(Child two)
-----Cont
232344343(Child One)
323233434(Child One)
232323242(Child One)
232324242(Child One)
----Cont..
In database I have one table having huge number of names of type "1234567891_w.jpg".
NOTE : Both number of data in database and number of photos are in lacs.
I need an effective and faster way to check the presence of each name from database table to the physical directory structure.
Ex : Whether any file with "1234567891_w.jpg" name is present in physical folder inside Photos (Root).*
Please let me know if I miss any information to be given here.
Update :
I know how to find a file name existance in a directory. But I am looking for an efficient way, as it will be too much resource consuming to check each filename (from lacs of record) existance in more than 40 GB data.

You can try to group data from the database based on the directory in which they are. Sort them somehow (based on the filename for instance) and then get the array of files within that directory
string[] filePaths = Directory.GetFiles(#"c:\MyDir\");. Now you only have to compare strings.

It might sound funny or Might be I was unclear or did not provide much information..
But from the directory pattern I got one nice way to handle it is :
AS the probability of existance of the file name is only in one location and that is :
Root/SubDir/filename
I should be using :
File.Exists(Root/SubDir/filename);
i.e - Photos/123456789/1234567891_w.jpg
And I think this will be O(1)

it would seem the files are uniquely named if that's the case you can do something like this
var fileNames = GetAllFileNamesFromDb();
var physicalFiles = Directory.GetFiles(rootDir,
string.Join(",",fileNames),
SearchOptions.AllDirectories)
.Select(f=>Path.GetFileName(f));
var setOfFiles = new Hashset<string>(physicalFiles);
var notPresent = from name in fileNames
where setOfFiles.Contains(name)
select name;
First get all the names of the files from the datatbase
Then search for all the files at once searching from the root and including all subdirectories to get all the physical files
Create a Hashset for fast lookup
Then match the fileNames to the set those not in the set are selected.
the Hashset is basically just a set. That is a collection that can only incude an item once (Ie there's no duplicates) equality in the Hashset is based on HashCode and the lookup to determine if an item is in the set is O(1).
This approach requires you to store a potentially hugh Hashset in memory and depending on the size of that set it might affect the system to an extend where it's no longer optimizing the speed of the application but passes an optimum instead.
As is the case with most optimizations they are all trade offs and the key is finding the balance between all the trade offs in the context of the value the application is producing for the end user

Unfortunately their is no magic bullet which you could use to improve your performance. As always it will be a trade off between speed and memory. Also their are two sides which could lack on performance: The database site and the hdd drive i/o speed.
So to gain speed i would in a first step improve the performance of the database query to ensure that it can return the names for searching fast enough. So ensure that your query is fast and also maybe uses (im MS SQL case) keywords like READ SEQUENTIAL in this case you will already retrieve the first results while the query is still running and you don't have to wait till the query finished and gave you the names as a big block.
On the other hdd side you can either call Directory.GetFiles(), but this call would block till it iterated over all files and will give you back a big array containing all filenames. This would be the memory consuming path and take a while for the first search, but if you afterwards only work on that array you get speed improvements for all consecutive searches. Another approach would be to call Directory.EnumerateFiles() which would search the drive on the fly by every call and so maybe gain speed for the first search, but their won't happen any memory storage for the next search which improves memory footprint but costs speed, due to the fact that their is no array in your memory which could be searched. On the other hand the OS will also do some caching if detects that you iterate over the same files over and over again and some caching occurs on a lower level.
So for the check on hdd site use Directory.GetFiles() if the returned array won't blow your memory and do all your searches on this (maybe put it into a HashSet to further improve performance if filename only or full path depends on what you get from your database) and in the other case use Directory.EnumerateFiles() and hope the best for some caching done be the OS.
Update
After re-reading your question and comments, as far as i understand you have a name like 1234567891_w.jpg and you don't know which part of the name represents the directory part. So in this case you need to make an explicit search, cause iteration through all directories simply takes to much time. Here is some sample code, which should give you an idea on how to solve this in a first shot:
string rootDir = #"D:\RootDir";
// Iterate over all files reported from the database
foreach (var filename in databaseResults)
{
var fullPath = Path.Combine(rootDir, filename);
// Check if the file exists within the root directory
if (File.Exists(Path.Combine(rootDir, filename)))
{
// Report that the file exists.
DoFileFound(fullPath);
// Fast exit to continue with next file.
continue;
}
var directoryFound = false;
// Use the filename as a directory
var directoryCandidate = Path.GetFileNameWithoutExtension(filename);
fullPath = Path.Combine(rootDir, directoryCandidate);
do
{
// Check if a directory with the given name exists
if (Directory.Exists(fullPath))
{
// Check if the filename within this directory exists
if (File.Exists(Path.Combine(fullPath, filename)))
{
// Report that the file exists.
DoFileFound(fullPath);
directoryFound = true;
}
// Fast exit, cause we looked into the directory.
break;
}
// Is it possible that a shorter directory name
// exists where this file exists??
// If yes, we have to continue the search ...
// (Alternative code to the above one)
////// Check if a directory with the given name exists
////if (Directory.Exists(fullPath))
////{
//// // Check if the filename within this directory exists
//// if (File.Exists(Path.Combine(fullPath, filename)))
//// {
//// // Report that the file exists.
//// DoFileFound(fullPath);
//// // Fast exit, cause we found the file.
//// directoryFound = true;
//// break;
//// }
////}
// Shorten the directory name for the next candidate
directoryCandidate = directoryCandidate.Substring(0, directoryCandidate.Length - 1);
} while (!directoryFound
&& !String.IsNullOrEmpty(directoryCandidate));
// We did our best but we found nothing.
if (!directoryFound)
DoFileNotAvailable(filename);
}
The only furhter performance improvement i could think of, would be putting the directories found into a HashSet and before checking with Directory.Exists() use this to check for an existing directory, but maybe this wouldn't gain anything cause the OS already makes some caching in directory lookups and would then nearly as fast as your local cache. But for these things you simply have to measure your concrete problem.