We Keep Coding

Parsing log file, ambiguous delimiter

I have to parse a log file and not sure how to best take different pieces of each line. The problem I am facing is original developer used ':' to delimit tokens which was a bit idiotic since the line contains timestamp which itself contains ':'!
A sample line looks something like this:
transaction_date_time:[systemid]:sending_system:receiving_system:data_length:data:[ws_name]
2019-05-08 15:03:13:494|2019-05-08 15:03:13:398:[192.168.1.2]:ABC:DEF:67:cd71f7d9a546ec2b32b,AACN90012001000012,OPNG:[WebService.SomeName.WebServiceModule::WebServiceName]
I have no problem reading the log file and accessing each line but no sure how to get the pieces parsed?

Since the input string is not exactly splittable, because of the delimiter char is also part of the content, a simple regex expression can be used instead.
Simple but probably fast enough, even with the default settings.
The different parts of the input string can be separated with these capturing groups:
string pattern = #"^(.*?)\|(.*?):\[(.*?)\]:(.*?):(.*?):(\d+):(.*?):\[(.*)\]$";
This will give you 8 groups + 1 (Group[0]) which contains the whole string.
Using the Regex class, simply pass a string to parse (named line, here) and the regex (named pattern) to the Match() method, using default settings:
var result = Regex.Match(line, pattern);
The Groups.Value property returns the result of each capturing group. For example, the two dates:
var dateEnd = DateTime.ParseExact(result.Groups[1].Value, "yyyy-MM-dd hh:mm:ss:ttt", CultureInfo.InvariantCulture),
var dateStart = DateTime.ParseExact(result.Groups[2].Value, "yyyy-MM-dd hh:mm:ss:ttt", CultureInfo.InvariantCulture),
The IpAddress is extracted with: \[(.*?)\].
You could give a name to this grouping, so it's more clear what the value refers to. Simply add a string, prefixed with ? and enclosed in <> or single quotes ' to name the grouping:
...\[(?<IpAddress>.*?)\]...
Note, however, that naming a group will modify the Regex.Groups indexing: the un-named groups will be inserted first, the named groups after. So, naming only the IpAddress group will cause it to become the last item, Groups[8]. Of course you can name all the groups and the indexing will be preserved.
var hostAddress = IPAddress.Parse(result.Groups["IpAddress"].Value);
This patter should allow a medium machine to parse 130,000~150,000 strings per second.
You'll have to test it to find the perfect pattern. For example, the first match (corresposnding to the first date): (.*?)\|, is much faster if non-greedy (using the *? lazy quantifier). The opposite for the last match: \[(.*)\]. The pattern used by jdweng is even faster than the one used here.
See Regex101 for a detailed description on the use and meaning of each token.

Using Regex I was able to parse everything. It looks like the data came from excel because the faction of seconds has a colon instead of a period. c# does not like the colon so I had to replace colon with a period. I also parsed from right to left to get around the colon issues.
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Text.RegularExpressions;
using System.IO;
namespace ConsoleApplication3
{
class Program1
{
const string FILENAME = #"c:\temp\test.txt";
static void Main(string[] args)
{
string line = "";
int rowCount = 0;
StreamReader reader = new StreamReader(FILENAME);
string pattern = #"^(?'time'.*):\[(?'systemid'[^\]]+)\]:(?'sending'[^:]+):(?'receiving'[^:]+):(?'length'[^:]+):(?'data'[^:]+):\[(?'ws_name'[^\]]+)\]";
while ((line = reader.ReadLine()) != null)
{
line = line.Trim();
if (line.Length > 0)
{
if (++rowCount != 1) //skip header row
{
Log_Data newRow = new Log_Data();
Log_Data.logData.Add(newRow);
Match match = Regex.Match(line, pattern, RegexOptions.RightToLeft);
newRow.ws_name = match.Groups["ws_name"].Value;
newRow.data = match.Groups["data"].Value;
newRow.length = int.Parse(match.Groups["length"].Value);
newRow.receiving_system = match.Groups["receiving"].Value;
newRow.sending_system = match.Groups["sending"].Value;
newRow.systemid = match.Groups["systemid"].Value;
//end data is first then start date is second
string[] date = match.Groups["time"].Value.Split(new char[] {'|'}).ToArray();
string replacePattern = #"(?'leader'.+):(?'trailer'\d+)";
string stringDate = Regex.Replace(date[1], replacePattern, "${leader}.${trailer}", RegexOptions.RightToLeft);
newRow.startDate = DateTime.Parse(stringDate);
stringDate = Regex.Replace(date[0], replacePattern, "${leader}.${trailer}", RegexOptions.RightToLeft);
newRow.endDate = DateTime.Parse(stringDate );
}
}
}
}
}
public class Log_Data
{
public static List<Log_Data> logData = new List<Log_Data>();
public DateTime startDate { get; set; } //transaction_date_time:[systemid]:sending_system:receiving_system:data_length:data:[ws_name]
public DateTime endDate { get; set; }
public string systemid { get; set; }
public string sending_system { get; set; }
public string receiving_system { get; set; }
public int length { get; set; }
public string data { get; set; }
public string ws_name { get; set; }
}
}

c# Regex.Split returns multiple instances of the same word

String: WITH Date = {GETDATE()} AND Customer = 8824
Pattern: (([A-Za-z0-9#=><]+)|((\(([^)]+)\)))|(('[^,;]+'))|({[A-Za-z0-9#=><()]+}))+
Output:
WITH
WITH
Date
Date
=
=
{GETDATE()}
{GETDATE()}
AND
AND
Customer
Customer
=
=
8824
8824
Obviously, the desired output is one instance of each word and not multiple.
I haven't included any flags.
Is there anything wrong with the pattern or should I include any flag?
Thanks.

Why Regex?
https://regexr.com/3isdf --> ([{}()A-Za-z0-9#=><]+)
If you include parentheses or curly-/square brackets inside [] as first characters, they are treated to be literally.
Simpler would be a string.Split(..) like this:
using System;
public class Program
{
public static void Main()
{
var t =
"WITH Date = {GETDATE()} AND Customer = 8824"
.Split(" ".ToCharArray(), // add other unwanted chars to splitter
StringSplitOptions.RemoveEmptyEntries);
foreach(var part in t)
Console.WriteLine(part);
}
}
Output:
WITH
Date
=
{GETDATE()}
AND
Customer
=
8824

Parsing string with 3 hyphens C#

I have string COO70-123456789-12345-1. I need to parse based on the "-" not the length of the substrings and use the parsed values. I have tried using Regular expressions but having issues.Please suggest.
Also after I have split the values I need to use each values: string A = COO70, int B = 123456789, int C = 12345, short D = 1 . How do I get it in different variables A,B,C,D.
string[] results = UniqueId.Split('-');
string A = results[0];
string B = results[1];
string C = results[2];
int k_id = Convert.ToInt32(k_id);
string D = results[3];
short seq = Convert.ToInt16(seq);

string s = "COO70-123456789-12345-1";
string[] split = s.Split('-'); //=> {"COO70", "123456789", "12345", "1"}

Use indexOf
To find everything before the first hyphen use:
string original= "COO70-123456789-12345-1";
string toFirstHyphen=original.Substring(0,original.IndexOf("-"));
Or if you want every section use split like the above example.

You can verify whether the input is formatted as you want and then split to get the parts.
using System;
using System.Text.RegularExpressions;
class Program
{
static void Main()
{
string pattern = #"(?x)^(\w+-\w+-\w+-\w+)$";
Regex reg = new Regex(pattern);
string test = "word-6798-3401-001";
if((reg.Match(test).Success))
foreach (var x in test.Split(new char[] {'-'}))
Console.WriteLine(x);
}
}

So it sounds like you first want to split it, but then store it into your values.
I would do something like this:
var myString = "COO70-123456789-12345-1";
var stringSet = myString.Split("-"); // This returns an array of values.
Now we need to verify that we receive only 4 sub strings:
if (stringSet.Count != 4)
throw Exception e; // Throw a real exception, not this
From here we need to know what order our strings should be in and assign them:
var A = stringSet[0];
var B = stringSet[1];
var C = stringSet[2];
var D = stringSet[3];
While this should answer your question as posed, I would recommend you work with stringSet differently personally.

Set String.Format at runtime

I have an XML File that I want to allow the end user to set the format of a string.
ex:
<Viewdata>
<Format>{0} - {1}</Format>
<Parm>Name(property of obj being formatted)</Parm>
<Parm>Phone</Parm>
</Viewdata>
So at runtime I would somehow convert that to a String.Format("{0} - {1}", usr.Name, usr.Phone);
Is this even possible?

Of course. Format strings are just that, strings.
string fmt = "{0} - {1}"; // get this from your XML somehow
string name = "Chris";
string phone = "1234567";
string name_with_phone = String.Format(fmt, name, phone);
Just be careful with it, because your end user might be able to disrupt the program. Do not forget to FormatException.

I agree with the other posters who say you probably shouldn't be doing this but that doesn't mean we can't have fun with this interesting question. So first of all, this solution is half-baked/rough but it's a good start if someone wanted to build it out.
I wrote it in LinqPad which I love so Dump() can be replaced with console writelines.
void Main()
{
XElement root = XElement.Parse(
#"<Viewdata>
<Format>{0} | {1}</Format>
<Parm>Name</Parm>
<Parm>Phone</Parm>
</Viewdata>");
var formatter = root.Descendants("Format").FirstOrDefault().Value;
var parms = root.Descendants("Parm").Select(x => x.Value).ToArray();
Person person = new Person { Name = "Jack", Phone = "(123)456-7890" };
string formatted = MagicFormatter<Person>(person, formatter, parms);
formatted.Dump();
/// OUTPUT ///
/// Jack | (123)456-7890
}
public string MagicFormatter<T>(T theobj, string formatter, params string[] propertyNames)
{
for (var index = 0; index < propertyNames.Length; index++)
{
PropertyInfo property = typeof(T).GetProperty(propertyNames[index]);
propertyNames[index] = (string)property.GetValue(theobj);
}
return string.Format(formatter, propertyNames);
}
public class Person
{
public string Name { get; set; }
public string Phone { get; set; }
}

XElement root = XElement.Parse (
#"<Viewdata>
<Format>{0} - {1}</Format>
<Parm>damith</Parm>
<Parm>071444444</Parm>
</Viewdata>");
var format =root.Descendants("Format").FirstOrDefault().Value;
var result = string.Format(format, root.Descendants("Parm")
.Select(x=>x.Value).ToArray());

What about specify your format string with parameter names:
<Viewdata>
<Format>{Name} - {Phone}</Format>
</Viewdata>
Then with something like this:
http://www.codeproject.com/Articles/622309/Extended-string-Format
you can do the work.

Short answer is yes but it depends on the variety of your formatting options how difficult it is going to be.
If you have some formatting strings that accept 5 parameter and some other that accept only 3 that you need to take that into account.
I’d go with parsing XML for params and storing these into array of objects to pass to String.Format function.

You can use System.Linq.Dynamic and make entire format command editable:
class Person
{
public string Name;
public string Phone;
public Person(string n, string p)
{
Name = n;
Phone = p;
}
}
static void TestDynamicLinq()
{
foreach (var x in new Person[] { new Person("Joe", "123") }.AsQueryable().Select("string.Format(\"{0} - {1}\", it.Name, it.Phone)"))
Console.WriteLine(x);
}

Dollar $ symbol in mongodb

I have a couple of collections that has a string like this.
This is a cool stock. $AAPL. Let's buy it.
This is a cool stock. $MSFT. Let's buy it.
This is a cool stock. $GOOG. Let's buy it.
How do I find the APPL one.
i use something like this db.collection_name.find(fieldname: /$AAPL/) but it doesn't like the dollar symbol. If i run it without the $ in it, it works fine. But I only want the result when the $AAPL is in the text.
Cheers.

A complete C# example:
// sample class with a property that could contain the sample string
// in your example, "This is a cool stock. $MSFT"
public class Talk {
public string Message { get; set; }
}
var client = new MongoClient("mongodb://localhost");
var server = client.GetServer();
var database = server.GetDatabase("stocktalk");
var collection = database.GetCollection<Talk>("talk");
var query = Query<Talk>.EQ(m => m.Message,
new BsonRegularExpression(#"\$MSFT"));
// get all of the Talk objects that match
var matches = collection.FindAs<Talk>(query);
Also note that this is a very inefficient query in general as it would need to search through all documents in the collection to find a match. You might want to consider storing the stock ticker symbols in a distinct array property as part of the document and using $in to find them (you could then use an index for example and it would be very fast to find matching strings):
public class Talk {
public string Message { get; set; }
public string[] TickerSymbols { get; set; }
}
var query = Query<Talk>.In(m => m.TickerSymbols, new string[]{"$MSFT"});

$ is a special character in regular expressions; it matches the end of the original string.
To match a literal $ character, you need to escape it with a backslash:
db.collection_name.find(fieldname: /\$AAPL/)