We Keep Coding

OpenPop.net, extract just last message from chain

I am using Openpop.net with c#. I can get the emails but want to extract the body text not the full email chain text. Is there a way of just getting the message body without the RE: parts?
Thanks.

Unfortunately, there's no easy way to do what you want to do because the text body of an email is free-form text and each mail client can (and often does) use a different way of quoting the original message text that it is replying to.
Usually, but not always, the older message(s) in the conversation will be rewritten to have a "> " sequence at the start of each line.
If that is the case, you can filter those lines out to get the latest message text of the conversation.
Keep in mind, however, that many people will do what is called "replying inline" where there interlace their responses with the quoted questions from a previous email. For example:
Hi Joe,
My answers are below.
On Mon, Feb 8 2021, Joe Smith wrote:
>
> Jake,
>
> Have you sent in that TPS report yet?
Nope, not yet. I was hoping to send it in today.
> If not, I've got a few facts & figures that should probably be included.
Sure, come by my desk after lunch and we'll work those details into my final report.
- Jake

C# Regex: How to break up plain text string

I have a problem which I am wondering how to solve.
I have a String I read in from a pdf file that has a list of questions.
It's in the format of:
QUESTION NO: 1
xxxxxxx (question text)
A) xxxx (multiple choice) B) xxxx C) xxxx ...
Answer: xxxxx
QUESTION NO: 2
xxxxxxx (question text)
.... (etc)
There are about 200 questions in the list.
I am trying to use Regex to break up the text so each question can be in a separate string.
I've done this before with html and xml documents, but they were easy since there are a lot of identifying tags like double quotes, brackets, and parentheses.
But I am clueless as to how to do this with just text. I've tried a lot of combinations, but it just seems like I can't get the right format:
var questionPattern = #"QUESTION NO:(.*)QUESTION NO:";
var questionMatch = Regex.Matches(pdfText, questionPattern, RegexOptions.Singleline);
I was wondering, is there a way to do:
var questionPattern = #"(?<=QUESTION NO:)[^QUESTION NO:]*";
Where the [^QUESTION NO:]* reads everything after each Question header until it stops when it comes to the next Question header?
Obviously this is the wrong format, but I hope people will understand what I'm trying to get at.
Any help would be greatly appreciated.
Thanks!

This is probably the best you're going to get - dependent on Answer. Lookaheads would need to be conditional, and would break the entire expression.
(QUESTION NO: \d+[\S\s]*?Answer.*\n*)
Working example: http://regex101.com/r/nC6yA1

Extracting data from text using templates

I'm building a web service which receives emails from a number of CRM-systems. Emails typically contain a text status e.g. "Received" or "Completed" as well as a free text comment.
The formats of the incoming email are different, e.g. some systems call the status "Status: ZZZZZ" and some "Action: ZZZZZ". The free text sometimes appear before the status and somethings after. Status codes will be mapped to my systems interpretation and the comment is required too.
Moreover, I'd expect that the the formats change over time so a solution that is configurable, possibly by customers providing their own templates thru a web interface would be ideal.
The service is built using .NET C# MVC 3 but I'd be interested in general strategies as well as any specific libraries/tools/approaches.
I've never quite got my head around RegExp. I'll make a new effort in case it is indeed the way to go. :)

I would go with regex:
First example, if you had only Status: ZZZZZ- like messages:
String status = Regex.Match(#"(?<=Status: ).*");
// Explanation of "(?<=Status: ).*" :
// (?<= Start of the positive look-behind group: it means that the
// following text is required but won't appear in the returned string
// Status: The text defining the email string format
// ) End of the positive look-behind group
// .* Matches any character
Second example if you had only Status: ZZZZZ and Action: ZZZZZ - like messages:
String status = Regex.Match(#"(?<=(Status|Action): ).*");
// We added (Status|Action) that allows the positive look-behind text to be
// either 'Status: ', or 'Action: '
Now if you want to give the possibility to the user to provide its own format, you could come up with something like:
String userEntry = GetUserEntry(); // Get the text submitted by the user
String userFormatText = Regex.Escape(userEntry);
String status = Regex.Match(#"(?<=" + userFormatText + ").*");
That would allow the user to submit its format, like Status:, or Action:, or This is my friggin format, now please read the status -->...
The Regex.Escape(userEntry) part is important to ensure that the user doesn't break your regex by submitting special character like \, ?, *...
To know if the user submits the status value before or after the format text, you have several solutions:
You could ask the user where his status value is, and then build you regex accordingly:
if (statusValueIsAfter) {
// Example: "Status: Closed"
regexPattern = #"(?<=Status: ).*";
} else {
// Example: "Closed:Status"
regexPattern = #".*(?=:Status)"; // We use here a positive look-AHEAD
}
Or you could be smarter and introduce a system of tags for the user entry. For instance, the user submits Status: <value> or <value>=The status and you build the regex by replacing the tags string.

parsing email text reply/forward

I am creating a web based email client using c# asp.net.
What is confusing is that various email clients seem to add the original text in alot of different ways when replying by email.
What I was wondering is that, if there is some sort of standardized way, to disambiguate this process?
Thank you
-Theo

I was thinking:
public String cleanMsgBody(String oBody, out Boolean isReply)
{
isReply = false;
Regex rx1 = new Regex("\n-----");
Regex rx2 = new Regex("\n([^\n]+):([ \t\r\n\v\f]+)>");
Regex rx3 = new Regex("([0-9]+)/([0-9]+)/([0-9]+)([^\n]+)<([^\n]+)>");
String txtBody = oBody;
while (txtBody.Contains("\n\n")) txtBody = txtBody.Replace("\n\n", "\n");
while (new Regex("\n ").IsMatch(txtBody)) txtBody = (new Regex("\n ")).Replace(txtBody, "\n");
while (txtBody.Contains(" ")) txtBody = txtBody.Replace(" ", " ");
if (isReply = (isReply || rx1.IsMatch(txtBody)))
txtBody = rx1.Split(txtBody)[0]; // Maybe a loop through would be better
if (isReply = (isReply || rx2.IsMatch(txtBody)))
txtBody = rx2.Split(txtBody)[0]; // Maybe a loop through would be better
if (isReply = (isReply || rx3.IsMatch(txtBody)))
txtBody = rx3.Split(txtBody)[0]; // Maybe a loop through would be better
return txtBody;
}

There isn't a standardized way, but a sensible heuristic will get you a good distance.
Some algorithms classify lines based on their initial character(s) and by comparing the text to a corpus of marked up text, yielding a statistical probability for each line that it is a) part of the same block as the next/previous one and b) quoted text, a signature, new text, etc.
It'd be worth trying out some of the most popular e-mail clients and creating and comparing some sample messages to see what the differences are. Usenet newsgroups may also help you build a reasonable corpus of messages to work from. HTML e-mail adds an extra level of complexity of course, tthough most compliant mail clients will included the corresponding plain text as well. Different languages also cause issues, as clients which can parse "Paul wrote:" may fall over at "Pablo ha scritto:".

Not really, no.
The original RFC for Internet Message talks about the in-reply-to header, but doesn't specify the format of the body.
As you've found, different clients add the original text in different ways, implying there's not a standard, coupled with the fact that users will do things differently as well:
Plain text, "rich text", HTML will all have a different way of separating the reply from the original
In Outlook I can choose from the following options when replying to a message:
Do not include
Attach original message
Include original message text
Include and indent original message text
Prefix each line of the original message
On top of that, I often send and receive replies that state "Responses in-line" where my comments are intermingled with the original message, so the original message no longer exists in its original form anyway.

Some heuristics you can try are
-Any number of > characters
-Looking for "wrote: " (be very careful with this one)
Also you can try relating the Message ID field with the In Reply To field
And finally, if you cannot find a good library to do this, it is time to start this project. No more parsing emails the Cthulhu way :)

Is it possible to programmatically 'clean' emails?

Does anyone have any suggestions as to how I can clean the body of incoming emails? I want to strip out disclaimers, images and maybe any previous email text that may be also be present so that I am left with just the body text content. My guess is it isn't going to be possible in any reliable way, but has anyone tried it? Are there any libraries geared towards this sort of thing?

In email, there is couple of agreed markings that mean something you wish to strip. You can look for these lines using regular expressions. I doubt you can't really well "sanitize" your emails, but some things you can look for:
Line starting with "> " (greater than then whitespace) marks a quote
Line with "-- " (two hyphens then whitespace then linefeed) marks the beginning of a signature, see Signature block on Wikipedia
Multipart messages, boundaries start with --, beyond that you need to do some searching to separate the message body parts from unwanted parts (like base64 images)
As for an actual C# implementation, I leave that for you or other SOers.

A few obvious things to look at:
if the mail is anything but pure plain text, the message will be multi-part mime. Any part whose type is "image/*" (image/jpeg, etc), can probably be dropped. In all likelyhood any part whose type is not "text/*" can go.
A HTML message will probably have a part of type "multipart/alternative" (I think), and will have 2 parts, one "text/plain" and one "text/html". The two parts should be just about equivalent, so you can drop the HTML part. If the only part present is the HTML bit, you may have to do a HTML to plain text conversion.
The usual format for quoted text is to precede the text by a ">" character. You should be able to drop these lines, unless the line starts ">From", in which case the ">" has been inserted to prevent the mail reader from thinking that the "From " is the start of a new mail.
The signature should start with "-- \r\n", though there is a very good chance that the trailing space will be missing.

Version 3 of OSBF-Lua has a mail-parsing library that will handle the MIME and split a message into its MIME parts and so on. I currently have a mess of Lua scripts that do
stuff like ignore most non-text attachments, prefer plain text to HTML, and so on. (I also wrap long lines to 80 characters while trying to preserve quoting.)
As far as removing previously quoted mail, the suggestions above are all good (you must subscribe to some ill-mannered mailing lists).
Removing disclaimers reliably is probably going to be hard. My first cut would be simply to maintain a library of disclaimers that would be stripped off the end of each mail message; I would write a script to make it easy for me to add to the library. For something more sophisticated I would try some kind of machine learning.
I've been working on spam filtering since Feb 2007 and I've learned that anything to do with email is a mess. A good rule of thumb is that whatever you want to do is a lot harder than you think it is :-(

Given your question "Is it possible to programmatically ‘clean’ emails?", I'd answer "No, not reliably".
The danger you face isn't really a technological one, but a sociological one.
It's easy enough to spot, and filter out, some aspects of the messages - like images. Filtering out signatures and disclaimers is, likewise, possible to achieve (though more of a challenge).
The real problem is the cost of getting it wrong.
What happens if your filter happens to remove a critical piece of the message? Can you trace it back to find the missing piece, or is your filtering desctructive? Worse, would you even notice that the piece was missing?
There's a classic comedy sketch I saw years ago that illustrates the point. Two guys working together on a car. One is underneath doing the work, the other sitting nearby reading instructions from a service manual - it's clear that neither guy knows what he's doing, but they're doing their best.
Manual guy, reading aloud: "Undo the bold in the centre of the oil pan ..." [turns page]
Tool guy: "Ok, it's out."
Manual guy: "... under no circumstances."

If you creating your own application i'd look into Regex, to find text and replace it. To make the application a little nice, i'd create a class Called Email and in that class i have a property called RAW and a property called Stripped.
Just some hints, you'll gather the rest when you look into regex!

SigParser has an assembly you can use in .NET. It gives you the body back in both HTML and text forms with the rest of the stuff stripped out. If you give it an HTML email it will convert the email to text if you need that.
var parser = new SigParser.EmailParsing.EmailParser();
var result = await parser.GetCleanedBodyAsync(new SigParser.EmailParsing.Models.CleanedBodyInput {
FromEmailAddress = "john.smith#example.com",
FromName = "John Smith",
TextBody = #"Hi Mark,
This is my message.
Thanks
John Smith
888-333-4434"
});
// This would print "Hi Mark,\r\nThis is my message."
Console.WriteLine(result.CleanedBodyPlain);