We Keep Coding

Send XML message to MSMQ without any formatting

I need to send an XMLDocument object to MSMQ. I don’t have a class to deserialize it into (the xml may vary). The default formatter, XMLMessageFormatter, will “pretty print” the object. This causes a problem since
<text></text>
Will be converted to
<text>
</text>
(I.e. cr + spaces). The message is being read by a process using the default XMLMessageFormatter and hasn’t been an issue whilst nodes have data in them. This is an issue, however, further down the line, as a process (out of my control) will interpret these new characters as data and cause an error.
I know I could write some code to convert them using IsEmpty = true giving <text /> but I’d like a solution that doesn’t alter the object at all.
BinaryMessageFormatter will prefix the data with BOM data (receiver is not expecting that) and ActiveXMessageFormatter will double byte the string (again causing issues the other end).
I would rather like to avoid having to write a custom message formatter. I’ve tried including some options in the XMLMessageFormatter but they’ve had little effect. Any ideas would be very much appreciated.

MSMQ operates on raw blobs. You do not have to use a formatter unless you want to.
To send a message and get it back byte-for-byte identical, use the BodyStream property.
Example:
var queue = new MessageQueue(#".\private$\queueName");
var msg = new Message();
msg.BodyStream = new MemoryStream(Encoding.UTF8.GetBytes("<root><test></test></root>"));
queue.Send(msg);
Resultant message:

IBM MQ server encodes messages

I'm sending XML to an IBM MQ Queue that contains a CDATA section. That CDATA section contains these special characters: $§)#ÜÖ&!^. For some reason, they are showing up within the MQ Queue as $�)#��&!^. This causes the other send to take it off the queue with these characters and ending up having an invalid signature because the messages no longer match up.
We've verified that the message when we do a .Put() does contain an XML string with those special characters. I've ensured that the message has .CharacterSet property assigned to it that matches what we will eventually pull off the queue.
What other places can possibly be auto-encoding the special characters when it's put on the queue? Our application is in a .NET windows environment, but the MQ server is on a Linux box. Is this something to consider?
string xmlMsg = "<message><data><![CDATA[<value>$§)#ÜÖ&!^</value>]]</data></message>"; // This is in a CDATA section.
mQMessage = new MQMessage
{
CharacterSet = 1208,
};
mQMessage.WriteBytes(xmlMsg);
_queue.Put(mQMessage);

By default MQ doesn't change the character set of your message. So by default it is the responsibility of the sending and receiving applications to agree and maintain a character set that suits both.
You can request MQ to do character set conversion either in the receiving application, when that calls a get, or on the sender channels when the message is transmitted between queue managers. But even if you request character set conversion from MQ, it is still the sending applications responsibility to actually write the data into the message using the character set the application is setting on the MQ message header.
Based on your code it seems your sending application doesn't use the correct character set when it writes the bytes to the message. If you use WriteBytes, you need to manually convert the string into bytes using the desired character set.
I'd suggest you to use the WriteString method, which is designed to use the chracter set specified in the CharacterSet property:
The WriteString method converts from Unicode to the character set encoded in CharacterSet. If CharacterSet is set to its default value, MQC.MQCCSI_Q_MGR, which is 0, no conversion takes place and CharacterSet is set to 1200. If you set CharacterSet to some other value, WriteString converts from Unicode to the alternate value.
https://www.ibm.com/support/knowledgecenter/SSFKSJ_7.5.0/com.ibm.mq.ref.dev.doc/q111220_.htm
And by the way, for debugging character set issues you have to be very careful what tools you use to check the message, as your tool needs to be able to interpret the character set of the message. For example MQ Explorer uses the character set of your workstation where you run it, so it will show every message with that one character set, so is not suitable to debug these issues. The best is to get the message off the queue without asking the QM for conversion with rfhutil for example, save it to a file and look at it with a hex editor.

Deserializing ServiceBus content in Azure Logic App

I'm trying to read the content body of a message in an Azure Logic App, but I'm not having much success. I have seen a lot of suggestions which say that the body is base64 encoded, and suggest using the following to decode:
#{json(base64ToString(triggerBody()?['ContentData']))}
The base64ToString(...) part is decoding the content into a string correctly, but the string appears to contain a prefix with some extra serialization information at the start:
#string3http://schemas.microsoft.com/2003/10/Serialization/�3{"Foo":"Bar"}
There are also some extra characters in that string that are not being displayed in my browser. So the json(...) function doesn't accept the input, and gives an error instead.
InvalidTemplate. Unable to process template language expressions in
action 'HTTP' inputs at line '1' and column '2451': 'The template
language function 'json' parameter is not valid. The provided value
#string3http://schemas.microsoft.com/2003/10/Serialization/�3{"Foo":"bar" }
cannot be parsed: Unexpected character encountered while parsing value: #. Path '', line 0, position 0.. Please see https://aka.ms/logicexpressions#json for usage details.'.
For reference, the messages are added to the topic using the .NET service bus client (the client shouldn't matter, but this looks rather C#-ish):
await TopicClient.SendAsync(new BrokeredMessage(JsonConvert.SerializeObject(item)));
How can I read this correctly as a JSON object in my Logic App?

This is caused by how the message is placed on the ServiceBus, specifically in the C# code. I was using the following code to add a new message:
var json = JsonConvert.SerializeObject(item);
var message = new BrokeredMessage(json);
await TopicClient.SendAsync(message);
This code looks fine, and works between different C# services no problem. The problem is caused by the way the BrokeredMessage(Object) constructor serializes the payload given to it:
Initializes a new instance of the BrokeredMessage class from a given object by using DataContractSerializer with a binary XmlDictionaryWriter.
That means the content is serialized as binary XML, which explains the prefix and the unrecognizable characters. This is hidden by the C# implementation when deserializing, and it returns the object you were expecting, but it becomes apparent when using a different library (such as the one used by Azure Logic Apps).
There are two alternatives to handle this problem:
Make sure the receiver can handle messages in binary XML format
Make sure the sender actually uses the format we want, e.g. JSON.
Paco de la Cruz's answer handles the first case, using substring, indexOf and lastIndexOf:
#json(substring(base64ToString(triggerBody()?['ContentData']), indexof(base64ToString(triggerBody()?['ContentData']), '{'), add(1, sub(lastindexof(base64ToString(triggerBody()?['ContentData']), '}'), indexof(base64ToString(triggerBody()?['ContentData']), '}')))))
As for the second case, fixing the problem at the source simply involves using the BrokeredMessage(Stream) constructor instead. That way, we have direct control over the content:
var json = JsonConvert.SerializeObject(item);
var bytes = Encoding.UTF8.GetBytes(json);
var stream = new MemoryStream(bytes);
var message = new BrokeredMessage(stream, true);
await TopicClient.SendAsync(message);

You can use the substring function together with indexOf and lastIndexOf to get only the JSON substring.
Unfortunately, it's rather complex, but it should look something like this:
#json(substring(base64ToString(triggerBody()?['ContentData']), indexof(base64ToString(triggerBody()?['ContentData']), '{'), add(1, sub(lastindexof(base64ToString(triggerBody()?['ContentData']), '}'), indexof(base64ToString(triggerBody()?['ContentData']), '}')))))
More info on how to use these functions here.
HTH

Paco de la Cruz solution worked for me, though I had to swap out the last '}' in the expression for a '{', otherwise it finds the wrong end of the data segment.
I also split it into two steps to make it a little more manageable.
First I get the decoded string out of the message into a variable (that I've called MC) using:
#{base64ToString(triggerBody()?['ContentData'])}
then in another logic app action do the substring extraction:
#{substring(variables('MC'),indexof(variables('MC'),'{'),add(1,sub(lastindexof(variables('MC'),'}'),indexof(variables('MC'),'{'))))}
Note that the last string literal '{' is reversed from Paco's solution.
This is working for my test cases, but I'm not sure how robust this is.
Also, I've left it as a String, I do the conversion to JSON later in my logic app.
UPDATE
We have found that just occasionally (2 in several hundred runs) the text that we want to discard can contain the '{' character.
I have modified our expression to explicitly locate the start of the data segment, which for me is:
'{"IntegrationRequest"'
so the substitution becomes:
#{substring(variables('MC'),indexof(variables('MC'),'{"IntegrationRequest"'),add(1,sub(lastindexof(variables('MC'),'}'),indexof(variables('MC'),'{"IntegrationRequest"'))))}

Check if a Outlook Mail (*.msg) is Digitaly Signed or Encrypted

I have a Web application that allows to upload Outlook Mails (*.msg) with File Upload.
The customer want to forbid the store Mails that are Digitaly Signed or Encrypted.
So after uploading the Mail I should someshow check the mail if it's Signed or Encrypted.
If there a way to check that? Like a pattern in the Stream of the File?

Checking for the English words not only is valid, but it is the actual documented way.
Refer to the authority:
2.1.3.1.3 Recognizing a Message Object that Represents a Clear-Signed Message
If a Message object has a message class (PidTagMessageClass property
([MS-OXCMSG] section 2.2.1.3)) value of
"IPM.Note.SMIME.MultipartSigned" and contains exactly one Attachment
object, it SHOULD be treated as a clear-signed message. Additional
verification steps can be performed to verify that the Attachment
object is marked with the appropriate media type (for example, the
PidTagAttachMimeTag property ([MS-OXPROPS] section 2.680) has a value
of "multipart/signed") and represents a valid multipart/signed MIME
entity as specified in [RFC1847]. If the message class value is not
"IPM.Note.SMIME.MultipartSigned" but it ends with the suffix
".SMIME.MultipartSigned", the Message object MAY<7><8> be treated as a
clear-signed message.
If a Message object with a message class value of
"IPM.Note.SMIME.MultipartSigned" does not have the structure specified
in section 2.1.3.1, the behavior is undefined.
2.1.3.2.3 Recognizing a Message Object that Represents an Opaque-Signed or Encrypted S/MIME
If a Message object has the message class (PidTagMessageClass property
([MS-OXCMSG] section 2.2.1.3)) value of "IPM.Note.SMIME" and contains
exactly one Attachment object, it SHOULD be treated as an
opaque-signed message or an encrypted message. Additional verification
steps can be performed to verify that the Attachment object is marked
with the appropriate media type (for example, the PidTagAttachMimeTag
property ([MS-OXPROPS] section 2.680) is either
"application/pkcs7-mime" or "application/x-pkcs7-mime", or it is
"application/octet-stream" and filename, as specified by the
PidTagAttachFilename property ([MS-OXPROPS] section 2.671), and has a
file extension ".p7m") and represents a valid encrypted or
opaque-signed message, as specified in [RFC3852]. If the value of the
message class is not "IPM.Note.SMIME", but ends with the suffix
".SMIME", then the Message object MAY<11> be treated as an
opaque-signed message or an encrypted message.
The message class value "IPM.Note.SMIME" can be ambiguous.<12>
If a Message object has a message class value of "IPM.Note.SMIME" does
not have the appropriate structure or content as specified in section
2.1.3.2, then the behavior is undefined.
EDIT:
To be more specific, yes, you SHOULD look for a "pattern in the Stream of the file".
Specifically, if the MSG is unicode, you would scan the "__substg1.0_001A001F" stream, and check for the patterns mentioned above.
The MSG file is an OLE Structured Storage file that contains streams and storages. To get at the streams, use an OLE Storage library like OpenMCDF if you are in the C# world. There are similar ones for java, python, etc.

This blog post describes the format pretty well and another post by the same author describes exactly what you're after, which is information on rights managed mail messages.
Essentially as long as the message conforms to the file format these posts and specifications should give you all you need to check for signatures and encryption.
Checking for English words is a bad idea. What if users don't write in English and what if a psuedo-random stream of encrypted data happens to create words like "or" or "and" in some encoding they're using? It's just not reliable.
EDIT:
To clarify what I mean when I say that checking for English words is a bad idea, I mean to say that simply scanning over the file and verifying if a certain set of words is present is the bad idea. Since someone down voted this solution, I feel that they may have misunderstood what I was saying because of this ambiguity.
As another user indicated in their answer, parsing the object out and actually handling conditions in the data is fine. You can see from their post that it is the documented method and works fine, because it's based on the standards. This is similar to the information I gave here with the two posts and the format specification.

To open the message and look into, I suggest you to use Outlook Redemption. This is what I use and it works without Outlook installed on the server.
If the GetMessageFromMsgFile method returns an RDOEncryptedMessage, it means your mail is encrypted or signed.

How do I safely create an XPathNavigator against a Stream in C#?

Given a Stream as input, how do I safely create an XPathNavigator against an XML data source?
The XML data source:
May possibly contain invalid hexadecimal characters that need to be removed.
May contain characters that do not match the declared encoding of the document.
As an example, some XML data sources in the cloud will have a declared encoding of utf-8, but the actual encoding is windows-1252 or ISO 8859-1, which can cause an invalid character exception to be thrown when creating an XmlReader against the Stream.
From the StreamReader.CurrentEncoding property documentation: "The current character encoding used by the current reader. The value can be different after the first call to any Read method of StreamReader, since encoding autodetection is not done until the first call to a Read method." This seems indicate that CurrentEncoding can be checked after the first read, but are we stuck storing this encoding when we need to write out the XML data to a Stream?
I am hoping to find a best practice for safely creating an XPathNavigator/IXPathNavigable instance against an XML data source that will gracefully handle encoding an invalid character issues (in C# preferably).

I had a similar issue when some XML fragments were imported into a CRM system using the wrong encoding (there was no encoding stored along with the XML fragments).
In a loop I created a wrapper stream using the current encoding from a list. The encoding was constructed using the DecoderExceptionFallback and EncoderExceptionFallback options (as mentioned by #Doug). If a DecoderFallbackException was thrown during processing the original stream is reset and the next-most-likely encoding is used.
Our encoding list was something like UTF-8, Windows-1252, GB-2312 and US-ASCII. If you fell off the end of the list then the stream was really bad and was rejected/ignored/etc.
EDIT:
I whipped up a quick sample and basic test files (source here). The code doesn't have any heuristics to choose between code pages that both match the same set of bytes, so a Windows-1252 file may be detected as GB2312, and vice-versa, depending on file content, and encoding preference ordering.

It's possible to use the DecoderFallback class (and a few related classes) to deal with bad characters, either by skipping them or by doing something else (restarting with a new encoding?).

When using a XmlTextReader or something similiar, the reader itself will figure out the encoding declared in the xml file.