We Keep Coding

IBM MQ server encodes messages

I'm sending XML to an IBM MQ Queue that contains a CDATA section. That CDATA section contains these special characters: $§)#ÜÖ&!^. For some reason, they are showing up within the MQ Queue as $�)#��&!^. This causes the other send to take it off the queue with these characters and ending up having an invalid signature because the messages no longer match up.
We've verified that the message when we do a .Put() does contain an XML string with those special characters. I've ensured that the message has .CharacterSet property assigned to it that matches what we will eventually pull off the queue.
What other places can possibly be auto-encoding the special characters when it's put on the queue? Our application is in a .NET windows environment, but the MQ server is on a Linux box. Is this something to consider?
string xmlMsg = "<message><data><![CDATA[<value>$§)#ÜÖ&!^</value>]]</data></message>"; // This is in a CDATA section.
mQMessage = new MQMessage
{
CharacterSet = 1208,
};
mQMessage.WriteBytes(xmlMsg);
_queue.Put(mQMessage);

By default MQ doesn't change the character set of your message. So by default it is the responsibility of the sending and receiving applications to agree and maintain a character set that suits both.
You can request MQ to do character set conversion either in the receiving application, when that calls a get, or on the sender channels when the message is transmitted between queue managers. But even if you request character set conversion from MQ, it is still the sending applications responsibility to actually write the data into the message using the character set the application is setting on the MQ message header.
Based on your code it seems your sending application doesn't use the correct character set when it writes the bytes to the message. If you use WriteBytes, you need to manually convert the string into bytes using the desired character set.
I'd suggest you to use the WriteString method, which is designed to use the chracter set specified in the CharacterSet property:
The WriteString method converts from Unicode to the character set encoded in CharacterSet. If CharacterSet is set to its default value, MQC.MQCCSI_Q_MGR, which is 0, no conversion takes place and CharacterSet is set to 1200. If you set CharacterSet to some other value, WriteString converts from Unicode to the alternate value.
https://www.ibm.com/support/knowledgecenter/SSFKSJ_7.5.0/com.ibm.mq.ref.dev.doc/q111220_.htm
And by the way, for debugging character set issues you have to be very careful what tools you use to check the message, as your tool needs to be able to interpret the character set of the message. For example MQ Explorer uses the character set of your workstation where you run it, so it will show every message with that one character set, so is not suitable to debug these issues. The best is to get the message off the queue without asking the QM for conversion with rfhutil for example, save it to a file and look at it with a hex editor.

Create Variable from part of file

Hi having trouble working out where to start with creating a variable from a file created by Windows Remote Assistance. I need to extract the port from the text file so i can create an ssh tunnel allowing remote assistance from anywhere.
the port appears after the ip address in 'RCTICKET="65538,1,192.168.9.22:7532,' The colon is the first one in the whole file so I think I need to search for the first ":" and then copy the 4 digits that come after it unless the port is 5 digits ( I think here checking if the 5th character is a comma meaning a 4 digit port of if its a number meaning a 5 digit port )
Any help where to start with this I've been googling for hours just can't think how to put this in a search term.
Below is an example of test.msrcincident the file created by Microsoft Remote Assistance that i need to extract the port from
<?xml version="1.0"?>
<UPLOADINFO TYPE="Escalated"><UPLOADDATA USERNAME="jon" LHTICKET="BDF9C9782B31A1BC276C029A169930ABB4490E2088169FA45A3A095258F5C54D345F4D793363E2C9 B924C5D6A38210AF2E86B3E3D33E5BEB3E35729ECDA88D5F5CE23879899768432726AF419FA2147194F4358BA2A0F245C4307EC8CAB882E2B670977562E5423C90EC336A15BA3DC57496F1EBB26B55B449B45FBD317CD4E422186EA7989F78C6FC3019BCF5831B1E060B174C5254D92448992A543079E576A66617F8B5BEA4C5961FC75C0B67F28B996CD4F1247DBC1C725B9D69B094B53AE24A533501A607CF119ED99C34F0C7210376C6564A48E25871AA32934409D981CF63F60DA956B0877AFBD669DFC321D16D55A34B9949AE0B26B6EEB473915AC416ABFC1129C08021F4011F1F0D1869BB86842C0218C03286C956FC7897B319E0B3A495EBA8ED41835E84E6BAD6B30199F6ACF191B6529DF2C5A264F578AF3B31A84997DA9C4BF1F8AD9E4931F99AE94A0E66D941F050AC0B025523148A95D24E60A6C548341C486BB40089B2088F5FE49AC966D65B728E36E0D7D76C98827335983BEC912DFC0B714DBBBFA060DE62658E7BABDB9BEB45486138950548DA62FDFD6437D0798A67D20CA1911880F58FCDA5F98FA5E0CAEF643171FE9DA8AF046" RCTICKET="65538,1,192.168.9.22:7532,*,U15FphW2EDtpPVdlHmafYLmnO/aVc+YFoFEw30tpjJ+6vJ+LspOTtaqgFoDt3bsp,*,*,P1ooZJPDyfMMTXqlz5hACdwD8F4=" PassStub="TE*0ViGNuB2T6I" RCTICKETENCRYPTED="1" DtStart="1379526042" DtLength="360" L="0"/></UPLOADINFO>
Thank you for reading

Something simple like this would get you the data you need, to some extent:
var reader = XDocument.Load("path to XML file");
var data = reader.Element("UPLOADINFO")
.Element("UPLOADDATA")
.Attribute("RCTICKET")
var values = data.Split(',');
You will need to work with that RCTICKET string to extract the value you need. It would be a bit safer to work with commas, colons, and whatnot in the context of a single attribute instead of the whole file. Caveat: When I generated an incident file, I ended up with multiple IP addresses in the RCTICKET field. I have multiple VPNs and ethernet adapters in my machine. You will have to pick the right one.
You will also want to handle failures if the XML isn't in the format we expect, or if the file is otherwise inaccessible. You can do this with a try/catch and/or checking for nulls.

Is XSS possible through the MailAddress class?

Considering I parse user input, which is supposed to be an email address, into the MailAdress class:
var mailString = Request.QueryString["mail"];
var mail = new MailAddress(mailString);
Is there any possibility left for a cross-site-scripting attack if I output the MailAddress object later in any way? For example through a Literal control in WebForms:
litMessage.Text = "Your mail address is " + mail.Address;
Is it necessary to sanitize the outpout even though I made sure that the address is a valid email address by parsing the string?
From what I could gather the RFC for mail addresses is pretty complicated, so I am unsure if cross site scripts can be hidden in a mail address considered valid by .NET.
EDIT:
MSDN says that > and < brackets are allowed in an email address:
The address parameter can contain a display name and the associated e-mail address if you enclose the address in angle brackets. For example: "Tom Smith <tsmith#contoso.com>"
So the question remains if this is enough for an XSS attack and/or if the MailMessage class does anything to escape dangerous parts.

Generally speaking, you shouldn't need to validate the output later. However, I always recommend that you do so for the following reasons:
There may be a hole somewhere in your app that doesn't validate the input properly. This could be discovered by an attacker and used for XSS. This is especially possible when many different devs are working on the app.
There may be old data in the database that was stored before implementing/updating your filter on the input. This could contain malicious code that could be used for XSS.
Attackers are very clever and can usually figure out a way to beat a filter. Microsoft puts a lot of attention on preventing this, but it's never going to perfect. It makes the attackers job that much harder if they face and outgoing filter as well and as incoming filter.
I know it's a pain to constantly filter, but there is a lot of value in doing so. A Defense-in-Depth strategy is necessary in today's world.
Edit:
Sorry I didn't really answer the second part of your question. Based on the documentation I don't get the impression that the API is focused on sanitizing as much as it is on verifying valid formatting. Therefore I don't know that it is safe to rely on it for security purposes.
However, writing your own sanitizer isn't terribly hard, and you can update it immediately if you find flaws. First run the address through a good RegEx filter (see: Regex Email validation), then recursively remove every nonvalid character in an email address (these shouldn't get through at this point but do this for comprehensiveness and in case you want to reuse the class elsewhere), then escape every character with HTML meaning. I emphasize the recursive application of the filter because attackers can take advantage of a non-recursive filter with stuff like this:
<scr<script>ipt>
Notice that a nonrecursive filter would remove the middle occurence of <script> and leave the outer occurrence in tact.

Is it necessary to sanitize the outpout
You don't 'sanitise' output, you encode it. Every string that you output into an HTML document needs to be HTML-encoded, so if there was a < character in the mail address it wouldn't matter - you'd get < in the HTML source as a result and that would display correctly as a literal < on the page.
Many ASP.NET controls automatically take care of HTML-escaping for you, but Literal does not by default because it can be used to show markup. But if you set the Mode property of the Literal control to Encode then setting the Text like you're doing is perfectly fine.
You should make sure you always use safe HTML-encoded output every time you put content into an HTML page, regardless of whether you think the values you're using will ever be able to include a < character. This is a separation-of-concerns issue: HTML output code knows all about HTML formatting, but it shouldn't know anything about what characters are OK in an e-mail address or other application field.
Leaving out an escape because you think the value is 'safe' introduces an implicit and fragile coupling between the output stage and the input stage, making it difficult to verify that the code is safe and easy to make it unsafe when you make changes.

Validate and correct invalid emailid formats in C#

Is there a way we can validate and correct invalid format emailids in C#.I got a function which can only validate but not correction.Some emailds like "abc#def.com." can be corrected.I`m fetching all emailids from database and sending them a mail,if I just remove invalid emailids,the person may loose info,so instead of removing I thought of correcting the mailid and send him the mail.
Is there a way?Or a function to do this.???
Thanks in advance.

If you have the email address as a string, then you can manipulate the string. In your example, that would be removal of the trailing period. Other than this simple example, I suggest that you think long and hard about how useful this will be. What is the context? Can you pass the mail address back to a user to get the correct address, as opposed to your best guess?Adding code will clarify your question. From your question, I don't know why you assume you can only validate, as opposed to correcting the mail address string.

You could check wheather the Mail domain exists for example like this, you can check if the Email ends with an unvalid char like "." or "," and remove this if found but you can not really "correct" wrong Emails by trying to change each char and check if the Email exists or not, and its not desired sicne you would find probably for each change you make an exissting Email adress which is not the one you really wish to reach.

No. There is no way to do this. You may have a built-in guess system that will take care of common mistakes though.
For instance, if I type my email id as abc#gmali.com, you may change it to abc#gmail.com. This still does not guarantee that the email is is now correct.
Assume I had an email id as abc#gmail.com and intentionally I typed in asd#gmail.com. Now, there is no way you can correct it. With the same intention, if I type asd#gmial.com, your code might make it correct email id as asd#gmail.com which still is incorrect.
Essentially what you are looking for is called client side validation. What ever front end you have, place validation that check if email address is correct as per syntax. For verifying if the user has given his real email, send a mail to the given address with activation link and ask them to click on it if they want to use applciation.
Edit:
If you need to just format the emails in database, you can check for common mistakes using queries/external executable. These will validate the data against a valid format which then, can be changed. What are the options you have, technology wise, for doing this?

Check if a Outlook Mail (*.msg) is Digitaly Signed or Encrypted

I have a Web application that allows to upload Outlook Mails (*.msg) with File Upload.
The customer want to forbid the store Mails that are Digitaly Signed or Encrypted.
So after uploading the Mail I should someshow check the mail if it's Signed or Encrypted.
If there a way to check that? Like a pattern in the Stream of the File?

Checking for the English words not only is valid, but it is the actual documented way.
Refer to the authority:
2.1.3.1.3 Recognizing a Message Object that Represents a Clear-Signed Message
If a Message object has a message class (PidTagMessageClass property
([MS-OXCMSG] section 2.2.1.3)) value of
"IPM.Note.SMIME.MultipartSigned" and contains exactly one Attachment
object, it SHOULD be treated as a clear-signed message. Additional
verification steps can be performed to verify that the Attachment
object is marked with the appropriate media type (for example, the
PidTagAttachMimeTag property ([MS-OXPROPS] section 2.680) has a value
of "multipart/signed") and represents a valid multipart/signed MIME
entity as specified in [RFC1847]. If the message class value is not
"IPM.Note.SMIME.MultipartSigned" but it ends with the suffix
".SMIME.MultipartSigned", the Message object MAY<7><8> be treated as a
clear-signed message.
If a Message object with a message class value of
"IPM.Note.SMIME.MultipartSigned" does not have the structure specified
in section 2.1.3.1, the behavior is undefined.
2.1.3.2.3 Recognizing a Message Object that Represents an Opaque-Signed or Encrypted S/MIME
If a Message object has the message class (PidTagMessageClass property
([MS-OXCMSG] section 2.2.1.3)) value of "IPM.Note.SMIME" and contains
exactly one Attachment object, it SHOULD be treated as an
opaque-signed message or an encrypted message. Additional verification
steps can be performed to verify that the Attachment object is marked
with the appropriate media type (for example, the PidTagAttachMimeTag
property ([MS-OXPROPS] section 2.680) is either
"application/pkcs7-mime" or "application/x-pkcs7-mime", or it is
"application/octet-stream" and filename, as specified by the
PidTagAttachFilename property ([MS-OXPROPS] section 2.671), and has a
file extension ".p7m") and represents a valid encrypted or
opaque-signed message, as specified in [RFC3852]. If the value of the
message class is not "IPM.Note.SMIME", but ends with the suffix
".SMIME", then the Message object MAY<11> be treated as an
opaque-signed message or an encrypted message.
The message class value "IPM.Note.SMIME" can be ambiguous.<12>
If a Message object has a message class value of "IPM.Note.SMIME" does
not have the appropriate structure or content as specified in section
2.1.3.2, then the behavior is undefined.
EDIT:
To be more specific, yes, you SHOULD look for a "pattern in the Stream of the file".
Specifically, if the MSG is unicode, you would scan the "__substg1.0_001A001F" stream, and check for the patterns mentioned above.
The MSG file is an OLE Structured Storage file that contains streams and storages. To get at the streams, use an OLE Storage library like OpenMCDF if you are in the C# world. There are similar ones for java, python, etc.

This blog post describes the format pretty well and another post by the same author describes exactly what you're after, which is information on rights managed mail messages.
Essentially as long as the message conforms to the file format these posts and specifications should give you all you need to check for signatures and encryption.
Checking for English words is a bad idea. What if users don't write in English and what if a psuedo-random stream of encrypted data happens to create words like "or" or "and" in some encoding they're using? It's just not reliable.
EDIT:
To clarify what I mean when I say that checking for English words is a bad idea, I mean to say that simply scanning over the file and verifying if a certain set of words is present is the bad idea. Since someone down voted this solution, I feel that they may have misunderstood what I was saying because of this ambiguity.
As another user indicated in their answer, parsing the object out and actually handling conditions in the data is fine. You can see from their post that it is the documented method and works fine, because it's based on the standards. This is similar to the information I gave here with the two posts and the format specification.

To open the message and look into, I suggest you to use Outlook Redemption. This is what I use and it works without Outlook installed on the server.
If the GetMessageFromMsgFile method returns an RDOEncryptedMessage, it means your mail is encrypted or signed.