We Keep Coding

IBM MQ server encodes messages

I'm sending XML to an IBM MQ Queue that contains a CDATA section. That CDATA section contains these special characters: $§)#ÜÖ&!^. For some reason, they are showing up within the MQ Queue as $�)#��&!^. This causes the other send to take it off the queue with these characters and ending up having an invalid signature because the messages no longer match up.
We've verified that the message when we do a .Put() does contain an XML string with those special characters. I've ensured that the message has .CharacterSet property assigned to it that matches what we will eventually pull off the queue.
What other places can possibly be auto-encoding the special characters when it's put on the queue? Our application is in a .NET windows environment, but the MQ server is on a Linux box. Is this something to consider?
string xmlMsg = "<message><data><![CDATA[<value>$§)#ÜÖ&!^</value>]]</data></message>"; // This is in a CDATA section.
mQMessage = new MQMessage
{
CharacterSet = 1208,
};
mQMessage.WriteBytes(xmlMsg);
_queue.Put(mQMessage);

By default MQ doesn't change the character set of your message. So by default it is the responsibility of the sending and receiving applications to agree and maintain a character set that suits both.
You can request MQ to do character set conversion either in the receiving application, when that calls a get, or on the sender channels when the message is transmitted between queue managers. But even if you request character set conversion from MQ, it is still the sending applications responsibility to actually write the data into the message using the character set the application is setting on the MQ message header.
Based on your code it seems your sending application doesn't use the correct character set when it writes the bytes to the message. If you use WriteBytes, you need to manually convert the string into bytes using the desired character set.
I'd suggest you to use the WriteString method, which is designed to use the chracter set specified in the CharacterSet property:
The WriteString method converts from Unicode to the character set encoded in CharacterSet. If CharacterSet is set to its default value, MQC.MQCCSI_Q_MGR, which is 0, no conversion takes place and CharacterSet is set to 1200. If you set CharacterSet to some other value, WriteString converts from Unicode to the alternate value.
https://www.ibm.com/support/knowledgecenter/SSFKSJ_7.5.0/com.ibm.mq.ref.dev.doc/q111220_.htm
And by the way, for debugging character set issues you have to be very careful what tools you use to check the message, as your tool needs to be able to interpret the character set of the message. For example MQ Explorer uses the character set of your workstation where you run it, so it will show every message with that one character set, so is not suitable to debug these issues. The best is to get the message off the queue without asking the QM for conversion with rfhutil for example, save it to a file and look at it with a hex editor.

SQL Service Broker Queue Handling Foreign Characters in the message

Setup: I have a form built in asp.net/c# that, on submit, XML serializes it's object model and calls a stored procedure with that XML serialized data as the sole parameter. The stored procedure sends that data to a sql broker queue. The message sent to the broker queue must be valid XML that obeys the message contract set on the queue. That message is picked up by BizTalk and processed accordingly.
Problem: Originally the data submitted to me was just regular English characters (essentially held to ASCII charset) but a requirement is on the horizon to support foreign characters as well. In my testing, I've noticed that if I try to submit something with foreign characters (chinese, arabic, etc), I get an error in the queue and the message that gets to BizTalk ends up with "?????" in place of the foreign characters. I've added the utf=16 xml header to the top of the document, but that doesn't seem to help.
Question: Is there a way I can cast the incoming XML message as nvarchar and still have it be considered valid XML by the queue? I don't want to change the actual type on the queue or recreate it. I'd prefer to change the message in the stored proc alone in some way that allows it to get on the queue.
Thanks in advance for your help.

I ended up handling this by encoding the characters using HTML5 and then security escaping them. I ran into some issues using the HttpUtility library to handle this encoding so I added the method that I used to handle the encoding.
I wish I could give direct credit for this, I can't remember where I found this but thank you to whomever it was:
private string EncodeToHTML(string text)
{
// call the normal HtmlEncode first
char[] chars = HttpUtility.HtmlEncode(text).ToCharArray();
StringBuilder encodedValue = new StringBuilder();
foreach (char c in chars)
{
if ((int)c > 127) // above normal ASCII
encodedValue.Append("&#" + (int)c + ";");
else
encodedValue.Append(c);
}
return encodedValue.ToString();
}

How can I decode PDU with GSMComm or PduBitPacker?

I want to Decode PDU Text Retrieved from GSM Modem in C#, How can I decode it with GSMComm or PduBitPacker?
Please answer me with a code
TY
(P.S. I have encoded text and I do not need to get it from Modem)

Make sure you include PDUConverter.dll in your project.
IncomingSmsPdu sms = IncomingSmsPdu.Decode("0791893905004100640C9189398978168400003160915151238110050003110202C26735B94D87DF41", true);
Console.WriteLine(sms.UserDataText);
The first argument of Decode is your RAW PDU string including header at the end, the second argument is a flag telling the decoder that a header is present.
Inside of UserDataText you will find the SMS text.
If you see estrange characters at the beginning, it is because you have a "Smart Message", that is, a multi-part SMS that should be concatenated into a single large message, this is a trick invented by Nokia.
The class GsmComm.PduConverter.SmartMessaging.SmartMessageDecoder could be of help if you want to deal with this smart messaging thing.

Encoding/Decoding PDU 7bit septets GSM long messages

I'm looking for Encoding/Decoding algorithm.
I have tried this:
http://codeproblem.hamaraquetta.com/articles/languages/81-net-framework/76-encoding-sms-in-pdu-format-in-net?showall=&start=1
and no luck. :(
Here is what I'm trying
This is the text:
This is a long text message greater than 160 characters. You can encode it to PDU format using the SMS-PDU lib for .NET, It also supports UCS-2 encoding, and special characters like { [ ] } are also supported. Its quite simple to use in your code.
From this text there should 2 messages encoded to septets and after I should be able to submit the message.
This is the result i get:
Part 1:
0041000C917952205197720000A00500033F0201A8E8F41C949E83C220F6DB7D06D1CB783AA85D9ECFC3E732E82C2F87E96539888E0EBB41311B0C344687E5E131BD2C9FBB40D9771D340EBB4165F7F84D2E83D27410FD0D8212AB20F35BDE0ED341F579DA7D06D1D165D0B4396D418955103B2D0699DF7290CB59A4B240493A28CC9EBF41F33A1CFE96D3E7A0EA70DA9281CAEEF19B9C769F59
Part 2:
0041000C917952205197720000690500033F020240613719348797C7E9301B344687E5E131BD2C9F83D8E97519B44181363CD0C607DAA4406179191466CFDFA0791D0E7FCBE965B20B94A4CF41F17A9A5E06CDD36D38BB0CA2BF41F57919947683F2EFBA1C347E93CB2E
this is doesn't work.
How do I solve this?
Btw: this is the phonenumber i know it's important.
+972502157927

Library works completely correctly. ComposeLongSms() returns a string array of PDUs and you should send("submit" as you said) all these PDUs to your GSM modem like separate SMSes. Any concatenating won't work, you can notice that each PDU starts with the same part, which contains encoded additional information for outgoing SMS. You can verify your PDUs here

What is the best way to correctly read a HTTP POST request from a socket byte stream?

I am accepting a POST request like so:
Socket connection = m_connection;
Byte[] receive = new Byte[1024];
int received = connection.Receive(receive);
Console.WriteLine(received.ToString());
string request = Encoding.ASCII.GetString(receive);
Console.WriteLine(request);
The post values end up being weird, if I post text values a lot of times they end up with a lot of +'s behind them. If I post C:\Users\John Doe\wwwroot, it ends up being: C%3A%5CUsers%5John+Doe%5Cwwwroot
index.html becomes index.html++++++++++++++++++++++++++++++++
It seems I am getting the Encoding wrong somehow, however I tried multiple encodings, and they have same weirdness. What is the best way to correctly read a HTTP POST request from a socket byte stream?

You need to trim the byte array receive that you are passing to the GetString method. Right now, you are passing all 1024 bytes, so the GetString method is trying to encode those as best it can.
You need to use the received variable to indicate the bounds for the string you are encoding.

You should use System.Web.HttpUtility.UrlDecode not Encoding.ASCII to peform the decoding.
You will probably get away with passing Encoding.Default as the second parameter to this static method.
Your are seeing the result of a HTML form POST which encodes the values as if they were being appended to a URL as a search string. Hence it is a & delimited set of name=value pairs. Any out-of-band characters are encoded to their hex value %xx.
The UrlDecode method will decode all this for you.
As other have stated you really need to chunk the stream in, it may be bigger that 1K.
Strictly speaking you should check the Content-Type header for any ;CharSet= attribute. If present you need to ensure the character encode you pass to UrlDecode is appropriate to that CharSet (e.g., if CharSet=UTF-8 then use Encoding.UTF8).

First of, you don't need to decode the input, HTTP is ASCII and it be faster to work with just bytes. Now, what you'll want to do is that you'll define a maximum HTTP request header size, say 4K? and then you'll keep reading bytes until you hit \r\n\r\n this signals the end of the HTTP request. You'll need to enforce this maximum header size limit otherwise a single malicious users could send a infinite HTTP request and your server would run out of memory.
You should read the HTTP specification.
Depending on your HTTP request the HTTP content can be many things and you need to act accordingly. The HTTP protocol itself is always ASCII so you can treat it as just bytes but the content can be encoded very differently. This is generally explained by the Content-Type: header. But again, read the HTTP specification.