We Keep Coding

Advantage in using SerialPort.ReadByte over ReadChar?

Of all the example codes I have read online regarding SerialPorts all uses ReadByte then convert to Character instead of using ReadChar in the first place.
Is there a advantage in doing this?

The SerialPort.Encoding property is often misunderstood. The default is ASCIIEncoding, it will produce ? for byte values 0x80..0xFF. So they don't like getting these question marks. If you see such code then converting the byte to char directly then they are getting it really wrong, Unicode has lots of unprintable codepoints in that byte range and the odds that the device actually meant to send these characters are zero. A string tends to be regarded as easier to handle than a byte[], it is.

When you use ReadChar it is based on the encoding you are using, like #Preston Guillot said. According to the docu of ReadChar:
This method reads one complete character based on the encoding.
Use caution when using ReadByte and ReadChar together. Switching
between reading bytes and reading characters can cause extra data to
be read and/or other unintended behavior. If it is necessary to switch
between reading text and reading binary data from the stream, select a
protocol that carefully defines the boundary between text and binary
data, such as manually reading bytes and decoding the data.

How to double-decode UTF-8 bytes C#

I have a problem.
Unicode 2019 is this character:
’
It is a right single quote.
It gets encoded as UTF8.
But I fear it gets double-encoded.
>>> u'\u2019'.encode('utf-8')
'\xe2\x80\x99'
>>> u'\xe2\x80\x99'.encode('utf-8')
'\xc3\xa2\xc2\x80\xc2\x99'
>>> u'\xc3\xa2\xc2\x80\xc2\x99'.encode('utf-8')
'\xc3\x83\xc2\xa2\xc3\x82\xc2\x80\xc3\x82\xc2\x99'
>>> print(u'\u2019')
’
>>> print('\xe2\x80\x99')
’
>>> print('\xc3\xa2\xc2\x80\xc2\x99')
’
>>> '\xc3\xa2\xc2\x80\xc2\x99'.decode('utf-8')
u'\xe2\x80\x99'
>>> '\xe2\x80\x99'.decode('utf-8')
u'\u2019'
This is the principle used above.
How can I do the bolded parts, in C#?
How can I take a UTF8-Encoded string, conver to byte array, convert THAT to a string in, and then do decode again?
I tried this method, but the output is not suitable in ISO-8859-1, it seems...
string firstLevel = "’";
byte[] decodedBytes = Encoding.UTF8.GetBytes(firstLevel);
Console.WriteLine(Encoding.UTF8.GetChars(decodedBytes));
// ’
Console.WriteLine(decodeUTF8String(firstLevel));
//â�,��"�
//I was hoping for this:
//’
Understanding Update:
Jon's helped me with my most basic question: going from "’" to "’ and thence to "’" But I want to honor the recommendations at the heart of his answer:
understand what is happening
fix the original sin
I made an effort at number 1.
Encoding/Decoding
I get so confused with terms like these.
I confuse them with terms like Encrypting/Decrypting, simply because of "En..." and "De..."
I forget what they translate from, and what they translate to.
I confuse these start points and end points; could it be related to other vague terms like hex, character entities, code points, and character maps.
I wanted to settle the definition at a basic level.
Encoding and Decoding in the context of this question is:
Decode
Corresponds to C# {Encoding}.'''GetString'''(bytesArray)
Corresponds to Python stringObject.'''decode'''({Encoding})
Takes bytes as input, and converts to string representation as output, according to some conversion scheme called an "encoding", represented by {Encoding} above.
Bytes -> String
Encode
Corresponds to C# {Encoding}.'''GetBytes'''(stringObject)
Corresponds to Python stringObject.'''encode'''({Encoding})
The reverse of Decode.
String -> Bytes (except for Python)
Bytes vs Strings in Python
So Encode and Decode take us back and forth between bytes and strings.
While Python helped me understand what was going wrong, it could also confuse my understanding of the "fundamentals" of Encoding/Decoding.
Jon said:
It's a shame that Python hides [the difference between binary data and text data] to a large extent
I think this is what PEP means when it says:
Python's current string objects are overloaded. They serve to hold both sequences of characters and sequences of bytes. This overloading of purpose leads to confusion and bugs.
Python 3.* does not overload strings in this way.:
Python 2.7
>>> #Encoding example. As a generalization, "Encoding" produce bytes.
>>> #In Python 2.7, strings are overloaded to serve as bytes
>>> type(u'\u2019'.encode('utf-8'))
<type 'str'>
Python 3.*
>>> #In Python 3.*, bytes and strings are distinct
>>> type('\u2019'.encode('utf-8'))
<class 'bytes'>
Another important (related) difference between Python 2 and 3, is their default encoding:
>>>import sys
>>>sys.getdefaultencoding()
Python 2
'ascii'
Python 3
'utf-8'
And while Python 2 says 'ascii', I think it means a specific type of ASCII;
It does '''not''' mean ISO-8859-1, which supports range(256), which is what Jon uses to decode (discussed below)
It means ASCII, the plainest variety, which are only range(128)
And while Python 3 no longer overloads string as both bytes, and strings, the interpreter still makes it easy to ignore what's happening and move between types. i.e.
just put a 'u' before a string in Python 2.* and it's a Unicode literal
just put a 'b' before a string in Python 3.* and it's a Bytes literal
Encoding and C
Jon points out that C# uses UTF-16, to correct my "UTF-8 Encoded String" comment, above;
Every string is effectively UTF-16.
My understanding of is: if C# has a string object "s", the computer memory actually has bytes corresponding to that character in the UTF-16 map. That is, (including byte-order-mark??) feff0073.
He also uses ISO-8859-1 in the hack method I requested.
I'm not sure why.
My head is hurting at the moment, so I'll return when I have some perspective.
I'll return to this post. I hope I'm explaining properly. I'll make it a Wiki?

You need to understand that fundamentally this is due to someone misunderstanding the difference between binary data and text data. It's a shame that Python hides that difference to a large extent - it's quite hard to accidentally perform this particular form of double-encoding in C#. Still, this code should work for you:
using System;
using System.Text;
class Test
{
static void Main()
{
// Avoid encoding issues in the source file itself...
string firstLevel = "\u00c3\u00a2\u00c2\u0080\u00c2\u0099";
string secondLevel = HackDecode(firstLevel);
string thirdLevel = HackDecode(secondLevel);
Console.WriteLine("{0:x}", (int) thirdLevel[0]); // 2019
}
// Converts a string to a byte array using ISO-8859-1, then *decodes*
// it using UTF-8. Any use of this method indicates broken data to start
// with. Ideally, the source of the error should be fixed.
static string HackDecode(string input)
{
byte[] bytes = Encoding.GetEncoding(28591)
.GetBytes(input);
return Encoding.UTF8.GetString(bytes);
}
}

how to write with a single byte character encoding?

I have a webservice that returns the config file to a low level hardware device.
The manufacturer of this device tells me he only supports single byte charactersets for this config file.
On this wiki page I found out that the following should be single byte character sets:
ISO 8859
ISO/IEC 646 (I could not find this one here)
various Microsoft/IBM code pages
But when I call Encoding.GetMaxByteCount(1) on these character sets it always returns 2.
I also tried various other encodings (for instance IBM437), but GetMaxByteCount also returns 2 for other character sets.
The method Endoding.IsSingleByte seems unreliable according to this
You should be careful in what your application does with the value for
IsSingleByte. An assumption of how an Encoding will proceed may still
be wrong. For example, Windows-1252 has a value of true for
Encoding.IsSingleByte, but Encoding.GetMaxByteCount(1) returns 2. This
is because the method considers potential leftover surrogates from a
previous decoder operation.
Also the method Encoding.GetMaxByteCount has some of the same issues according to this
Note that GetMaxByteCount considers potential leftover surrogates from
a previous decoder operation. Because of the decoder, passing a value
of 1 to the method retrieves 2 for a single-byte encoding, such as
ASCII. Your application should use the IsSingleByte property if this
information is necessary.
Because of this I am not sure anymore on what to use.
Further reading.

Basically, GetMaxByteCount considers an edge-case that you will probably never need in regular code, specifically what it says about the decoder and surrogates. The point here is that some code-points are encoded as surrogate pairs, which in unfortunate cases can mean that it straddles two calls to GetBytes() / GetChars (on the encoder/decoder). As a consequence, the implementation may theoretically have a single byte/character still buffered and waiting to be processed, therefore GetMaxByteCount needs to warn about this.
However! All of this only makes sense if you are using the encoder/decoder directly. If you are using operations on the Encoding, such as Encoding.GetBytes, then all of this is abstracted away from you and you will never need to know. In which case, just use IsSingleByte and you'll be fine.

Maybe you should use the example from Encoding.Convert Method page on MSDN
The Encoding.Convert method should provide an ASCII encoded string. Hopefully single byte..

Compress small string

Maybe there are any way to compress small strings(86 chars) to something smaller?
#a#1\s\215\c\6\-0.55955,-0.766462,0.315342\s\1\x\-3421.-4006,3519.-4994,3847.1744,sbs
The only way I see is to replace the recurring characters on a unique character.
But i can't find something about that in google.
Thanks for any reply.

http://en.wikipedia.org/wiki/Huffman_coding
Huffman coding would probably be pretty good start. In general the idea is to replace individual characters with the smallest bit pattern needed to replicate the original string or dataset.
You'll want to run statistical analysis on a variety of 'small strings' to find the most common characters so that the more common characters will be represented with the smallest unique bit patterns. And possibly makeup a 'example' small string with every character that will need to be represented (like a-z0-9#.0-)

I took your example string of 85 bytes (not 83 since it was copied verbatim from the post, perhaps with some intended escapes not processed). I compressed it using raw deflate, i.e. no zlib or gzip headers and trailers, and it compressed to 69 bytes. This was done mostly by Huffman coding, though also with four three-byte backward string references.
The best way to compress this sort of thing is to use everything you know about the data. There appears to be some structure to it and there are numbers coded in it. You could develop a representation of the expected data that is shorter. You can encode it as a stream of bits, and the first bit could indicate that what follows is straight bytes in the case that the data you got was not what was expected.
Another approach would be to take advantage of previous messages. If this message is one of a stream of messages, and they all look similar to each other, then you can make a dictionary of previous messages to use as a basis for compression, which can be reconstructed at the other end by the previous messages received. That may offer dramatically improved compression if they messages really are similar.

You should look up RUN-LENGTH ENCODING. Here is a demonstration
rrrrrunnnnnn BECOMES 5r1u6n WHAT? truncate repetitions: for x consecutive r use xr
Now what if some of the characters are digits? Then instead of using x, use the character whose ASCII value is x. for example,
if you have 43 consecutive P, write +P because '+' has ASCII code 43. If you have 49 consecutive y, write 1y because '1' has ASCII code 49.
Now the catch, which you will find with all compression algorithms, is if you have a string with little or no repetitions. Then in that case your code may be longer than the original word. But that's true for all compression algorithms.
NOTE:
I don't encourage using Huffman coding because even if you use the Ziv-Lempel implementation, it's still a lot of work to get it right.

C#: String -> MD5 -> Hex

in languages like PHP or Python there are convenient functions to turn an input string into an output string that is the HEXed representation of it.
I find it a very common and useful task (password storing and checking, checksum of file content..), but in .NET, as far as I know, you can only work on byte streams.
A function to do the work is easy to put on (eg http://blog.stevex.net/index.php/c-code-snippet-creating-an-md5-hash-string/), but I'd like to know if I'm missing something, using the wrong pattern or there is simply no such thing in .NET.
Thanks

The method you linked to seems right, a slightly different method is showed on the MSDN C# FAQ
A comment suggests you can use:
System.Web.Security.FormsAuthentication.HashPasswordForStoringInConfigFile(string, "MD5");

Yes you can only work with bytes (as far as I know). But you can turn those bytes easily into their hex representation by looping through them and doing something like:
myByte.ToString("x2");
And you can get the bytes that make up the string using:
System.Text.Encoding.UTF8.GetBytes(myString);
So it could be done in a couple lines.

One problem is with the very concept of "the HEXed representation of [a string]".
A string is a sequence of characters. How those characters are represented as individual bits depends on the encoding. The "native" encoding to .NET is UTF-16, but usually a more compact representation is achieved (while preserving the ability to encode any string) using UTF-8.
You can use Encoding.GetBytes to get the encoded version of a string once you've chosen an appropriate encoding - but the fact that there is that choice to make is the reason that there aren't many APIs which go straight from string to base64/hex or which perform encryption/hashing directly on strings. Any such APIs which do exist will almost certainly be doing the "encode to a byte array, perform appropriate binary operation, decode opaque binary data to hex/base64".
(That makes me wonder whether it wouldn't be worth writing a utility class which could take an encoding, a Func<byte[], byte[]> and an output format such as hex/base64 - that could represent an arbitrary binary operation applied to a string.)