We Keep Coding

uGUI Text Field, How to Remove "Replacement Characters" (uFFFD aka �)?

Using the uGUI Text component, I'm getting "replacement characters" aka � and I can't find a way to remove them.
I'm getting a string from the Instagram api which contains unicode characters for both non-alphabet language characters (for Japanese for example) which I need.
However, the unicode characters for the emojis come in as replacement characters aka �.
I don't require the emojis and they can be stripped out however, I can't find a method to do this.
I'm unable to use TextMeshPro as I'm unable generate a font asset with all the unicode characters need to display the various languages (this could be user error but when I try the process hangs).
I notice these � characters don't appear in the Inspector or console so there must be a way to ignore or remove them.
I'm setting the string like this
body.text = System.Uri.UnescapeDataString(postData.text);
I've tried a number of things that haven't worked including
body.text = body.text.Replace('\uFFFD','\'');//doesn't work
body.text = Regex.Replace(body.text, #"^[\ufffd]", string.Empty);//doesn't work
I've also tried breaking up the string as a char array. When I try to print to console I get this error when it hits a replacement character:
foreach (char item in postData.text.ToCharArray())
print(item); //Error: UTF-16 to UTF-8 conversion failed because the input string is invalid
Any help with this would be greatly appreciated!
Thank you.
Unity 2018.4.4, c#

Found the answer!
This post provided a solution: How do I remove emoji characters from a string?
body.text = Regex.Replace(body.text, #"\p{Cs}", "");

Send plain text mixed with Emojis as Unicode data via SMS by using C# in Unity

I am writing an Android app in Unity using C#. The app will send SMS text messages that include a mixture of text and emojis.
My initial thought is to send the Unicode values of the respective emojis inline with any plain text. I have searched StackOverflow and I haven't found a concise example that solves this problem.
Here is code I have tried:
string mobile_num = "+18007671111; //Placeholder
string text = "Test: \\uFFFd\\uFFFd"; //(smile emoji Unicode value)
char[] chars = text.ToCharArray();
byte[] bytes = Encoding.UTF8.GetBytes(chars);
string message = HttpUtility.UrlEncode(bytes);
string sms = string.Format("sms:{0}?body={1}", mobile_num, message);
Application.OpenURL(sms);
I need to know:
1. Is this the correct approach?
a. if not, please help me correctly encode text + emoji data
b. What is the step required to covert so that the final message can be sent via SMS?

So after much searching, I found the simplest way in C# is to use:
\U########
Where:
\ is an escape character
U is a constant to define a Unicode sequence follows
## is the hex value of the emoticon encoded in exactly 8 characters left filled with zeros if necessary.
For example:
string u = "Smile: \U0001F601";
Will send:
Smile: 😁
Thank you Jeppe Stig Nielsen for your insight. For the full discussion follow this link:
How to convert numbers between hexadecimal and decimal

How do I print the graphical representation of a Unicode character (BELL) using its byte value? [duplicate]

I'm new to programming and self taught. I'm trying to output the astrological symbol for Taurus, which is supposed to be U+2649 in Unicode. Here is the code I'm using...
string myString = "\u2649";
byte[] unicode = System.Text.Encoding.Unicode.GetBytes(myString);
Console.WriteLine(unicode.Length);
The result I'm getting is the number 2 instead of the symbol or font. I'm sure I'm doing something wrong.

Why are you converting it to unicode, this will not do anything.. lose the conversion and do the following:
string a ="\u2649" ;
Console.write(a) ;

You need to have a font which displays that glyph. If you do, then:
Console.WriteLine(myString);
is all you need.
EDIT: Note, the only font I could find which has this glyph is "MS Reference Sans Serif".

The length of the Unicode character, in bytes, is 2 and you are writing the Length to the Console.
Console.WriteLine(unicode.Length);
If you want to display the actual character, then you want:
Console.WriteLine(myString);
You must be using a font that has that Unicode range for it to display properly.
UPDATE:
Using default console font the above Console.WriteLine(myString) will output a ? character as there is no \u2649. As far I have so far googled, there is no easy way to make the console display Unicode characters that are not already part of the system code pages or the font you choose for the console.
It may be possible to change the font used by the console: Changing Console Fonts

You are outputting the length of the character, in bytes. The Console doesn't support unicode output, however, so it will come out as an '?' character.

Best way to convert arabic to hex in custom code table in c#

I'm currently working on a new program to print receipts which contain arabic text. The printer can handle these characters, but uses a special code table to print them. Therefore all information sent to the printer must be in hex. Information sent to the printer can have a mix of both arabic and non arabic characters.
The code table is here(page 133 http://support.epostraders.co.uk/support-files/documents/3/dwY-TM-T88V_TechRefGuide.pdf)
For example, ق = E7, ك = E8
With a standard hex conversion (below) the first 128 latin alphanumeric characters work just fine, but arabic are displayed as question marks.
byte[] ba = Encoding.Default.GetBytes(textBox1.Text);
var hexstring = BitConverter.ToString(ba);
Does anyone have any suggestions for the best way to convert to the correct hex?

Answering this myself in case anyone else has a similar issue.
So the scenario was trying to send Arabic characters to a TM-T88iV Epson printer. Firstly, You need the TM-T88V or later to support Arabic.
Secondly, characters must be sent to the printer as Bytes using a Write command rather than as a String in a WriteLine. The Write command needs to terminate with a CRLF in order for the printer to print. The printer has no Right-to-Left funtion, so the Bytes sent to the printer need to be inverted first: Array.Reverse(byte[])
In order to get visual studio to convert arabic characters to the correct hex values, you need to change the windows code table. This is usually done in Control panel > region & language > administrative > language for non-unicode.
Windows default for UK was 850. Arabic (U.A.E) is 720. You can double check by running cmd and typing chcp. As it turns out, Arabic U.A.E 720 is NOT correct, you need 1256. I couldn't find the matching language in control panel, so I changed it manually in cmd using the command 'chcp 1256'
Finally, you need to change the internal code table of the printer. To do this, I used the TM-T88V utility (https://download.epson-biz.com/modules/pos/index.php?page=single_soft&cid=4100&pcat=3&scat=42). I believe you can do this via direct commands sent via serial too, but it proved too fiddly.
Fingers crossed this should all work now. Happy printing.

Encode data for easy passing over phone

I need to encode some data (text) so that it can easily be passed by the user over phone.
The text contains random characters and is normally not longer than 100 chars. Example:
"37-b,kA.sZ:Bb9--10.y<§"
I'd like to encode this text into more human readable form so that it can easily be passed over phone.
Base36 produces a text that can easily be passed over phone, but I don't see how to encode/decode this correctly.
Any ideas or alternatives?
(Platform is .net 3.5 SP1)

Base 36 sounds like a good choice (when using symbols a-z and 0-9, it is the largest space of characters, that can be easily passed over the phone). I would suggest you make the output contain blocks of 6 or 8 characters, to make it easier to read. Also; consider adding a checksum in the end, so you can verify there are no errors in the data.
100 characters in this encoding will still not be easy to read over the phone and get right the first time. Have you considered another delivery mechanism ? Text message (SMS) ?
On Wikipedia, there is an example of encoding Base36 in Python - shouldn't be too hard to convert to C#.