We Keep Coding

XmlSerializer escapes an added escape character

I'm using the XmlSerializer to output a class to a .xml file. For the most part, this is working as expected and intended. However, as a requirement, certain characters need to be removed from the values of the data and replaced with their proper escape characters.
In the elements I need to replace values in, I'm using the Replace() method and returning the updated string. The code below shows this string replacement; the lines commented out are because the XmlSerializer already escapes those particular characters.
I have a requirement from a third-party to escape &, <, >, ', and " characters when they appear within the values of the XML elements. Currently the characters &, <, and > are being escaped appropriately through the XmlSerializer.
The error received when these characters are present is:
Our system has detected a potential threat in the request message attachment.
However, when I serialize the XML Document after performing the string replace, the XmlSerializer sees the & character in &apos; and makes it &apos;. I think this is a correct functionality of the XmlSerializer object. However, I would like the serializer to either a.) ignore the escape characters; or b.) serialize the other characters which are necessary to escape.
Can anyone shed some light on, specifically, how to accomplish either of these?
String Replacement Method
public static string CheckValueOfProperty(string str)
{
string trimmedString = str.Trim();
if (string.IsNullOrEmpty(trimmedString))
return null;
else
{
// Commented out because the Serializer already transforms a '&' character into the appropriate escape character.
//trimmedString = trimmedString .Replace("&", "&");
//trimmedString = trimmedString.Replace("<", "<");
//trimmedString = trimmedString.Replace(">", ">");
trimmedString = trimmedString.Replace("'", "&apos;");
trimmedString = trimmedString.Replace("\"", """);
return trimmedString;
}
}
XmlSerializer Code
public static void SerializeAndOutput(object obj, string outputFilePath, XmlSerializerNamespaces ns = null)
{
XmlSerializer x = new XmlSerializer(obj.GetType());
// If the Output File already exists, delete it.
if (File.Exists(outputFilePath))
{
File.Delete(outputFilePath);
}
// Then, Create the Output File and Serialize the parameterized object as Xml to the Output File
using (TextWriter tw = File.CreateText(outputFilePath))
{
if (ns == null)
{
x.Serialize(tw, obj);
}
else { x.Serialize(tw, obj, ns); }
}
// =====================================================================
// The code below here is no longer needed, was used to force "utf-8" to
// UTF-8" to ensure the result was what was being expected.
// =====================================================================
// Create a new XmlDocument object, and load the contents of the OutputFile into the XmlDocument
// XmlDocument xdoc = new XmlDocument() { PreserveWhitespace = true };
// xdoc.Load(outputFilePath);
// Set the Encoding property of each XmlDeclaration in the document to "UTF-8";
// xdoc.ChildNodes.OfType<XmlDeclaration>().ToList().ForEach(d => d.Encoding = "UTF-8");
// Save the XmlDocument to the Output File Path.
// xdoc.Save(outputFilePath);
}

The single and double quote characters do not need to be escaped when used inside the node content in XML. The single quote or double quote characters only need to be escaped when used in a value of a node attribute. That's why the XMLSerializer does not escape them. And you also do not need to escape them.
See this question and answer for reference.
BTW: The way you set the Encoding to UTF-8 afterwards, is awkward as well. You can specify the encoding with the StreamWriter and then the XMLSerializer will automatically use that encoding and also specify it in the XML declaration.

Here's the solution I came up with. I have only tested it with a sample XML file and not the actual XML file I'm creating, so performance may take a hit; however, this seems to be working.
I'm reading the XML file line-by-line as a string, and replacing any of the defined "special" characters found in the string with their appropriate escape characters. It should process in the order of the specialCharacterList Dictionary<string, string> variable, which means the & character should process first. When processing <, > and " characters, it will only look at the value of the XML element.
using System;
using System.Collections.Generic;
using System.IO;
namespace testSerializer
{
class Program
{
private static string filePath = AppDomain.CurrentDomain.BaseDirectory + "testFile.xml";
private static string tempFile = AppDomain.CurrentDomain.BaseDirectory + "tempFile.xml";
private static Dictionary<string, string> specialCharacterList = new Dictionary<string, string>()
{
{"&","&"}, {"<","<"}, {">",">"}, {"'","&apos;"}, {"\"","""}
};
static void Main(string[] args)
{
ReplaceSpecialCharacters();
}
private static void ReplaceSpecialCharacters()
{
string[] allLines = File.ReadAllLines(filePath);
using (TextWriter tw = File.CreateText(tempFile))
{
foreach (string strLine in allLines)
{
string newLineString = "";
string originalString = strLine;
foreach (var item in specialCharacterList)
{
// Since these characters are all valid characters to be present in the XML,
// We need to look specifically within the VALUE of the XML Element.
if (item.Key == "\"" || item.Key == "<" || item.Key == ">")
{
// Find the ending character of the beginning XML tag.
int firstIndexOfCloseBracket = originalString.IndexOf('>');
// Find the beginning character of the ending XML tag.
int lastIndexOfOpenBracket = originalString.LastIndexOf('<');
if (lastIndexOfOpenBracket > firstIndexOfCloseBracket)
{
// Determine the length of the string between the XML tags.
int lengthOfStringBetweenBrackets = lastIndexOfOpenBracket - firstIndexOfCloseBracket;
// Retrieve the string that is between the element tags.
string valueOfElement = originalString.Substring(firstIndexOfCloseBracket + 1, lengthOfStringBetweenBrackets - 1);
newLineString = originalString.Substring(0, firstIndexOfCloseBracket + 1) + valueOfElement.Replace(item.Key, item.Value) + originalString.Substring(lastIndexOfOpenBracket);
}
}
// For the ampersand (&) and apostrophe (') characters, simply replace any found with the escape.
else
{
newLineString = originalString.Replace(item.Key, item.Value);
}
// Set the "original" string to the new version.
originalString = newLineString;
}
tw.WriteLine(newLineString);
}
}
}
}
}

How can I replace line breaks with nothing/an empty string using the DocX Library?

I need to retain paragraph breaks in a .docx file, but get rid of linebreaks which are often in the wrong place when copying from one file to another (due to different page sizes, and when the font is changed).
Using the DocX Library, I'm trying this:
private void ReplaceLineBreaksWithBoo(string filename)
{
List<string> lineBreaks;
using (DocX document = DocX.Load(filename))
{
lineBreaks = document.FindUniqueByPattern("\n", System.Text.RegularExpressions.RegexOptions.None);
if (lineBreaks.Count > 0)
{
foreach (string s in lineBreaks)
{
document.ReplaceText(s, string.empty); // <-- or a space?
}
}
document.Save();
}
}
...but it doesn't work - "\n" is not the right thing to pass, I reckon; I don't know what I need for that first arg to the FindUniqueByPattern() method. Documentation is nil and the discussion forum there resembles Bodie, California:

I guess you can't do it using FindUniqueByPattern or FindAll. Newline is not represented by any symbol but stored as a paragraph with empty text. You can peek document representation in xml format from document.Xml property, there you'll see empty line stored as single <w:p> element.
Therefore you can search for Paragraphs with empty text instead of searching for newline character :
using (DocX document = DocX.Load(filename))
{
var emptyLines = document.Paragraphs.Where(o => string.IsNullOrEmpty(o.Text));
foreach (var paragraph in emptyLines)
{
paragraph.Remove(false);
}
document.Save();
}

XmlException when loading an XML file with certain characters

I need to use the XmlDocument class to load an XML file:
var doc = new XmlDocument();
doc.Load(filename);
Unfortunately I get an XmlException when in my XML there are specifc characters that I use to rappresent my data, in particular I have a node like the following:
<rect data="string with invalid characters: † ¶"/>
So, the forbidden characters are: † and ¶.
How can I load the file without exceptions and leaving these characters in my XML file?

You'll need to replace those characters with a numerical character reference. Similar to how you replace > and < with & gt; and & lt;, you would replace those characters with something like & #931; or whatever references those specific characters.
edit: I had to add a space after the & to avoid the editor actually picking up and interpreting the character. Just remove the space in use - you get the idea.

Alternatively, if you have no control over the source of the XML and just need to read all of the values in to a database or something, you could use an XmlTextReader to read through the xml line by line, stop on the element you know may contain bad data, and read the chars of that element. I've had to do this in the past. Something like this
static void Main(string[] args)
{
var xtr = new XmlTextReader("");
xtr.Normalization = false;
while (xtr.Read())
{
if(xtr.IsStartElement("Row")) // My xml doc contains many row elements
{
var fields = new string[6];
while(xtr.Read())
{
for (int i = 0; i < 6; i++) // I know my xml only has six child elements per row
{
while(!xtr.IsStartElement())
{
xtr.Read(); // We're not interested in hitting the end elements
}
if(i == 1) // I know my special characters are in the second child element of my row
{
var charBuff = new char[255];
xtr.ReadChars(charBuff, 0, 255); // I know there will be a maximum of 255 characters
fields[i] = new string(charBuff);
}
else
{
fields[i] = xtr.ReadElementContentAsString();
}
}
}
}
}
}

Embedding an xml document inside an xml string [duplicate]

This question already has answers here:
Best way to encode text data for XML
(13 answers)
Closed 2 years ago.
I have a web service that returns an xml string as results. The return string is in this format:
<ReturnValue>
<ErrorNumber>0
</ErrorNumber>
<Message>my message</Message>
</ReturnValue>
The data that I want to insert into the "message" tag is a serialized version of a custom object. The serialized format of that object contains xml and namespace declarations post serialization. When that gets thrown into the "message" tag of my return xml string, XmlSpy says that it's not well-formed. How should I get rid of the namespace declarations, or is there a different way to imbed a serialized object into an xml string?

Wrap the string in CDATA like so:
<![CDATA[your xml, which can be multi-line]]>
CDATA will inform a validator to treat the CDATA contents as ignored text. It's often the most expedient way to embed XML (or taggy non-XML content) as a string. You can run into problems if your embedded XML contains its own CDATA, but otherwise it's a simple fix.

Just make sure that your <Message> XML is encoded so that <, >, ", and & show up as <, >, " and &, respectively.
There are few built-in ways to encode the characters:
string message = System.Web.HttpUtility.HtmlEncode(serializedXml);
string message = System.Security.SecurityElement.Escape(serializedXml);
Using an XmlTextWriter to do the work for you
Use CDATA to wrap your XML
Also, this is probably a duplicate of:
Best way to encode text data for XML

Think of XML as a document not a string.
Create a node named "wrapper", and store the content of your file in it as a Base64 encoded string. The results will look like this.
<ReturnValue>
<ErrorNumber>0</ErrorNumber>
<Message>my message</Message>
<wrapper type="bin.base64">PD94bWwgdmVyc2lvbj0iMS4wIj8+PHhzbDpzdHlsZXNoZWV0IHZ
lcnNpb249IjEuMCIgeG1sbnM6eHNsPSJodHRwOi8vd3d3LnczLm9yZy8xOTk5L1hTTC9UcmFuc2Zvcm0
iIHhtbG5zOm1zeHNsPSJ1cm46c2NoZW1hcy1taWNyb3NvZnQtY29tOnhzbHQiPjx4c2w6b3V0cHV0IG1
ldGhvZD0ieG1sIiAvPjx4c2w6dGVtcGxhdGUgbWF0Y2g9Ii8iPjwveHNsOnRlbXBsYXRlPjwveHNsOnN
0eWxlc2hlZXQ+</wrapper>
</ReturnValue>
The following code shows how to add the wrapper, encode the content. Then it reverses the process to show that it all "works".
Using Base64 in XML has a number of other applications as well. For example embedding images, or other documents in XML content.
using System;
using System.IO;
using System.Xml;
public class t
{
static public string EncodeTo64(string toEncode) {
byte[] toEncodeAsBytes = System.Text.ASCIIEncoding.ASCII.GetBytes(toEncode);
string returnValue = System.Convert.ToBase64String(toEncodeAsBytes);
return returnValue;
}
static public string DecodeFrom64(string encodedData) {
byte[] encodedDataAsBytes = System.Convert.FromBase64String(encodedData);
string returnValue = System.Text.ASCIIEncoding.ASCII.GetString(encodedDataAsBytes);
return returnValue;
}
public static void Main() {
try {
//Create the XmlDocument.
XmlDocument doc = new XmlDocument();
doc.LoadXml( #"
<ReturnValue>
<ErrorNumber>0</ErrorNumber>
<Message>my message</Message>
</ReturnValue>
");
XmlNode nodeMessage = doc.SelectSingleNode( "/ReturnValue/Message" );
if( nodeMessage != null ) {
XmlDocument docImport = new XmlDocument();
docImport.Load( "docwithnamespace.xml" );
// create a wrapper element for the file, then import and append it after <Message>
XmlElement nodeWrapper = (XmlElement)doc.CreateElement( "wrapper" );
nodeWrapper.SetAttribute( "type", "bin.base64" );
nodeWrapper = (XmlElement)doc.ImportNode( nodeWrapper, true );
XmlNode ndImport = nodeMessage.ParentNode.AppendChild( nodeWrapper.CloneNode( true ) );
ndImport.InnerText = EncodeTo64( docImport.OuterXml );
doc.Save( "wrapperadded.xml" );
// Next, let's test un-doing the wrapping
// Re-load the "wrapped" document
XmlDocument docSaved = new XmlDocument();
docSaved.Load( "wrapperadded.xml" );
// Get the wrapped element, decode from base64 write to disk
XmlNode node = doc.SelectSingleNode( "/ReturnValue/wrapper" );
if( node != null ) {
// Load the content, and save as a new XML
XmlDocument docUnwrapped = new XmlDocument();
docUnwrapped.LoadXml( DecodeFrom64( node.InnerText ) );
docUnwrapped.Save( "unwrapped.xml" );
Console.WriteLine( "Eureka" );
}
}
} catch( Exception e ) {
Console.WriteLine(e.Message);
}
}
}

Convert character entities to their unicode equivalents

I have html encoded strings in a database, but many of the character entities are not just the standard & and <. Entities like “ and —. Unfortunately we need to feed this data into a flash based rss reader and flash doesn't read these entities, but they do read the unicode equivalent (ex “).
Using .Net 4.0, is there any utility method that will convert the html encoded string to use unicode encoded character entities?
Here is a better example of what I need. The db has html strings like: <p>John & Sarah went to see $ldquo;Scream 4$rdquo;.</p> and what I need to output in the rss/xml document with in the <description> tag is: <p>John &#38; Sarah went to see &#8220;Scream 4&#8221;.</p>
I'm using an XmlTextWriter to create the xml document from the database records similar to this example code http://www.dotnettutorials.com/tutorials/advanced/rss-feed-asp-net-csharp.aspx
So I need to replace all of the character entities within the html string from the db with their unicode equivilant because the flash based rss reader doesn't recognize any entities beyond the most common like &.

My first thought is, can your RSS reader accept the actual characters? If so, you can use HtmlDecode and feed it directly in.
If you do need to convert it to the numeric representations, you could parse out each entity, HtmlDecode it, and then cast it to an int to get the base-10 unicode value. Then re-insert it into the string.
EDIT:
Here's some code to demonstrate what I mean (it is untested, but gets the idea across):
string input = "Something with — or other character entities.";
StringBuilder output = new StringBuilder(input.Length);
for (int i = 0; i < input.Length; i++)
{
if (input[i] == '&')
{
int startOfEntity = i; // just for easier reading
int endOfEntity = input.IndexOf(';', startOfEntity);
string entity = input.Substring(startOfEntity, endOfEntity - startOfEntity);
int unicodeNumber = (int)(HttpUtility.HtmlDecode(entity)[0]);
output.Append("&#" + unicodeNumber + ";");
i = endOfEntity; // continue parsing after the end of the entity
}
else
output.Append(input[i]);
}
I may have an off-by-one error somewhere in there, but it should be close.

would HttpUtility.HtmlDecode work for you?
I realize it doesn't convert to unicode equivalent entities, but instead converts it to unicode. Is there a specific reason you want the unicode equivalent entities?
updated edit
string test = "<p>John & Sarah went to see “Scream 4”.</p>";
string decode = HttpUtility.HtmlDecode(test);
string encode = HttpUtility.HtmlEncode(decode);
StringBuilder builder = new StringBuilder();
foreach (char c in encode)
{
if ((int)c > 127)
{
builder.Append("&#");
builder.Append((int)c);
builder.Append(";");
}
else
{
builder.Append(c);
}
}
string result = builder.ToString();

you can download a local copy of the appropriate HTML and/or XHTML DTDs from the W3C. Then set up an XmlResolver and use it to expand any entities found in the document.
You could use a regular expression to find/expand the entities, but that won't know anything about context (e.g., anything in a CDATA section shouldn't be expanded).

this might help you put input path in textbox
try
{
FileInfo n = new FileInfo(textBox1.Text);
string initContent = File.ReadAllText(textBox1.Text);
int contentLength = initContent.Length;
Match m;
while ((m = Regex.Match(initContent, "[^a-zA-Z0-9<>/\\s(&#\\d+;)-]")).Value != String.Empty)
initContent = initContent.Remove(m.Index, 1).Insert(m.Index, string.Format("&#{0};", (int)m.Value[0]));
File.WriteAllText("outputpath", initContent);
}
catch (System.Exception excep)
{
MessageBox.Show(excep.Message);
}
}