I recognized that based on a context in which I want to use some parameters, there are at least 4 kinds of encoding that are necessary to avoid corrupted code being executed :
Javascript encoding when constructing a javascript code, e.g.
var a = "what's up ?"
var b = "alert('" + a + "');"
eval(b); // or anything else that executes b as code
URL encoding when using a string as a parameter into the url, e.g.
var a = "Bonnie & Clyde";
var b = "mypage.html?par=" + a;
window.location.href = b; // or anything else that tries to use b as URL
HTML encoding when using a string as an HTML source of some element, e.g.
var a = "<script>alert('hi');</script>";
b.innerHTML = a; // or anything else that interprets a directly
HTML attribute encoding when using a string as a value of an attribute, e.g.
var a = 'alert("hello")';
var b = '<img onclick="' + a + '" />'; // or anything else that uses a as a (part of) a tag's attribute
While in the ASP.NET codebehind I'm aware of ways to encode the string in all 4 cases (using e.g. DataContractJsonSerializer, HttpUtility.UrlEncode, HttpUtility.HtmlEncode and HttpUtility.HtmlAttributeEncode), it would be quite interesting to know whether there are some utilities that I could use directly from javascript to encode / decode strings in these 4 cases.
Case 2 can be dealt with using encodeURIComponent(), as danp suggested.
Case 3 won't execute the script in most browsers. If you want the output to the document to be <script>...</script>, you should edit the text content of the element instead:
var a = "<script>alert('hi');</script>";
if ("textContent" in b)
b.textContent = a; // W3C DOM
else
b.innerText = a; // Internet Explorer <=8
Cases 1, and 4 aren't really encoding issues, they're sanitation issues. Encoding the strings passed to these functions would probably cause a syntax error or just result in a string value that isn't assigned to anything. Sanitizing usually involves looking for certain patterns and either allowing the action or disallowing it - it's safer to have a whitelist than a blacklist (that sounds terrible!).
Internet Explorer 8 has an interesting function called window.toStaticHTML() that will remove any script content from a HTML string. Very useful for sanitizing HTML before inserting into the DOM. Unfortunately, it's proprietary so you won't find this function in other browsers.
You can use the javascript function escape(..) for some of these purposes.
e: actually forget! sorry, it's a deprecated function - encodeURI(), decodeURI() etc are the way forward! Details here.
escape and unescape functions do not
work properly for non-ASCII characters
and have been deprecated. In
JavaScript 1.5 and later, use
encodeURI, decodeURI,
encodeURIComponent, and
decodeURIComponent.
The escape and unescape functions let
you encode and decode strings. The
escape function returns the
hexadecimal encoding of an argument in
the ISO Latin character set. The
unescape function returns the ASCII
string for the specified hexadecimal
encoding value.encoding value.
Related
Can anyone tell me why this is not encoding using htmlencode
any string that has < before the string ie
<something or &something
is not being displayed back to the html page when looking at the encoding the < and & is not being encoded. I would have expected these characters to be encoded to < or &
edit: this is the code I use to encode the string:
var replacedHtml = Regex.Replace(html,
#"</?(\w*)[^>]*>",
me => AllowedTags.Any(s => s.Equals(me.Groups[1].Value, StringComparison.OrdinalIgnoreCase))
? me.Value
: HttpUtility.HtmlEncode(me.Value), RegexOptions.Singleline);
return replacedHtml;
edit: i think the issue is not on the server side but rather on the angular side. the ng-bind-html
<span ng-bind-html="ctl.linkGroup.Notes | TextToHtmlSafe">
angular.module('CPSCore.Filters')
.filter('TextToHtmlSafe', ['$sce',function ($sce) {
return function (text) {
if (!text)
return text;
var htmlText = text.replace(/\n/g, '<br />');
return $sce.trustAsHtml(htmlText);
};
}]);
is declaring that
<something
without the closing tag is not safe and therefore removes it from the view
Try System.Net.WebUtility.HtmlDecode to properly decode special characters. Using this, < changes to < and & changes to & which is properly displayed html pages.
In HTML, the ampersand character (“&”) declares the beginning of an entity reference (a special character). If you want one to appear in text on a web page you should use the encoded named entity “&”—more technical mumbo-jumbo at w3c.org. While most web browsers will let you get away without encoding them, stuff can get dicey in weird edge cases and fail completely in XML.
The other main characters to remember to encode are < (<) and > (>), you don’t want to confuse your browser about where HTML tags start and end
I am setting a value in the cookie using JavaScript and getting the contents of the cookie in the code behind.
But the problem is if I am storing the string with some special characters or whitespace characters, when I am retrieving the contents of the cookie the special symbols are getting converted into ASCII equivalent.
For example, if I want to store Adam - (SET) in cookie , its getting converted into Adam%20-%20%28SET%29 and getting stored and when I am retrieving it I get the same Adam%20-%20%28SET%29. But I wan tot get this Adam - (SET) in the code behind.
How I get this. Please help.
In C#
Use:
String decoded = HttpUtility.UrlDecode(EncodedString);
HttpUtility.UrlDecode() is the underlying function used by most of the other alternatives you can use in the .NET Framwework (see below).
You may want to specify an encoding, if necessary.
Or:
String decoded = Uri.UnescapeDataString(s);
See Uri.UnescapeDataString()'s documentation for some caveats.
In JavaScript
var decoded = decodeURIComponent(s);
Before jumping on using unescape as recommended in other questions, read decodeURIComponent vs unescape, what is wrong with unescape? . You may also want to read What is the difference between decodeURIComponent and decodeURI? .
You can use the unescape function in JS to do that.
var str = unescape('Adam%20-%20%28SET%29');
You are looking for HttpUtility.UrlDecode() (in the System.Web namespace, I think)
In javasrcipt you can use the built-in decodeURIComponent, but I suspect that the string encoding is happening when the value is sent to server so the C# answers are what you want.
I am using silverlight / ASP .NET and C#. What if I want to do this from silverlight for instance,
// I have left out the quotes to show you literally what the characters
// are that I want to use
string password = vtakyoj#"5
string encodedPassword = HttpUtility.UrlEncode(encryptedPassword, Encoding.UTF8);
// encoded password now = vtakyoj%23%225
URI uri = new URI("http://www.url.com/page.aspx#password=vtakyoj%23%225");
HttpPage.Window.Navigate(uri);
If I debug and look at the value of uri it shows up as this (we are still inside the silverlight app),
http://www.url.com?password=vtakyoj%23"5
So the %22 has become a quote for some reason.
If I then debug inside the page.aspx code (which of course is ASP .NET) the value of Request["password"] is actually this,
vtakyoj#"5
Which is the original value. How does that work? I would have thought that I would have to go,
HttpUtility.UrlDecode(Request["password"], Encoding.UTF8)
To get the original value.
Hope this makes sense?
Thanks.
First lets start with the UTF8 business. Esentially in this case there isn't any. When a string contains characters with in the standard ASCII character range (as your password does) a UTF8 encoding of that string is identical to a single byte ASCII string.
You start with this:-
vtakyoj#"5
The HttpUtility.UrlEncode somewhat aggressively encodes it to:-
vtakyoj%23%225
Its encoded the # and " however only # has special meaning in a URL. Hence when you view string value of the Uri object in Silverlight you see:-
vtakyoj%23"5
Edit (answering supplementary questions)
How does it know to decode it?
All data in a url must be properly encoded thats part of its being valid Url. Hence the webserver can rightly assume that all data in the query string has been appropriately encoded.
What if I had a real string which had %23 in it?
The correct encoding for "%23" would be "%3723" where %37 is %
Is that a documented feature of Request["Password"] that it decodes it?
Well I dunno, you'd have check the documentation I guess. BTW use Request.QueryString["Password"] the presence of this same indexer directly on Request was for the convenience of porting classic ASP to .NET. It doesn't make any real difference but its better for clarity since its easier to make the distinction between QueryString values and Form values.
if I don't use UFT8 the characters are being filtered out.
Aare you sure that non-ASCII characters may be present in the password? Can you provide an example you current example does not need encoding with UTF-8?
If Request["password"] is to work, you need "http://url.com?password=" + HttpUtility.UrlEncode("abc%$^##"). I.e. you need ? to separate the hostname.
Also the # syntax is username:password#hostname, but it has been disabled in IE7 and above IIRC.
I have been tracking down a bug on a Url Rewriting application. The bug showed up as an encoding problem on some diacritic characters in the querystring.
Basically, the problem was that a request which was basically /search.aspx?search=heřmánek was getting rewritten with a querystring of "search=he%c5%99m%c3%a1nek"
The correct value (using some different, working code) was a rewrite of the querystring as "search=he%u0159m%u00e1nek"
Note the difference between the two strings. However, if you post both you'll see that the Url Encoding reproduces the same string. It's not until you use the context.Rewrite function that the encoding breaks. The broken string returns 'heÅmánek' (using Request.QueryString["Search"] and the working string returns 'heřmánek'. This change happens after the call to the rewrite function.
I traced this down to one set of code using Request.QueryString (working) and the other using Request.Url.Query (request.Url returns a Uri instance).
While I have worked out the bug there is a hole in my understanding here, so if anyone knows the difference, I'm ready for the lesson.
Your question really sparked my interest, so I've done some reading for the past hour or so. I'm not absolutely positive I've found the answer, but I'll throw it out there to see what you think.
From what I've read so far, Request.QueryString is actually "a parsed version of the QUERY_STRING variable in the ServerVariables collection" [reference] , where as Request.Url is (as you stated) the raw URL encapsulated in the Uri object. According to this article, the Uri class' constructor "...parses the [url string], puts it in canonical format, and makes any required escape encodings."
Therefore, it appears that Request.QueryString uses a different function to parse the "QUERY_STRING" variable from the ServerVariables constructor. This would explain why you see the difference between the two. Now, why different encoding methods are used by the custom parsing function and the Uri object's parsing function is entirely beyond me. Maybe somebody a bit more versed on the aspnet_isapi DLL could provide some answers with that question.
Anyway, hopefully my post makes sense. On a side note, I'd like to add another reference which also provided for some very thorough and interesting reading: http://download.microsoft.com/download/6/c/a/6ca715c5-2095-4eec-a56f-a5ee904a1387/Ch-12_HTTP_Request_Context.pdf
What you indicated as the "broken" encoded string is actually the correct encoding according to standards. The one that you indicated as "correct" encoding is using a non-standard extension to the specifications to allow a format of %uXXXX (I believe it's supposed to indicate UTF-16 encoding).
In any case, the "broken" encoded string is ok. You can use the following code to test that:
Uri uri = new Uri("http://www.example.com/test.aspx?search=heřmánek");
Console.WriteLine(uri.Query);
Console.WriteLine(HttpUtility.UrlDecode(uri.Query));
Works fine. However... on a hunch, I tried UrlDecode with a Latin-1 codepage specified, instead of the default UTF-8:
Console.WriteLine(HttpUtility.UrlDecode(uri.Query,
Encoding.GetEncoding("iso-8859-1")));
... and I got the bad value you specified, 'heÅmánek'. In other words, it looks like the call to HttpContext.RewritePath() somehow changes the urlencoding/decoding to use the Latin-1 codepage, rather than UTF-8, which is the default encoding used by the UrlEncode/Decode methods.
This looks like a bug if you ask me. You can look at the RewritePath() code in reflector and see that it is definitely playing with the querystring - passing it around to all kinds of virtual path functions, and out to some unmanaged IIS code.
I wonder if somewhere along the way, the Uri at the core of the Request object gets manipulated with the wrong codepage? That would explain why Request.Querystring (which is simply the raw values from the HTTP headers) would be correct, while the Uri using the wrong encoding for the diacriticals would be incorrect.
I have done a bit of research over the past day or so and I think I have some information on this.
When you use Request.Querystring or HttpUtility.UrlDecode (or Encode) it is using the Encoding that is specified in the element (specifically the requestEncoding attribute) of the web.config (or the .config hierarchy if you haven't specified) ---NOT the Encoding.Default which is the default encoding for your server.
When you have the encoding set to UTF-8, a single unicode character can be encoded as 2 %xx hex values. It will also be decoded that way when given the whole value.
If you are UrlDecoding with a different Encoding than the url was encoded with, you will get a different result.
Since HttpUtility.UrlEncode and UrlDecode can take an encoding parameter, its tempting to try to encode using an ANSI codepage, but UTF-8 is the right way to go if you have the browser support (apparently old versions don't support UTF-8). You just need to make sure that the is properly set and both sides will work fine.
UTF-8 Seems to be the default encoding: (from .net reflector System.Web.HttpRequest)
internal Encoding QueryStringEncoding
{
get
{
Encoding contentEncoding = this.ContentEncoding;
if (!contentEncoding.Equals(Encoding.Unicode))
{
return contentEncoding;
}
return Encoding.UTF8;
}
}
Following the path to find out the this.ContentEncoding leads you to (also in HttpRequest)
public Encoding ContentEncoding
{
get
{
if (!this._flags[0x20] || (this._encoding == null))
{
this._encoding = this.GetEncodingFromHeaders();
if (this._encoding == null)
{
GlobalizationSection globalization = RuntimeConfig.GetLKGConfig(this._context).Globalization;
this._encoding = globalization.RequestEncoding;
}
this._flags.Set(0x20);
}
return this._encoding;
}
set
{
this._encoding = value;
this._flags.Set(0x20);
}
}
To answer your specific question on the difference betwen Request.Url.Quer and Request.QueryString... here is how HttpRequest builds its Url Property:
public Uri Url
{
get
{
if ((this._url == null) && (this._wr != null))
{
string queryStringText = this.QueryStringText;
if (!string.IsNullOrEmpty(queryStringText))
{
queryStringText = "?" + HttpEncoder.CollapsePercentUFromStringInternal(queryStringText, this.QueryStringEncoding);
}
if (AppSettings.UseHostHeaderForRequestUrl)
{
string knownRequestHeader = this._wr.GetKnownRequestHeader(0x1c);
try
{
if (!string.IsNullOrEmpty(knownRequestHeader))
{
this._url = new Uri(this._wr.GetProtocol() + "://" + knownRequestHeader + this.Path + queryStringText);
}
}
catch (UriFormatException)
{
}
}
if (this._url == null)
{
string serverName = this._wr.GetServerName();
if ((serverName.IndexOf(':') >= 0) && (serverName[0] != '['))
{
serverName = "[" + serverName + "]";
}
this._url = new Uri(this._wr.GetProtocol() + "://" + serverName + ":" + this._wr.GetLocalPortAsString() + this.Path + queryStringText);
}
}
return this._url;
}
}
You can see it is using the HttpEncoder class to do the decoding, but it uses the same QueryStringEncoding value.
Since I am already posting a lot of code here and anyone can get .NET Reflector, I'm going to snippet up the rest. The QueryString property comes from the HttpValueCollection which uses FillFromEncodedBytes method to eventually call HttpUtility.UrlDecode (with the QueryStringEncoding value set above), which eventually calls the HttpEncoder to decode it.
They do seem to use different methodology to decode the actual bytes of the querystring, but the encoding they use to do it seems to be the same.
It is interesting to me that the HttpEncoder has so many functions that seem to do the same thing, so its possible there are differences in those methods which can cause an issue.
I have an MVC route like this www.example.com/Find?Key= with the Key being a Base64 string. The problem is that the Base64 string sometimes has a trailing equal sign (=) such as:
huhsdfjbsdf2394=
When that happens, for some reason my route doesn't get hit anymore.
What should I do to resolve this?
My route:
routes.MapRoute(
"FindByKeyRoute",
"Find",
new { controller = "Search", action = "FindByKey" }
);
If I have http://www.example.com/Find?Key=bla then it works.
If I have http://www.example.com/Find?Key=bla= then it doesn't work anymore.
Important Addition:
I'm writing against an IIS7 instance that doesn't allow % or similar encoding. That's why I didn't use UrlEncode to begin with.
EDIT: Original suggestion which apparently doesn't work
I'm sure the reason is that it thinks it's a query parameter called Key. Could you make it a parameter, with that part being the value, e.g.
www.example.com/Find?Route=Key=
I expect that would work (as the parser would be looking for an & to start the next parameter) but it's possible it'll confuse things still.
Suggestion which I believe will work
Alternatively, replace "=" in the base64 encoded value with something else on the way out, and re-replace it on the way back in, if you see what I mean. Basically use a different base64 decodabet.
Alternative suggestion which should work
Before adding base64 to the URL:
private static readonly char[] Base64Padding = new char[] { '=' };
...
base64 = base64.TrimEnd(Base64Padding);
Then before calling Convert.FromBase64String() (which is what I assume you're doing) on the inbound request:
// Round up to a multiple of 4 characters.
int paddingLength = (4 - (base64.Length % 4)) % 4;
base64 = base64.PadRight(base64.Length + paddingLength, '=');
IF you're passing data in the URL you should probably URL Encode it which would take care of the trailing =.
http://www.albionresearch.com/misc/urlencode.php
UrlEncode the encrypted (it is encrypted, right?) parameter.
If it is an encrypted string, beware that spaces and the + character will also get in your way.
Ok, so IIS 7 won't allow some special characters as part of your path. However, it would allow them if they were part of the querystring.
It is apparently, possible, to change this with a reg hack, but I wouldn't recommend that.
What I would suggest, then, is to use an alternate token, as suggested by Mr Skeet, or simply do not use it in your path, use it as querystring, where you CAN url encode it.
If it is an encrypted string, you haven't verified that it is or is not, you may in some cases get other 'illegal' characters. Querystring really would be the way to go.
Except your sample shows it as querystring... So what gives? Where did you find an IIS that won't allow standard uri encoding as part of the querystring??
Ok then. Thanks for the update.
RequestFiltering?
I see. Still that mentions double-encoded values that it blocks. Someone created a URL Sequence to deny any request with the '%' characters? At that point you might want to not use the encrypted string at all, but generate a GUID or something else that is guaranteed to not contain special characters, yet is not trivial to guess.