I have a .NET Web API v2 and need to define a route that can contain forward slashes at any point in the route. I have it configured (something) like this in WebApiConfig:
config.Routes.MapHttpRoute(
name: "SomeRouteName",
routeTemplate: "api/summary/{*token}",
defaults: new { controller = "Summary" });
Unfortunately the token contains slashes and there's nothing I can do about it at this point. The above route works in most cases if the slash is not at the beginning of the token. So that takes care of most cases. But if token begins with slash(es), it doesn't work because that first slash gets interpreted as part of the URL I assume and gets eaten. So in my controller action, I have the following code (admittedly a hack that I'm trying to avoid):
if (summary == null)
{
summary = _repo.GetSummary($"/{token}");
}
Obviously, this will only work for a single slash. I could do a loop and add more, but there isn't way to know how many it could be. Currently no tokens in my DB begin with two slashes, so this bad code works for now. It was implemented as a band-aid until a better solution is found.
Edit: This references the * route, which mostly fixed my original issue, but still doesn't match the first slashes: URLs with slash in parameter?
Since OP said in some comment:
There is no purpose -- but the tokens are generated and not
necessarily by my code. So I don't have control over the token
generation.
and also:
I've attempted to UrlEncode /Decode to see if this works, but the
slash (encoded as %2F) still gets eaten for whatever reason. As a side
note, I can say that Base64 encoding it will fix this but that I can't
change this at this point because it would break the API for existing
apps.
I would say that one of best choices to avoid issues with special characters is firstly encoding the whole token as a base 64 string, and then url-encode it to encode possible characters like =.
That is, you can configure a route where you don't need {token*} but just {token}. If you need to simplify the token decoding, you can implement an ActionFilterAttribute that decodes the so-called bound parameter under the hoods.
Others have already done this way...
OAuth2 already sends basic authentication's credentials encoding them as a base64 string (you convert user:password to the whole base 64 string).
It's a common way of avoding these issues and base 64 strings can be decoded in every modern client and server platform.
Related
Running this query:
"https://someurl.com/graph/users/%24myuser%40myorg.org/licenseDetails?$select=skuId,skuPartNumber"
Results in:
"The request URI is not valid. Since the segment 'users' refers to a collection, this must be the last segment in the request URI or it must be followed by an function or action that can be bound to it otherwise all intermediate segments must refer to a single resource"
After some playing around, I managed to make it work by adding a second dollarsign (%24) before myuser.
This seems to be poorly documented, even here https://docs.oasis-open.org/odata/odata/v4.0/errata02/os/complete/abnf/odata-abnf-construction-rules.txt.
Is there a c# library that correctly escapes a given literal string for odata instead of everyone manually hitting problems from time to time ?
Shouldn't the doc be improved for these special cases that don't seem to be documented ?
A $ prefix in an OData url typically designates that your are about to use a System Query Option, such as $filter, $select or $expand. This is likely causing your key to be interpreted as such.
Since you're using an entity key to access a specific entity then navigate to one of it's navigation properties, you can typically provide the key directly within parenthesis on the entity. This is simple when you don't have a composite key, and looks like this:
/users('my-key-value')
You can then expand a particular navigation property and select down to specific properties (assuming the implementation is compliant with this) in the same way you were previously attempting, leading to
/users('$myuser#myorg.org')/licenseDetails?$select=skuId,skuPartNumber
As stated by Jonathon Chase, you can use /users('$myuser#myorg.org')/licenseDetails as an alternative syntax.
I have the following configured route:
routes.MapHttpRoute("oneKey",
"{controller}/{id}");
If I go to the endpoint /users/Maticicero, the route will be correctly delegated to the GET method in my UsersController class.
However I am getting the id parameter just as it was typed, like Maticicero.
Since this is the primary key of my database, and the comparision is case sensitive, I need to normalize my ids to lowercase, that is:
id = id.ToLower()
I have to do this in every controller and in every method that takes the id parameter.
Is there a way to tell my Web Api (or route handler) to automatically lower case these parameters (or the whole url)?
Try to use routes.LowercaseUrls = true; when you register your routes. MSDN
RouteCollection.LowercaseUrls:
Gets or sets a value that indicates whether URLs are converted to
lower case when virtual paths are normalized.
But be careful when you use your requests like this http://host/controller?id=Maticicero
If a query string is included in the URL, that part of the URL is not
converted to lower case.
Also take a look at this SO question: How can I have lowercase routes in ASP.NET MVC?
You can trap the URL in the Application_BeginRequest event, parse the URL, and use HttpContext.RewritePath to modify the original URL.
https://msdn.microsoft.com/en-us/library/sa5wkk6d(v=vs.110).aspx
We have a high security application and we want to allow users to enter URLs that other users will see.
This introduces a high risk of XSS hacks - a user could potentially enter javascript that another user ends up executing. Since we hold sensitive data it's essential that this never happens.
What are the best practices in dealing with this? Is any security whitelist or escape pattern alone good enough?
Any advice on dealing with redirections ("this link goes outside our site" message on a warning page before following the link, for instance)
Is there an argument for not supporting user entered links at all?
Clarification:
Basically our users want to input:
stackoverflow.com
And have it output to another user:
stackoverflow.com
What I really worry about is them using this in a XSS hack. I.e. they input:
alert('hacked!');
So other users get this link:
stackoverflow.com
My example is just to explain the risk - I'm well aware that javascript and URLs are different things, but by letting them input the latter they may be able to execute the former.
You'd be amazed how many sites you can break with this trick - HTML is even worse. If they know to deal with links do they also know to sanitise <iframe>, <img> and clever CSS references?
I'm working in a high security environment - a single XSS hack could result in very high losses for us. I'm happy that I could produce a Regex (or use one of the excellent suggestions so far) that could exclude everything that I could think of, but would that be enough?
If you think URLs can't contain code, think again!
https://owasp.org/www-community/xss-filter-evasion-cheatsheet
Read that, and weep.
Here's how we do it on Stack Overflow:
/// <summary>
/// returns "safe" URL, stripping anything outside normal charsets for URL
/// </summary>
public static string SanitizeUrl(string url)
{
return Regex.Replace(url, #"[^-A-Za-z0-9+&##/%?=~_|!:,.;\(\)]", "");
}
The process of rendering a link "safe" should go through three or four steps:
Unescape/re-encode the string you've been given (RSnake has documented a number of tricks at http://ha.ckers.org/xss.html that use escaping and UTF encodings).
Clean the link up: Regexes are a good start - make sure to truncate the string or throw it away if it contains a " (or whatever you use to close the attributes in your output); If you're doing the links only as references to other information you can also force the protocol at the end of this process - if the portion before the first colon is not 'http' or 'https' then append 'http://' to the start. This allows you to create usable links from incomplete input as a user would type into a browser and gives you a last shot at tripping up whatever mischief someone has tried to sneak in.
Check that the result is a well formed URL (protocol://host.domain[:port][/path][/[file]][?queryField=queryValue][#anchor]).
Possibly check the result against a site blacklist or try to fetch it through some sort of malware checker.
If security is a priority I would hope that the users would forgive a bit of paranoia in this process, even if it does end up throwing away some safe links.
Use a library, such as OWASP-ESAPI API:
PHP - http://code.google.com/p/owasp-esapi-php/
Java - http://code.google.com/p/owasp-esapi-java/
.NET - http://code.google.com/p/owasp-esapi-dotnet/
Python - http://code.google.com/p/owasp-esapi-python/
Read the following:
https://www.golemtechnologies.com/articles/prevent-xss#how-to-prevent-cross-site-scripting
https://www.owasp.org/
http://www.secbytes.com/blog/?p=253
For example:
$url = "http://stackoverflow.com"; // e.g., $_GET["user-homepage"];
$esapi = new ESAPI( "/etc/php5/esapi/ESAPI.xml" ); // Modified copy of ESAPI.xml
$sanitizer = ESAPI::getSanitizer();
$sanitized_url = $sanitizer->getSanitizedURL( "user-homepage", $url );
Another example is to use a built-in function. PHP's filter_var function is an example:
$url = "http://stackoverflow.com"; // e.g., $_GET["user-homepage"];
$sanitized_url = filter_var($url, FILTER_SANITIZE_URL);
Using filter_var allows javascript calls, and filters out schemes that are neither http nor https. Using the OWASP ESAPI Sanitizer is probably the best option.
Still another example is the code from WordPress:
http://core.trac.wordpress.org/browser/tags/3.5.1/wp-includes/formatting.php#L2561
Additionally, since there is no way of knowing where the URL links (i.e., it might be a valid URL, but the contents of the URL could be mischievous), Google has a safe browsing API you can call:
https://developers.google.com/safe-browsing/lookup_guide
Rolling your own regex for sanitation is problematic for several reasons:
Unless you are Jon Skeet, the code will have errors.
Existing APIs have many hours of review and testing behind them.
Existing URL-validation APIs consider internationalization.
Existing APIs will be kept up-to-date with emerging standards.
Other issues to consider:
What schemes do you permit (are file:/// and telnet:// acceptable)?
What restrictions do you want to place on the content of the URL (are malware URLs acceptable)?
Just HTMLEncode the links when you output them. Make sure you don't allow javascript: links. (It's best to have a whitelist of protocols that are accepted, e.g., http, https, and mailto.)
You don't specify the language of your application, I will then presume ASP.NET, and for this you can use the Microsoft Anti-Cross Site Scripting Library
It is very easy to use, all you need is an include and that is it :)
While you're on the topic, why not given a read on Design Guidelines for Secure Web Applications
If any other language.... if there is a library for ASP.NET, has to be available as well for other kind of language (PHP, Python, ROR, etc)
For Pythonistas, try Scrapy's w3lib.
OWASP ESAPI pre-dates Python 2.7 and is archived on the now-defunct Google Code.
How about not displaying them as a link? Just use the text.
Combined with a warning to proceed at your own risk may be enough.
addition - see also Should I sanitize HTML markup for a hosted CMS? for a discussion on sanitizing user input
There is a library for javascript that solves this problem
https://github.com/braintree/sanitize-url
Try it =)
In my project written in JavaScript I use this regex as white list:
url.match(/^((https?|ftp):\/\/|\.{0,2}\/)/)
the only limitation is that you need to put ./ in front for files in same directory but I think I can live with that.
Using Regular Expression to prevent XSS vulnerability is becoming complicated thus hard to maintain over time while it could leave some vulnerabilities behind. Having URL validation using regular expression is helpful in some scenarios but better not be mixed with vulnerability checks.
Solution probably is to use combination of an encoder like AntiXssEncoder.UrlEncode for encoding Query portion of the URL and QueryBuilder for the rest:
public sealed class AntiXssUrlEncoder
{
public string EncodeUri(Uri uri, bool isEncoded = false)
{
// Encode the Query portion of URL to prevent XSS attack if is not already encoded. Otherwise let UriBuilder take care code it.
var encodedQuery = isEncoded ? uri.Query.TrimStart('?') : AntiXssEncoder.UrlEncode(uri.Query.TrimStart('?'));
var encodedUri = new UriBuilder
{
Scheme = uri.Scheme,
Host = uri.Host,
Path = uri.AbsolutePath,
Query = encodedQuery.Trim(),
Fragment = uri.Fragment
};
if (uri.Port != 80 && uri.Port != 443)
{
encodedUri.Port = uri.Port;
}
return encodedUri.ToString();
}
public static string Encode(string uri)
{
var baseUri = new Uri(uri);
var antiXssUrlEncoder = new AntiXssUrlEncoder();
return antiXssUrlEncoder.EncodeUri(baseUri);
}
}
You may need to include white listing to exclude some characters from encoding. That could become helpful for particular sites.
HTML Encoding the page that render the URL is another thing you may need to consider too.
BTW. Please note that encoding URL may break Web Parameter Tampering so the encoded link may appear not working as expected.
Also, you need to be careful about double encoding
P.S. AntiXssEncoder.UrlEncode was better be named AntiXssEncoder.EncodeForUrl to be more descriptive. Basically, It encodes a string for URL not encode a given URL and return usable URL.
You could use a hex code to convert the entire URL and send it to your server. That way the client would not understand the content in the first glance. After reading the content, you could decode the content URL = ? and send it to the browser.
Allowing a URL and allowing JavaScript are 2 different things.
I have an MVC route like this www.example.com/Find?Key= with the Key being a Base64 string. The problem is that the Base64 string sometimes has a trailing equal sign (=) such as:
huhsdfjbsdf2394=
When that happens, for some reason my route doesn't get hit anymore.
What should I do to resolve this?
My route:
routes.MapRoute(
"FindByKeyRoute",
"Find",
new { controller = "Search", action = "FindByKey" }
);
If I have http://www.example.com/Find?Key=bla then it works.
If I have http://www.example.com/Find?Key=bla= then it doesn't work anymore.
Important Addition:
I'm writing against an IIS7 instance that doesn't allow % or similar encoding. That's why I didn't use UrlEncode to begin with.
EDIT: Original suggestion which apparently doesn't work
I'm sure the reason is that it thinks it's a query parameter called Key. Could you make it a parameter, with that part being the value, e.g.
www.example.com/Find?Route=Key=
I expect that would work (as the parser would be looking for an & to start the next parameter) but it's possible it'll confuse things still.
Suggestion which I believe will work
Alternatively, replace "=" in the base64 encoded value with something else on the way out, and re-replace it on the way back in, if you see what I mean. Basically use a different base64 decodabet.
Alternative suggestion which should work
Before adding base64 to the URL:
private static readonly char[] Base64Padding = new char[] { '=' };
...
base64 = base64.TrimEnd(Base64Padding);
Then before calling Convert.FromBase64String() (which is what I assume you're doing) on the inbound request:
// Round up to a multiple of 4 characters.
int paddingLength = (4 - (base64.Length % 4)) % 4;
base64 = base64.PadRight(base64.Length + paddingLength, '=');
IF you're passing data in the URL you should probably URL Encode it which would take care of the trailing =.
http://www.albionresearch.com/misc/urlencode.php
UrlEncode the encrypted (it is encrypted, right?) parameter.
If it is an encrypted string, beware that spaces and the + character will also get in your way.
Ok, so IIS 7 won't allow some special characters as part of your path. However, it would allow them if they were part of the querystring.
It is apparently, possible, to change this with a reg hack, but I wouldn't recommend that.
What I would suggest, then, is to use an alternate token, as suggested by Mr Skeet, or simply do not use it in your path, use it as querystring, where you CAN url encode it.
If it is an encrypted string, you haven't verified that it is or is not, you may in some cases get other 'illegal' characters. Querystring really would be the way to go.
Except your sample shows it as querystring... So what gives? Where did you find an IIS that won't allow standard uri encoding as part of the querystring??
Ok then. Thanks for the update.
RequestFiltering?
I see. Still that mentions double-encoded values that it blocks. Someone created a URL Sequence to deny any request with the '%' characters? At that point you might want to not use the encrypted string at all, but generate a GUID or something else that is guaranteed to not contain special characters, yet is not trivial to guess.
In my web app, my parameters can contain all sorts of crazy characters (russian chars, slashes, spaces etc) and can therefor not always be represented as-is in a URL.
Sending them on their merry way will work in about 50% of the cases. Some things like spaces are already encoded somewhere (I'm guessing in the Html.BuildUrlFromExpression does). Other things though (like "/" and "*") are not.
Now I don't know what to do anymore because if I encode them myself, my encoding will get partially encoded again and end up wrong. If I don't encode them, some characters will not get through.
What I did is manually .replace() the characters I had problems with.
This is off course not a good idea.
Ideas?
--Edit--
I know there are a multitude of encoding/decoding libraries at my disposal.
It just looks like the mvc framework is already trying to do it for me, but not completely.
<a href="<%=Html.BuildUrlFromExpression<SearchController>(c=>c.Search("", 1, "a \v/&irdStr*ng"))%>" title="my hat's awesome!">
will render me
<a href="/Search.mvc/en/Search/1/a%20%5Cv/&irdStr*ng" title="my hat's awesome!">
Notice how the forward slash, asterisk and ampersand are not escaped.
Why are some escaped and others not? How can I now escape this properly?
Am I doing something wrong or is it the framework?
Parameters should be escaped using Uri.EscapeDataString:
string url = string.Format("http://www.foo.bar/page?name={0}&address={1}",
Uri.EscapeDataString("adlknad /?? lkm#"),
Uri.EscapeDataString(" qeio103 8182"));
Console.WriteLine(url);
Uri uri = new Uri(url);
string[] options = uri.Query.Split('?','&');
foreach (string option in options)
{
string[] parts = option.Split('=');
if (parts.Length == 2)
{
Console.WriteLine("{0} = {1}",parts[0],
Uri.UnescapeDataString(parts[1]));
}
}
AS others have mentioned, if you encode your string first you aviod the issue.
The MVC Framework is encoding characters that it knows it needs to encode, but leaving those that are valid URL characters (e.g. & % ? * /). This is because these are valid URL characters, although they are special chracters in a URL that might not acheive the result you are after.
Try using the Microsoft Anti-Cross Site Scripting library. It contains several Encode methods, which encode all the characters (including #, and characters in other languages). As for decoding, the browser should handle the encoded Url just fine, however if you need to manually decode the Url, use Uri.UnescapeDataString
Hope that helps.
Escaping of forward slahes and dots in path part of url is prohibited by security reason (althrough, it works in mono).
Html.BuildUrlFromExpression needs to be fixed then, would submit this upstream to the MVC project... alternatively do the encoding to the string before passing to BuildUrlFromExpression, and decode it when it comes back out on the other side.
It may not be readily fixable, as IIS may be handling the decoding of the url string beforehand... may need to do some more advanced encoding/decoding for alternative path characters in the utility methods, and decode on your behalf coming out.
I've seen similar posts on this. Too me, it looks like a flaw in MVC. The function would be more appropriately named "BuildUrlFromEncodedExpression". Whats worse, is that the called function needs to decode its input parameters. Yuk.
If there is any overlap between the characters encoded BuildUrlFromExpression() and the characters encoded by the caller (who, I think might fairly just encode any non-alphanumeric for simplicities sake) then you have potential for nasty bugs.
Server.URLEncode or HttpServerUtility.UrlEncode
I see what you're saying now - I didn't realize the question was specific to MVC. Looks like a limitation of that part of the MVC framework - particularly BuildUrlFromExpression is doing some URL encoding, but it knows that also needs some of those punctation as part of the framework URLs.
And also unfortunately, URLEncoding doesn't produce an invariant, i.e.
URLEncode(x) != URLEncode(URLEncode(x))
Wouldn't that be nice. Then you could pre-encode your variables and they wouldn't be double encoded.
There's probably an ASP.NET MVC framework best practice for this. I guess another thing you could do is encode into base64 or something that is URLEncode-invariant.
Have you tried using the Server.UrlEncode() method to do the encoding, and the Server.UrlDecode() method to decode?
I have not had any issues with using it for passing items.