I'm trying to post simple request using Flurl. But there is an exception, because takenPicture string is too large. Is there any way to post large string using Flurl?
System.UriFormatException: Uri is longer than the maximum 32766 characters.
var postData = BaseAdress.PostUrlEncodedAsync(new {
text = Uri.EscapeDataString(body),
subject = Uri.EscapeDataString(subject),
from_email = from,
recipient_email = to,
picture = takenPicture //Base64 string
});
This limitation, while a little arbitrary, is by design in Microsoft's Uri.EscapeDataString method, which is commonly used by applications and libraries (including Flurl) to encode data for both URL queries and URL-encoded request bodies. Both Xamarin and RestSharp Portable have dealt with this so I'm sure a work-around is possible and I might consider this in Flurl if you want to create an issue.
However, keep in mind that URL-encoding binary data like images is very unusual, and there are good reasons to avoid it if possible. Of course, if this is a 3rd-party API you're working with, you don't have much of a choice. But if you have control of the server-side code, I would suggest refactoring that to accept multipart/form-data instead.
Related
Is there a way to use https://translate.google.co.za/ in code?
Maybe make use of Encoding, WebClients and Uri's, but I'm not sure on the correct way to do this.
In code I can get the translate to language and translate from language as well as the content, but how can I incorporate those parameters into the url and then display the end result?
Please Help
Code attempt:
UnicodeEncoding tmpEncoding = new UnicodeEncoding();
string url = String.Format("http://translate.google.co.za/#{0}/{1}/{2}", languageFrom, languageTo, content);
WebClient tmpClient = new WebClient();
tmpClient.Encoding = System.Text.Encoding.ASCII;
string result = tmpEncoding.GetString(tmpClient.DownloadData(url));
The result it gives me is a list of chinese or japanese characters. I dont know what I doing wrong. Maybe the Encoding?
Take a look at the following website click here
You can use the official Google Translate API for this
Take note that it will cost money to translate. Also take a look at other translate api which can be used inside .net
I Did some searching for ya, Bing translator service is a free API for a maximum of 2M characters a monthe from there on you have to pay for it. It also has a nice SDK to go with it.
I found an answer courtesy of Rick Strahl's Web Log (http://weblog.west-wind.com/posts/2011/Aug/06/Translating-with-Google-Translate-without-API-and-C-Code)
Although I didnt use the JavaScriptSerializer it gave me what I wanted. In the form of (\"content\").
So just a bit of string manipulation and I'm golden.
EDIT:
I ended up using the Serializer as the other way didn't give the special characters that form some words, i.e words from french woudnt have those characters that make the words frenchy. Instead it would give a question mark surrounded by a white diamond.
In a WinForms application targeting .NET 4, I am using the WebClient.UploadValues(Uri, "POST", NameValueCollection) to send values to an instance of HttpListener. On the listener side, when the HttpListener.GetContext() method returns, I can access the sent data as a byte [].
I can convert this data to text using EncodingXXX.GetString(buffer) which returns the following:
Key1=Value1
Key2=Value2
...
Each item in the string is delimited by the ampersand sign &. I have to manually split the data into pairs and further split the pairs into keys and values.
I'd like to know if the same data can be accessed similar to ASP .NET WebForms (var value = this.Form["Key"])? Of course, I know we don't have the WebForms context here but I'd prefer to use existing infrastructure for this kind of processing wherever possible.
As per comments, I'm posting this as an answer.
I believe it should work in all cases, let me know if it doesn't, but HttpUtility.ParseQueryString(string) should be a perfect match for this.
It is, as is suggested by the name, meant for parsing URL query strings, but from what I can think, it should be a clean conversion from anything that's permissible in a datastring to anything that's permissible in a URL (of course, not vice-versa).
We have a high security application and we want to allow users to enter URLs that other users will see.
This introduces a high risk of XSS hacks - a user could potentially enter javascript that another user ends up executing. Since we hold sensitive data it's essential that this never happens.
What are the best practices in dealing with this? Is any security whitelist or escape pattern alone good enough?
Any advice on dealing with redirections ("this link goes outside our site" message on a warning page before following the link, for instance)
Is there an argument for not supporting user entered links at all?
Clarification:
Basically our users want to input:
stackoverflow.com
And have it output to another user:
stackoverflow.com
What I really worry about is them using this in a XSS hack. I.e. they input:
alert('hacked!');
So other users get this link:
stackoverflow.com
My example is just to explain the risk - I'm well aware that javascript and URLs are different things, but by letting them input the latter they may be able to execute the former.
You'd be amazed how many sites you can break with this trick - HTML is even worse. If they know to deal with links do they also know to sanitise <iframe>, <img> and clever CSS references?
I'm working in a high security environment - a single XSS hack could result in very high losses for us. I'm happy that I could produce a Regex (or use one of the excellent suggestions so far) that could exclude everything that I could think of, but would that be enough?
If you think URLs can't contain code, think again!
https://owasp.org/www-community/xss-filter-evasion-cheatsheet
Read that, and weep.
Here's how we do it on Stack Overflow:
/// <summary>
/// returns "safe" URL, stripping anything outside normal charsets for URL
/// </summary>
public static string SanitizeUrl(string url)
{
return Regex.Replace(url, #"[^-A-Za-z0-9+&##/%?=~_|!:,.;\(\)]", "");
}
The process of rendering a link "safe" should go through three or four steps:
Unescape/re-encode the string you've been given (RSnake has documented a number of tricks at http://ha.ckers.org/xss.html that use escaping and UTF encodings).
Clean the link up: Regexes are a good start - make sure to truncate the string or throw it away if it contains a " (or whatever you use to close the attributes in your output); If you're doing the links only as references to other information you can also force the protocol at the end of this process - if the portion before the first colon is not 'http' or 'https' then append 'http://' to the start. This allows you to create usable links from incomplete input as a user would type into a browser and gives you a last shot at tripping up whatever mischief someone has tried to sneak in.
Check that the result is a well formed URL (protocol://host.domain[:port][/path][/[file]][?queryField=queryValue][#anchor]).
Possibly check the result against a site blacklist or try to fetch it through some sort of malware checker.
If security is a priority I would hope that the users would forgive a bit of paranoia in this process, even if it does end up throwing away some safe links.
Use a library, such as OWASP-ESAPI API:
PHP - http://code.google.com/p/owasp-esapi-php/
Java - http://code.google.com/p/owasp-esapi-java/
.NET - http://code.google.com/p/owasp-esapi-dotnet/
Python - http://code.google.com/p/owasp-esapi-python/
Read the following:
https://www.golemtechnologies.com/articles/prevent-xss#how-to-prevent-cross-site-scripting
https://www.owasp.org/
http://www.secbytes.com/blog/?p=253
For example:
$url = "http://stackoverflow.com"; // e.g., $_GET["user-homepage"];
$esapi = new ESAPI( "/etc/php5/esapi/ESAPI.xml" ); // Modified copy of ESAPI.xml
$sanitizer = ESAPI::getSanitizer();
$sanitized_url = $sanitizer->getSanitizedURL( "user-homepage", $url );
Another example is to use a built-in function. PHP's filter_var function is an example:
$url = "http://stackoverflow.com"; // e.g., $_GET["user-homepage"];
$sanitized_url = filter_var($url, FILTER_SANITIZE_URL);
Using filter_var allows javascript calls, and filters out schemes that are neither http nor https. Using the OWASP ESAPI Sanitizer is probably the best option.
Still another example is the code from WordPress:
http://core.trac.wordpress.org/browser/tags/3.5.1/wp-includes/formatting.php#L2561
Additionally, since there is no way of knowing where the URL links (i.e., it might be a valid URL, but the contents of the URL could be mischievous), Google has a safe browsing API you can call:
https://developers.google.com/safe-browsing/lookup_guide
Rolling your own regex for sanitation is problematic for several reasons:
Unless you are Jon Skeet, the code will have errors.
Existing APIs have many hours of review and testing behind them.
Existing URL-validation APIs consider internationalization.
Existing APIs will be kept up-to-date with emerging standards.
Other issues to consider:
What schemes do you permit (are file:/// and telnet:// acceptable)?
What restrictions do you want to place on the content of the URL (are malware URLs acceptable)?
Just HTMLEncode the links when you output them. Make sure you don't allow javascript: links. (It's best to have a whitelist of protocols that are accepted, e.g., http, https, and mailto.)
You don't specify the language of your application, I will then presume ASP.NET, and for this you can use the Microsoft Anti-Cross Site Scripting Library
It is very easy to use, all you need is an include and that is it :)
While you're on the topic, why not given a read on Design Guidelines for Secure Web Applications
If any other language.... if there is a library for ASP.NET, has to be available as well for other kind of language (PHP, Python, ROR, etc)
For Pythonistas, try Scrapy's w3lib.
OWASP ESAPI pre-dates Python 2.7 and is archived on the now-defunct Google Code.
How about not displaying them as a link? Just use the text.
Combined with a warning to proceed at your own risk may be enough.
addition - see also Should I sanitize HTML markup for a hosted CMS? for a discussion on sanitizing user input
There is a library for javascript that solves this problem
https://github.com/braintree/sanitize-url
Try it =)
In my project written in JavaScript I use this regex as white list:
url.match(/^((https?|ftp):\/\/|\.{0,2}\/)/)
the only limitation is that you need to put ./ in front for files in same directory but I think I can live with that.
Using Regular Expression to prevent XSS vulnerability is becoming complicated thus hard to maintain over time while it could leave some vulnerabilities behind. Having URL validation using regular expression is helpful in some scenarios but better not be mixed with vulnerability checks.
Solution probably is to use combination of an encoder like AntiXssEncoder.UrlEncode for encoding Query portion of the URL and QueryBuilder for the rest:
public sealed class AntiXssUrlEncoder
{
public string EncodeUri(Uri uri, bool isEncoded = false)
{
// Encode the Query portion of URL to prevent XSS attack if is not already encoded. Otherwise let UriBuilder take care code it.
var encodedQuery = isEncoded ? uri.Query.TrimStart('?') : AntiXssEncoder.UrlEncode(uri.Query.TrimStart('?'));
var encodedUri = new UriBuilder
{
Scheme = uri.Scheme,
Host = uri.Host,
Path = uri.AbsolutePath,
Query = encodedQuery.Trim(),
Fragment = uri.Fragment
};
if (uri.Port != 80 && uri.Port != 443)
{
encodedUri.Port = uri.Port;
}
return encodedUri.ToString();
}
public static string Encode(string uri)
{
var baseUri = new Uri(uri);
var antiXssUrlEncoder = new AntiXssUrlEncoder();
return antiXssUrlEncoder.EncodeUri(baseUri);
}
}
You may need to include white listing to exclude some characters from encoding. That could become helpful for particular sites.
HTML Encoding the page that render the URL is another thing you may need to consider too.
BTW. Please note that encoding URL may break Web Parameter Tampering so the encoded link may appear not working as expected.
Also, you need to be careful about double encoding
P.S. AntiXssEncoder.UrlEncode was better be named AntiXssEncoder.EncodeForUrl to be more descriptive. Basically, It encodes a string for URL not encode a given URL and return usable URL.
You could use a hex code to convert the entire URL and send it to your server. That way the client would not understand the content in the first glance. After reading the content, you could decode the content URL = ? and send it to the browser.
Allowing a URL and allowing JavaScript are 2 different things.
I'm having trouble figuring out the best way to encode the POST parameters to a server call. I writing a C# client that will be served by a PHP server. I want to allow a great deal of flexibility in the parameters, so my current plan is to have a single parameter that I use JSON to encode. For example:
params = {"object":"Main","function":"doecho","params":["echothis...."]}
I'm using the C# WebRequest object and contentType of "application/x-www-form-urlencoded; charset=UTF-8". The data gets to the server and everything works as expected until I add illegal JSON characters in the data.
For example, if I use the following data, then I can't do a json_decode on it on the server side. The server appears to automatically turn the %40 into a double-quote (") when I read it with $this->getRequest()->getParams(); (Zend_Framework).
params = {"object":"Main","function":"doecho","params":["echothis%25%5d%22%40%3d%26...."]}
What is the best practice here? Do I need to base64 encode the data? Is there something obvious I'm missing with the content type or a php setting?
I have full control over the client and server, so I'd like to know what is the right/best thing to do.
While potentially any content-type can be used to upload to HTTP, there are three used in practice:
One specifically set by a given service's documentation.
application/x-www-form-urlencoded - the default used by HTML forms.
multipart/form-data - the other form used by HTML forms, required when it includes form uploads.
Due to 2 and 3 being so commonly used (as they are supported by all browser's for submitting forms), pretty much all server-side tech has things to handle them. So unless the PHP part does something strange, you should be able to use either.
application/x-www-form-urlencoded isn't appropriate for some data, but is the simplest for what it is used for. It's pretty much the same as the way query-strings are created for GET form requests, but as POST content.
Hence you want your content to be:
"params=" + Uri.EscapeDataString(paramData)
As such the first becomes:
params=%7B%22object%22%3A%22Main%22%2C%22function%22%3A%22doecho%22%2C%22params%22%3A%5B%22echothis....%22%5D%7D
And the second:
params=%7B%22object%22%3A%22Ccmes_Main%22%2C%22function%22%3A%22doecho%22%2C%22params%22%3A%5B%22echothis%2525%255d%2522%2540%253d%2526....%22%5D%7D
Both of which PHP's built-ins will turn back into the forms in your question.
My first thought was encoding it as base64. I think this should be the simplies way.
From arcanecode.com:
static public string EncodeTo64(string toEncode)
{
byte[] toEncodeAsBytes = System.Text.ASCIIEncoding.ASCII.GetBytes(toEncode);
string returnValue = System.Convert.ToBase64String(toEncodeAsBytes);
return returnValue;
}
I'm working on a quick captcha generator for a simple site I'm putting together, and I'm hoping to pass an encrypted key in the url of the page. I could probably do this as a query string parameter easy enough, but I'm hoping not too (just because nothing else runs off the query string)...
My encryption code produces a byte[], which is then transformed using Convert.ToBase64String(byte[]) into a string. This string, however, is still not quite url friendly, as it can contain things like '/' and '='. Does anyone know of a better function in the .NET framework to convert a byte array to a url friendly string?
I know all about System.Web.HttpUtility.UrlEncode() and its equivalents, however, they only work properly with query string parameters. If I url encode an '=' inside of the path, my web server brings back a 400 Bad Request error.
Anyways, not a critical issue, but hoping someone can give me a nice solution
**EDIT: Just to be absolutely sure exactly what I'm doing with the string, I figured I would supply a little more information.
The byte[] that results from my encryption algorithm should be fed through some sort of algorithm to make it into a url friendly string. After this, it becomes the content of an XElement, which is then used as the source document for an XSLT transformation, and is used as a part of the href attribute for an anchor. I don't believe the xslt transformation is causing the issues, since what is coming through on the path appears to be an encoded query string parameter, but causes the HTTP 400
I've also tried HttpUtility.UrlPathEncode() on a base64 string, but that doesn't seem to do the trick either (I still end up with '/'s in my url)**
You're looking for HttpServerUtility.UrlTokenEncode and HttpServerUtility.UrlTokenDecode, in System.Web.
They encode in base64, replacing the potentially dangerous '+' and '/' chars with '-' and '_' instead.
MSDN documentation
For ASP.NET Core 6.0+ use Microsoft.AspNetCore.WebUtilities.WebEncoders:
byte[] bytes = RandomNumberGenerator.GetBytes(64);
string encoded = WebEncoders.Base64UrlEncode(bytes);
byte[] decoded = WebEncoders.Base64UrlDecode(encoded);
Have a look at System.BitConverter.ToString(myByteArray)
Handy for one way encoding for things like hashes but as pointed out by ssg it's not very efficient. I wouldn't recommend it for large amounts of data.
HttpServerUtility.UrlTokenEncode and similar are no longer available in .NET Core as of .NET 6.
The following method should provide an RFC4648ยง5-compliant base64 string based on a quick reading.
This code allocates at least four objects on the heap, so ultra-high performance use cases should tune it a bit and perhaps wait for .NET 7, which is expected to introduce a means to get the random bytes into a Span for stack allocation.
private string GetUrlSafeRandomBytes(int byteCount)
{
var salt = new byte[byteCount];
using var rng = RandomNumberGenerator.Create();
rng.GetBytes(salt);
var asBase64 = Convert.ToBase64String(salt);
var asUrlSafeString = asBase64.Replace('+', '-').Replace('/', '_');
return asUrlSafeString;
}