We Keep Coding

WebRequest POST long value?

I have a string with 300 rows, there is a way to do it POST?
Here is my code, is currently working on a limited amount of short letters:
WebRequest req = (HttpWebRequest)WebRequest.Create(
"http://thisisurl/test.php?ad=test&f=" + information_data);
req.Method = "POST";
WebResponse res = req.GetResponse();

I'm going to explain your problem right now and go ahead and give you a possible solution at the end.
You are hitting the character limit for url length / query parameter length. IE limits it as low as 2,083.
The data you are providing should be sent in the body of the http request, not the URL parameters.
A Post Request Normally is done in the following format(Code from the link).
using (var wb = new WebClient())
{
var data = new NameValueCollection();
data["username"] = "myUser";
data["password"] = "myPassword";
var response = wb.UploadValues(url, "POST", data);
}
This thread should have enough info if you want to use the WebRequest class instead:
HTTP request with post

HttpWebRequest.GetResponse() keeps getting timed out

i wrote a simple C# function to retrieve trade history from MtGox with following API call:
https://data.mtgox.com/api/1/BTCUSD/trades?since=<trade_id>
documented here: https://en.bitcoin.it/wiki/MtGox/API/HTTP/v1#Multi_currency_trades
here's the function:
string GetTradesOnline(Int64 tid)
{
Thread.Sleep(30000);
// communicate
string url = "https://data.mtgox.com/api/1/BTCUSD/trades?since=" + tid.ToString();
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
StreamReader reader = new StreamReader(response.GetResponseStream());
string json = reader.ReadToEnd();
reader.Close();
reader.Dispose();
response.Close();
return json;
}
i'm starting at tid=0 (trade id) to get the data (from the very beginning). for each request, i receive a response containing 1000 trade details. i always send the trade id from the previous response for the next request. it works fine for exactly 4 requests & responses. but after that, the following line throws a "System.Net.WebException", saying that "The operation has timed out":
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
here are the facts:
catching the exception and retying keeps causing the same exception
the default HttpWebRequest .Timeout and .ReadWriteTimeout are already high enough (over a minute)
changing HttpWebRequest.KeepAlive to false didn't solve anything either
it seems to always work in the browser even while the function is failing
it has no problems retrieveing the response from https://www.google.com
the amount of successful responses before the exceptions varies from day to day (but browser always works)
starting at the trade id that failed last time causes the exception immediately
calling this function from the main thread instead still caused the exception
running it on a different machine didn't work
running it from a different IP didn't work
increasing Thread.Sleep inbetween requests does not help
any ideas of what could be wrong?

I had the very same issue.
For me the fix was as simple as wrapping the HttpWebResponse code in using block.
using (HttpWebResponse response = (HttpWebResponse) request.GetResponse())
{
// Do your processings here....
}
Details: This issue usually happens when several requests are made to the same host, and WebResponse is not disposed properly. That is where using block will properly dispose the WebResponse object properly and thus solving the issue.

There are two kind of timeouts. Client timeout and server timeout. Have you tried doing something like this:
request.Timeout = Timeout.Infinite;
request.KeepAlive = true;
Try something like this...

I just had similar troubles calling a REST Service on a LINUX Server thru ssl. After trying many different configuration scenarios I found out that I had to send a UserAgent in the http head.
Here is my final method for calling the REST API.
private static string RunWebRequest(string url, string json)
{
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
// Header
request.ContentType = "application/json";
request.Method = "POST";
request.AllowAutoRedirect = false;
request.KeepAlive = false;
request.Timeout = 30000;
request.ReadWriteTimeout = 30000;
request.UserAgent = "test.net";
request.Accept = "application/json";
request.ProtocolVersion = HttpVersion.Version11;
request.Headers.Add("Accept-Language","de_DE");
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls;
ServicePointManager.ServerCertificateValidationCallback = delegate { return true; };
byte[] bytes = Encoding.UTF8.GetBytes(json);
request.ContentLength = bytes.Length;
using (var writer = request.GetRequestStream())
{
writer.Write(bytes, 0, bytes.Length);
writer.Flush();
writer.Close();
}
var httpResponse = (HttpWebResponse)request.GetResponse();
using (var streamReader = new StreamReader(httpResponse.GetResponseStream()))
{
var jsonReturn = streamReader.ReadToEnd();
return jsonReturn;
}
}

This is not a solution, but just an alternative:
These days i almost only use WebClient instead of HttpWebRequest. Especially WebClient.UploadString for POST and PUT and WebClient.DownloadString. These simply take and return strings. This way i don't have to deal with streams objects, except when i get a WebException. i can also set the content type with WebClient.Headers["Content-type"] if necessary. The using statement also makes life easier by calling Dispose for me.
Rarely for performance, i set System.Net.ServicePointManager.DefaultConnectionLimit high and instead use HttpClient with it's Async methods for simultaneous calls.
This is how i would do it now
string GetTradesOnline(Int64 tid)
{
using (var wc = new WebClient())
{
return wc.DownloadString("https://data.mtgox.com/api/1/BTCUSD/trades?since=" + tid.ToString());
}
}
2 more POST examples
// POST
string SubmitData(string data)
{
string response;
using (var wc = new WebClient())
{
wc.Headers["Content-type"] = "text/plain";
response = wc.UploadString("https://data.mtgox.com/api/1/BTCUSD/trades", "POST", data);
}
return response;
}
// POST: easily url encode multiple parameters
string SubmitForm(string project, string subject, string sender, string message)
{
// url encoded query
NameValueCollection query = HttpUtility.ParseQueryString(string.Empty);
query.Add("project", project);
query.Add("subject", subject);
// url encoded data
NameValueCollection data = HttpUtility.ParseQueryString(string.Empty);
data.Add("sender", sender);
data.Add("message", message);
string response;
using (var wc = new WebClient())
{
wc.Headers[HttpRequestHeader.ContentType] = "application/x-www-form-urlencoded";
response = wc.UploadString( "https://data.mtgox.com/api/1/BTCUSD/trades?"+query.ToString()
, WebRequestMethods.Http.Post
, data.ToString()
);
}
return response;
}
Error handling
try
{
Console.WriteLine(GetTradesOnline(0));
string data = File.ReadAllText(#"C:\mydata.txt");
Console.WriteLine(SubmitData(data));
Console.WriteLine(SubmitForm("The Big Project", "Progress", "John Smith", "almost done"));
}
catch (WebException ex)
{
string msg;
if (ex.Response != null)
{
// read response HTTP body
using (var sr = new StreamReader(ex.Response.GetResponseStream())) msg = sr.ReadToEnd();
}
else
{
msg = ex.Message;
}
Log(msg);
}

For what it's worth, I was experiencing the same issues with timeouts every time I used it, even though calls went through to the server I was calling. The problem in my case was that I had Expect set to application/json, when that wasn't what the server was returning.

WebRequest extremely slow

My method looks like this:
public string Request(string action, NameValueCollection parameters, uint? timeoutInSeconds = null)
{
parameters = parameters ?? new NameValueCollection();
ProvideCredentialsFor(ref parameters);
var data = parameters.ToUrlParams(); // my extension method converts the collection to a string, works well
byte[] dataStream = Encoding.UTF8.GetBytes(data);
string request = ServiceUrl + action;
var webRequest = (HttpWebRequest)WebRequest.Create(request);
webRequest.AllowAutoRedirect = false;
webRequest.Method = "POST";
webRequest.ContentType = "application/x-www-form-urlencoded";
webRequest.ContentLength = dataStream.Length;
webRequest.Timeout = (int)(timeoutInSeconds == null ? DefaultTimeoutMs : timeoutInSeconds * 1000);
webRequest.Proxy = null; // should make it faster...
using (var newStream = webRequest.GetRequestStream())
{
newStream.Write(dataStream, 0, dataStream.Length);
}
var webResponse = (HttpWebResponse)webRequest.GetResponse();
string uri = webResponse.Headers["Location"];
string result;
using (var sr = new StreamReader(webResponse.GetResponseStream()))
{
result = sr.ReadToEnd();
}
return result;
}
The server sends JSON in response. It works fine for small JSON, but when I request a large one - something goes wrong. By large one I mean something that takes 1-2 minutes to appear in a browser (google chrome, including server side generation time). It's actually 412KB of text. When I try to ask for the same JSON with the method above I get a web exception (timeout). I changed the timeout to 10 minutes (at least 5 times longer than chrome). Still the same.
Any ideas?
EDIT
This seems to have something to do with MS technologies. On IE this JSON also won't load.

Make sure you close your request. Otherwise, once you hit the maximum number of allowed connections (as low as four, for me, on one occasion), you have to wait for the earlier ones to time out. This is best done using
using (var response = webRequest.GetResponse()) {...

Using google translate api POST method, returns 404

So basically I have some french text and want to traduce it in english using c#.
I'm using google translate api, which was working fine until i had a text.length > 1000 .... then I realized that I must use POST method.
Since I use the post method, it returns me 404.
btw i know the api is deprecated, I though it would be cool anyways but I'm starting to realize maybe i should use bing ?
string fromLanguage = "fr";
string toLanguage = "en";
String apiKey = "AIzasdfasdfJvWKNioZwLg-3kyYsm4_dao";
String apiUrl = "https://www.googleapis.com/language/translate/v2";
string tmpTranslatedContent = Translate(apiUrl, "salut la planete", apiKey, fromLanguage, toLanguage);
public string Translate(string url, string text, string key, string fromLanguage, string toLanguage)
{
PostSubmitter post = new PostSubmitter();
post.Url = url;
post.PostItems.Add("key", key);
post.PostItems.Add("source", fromLanguage);
post.PostItems.Add("target", toLanguage);
post.PostItems.Add("q", text);
post.Type = PostSubmitter.PostTypeEnum.Post;
string result = post.Post();
return result;
}
PostSubmitter is a class i found when searching google
Comments on the site are saying it works.....
the main part of the class looks like this
HttpWebRequest request=null;
if (m_type==PostTypeEnum.Post)
{
Uri uri = new Uri(url);
request = (HttpWebRequest) WebRequest.Create(uri);
request.Method = "POST";
request.ContentType = "application/x-www-form-urlencoded";
request.ContentLength = postData.Length;
using(Stream writeStream = request.GetRequestStream())
{
UTF8Encoding encoding = new UTF8Encoding();
byte[] bytes = encoding.GetBytes(postData);
writeStream.Write(bytes, 0, bytes.Length);
}
thanks.

This is a little old, but I just ran into a similar problem but with PHP instead of C# and the fix should be quite similar.
Basically, even though you are using POST, you still need to tell Google that from a REST point of view you are actually doing a GET. This can be done with the X-HTTP-Method-Override header, setting it to be: X-HTTP-Method-Override: GET
Google tells me that as ASP.NET MVC, version 2, there is a method HttpHelper.HttpMethodOverride that will allow you to do this.
According to the Google Translate API however, text is still limited to 5k even when posting.

C# HttpWebRequest of type "application/x-www-form-urlencoded" - how to send '&' character in content body?

I'm writing a small API-connected application in C#.
I connect to a API which has a method that takes a long string, the contents of a calendar(ics) file.
I'm doing it like this:
HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create(URL);
request.Method = "POST";
request.AllowAutoRedirect = false;
request.CookieContainer = my_cookie_container;
request.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8";
request.ContentType = "application/x-www-form-urlencoded";
string iCalStr = GetCalendarAsString();
string strNew = "&uploadfile=true&file=" + iCalStr;
using (StreamWriter stOut = new StreamWriter(request.GetRequestStream(), System.Text.Encoding.ASCII))
{
stOut.Write(strNew);
stOut.Close();
}
This seems to work great, until I add some specific HTML in my calendar.
If I have a '&nbsp' somewhere in my calendar (or similar) the server only gets all the data up to the '&'-point, so I'm assuming the '&' makes it look like anything after this point belongs to a new parameter?
How can I fix this?

First install "Microsoft ASP.NET Web API Client" nuget package:
PM > Install-Package Microsoft.AspNet.WebApi.Client
Then use the following function to post your data:
public static async Task<TResult> PostFormUrlEncoded<TResult>(string url, IEnumerable<KeyValuePair<string, string>> postData)
{
using (var httpClient = new HttpClient())
{
using (var content = new FormUrlEncodedContent(postData))
{
content.Headers.Clear();
content.Headers.Add("Content-Type", "application/x-www-form-urlencoded");
HttpResponseMessage response = await httpClient.PostAsync(url, content);
return await response.Content.ReadAsAsync<TResult>();
}
}
}
And this is how to use it:
TokenResponse tokenResponse =
await PostFormUrlEncoded<TokenResponse>(OAuth2Url, OAuth2PostData);
or
TokenResponse tokenResponse =
(Task.Run(async ()
=> await PostFormUrlEncoded<TokenResponse>(OAuth2Url, OAuth2PostData)))
.Result
or (not recommended)
TokenResponse tokenResponse =
PostFormUrlEncoded<TokenResponse>(OAuth2Url, OAuth2PostData).Result;

Since your content-type is application/x-www-form-urlencoded you'll need to encode the POST body, especially if it contains characters like & which have special meaning in a form.
Try passing your string through HttpUtility.UrlEncode before writing it to the request stream.
Here are a couple links for reference.
http://en.wikipedia.org/wiki/Percent-encoding

As long as the server allows the ampresand character to be POSTed (not all do as it can be unsafe), all you should have to do is URL Encode the character. In the case of an ampresand, you should replace the character with %26.
.NET provides a nice way of encoding the entire string for you though:
string strNew = "&uploadfile=true&file=" + HttpUtility.UrlEncode(iCalStr);