I´m using .NET C# and want to use HTTPrequest to make a POST in the same way as it is written in Curl:
curl
--header ’Accept: text/html’
--user ’test1:password’
--Form ’wait=0’
--Form ’data=#data.csv;type=text/csv’
some URL address here
I´m getting Error 500 Internal error from the Internet server.
The data to be send is a CSV file.
My source code in C#:
string data = "Time, A, B\n20120101, 100, 24\n20120102\n, 101, 27";
// Create a request using a URL that can receive a post.
request = WebRequest.Create(url);
//Set authorization
request.Credentials = new NetworkCredential(username, password);
// Set the Method property of the request to POST.
request.Method = "POST";
// Create POST data and convert it to a byte array.
byte[] byteArray = Encoding.UTF8.GetBytes(data);
// Set the ContentType property of the WebRequest.
request.ContentType = "application/x-www-form-urlencoded";
// Set the ContentLength property of the WebRequest.
request.ContentLength = byteArray.Length;
// Get the request stream.
dataStream = request.GetRequestStream();
// Write the data to the request stream.
dataStream.Write(byteArray, 0, byteArray.Length);
// Close the Stream object.
dataStream.Close();
// Get the original response.
WebResponse response = request.GetResponse();
this.Status = ((HttpWebResponse)response).StatusDescription;
// Get the stream containing all content returned by the requested server.
dataStream = response.GetResponseStream();
// Open the stream using a StreamReader for easy access.
StreamReader reader = new StreamReader(dataStream);
// Read the content fully up to the end.
string responseFromServer = reader.ReadToEnd();
// Clean up the streams.
reader.Close();
dataStream.Close();
response.Close();
What am I doing wrong?
cURL uses the multipart/form-data content type to create its requests. I think this is required in your case as you are passing both a text parameter and a file. Sadly the .Net framework does not appear to have built in support for creating this content type.
There is a question on SO that gives example code for how to do it yourself here: Multipart forms from C# client
For reference the HTTP request generated by that cURL command looks something like this:
POST http://www.example.com/example HTTP/1.1
Authorization: Basic J3Rlc3Q6cGFzc3dvcmQn
User-Agent: curl/7.33.0
Host: www.example.com
Accept: text/html
Connection: Keep-Alive
Content-Length: 335
Expect: 100-continue
Content-Type: multipart/form-data; boundary=------------------------99aa46d554951aa6
--------------------------99aa46d554951aa6
Content-Disposition: form-data; name="'wait"
0'
--------------------------99aa46d554951aa6
Content-Disposition: form-data; name="'data"; filename="data.csv"
Content-Type: text/csv'
Time, A, B\n20120101, 100, 24\n20120102\n, 101, 27
--------------------------99aa46d554951aa6--
Doesn't data need to be URL encoded for a form post? http://en.wikipedia.org/wiki/POST_%28HTTP%29#Use_for_submitting_web_forms
Related
I need to replicate a HTTP request using C# and the HttpWebRequest, that looks like this:
HEADER: POST /upnp/control/x_contact HTTP/1.1 Host: fritz.box:49000
Connection: Keep-Alive User-Agent:...
So far I got everything working, except for the part after the POST
"/upnp/control/x_contact HTTP/1.1".
How would I have to modify the HttpWebRequest to get these additional data to the server?
PS: This is how the java code looks, that generates the correct request. I would like to port this code to C# and send the same request from there:
socket = new Socket(InetAddress.getByName("fritz.box"), 49000);
// Send header
writer = new BufferedWriter(new OutputStreamWriter(socket.getOutputStream(),"UTF-8"));
writer.write("POST /igdupnp/control/WANIPConn1 HTTP/1.1");
writer.write("Host: fritz.box:49000\r\n");
writer.write("SOAPACTION: \"urn:schemas-upnp-org:service:WANIPConnection:1#GetStatusInfo\"\r\n");
writer.write("Content-Type: text/xml; charset=\"utf-8\""+"\r\n");
writer.write("Content-Length: " + myData.length() + "\r\n");
writer.write("\r\n");
// Send data
writer.write(myData);
writer.flush();
And this is my C# code so far (producing an incomplete HTML request):
HttpWebRequest webRequest = (HttpWebRequest)WebRequest.Create (#"http://fritz.box:49000");
webRequest.Headers.Add ("SOAPACTION", "urn:schemas-upnp-org:service:WANIPConnection:1#GetStatusInfo");
webRequest.ContentType = "text/xml;charset=\"utf-8\"";
webRequest.Accept = "text/xml";
webRequest.Method = "POST";
I'm trying to send requests and get responses from MailChimp API . . so far, GET, POST and DELETE are working good however, PATCH always results to Bad Request can you identify the error in this code?
string data = "{\"name\": \"TestListTWOTWOTWO\"}";
HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create(uri);
request.Headers[HttpRequestHeader.Authorization] = accessToken;
request.Method = "PATCH";
request.ContentType = "text/plain;charset=utf-8";
System.Text.UTF8Encoding encoding = new System.Text.UTF8Encoding();
byte[] bytes = encoding.GetBytes(data);
request.ContentLength = bytes.Length;
using (Stream requestStream = request.GetRequestStream())
{
// Send the data.
requestStream.Write(bytes, 0, bytes.Length);
}
var response = (HttpWebResponse)request.GetResponse();
var responseString = new StreamReader(response.GetResponseStream()).ReadToEnd();
the error occus on the line with request.GetResponse();
it is an unhandled WebException saying The remote server returned an error: (400) Bad Request
after checking the error response, here's the what it says
"Your request doesn't appear to be valid JSON:
\nParse error on line 1:\nPATCH /3.0/lists/9bb\n^\n
Expected one of: 'STRING', 'NUMBER', 'NULL', 'TRUE', 'FALSE', '{', '['"
Many C# libraries seem to try to use the Expect: 100-Continue header, which MailChimp/Akamai has a problem with when combined with PATCH. You have two options.
Turn off Expect: 100-Continue in your HTTP library. In one C# library, you do that with a line of code like Client.DefaultRequestHeaders.ExpectContinue = False
Tunnel the PATCH request through HTTP POST using the X-Http-Method-Override header. Here's more details on that header.
Cause PATCH is a quite new RFC, so you would not expect that more then a few services support it at all. You'd better check that if the service supports it.
You send request using json format, but set content type to "text/plain" is that OK?
I would like to scrape some contents from a dynamic web page (seems it is developed in MVC).
Data scraping logics are done with HTML agility, but now the issue is,
HTML returned while requesting for URL from browser and web response of the URL from ASP.NET web request is different.
Mainly browser response has dynamic data I need (renders based on the value passed in query string), but the WebResponse result is different.
Could you please help me to get the actual content of the dynamic web page view WebRequest.
Below is the code I used to read:
WebRequest request = WebRequest.Create(sURL);
request.Method = "Get";
//Get the response
WebResponse response = request.GetResponse();
//Read the stream from the response
StreamReader reader = new StreamReader(response.GetResponseStream(), System.Text.Encoding.UTF8);
To get the content of any web page using HttpWebRequest...
// We will store the html response of the request here
string siteContent = string.Empty;
// The url you want to grab
string url = "http://google.com";
// Here we're creating our request, we haven't actually sent the request to the site yet...
// we're simply building our HTTP request to shoot off to google...
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
request.AutomaticDecompression = DecompressionMethods.GZip;
// Right now... this is what our HTTP Request has been built in to...
/*
GET http://google.com/ HTTP/1.1
Host: google.com
Accept-Encoding: gzip
Connection: Keep-Alive
*/
// Wrap everything that can be disposed in using blocks...
// They dispose of objects and prevent them from lying around in memory...
using(HttpWebResponse response = (HttpWebResponse)request.GetResponse()) // Go query google
using(Stream responseStream = response.GetResponseStream()) // Load the response stream
using(StreamReader streamReader = new StreamReader(responseStream)) // Load the stream reader to read the response
{
siteContent = streamReader.ReadToEnd(); // Read the entire response and store it in the siteContent variable
}
// magic...
Console.WriteLine (siteContent);
I am trying to send Facebook graph link to the AppEngine server. I receive "Malformed string exception". Here is my method sending json to server:
public async Task<string> SendJSONData(string urlToCall, string JSONData)
{
// server to POST to
string url = urlToCall;
// HTTP web request
var httpWebRequest = (HttpWebRequest)WebRequest.Create(url);
httpWebRequest.ContentType = "application/x-www-form-urlencoded";
httpWebRequest.Method = "POST";
// Write the request Asynchronously
using (var stream = await Task.Factory.FromAsync<Stream>(httpWebRequest.BeginGetRequestStream,
httpWebRequest.EndGetRequestStream, null))
{
//create some json string
string json = "action=" + JSONData;
// convert json to byte array
byte[] jsonAsBytes = Encoding.UTF8.GetBytes(json);
// Write the bytes to the stream
await stream.WriteAsync(jsonAsBytes, 0, jsonAsBytes.Length);
}
WebResponse response = await httpWebRequest.GetResponseAsync();
StreamReader requestReader = new StreamReader(response.GetResponseStream());
String webResponse = requestReader.ReadToEnd();
return webResponse; }
Here is what I sniff using Fiddler:
POST http://x.appspot.com/register HTTP/1.1
Accept: */*
Content-Length: 376
Accept-Encoding: identity
Content-Type: application/x-www-form-urlencoded
User-Agent: NativeHost
Host: x.appspot.com
Connection: Keep-Alive
Pragma: no-cache
action={
"mailFb": "mail#gmail.com",
"userName": "Michael",
"userSurname": "w00t",
"nickname": "Michael w00t",
"userSex": "male",
"userAvatar": "https://graph.facebook.com/myperfectid/picture?type=large&access_token=BlahblahblahblahToken"
}
So everything looks fine, but the problem is that i receive the following error in AppEngine log:
2013-03-02 17:52:10.431 /register 500 56ms 0kb NativeHost
W 2013-03-02 17:52:10.427 /register com.google.gson.JsonSyntaxException: com.google.gson.stream.MalformedJsonException: Unterminated string at line 7 column 79 at com.google.g
C 2013-03-02 17:52:10.429 Uncaught exception from servlet com.google.gson.JsonSyntaxException: com.google.gson.stream.MalformedJsonException: Unterminated string at line 7 colu
I managed to narrow the problem to its source, which is "&" character. So my question is, how to fix the code, so that it works with AppEngine.
Oh, here is how i read the received data on the server:
gson.fromJson(reader, User.class);
The problem is highlighted by the fact you're claiming you are sending "Content-Type: application/x-www-form-urlencoded"
But it isn't. Hence the error.
The correct encoding for & is &.
I tried to search previous discussion about this issue but I didn't find one, maybe it's because I didn't use right keywords.
I am writing a small program which posts data to a webpage and gets the response. The site I'm posting data to does not provide an API. After some Googling I came up to the use of HttpWebRequest and HttpWebResponse. The code looks like this:
HttpWebRequest httpRequest = (HttpWebRequest)WebRequest.Create("https://www.site.com/index.aspx");
CookieContainer cookie = new CookieContainer();
httpRequest.CookieContainer = cookie;
String sRequest = "SomeDataHere";
httpRequest.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8";
httpRequest.Headers.Add("Accept-Encoding: gzip, deflate");
httpRequest.Headers.Add("Accept-Language: en-us,en;q=0.5");
httpRequest.Headers.Add("Cookie: SomecookieHere");
httpRequest.Host = "www.site.com";
httpRequest.Referer = "https://www.site.com/";
httpRequest.UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:14.0) Gecko/20100101 Firefox/14.0.1";
httpRequest.ContentType = "application/x-www-form-urlencoded";
//httpRequest.Connection = "keep-alive";
httpRequest.ContentLength = sRequest.Length;
byte[] bytedata = Encoding.UTF8.GetBytes(sRequest);
httpRequest.ContentLength = bytedata.Length;
httpRequest.Method = "POST";
Stream requestStream = httpRequest.GetRequestStream();
requestStream.Write(bytedata, 0, bytedata.Length);
requestStream.Flush();
requestStream.Close();
HttpWebResponse httpWebResponse = (HttpWebResponse)httpRequest.GetResponse();
string sResponse;
using (Stream stream = httpWebResponse.GetResponseStream())
{
StreamReader reader = new StreamReader(stream, System.Text.Encoding.GetEncoding("iso-8859-1"));
sResponse = reader.ReadToEnd();
}
return sResponse;
I used firefox's firebug to get the header and data to post.
My question is, when I store and display the response using a string, all I got are garbled characters, like:
?????*??????xV?J-4Si1?]R?r)f?|??;????2+g???6?N-?????7??? ?6?? x???q v ??? j?Ro??_*?e*??tZN^? 4s?????? ??Pwc??3???|??_????_??9???^??#?Y??"?k??,?a?H?Lp?A?$ ;???C#????e6'?N???L7?j#???ph??y=?I??=(e?V?6C??
By reading the response header using FireBug I got the content type of response:
Content-Type text/html; charset=ISO-8859-1
And it is reflected in my code. I have even tried other encoding such as utf-8 and ascii, still no luck. Maybe I am in the wrong direction.
Please advise. A small code snippet will be even better.
Thanks you.
You're telling the server that you can accept compressed responses with httpRequest.Headers.Add("Accept-Encoding: gzip, deflate");. Try removing that line, and you should get a clear-text response.
HttpWebRequest does have built in support for gzip and deflate if you want to allow compressed responses. Remove the Accept-Encoding header line, and replace it with
httpRequest.AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate
This will take care of adding the appropriate Accept-Encoding header for you, and handle decompressing the content automatically when you receive it.