I tried to search previous discussion about this issue but I didn't find one, maybe it's because I didn't use right keywords.
I am writing a small program which posts data to a webpage and gets the response. The site I'm posting data to does not provide an API. After some Googling I came up to the use of HttpWebRequest and HttpWebResponse. The code looks like this:
HttpWebRequest httpRequest = (HttpWebRequest)WebRequest.Create("https://www.site.com/index.aspx");
CookieContainer cookie = new CookieContainer();
httpRequest.CookieContainer = cookie;
String sRequest = "SomeDataHere";
httpRequest.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8";
httpRequest.Headers.Add("Accept-Encoding: gzip, deflate");
httpRequest.Headers.Add("Accept-Language: en-us,en;q=0.5");
httpRequest.Headers.Add("Cookie: SomecookieHere");
httpRequest.Host = "www.site.com";
httpRequest.Referer = "https://www.site.com/";
httpRequest.UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:14.0) Gecko/20100101 Firefox/14.0.1";
httpRequest.ContentType = "application/x-www-form-urlencoded";
//httpRequest.Connection = "keep-alive";
httpRequest.ContentLength = sRequest.Length;
byte[] bytedata = Encoding.UTF8.GetBytes(sRequest);
httpRequest.ContentLength = bytedata.Length;
httpRequest.Method = "POST";
Stream requestStream = httpRequest.GetRequestStream();
requestStream.Write(bytedata, 0, bytedata.Length);
requestStream.Flush();
requestStream.Close();
HttpWebResponse httpWebResponse = (HttpWebResponse)httpRequest.GetResponse();
string sResponse;
using (Stream stream = httpWebResponse.GetResponseStream())
{
StreamReader reader = new StreamReader(stream, System.Text.Encoding.GetEncoding("iso-8859-1"));
sResponse = reader.ReadToEnd();
}
return sResponse;
I used firefox's firebug to get the header and data to post.
My question is, when I store and display the response using a string, all I got are garbled characters, like:
?????*??????xV?J-4Si1?]R?r)f?|??;????2+g???6?N-?????7??? ?6?? x???q v ??? j?Ro??_*?e*??tZN^? 4s?????? ??Pwc??3???|??_????_??9???^??#?Y??"?k??,?a?H?Lp?A?$ ;???C#????e6'?N???L7?j#???ph??y=?I??=(e?V?6C??
By reading the response header using FireBug I got the content type of response:
Content-Type text/html; charset=ISO-8859-1
And it is reflected in my code. I have even tried other encoding such as utf-8 and ascii, still no luck. Maybe I am in the wrong direction.
Please advise. A small code snippet will be even better.
Thanks you.
You're telling the server that you can accept compressed responses with httpRequest.Headers.Add("Accept-Encoding: gzip, deflate");. Try removing that line, and you should get a clear-text response.
HttpWebRequest does have built in support for gzip and deflate if you want to allow compressed responses. Remove the Accept-Encoding header line, and replace it with
httpRequest.AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate
This will take care of adding the appropriate Accept-Encoding header for you, and handle decompressing the content automatically when you receive it.
Related
You can see my code down there.This is going to show user id of a instagram user as the response but it is showing "�".
private void button18_Click(object sender, EventArgs e)
{
string username = ""; // your username
string password = ""; // your password
HttpWebRequest httpWebRequest = (HttpWebRequest)WebRequest.Create("https://i.instagram.com/api/v1/accounts/login/");
httpWebRequest.Headers.Add("X-IG-Connection-Type", "WiFi");
httpWebRequest.Method = "POST";
httpWebRequest.ContentType = "application/x-www-form-urlencoded; charset=UTF-8";
httpWebRequest.Headers.Add("X-IG-Capabilities", "AQ==");
httpWebRequest.Accept = "*/*";
httpWebRequest.UserAgent = "Instagram 10.9.0 Android (23/6.0.1; 944dpi; 915x1824; samsung; SM-T185; gts210velte; qcom; en_GB)";
httpWebRequest.Headers.Add("Accept-Encoding", "gzip, deflate");
httpWebRequest.Headers.Add("Accept-Language", "en;q=1, ru;q=0.9, ar;q=0.8");
httpWebRequest.Headers.Add("Cookie", "mid=qzejldb8ph9eipis9e6nrd1n457b;csrftoken=a7nd2ov9nbxgqy473aonahi58y21i8ee");
httpWebRequest.Host = "i.instagram.com";
httpWebRequest.KeepAlive = true;
byte[] bytes = new ASCIIEncoding().GetBytes("ig_sig_key_version=5&signed_body=5128c31533802ff7962073bb1ebfa9972cfe3fd9c5e3bd71fe68be1d02aa92c8.%7B%22username%22%3A%22"+ username + "%22%2C%22password%22%3A%22"+ password +"%22%2C%22_uuid%22%3A%22D26E6E86-BDB7-41BE-8688-1D60DE60DAF6%22%2C%22_uid%22%3A%22%22%2C%22device_id%22%3A%22android-a4d01b84202d6018%22%2C%22_csrftoken%22%3A%22a7nd2ov9nbxgqy473aonahi58y21i8ee%22%2C%22login_attempt_count%22%3A%220%22%7D");
httpWebRequest.ContentLength = (long)bytes.Length;
using (Stream requestStream = httpWebRequest.GetRequestStream())
{
requestStream.Write(bytes, 0, bytes.Length);
}
string result = new StreamReader(((HttpWebResponse)httpWebRequest.GetResponse()).GetResponseStream()).ReadToEnd();
textBox1.Text = result;
}
It's answered in a comment by #Dour, but I'm going to add a bit of detail that isn't mentioned.
This following line tells the server: Hey server! Please send me a compressed response (to reduce size).
httpWebRequest.Headers.Add("Accept-Encoding", "gzip, deflate");
so, the response you're getting is a compressed response.
Does that mean you just remove this line ?
Well, Removing it will fix your problem but that isn't the correct way to do that.
It's really better to get a reduced size response because that will need lower time to download it from the server.
What you really need to do is to make HttpWebRequest handles the decompression process for you.
You can do that by setting AutomaticDecompression property.
httpWebRequest.AutomaticDecompression = DecompressionMethods.Deflate | DecompressionMethods.GZip;
Note: With the previous line, you do NOT need to set Accept-Encoding yourself. HttpWebRequest will send it automatically for you. And when you call GetResponse, HttpWebRequest will handle the decompression process for you.
Besides of your problem:
I suggest that you use using statement for getting the response, because with your current code, I think it won't get disposed.
I am new in Programming. I am composing a straightforward application utilizing C# with android in which I need send information to server, information incorporates URL with two parameters. When I attempt to send information to server it just sends the URL however not parameters. Would you be able to help me with respect to this.
var postData = ("id1="+"123456");
postData += ("&id2="+"0123456789");
var request = (HttpWebRequest)WebRequest.Create("http://abc.xyz.com");
var data = Encoding.ASCII.GetBytes(postData);
request.Method = "POST"
request.ContentType = "multipart/form-data";
request.ContentLength = data.Length;
using (var stream = request.GetRequestStream())
{
stream.Write(data, 0, data.Length);
stream.Close();
}
var response = (HttpWebResponse)request.GetResponse();
var responseString = new StreamReader(response.GetResponseStream()).ReadToEnd();
response.Close();
You have set the content type to multipart/form-data but the data you are posting is formed as application/x-www-form-urlencoded.
A multipart content should look something like this:
--12345
Content-Disposition: form-data; name="sometext"
some text sent via post...
--12345--
And the content type must contain the boundary:
request.ContentType = "multipart/form-data; boundary=12345";
But you have this type of content:
id1=123456&id2=0123456789
So using application/x-www-form-urlencoded for the content type will solve the problem:
request.ContentType = "application/x-www-form-urlencoded";
c# code:
ASCIIEncoding encoding = new ASCIIEncoding();
CookieContainer cook = new CookieContainer();
byte[] data = encoding.GetBytes(postData);
HttpWebRequest myRequest =(HttpWebRequest)WebRequest.Create("http://website.com/index.php");
myRequest.Method = "POST";
myRequest.ContentType = "application/x-www-form-urlencoded";
//myRequest.UserAgent = "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)";
myRequest.ContentLength = data.Length;
Stream newStream = myRequest.GetRequestStream();
newStream.Write(data, 0, data.Length);
newStream.Close();
//myRequest.AllowAutoRedirect = false;
myRequest.CookieContainer = new CookieContainer();
WebResponse response = myRequest.GetResponse();
Console.WriteLine(((HttpWebResponse)response).StatusDescription);
newStream = response.GetResponseStream();
StreamReader reader = new StreamReader(newStream);
Console.WriteLine(reader.ReadToEnd());
index.php
$num= $_POST["num"];
echo $num;
the above code work for me before but suddenly it fail. I tried everything
adding CookieContainer(), setting Redirection to false, using other code for httprequest.
If I AllowAutoRedirect set to false the result would be:
302 Found
Found
The document has moved here
But when i tried this to other web-hosting server it works. So i was thinking its on my WHM/cPanel? Also i have already white-listed my Ip. Already check .htaccess and the options in my cpanelb(redirection).
So any ideas what might cause this?
note: i have already disable'd cloud flare protection
Already Found the solution. It was when they updated my WHM and some Rule was added in ModSecurity Tool that I was not aware off. I just disable that and everything was back to normal :)
I can't seem to get the hang of my HTTP POST methods. I have just learned how to do GET methods to retrieve webpages but now i'm trying to fill in information on the webpage and can't seem to get it working. The source code that comes back is always an invalid page (full of broken images/not the right information)
public static void jsonPOST(string url)
{
url = "http://treasurer.maricopa.gov/Parcel/TaxReceipt.aspx/GetTaxReceipt";
var httpWebRequest = (HttpWebRequest)WebRequest.Create(new Uri(url));
httpWebRequest.ContentType = "application/json; charset=utf-8";
httpWebRequest.Accept = "application/json, text/javascript, */*; q=0.01";
httpWebRequest.Headers.Add("Accept-Encoding: gzip, deflate");
httpWebRequest.CookieContainer = cookieJar;
httpWebRequest.Method = "POST";
httpWebRequest.Headers.Add("Accept-Language: en-US,en;q=0.5");
httpWebRequest.UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW65; Trident/7.0; MAM5; rv:11.0) like Gecko";
httpWebRequest.Referer = "http://treasurer.maricopa.gov/Parcel/TaxReceipt.aspx";
string postData = "{\"startDate\":\"1/1/2013\",\"parcelNumber\":\"17609419\"}";
byte[] bytes = System.Text.Encoding.ASCII.GetBytes(postData);
httpWebRequest.ContentLength = bytes.Length;
System.IO.Stream os = httpWebRequest.GetRequestStream();
os.Write(bytes, 0, bytes.Length); //Push it out there
os.Close();
System.Net.WebResponse resp = httpWebRequest.GetResponse();
if (resp == null)
{
Console.WriteLine("null");
}
System.IO.StreamReader sr = new System.IO.StreamReader(resp.GetResponseStream());
string source = sr.ReadToEnd().Trim();
}
EDIT: I updated the code to reflect my new problem. The problem i have now is that the source code is not what is coming back to me. I am getting just the raw JSON information in the source. Which i can use to deserialize the information i need to obtain, but i'm curious why the actual source code isn't coming back to me
The source code that comes back is always an invalid page (full of broken images/not the right information)
It sounds like you just get the Source code without thinking of relative paths. As long as there are relative paths on the site it will not show correctly at your copy. You have to replace all the relative paths before it is useful.
http://webdesign.about.com/od/beginningtutorials/a/aa040502a.htm
Remember crossdomain ajax can be a problem in that situation.
I´m using .NET C# and want to use HTTPrequest to make a POST in the same way as it is written in Curl:
curl
--header ’Accept: text/html’
--user ’test1:password’
--Form ’wait=0’
--Form ’data=#data.csv;type=text/csv’
some URL address here
I´m getting Error 500 Internal error from the Internet server.
The data to be send is a CSV file.
My source code in C#:
string data = "Time, A, B\n20120101, 100, 24\n20120102\n, 101, 27";
// Create a request using a URL that can receive a post.
request = WebRequest.Create(url);
//Set authorization
request.Credentials = new NetworkCredential(username, password);
// Set the Method property of the request to POST.
request.Method = "POST";
// Create POST data and convert it to a byte array.
byte[] byteArray = Encoding.UTF8.GetBytes(data);
// Set the ContentType property of the WebRequest.
request.ContentType = "application/x-www-form-urlencoded";
// Set the ContentLength property of the WebRequest.
request.ContentLength = byteArray.Length;
// Get the request stream.
dataStream = request.GetRequestStream();
// Write the data to the request stream.
dataStream.Write(byteArray, 0, byteArray.Length);
// Close the Stream object.
dataStream.Close();
// Get the original response.
WebResponse response = request.GetResponse();
this.Status = ((HttpWebResponse)response).StatusDescription;
// Get the stream containing all content returned by the requested server.
dataStream = response.GetResponseStream();
// Open the stream using a StreamReader for easy access.
StreamReader reader = new StreamReader(dataStream);
// Read the content fully up to the end.
string responseFromServer = reader.ReadToEnd();
// Clean up the streams.
reader.Close();
dataStream.Close();
response.Close();
What am I doing wrong?
cURL uses the multipart/form-data content type to create its requests. I think this is required in your case as you are passing both a text parameter and a file. Sadly the .Net framework does not appear to have built in support for creating this content type.
There is a question on SO that gives example code for how to do it yourself here: Multipart forms from C# client
For reference the HTTP request generated by that cURL command looks something like this:
POST http://www.example.com/example HTTP/1.1
Authorization: Basic J3Rlc3Q6cGFzc3dvcmQn
User-Agent: curl/7.33.0
Host: www.example.com
Accept: text/html
Connection: Keep-Alive
Content-Length: 335
Expect: 100-continue
Content-Type: multipart/form-data; boundary=------------------------99aa46d554951aa6
--------------------------99aa46d554951aa6
Content-Disposition: form-data; name="'wait"
0'
--------------------------99aa46d554951aa6
Content-Disposition: form-data; name="'data"; filename="data.csv"
Content-Type: text/csv'
Time, A, B\n20120101, 100, 24\n20120102\n, 101, 27
--------------------------99aa46d554951aa6--
Doesn't data need to be URL encoded for a form post? http://en.wikipedia.org/wiki/POST_%28HTTP%29#Use_for_submitting_web_forms