I need to extract data from an URL which returns HTML data. I can do it using Postman I just need the correct syntax to do it in ASP.NET MVC C#.
https://www.dsebd.org/ajax/load-news.php
Headers
accept:text/html, */*; q=0.01
accept-language:en-US,en;q=0.9
content-type:application/x-www-form-urlencoded; charset=UTF-8
sec-fetch-dest:empty
sec-fetch-mode:cors
sec-fetch-site:same-origin
x-requested-with:XMLHttpRequest
And the method is Post.
Can anyone please share me the syntax in ASP.NET MVC C#?
Guys I wanted the correct format. I never asked for the rules of web scraping.
var httpWebRequest = (HttpWebRequest)WebRequest.Create(String.Format(url));
httpWebRequest.Accept = "text/html, */*; q=0.01";
httpWebRequest.ContentType = "application/x-www-form-urlencoded; charset=UTF-8";
httpWebRequest.Headers.Add("accept-language", "en-US,en;q=0.9");
httpWebRequest.Headers.Add("sec-fetch-dest", "empty");
httpWebRequest.Headers.Add("sec-fetch-mode", "cors");
httpWebRequest.Headers.Add("sec-fetch-site", "same-origin");
httpWebRequest.Headers.Add("x-requested-with", "XMLHttpRequest");
httpWebRequest.Method = "POST";
This is all I asked. Thanks for your help though.
Related
I am consuming an API that expects a XML in the body request. First i consumed the api via Postman and it worked, then i used that tool of Postman to convert the request to RestCharp C# code, and then using that code the reponse that i was receiving was different compared to postman. After that, i used Fiddler to generate c# code with the postman request, and using that code that fiddler generated i was able to consume the API via code sucessfully. I am just trying to understand what is the difference between the code generated from postman and the code generated from Fiddler.
This is the code that is generated from Fiddler and it works:
HttpWebRequest request =
(HttpWebRequest)WebRequest.Create("http://x.x.x.x.x");
request.Accept = "*/*";
request.KeepAlive = true;
request.Method = "POST";
request.ServicePoint.Expect100Continue = false;
byte[] postBytes = System.Text.Encoding.UTF8.GetBytes(body);
request.ContentLength = postBytes.Length;
Stream stream = request.GetRequestStream();
stream.Write(postBytes, 0, postBytes.Length);
stream.Close();
response = (HttpWebResponse)request.GetResponse();
This is the code generated from Postman (slightly altered, but still the code that was generated from postman didn't work and i don't think that the changes that was made interfered with the result) using RestSharp that doesn't work:
var client = new RestClient("http://x.x.x.x.x");
client.ConfigureWebRequest((r) =>
{
r.ServicePoint.Expect100Continue = false;
r.KeepAlive = true;
});
var request = new RestRequest();
request.AddXmlBody(body);
IRestResponse response = client.Post(request);
return response;
I tried a lot of things in the RestSharp code, like adding a header with different content-types and encoding, for example
request.AddHeader("Content-Type", "text/xml;charset=utf-8");
but nothing worked. The response from the api when consumed by the RestSharp code says it got an error of NPE, which i believe it means NullPointerException, but since the api is working just fine via postman and the code generated by Fiddler, i don't think the problem is in the API. Btw, the parameter body in the code are the exact same in both codes.
It looks like the request body is not matching with expected content type by API. When the content does not match the content type expected by the API then you may get NPE error.
in your fiddler generated code you are sending XML string as text.
Please add the following code:
request.AddHeader("Content-Type", "text/plain");
request.AddParameter("undefined", "<YourXml></YourXml>", ParameterType.RequestBody);
or
request.AddHeader("Content-Type", "application/xml");
request.AddParameter("undefined", "<YourXml></YourXml>", ParameterType.RequestBody);
Instead of
request.AddXmlBody(body);
I'm having an issue with HttpWebRequest accepting cookies, this is my code:
HttpWebRequest req= (HttpWebRequest)WebRequest.Create("https://www.companyabc.com/security?action=authenticate");
req.CookieContainer = new CookieContainer();
req.CookieContainer.Add(new Uri("https://www.companyabc.com"), new CookieCollection());
string postData = "account_id=xxxx&password=xxxx";
req.KeepAlive = true;
byte[] send = Encoding.Default.GetBytes(postData);
req.Method = "POST";
req.ContentType = "application/x-www-form-urlencoded";
req.ContentLength = send.Length;
Stream sout = req.GetRequestStream();
sout.Write(send, 0, send.Length);
sout.Flush();
sout.Close();
The response I'm getting is:
Sorry...
We have detected that your browser is not set up to allow Session Cookies. Our platform uses cookies to help enhance your overall user experience. You cannot log in without them.
Please enable Session Cookies and try again. Contact us at ....
What am I doing wrong? Thank you in advanced.
FYI, I can access the web page and login without any issues from a browser. The issue comes when I try to automate the process.
Sounds like the url you're talking to is expecting some specific headers. Try setting the User-Agent header to Mozilla/5.0 (Windows NT 6.3; rv:36.0) Gecko/20100101 Firefox/36.0 (the default Firefox UserAgent).
Failing that, have a look at the request in Firefox (open the developer tools, and switch to the 'Network' tab before you press the login button) and see what other headers the request has.
Using fiddler to observe this URL:
http://opencaselist.paperlessdebate.com/bin/AllDocs?view=attachments#format=json?|t=allattachments&p=1&l=10&s=filename&d=asc
I find a nice JSON response like this
How do I get this response into a string that I can save in a txt file using C#? Is there a way to turn an HTTP Web Response to a string? Is there something is NewtonSoft JSON that can help me? Are there particular terms that will help me google this more effectively?
Every time I try I just get an HTML version of the web-page at the link and not the JSON data I'm trying to get:
string url = "http://opencaselist.paperlessdebate.com/bin/AllDocs?view=attachments#|t=allattachments&p=1&l=10&s=filename&d=asc";
HttpWebRequest httpWebRequest = (HttpWebRequest)WebRequest.Create(url);
httpWebRequest.Method = WebRequestMethods.Http.Get;
httpWebRequest.Accept = "text/json";
httpWebRequest = (HttpWebRequest)WebRequest.Create(url);
HttpWebResponse response = (HttpWebResponse)httpWebRequest.GetResponse();
response.Close();
Note that on that screenshot, there are some differences:
the URL is way different: AllAttachements vs AllDocs, but that's minor, I'm pointing it out "just in case"
the PARAMS are way different: the screenshot specifies xpage=plain&outputSyntax=plain and your code - not
the HEADERS are different: your code has Accept=text\json while screenshot has Accept: text/javascript
Have you tried using the same params and headers?
EDIT: also, I've opened up the page from your code, and it actually is a page. After loading, it generates additional requests to
http://opencaselist.paperlessdebate.com/bin/get/XWiki/AllAttachmentsResults?xpage=plain&outputSyntax=plain&offset=1&limit=10&reqNo=1&sort=filename&dir=asc
which, if you download, results in JSON data. No headers at all, simple GET. I've just got the JSON data by simply pasting that URL into Chrome.. I think that you simply use wrong URL.
When it comes to web development, I know very very little...
I have found some code and explanations from the following site. https://dev.twitter.com/docs/auth/implementing-sign-twitter
Ultimately, I want to implement login with twitter. But I am having trouble rewriting those POST web requests into a c# HttpWebRequest format that I can reuse in the rest of our apps. If we examine the first webrequest made...
POST /oauth/request_token HTTP/1.1
User-Agent: themattharris' HTTP Client
Host: api.twitter.com
Accept: */*
Authorization:
OAuth oauth_callback="http%3A%2F%2Flocalhost%2Fsign-in-with-twitter%2F",
oauth_consumer_key="cChZNFj6T5R0TigYB9yd1w",
oauth_nonce="ea9ec8429b68d6b77cd5600adbbb0456",
oauth_signature="F1Li3tvehgcraF8DMJ7OyxO4w9Y%3D",
oauth_signature_method="HMAC-SHA1",
oauth_timestamp="1318467427",
oauth_version="1.0"
I want to transform that into a working HttpWebRequest.
Thus far. My code looks like this...
HttpWebRequest httpReq = (HttpWebRequest)WebRequest.Create("https://api.twitter.com/oauth/request_token");
ASCIIEncoding encoding = new ASCIIEncoding();
httpReq.Method = "POST";
httpReq.ContentType = "application/x-www-form-urlencoded";
httpReq.Accept = "Accept=text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8";
This is unfortunately how far I did get... I don't know how these requests work. I need to include the rest of the data and make the call. But I am stuck. Any help would be greatly appreciated.
Try this :
ASCIIEncoding encoder = new ASCIIEncoding();
byte[] data = encoder.GetBytes(serializedObject); // the data you wanted to send
HttpWebRequest request = new WebRequest.Create("https://api.twitter.com/oauth/request_token") as HttpWebRequest;
request.Method = "POST";
request.ContentType = "application/x-www-form-urlencoded";
request.ContentLength = data.Length;
request.GetRequestCode().Write(data, 0, data.Length);
Also a possible dublicate (similar question) : Why I get 411 Length required error?
I'm trying to log in to my eBay account using the following code:
string signInURL = "https://signin.ebay.com/ws/eBayISAPI.dll?co_partnerid=2&siteid=0&UsingSSL=1";
string postData = String.Format("MfcISAPICommand=SignInWelcome&userid={0}&pass={1}", "username", "password");
string contentType = "application/x-www-form-urlencoded";
string method = "POST";
string userAgent = "Mozilla/5.0 (Windows; U; MSIE 7.0; Windows NT 6.0; en-US)";
CookieContainer cookieContainer = new CookieContainer();
HttpWebRequest req = (HttpWebRequest)HttpWebRequest.Create(signInURL);
req.CookieContainer = cookieContainer;
req.Method = method;
req.ContentType = contentType;
req.UserAgent = userAgent;
ASCIIEncoding encoding = new ASCIIEncoding();
byte[] loginDataBytes = encoding.GetBytes(postData);
req.ContentLength = loginDataBytes.Length;
Stream stream = req.GetRequestStream();
stream.Write(loginDataBytes, 0, loginDataBytes.Length);
stream.Close();
HttpWebResponse res = (HttpWebResponse)req.GetResponse();
StreamReader xsr = new StreamReader(res.GetResponseStream());
String responseText = xsr.ReadToEnd();
Obviously substituting my real username and password. When I look at the string responseText, I see that part of the response from eBay is
The browser you are using is rejecting cookies.
Any ideas what I'm doing wrong?
P.S. And yes, I am also using the eBay API, but this is for something slightly different than what I want to do with the API.
You're doing a direct http request. The Ebay site has functionality to talk to a browser (probably to store the session cookie). Unless you make the request code smart enough to use cookies correctly it won't work. You'll probably have to use the internet explorer object instead.
Before doing the POST you need to download the page with the form that you are submitting in your code, take the cookie they give you, put it in your CookieContainer (making sure you get the path right) and post it back up in your request.
To clarify, while you might be POSTing the correct data, you are not sending the cookie that needs to go with it. You will get this cookie from the login page.
You need to intercept the http traffic to see what exactly what had happened. I use Fiddler2. It is the good tools for debugging http. So I can know whos wrong, my application or the remote web server.
Using fiddler, you can see the request header, response header with its cookies as well as response content. It used in the middle of your app and the Ebay.
Based on my experience. I think it is because Ebay cookie sent to you is not send back to Ebay server. Fiddler will prove it whether yes or not.
Another thing, the response cookie you receive should be send back to next request by using the same CookieContainer.
You should notice that CookieContainer has a bug on .Add(Cookie) and .GetCookies(uri) method. You may not using it, but internal codes might use it.
See the details and fix here:
http://dot-net-expertise.blogspot.com/2009/10/cookiecontainer-domain-handling-bug-fix.html
CallMeLaNN