I have recently started using libcurl.net with one of my projects as a replacement to the HttpWebRequest and HttpWebResponse classes. The reason I chose to use libcurl.net instead of the managed classes is that libcurl.net mimics the behavior of cURL from PHP and I was porting over some code from PHP. I attempted to use the built-in managed classes, but the CookieContainer class was not capturing all of the cookies correctly from the website that I was trying to capture cookies from. I may end up going back to the managed classes if I can figure out how to capture the cookies correctly.
My PHP script works perfectly fine in capturing cookies so I ported most of the cURL functionality using libcurl.net to my C# project. The problem I'm having is when I have to send more than one request header with the CURLOPT_HTTPHEADER cURL option and I have to use an Slist datatype to pass in more than one header like so:
Slist headers = new Slist();
headers.Append("Content-Type: application/x-www-form-urlencoded");
headers.Append("X-Requested-With: XMLHttpRequest");
easy.SetOpt(CURLoption.CURLOPT_HTTPHEADER, headers);
I sometimes have to fake an AJAX request but it does not seem to pass the X-Requested-With: XMLHttpRequest header with the request as the website I'm scraping does not return any results for these "fake" AJAX requests. If I set the CURLOPT_HTTPHEADER do I need to set the Content-Type header or is that always defaulted to Content-Type: application/x-www-form-urlencoded?
It turns out that I was adding multiple headers correctly. I simply made an Slist object and added my headers to the request using the CURLOPT_HTTPHEADER option. In this way, one can "fake" AJAX requests or any other type of request sent by a web browser. The problem was that I wasn't sending the correct POST data with my request.
Related
I iam working on a tool that let users check their API. One of the features is to show the actual send request headers.
Iam having trouble getting these headers though as the Headers property doesnt seem to include them all. I tried looking at tracelisteners but these seem to be more oriented to debugging and the config is global so it applies to all webrequests send by the app which is not what I want.
When I run this code on net48 (in core I seem to get 0 headers back):
// Create a new 'HttpWebRequest' Object to the mentioned URL.
HttpWebRequest myHttpWebRequest=(HttpWebRequest)WebRequest.Create("http://www.contoso.com");
// Assign the response object of 'HttpWebRequest' to a 'HttpWebResponse' variable.
HttpWebResponse myHttpWebResponse=(HttpWebResponse)myHttpWebRequest.GetResponse();
Console.WriteLine("\nThe HttpHeaders are \n\n\tName\t\tValue\n{0}",myHttpWebRequest.Headers);
I get the following output
The HttpHeaders are
Name Value
Host: www.microsoft.com
However in fiddler and with trace listeners I see these headers:
Host: www.contoso.com
Connection: Keep-Alive
Why can't I see the Connection header?
Now I see there is some redirecting going on. WebRequest seems to only show the headers send in the LAST request which didn't had the Connection header.
I have defined a route in WordPress using register_rest_route, this is set as a route that accepts only POST requests:
register_rest_route( 'myNamespace', '/myRoute/', array(
'methods' => 'POST',
'callback' => 'myCallbackFunction',
) );
Making a POST request to this route using Postman returns the result I expect. However, making a request from a C# application using either System.Net.Http.HttpClient or System.Net.HttpWebRequest I receive a 404 response.
When I change my register_rest_route to accept GET requests, using the code below, my request is received, but fails because I need values posted in the body of the request.
register_rest_route( 'myNamespace', '/myRoute/', array(
'methods' => 'GET',
'callback' => 'myCallbackFunction',
) );
This is extremely peculiar and I cannot figure out why making the request from a C# application converts a HttpPost to a HttpGet request, but works perfectly fine when making the request from Postman.
I'm using .NET 4.8 SDK for my C# application and WordPress 5.3.2 for WordPress.
Right now I cannot figure out where the issue lies. At first I thought it was an issue with my C# application, so I changed the request tools from HttpWebRequest to HttpClient, but that did not fix the issue. Then I thought it was an issue with the PHP side of things, but Postman works fine.
I don't understand how the request method is changing in transit?
I have a solution with two ASP.NET Core MVC projects. One project (Client) is making a request to the other (Server) using HttpClient. When the action in Server receives the request, I want to get the URL of the thing that sent it. Every article I have read purports Request.Headers["Referer"] as the solution, but in my case Headers does not contain a "referer" key (or "referrer").
When receiving the request in Server, how should I find the URL of the Client that sent it?
That is how you you get the referring url for a request. But the referer isn't the thing that sent the request. The referer gets set in the headers by the browser when a person clicks on a link from one website to go to another website. When that request is made by the browser to the new website the request will typically have the Referer header which will contain the url of the prior website.
The receiving server can't get the url of the "client" making the request, remember a typical web browser client isn't at any url. All the receiving server can get is the IP address of the client typically.
Since you have control of the client software, if you wanted you could have the client put whatever info you want in the header of the request before it's sent to the server and the server could then get that info out of the header.
If you're using HttpClient, then it is up to the site making the request to add that header. It isn't added automatically in this case. So: change the code - or request that the code is changed - so as to add the header and value that you expect. If you are proxying through a request, you might get the value from the current request's Referer header, and add that.
Even in the general case of a browser making the request as part of a normal page cycle, you can't rely on it: the Referer header is often deliberately not sent; depending on the browser version, configuration, whether you're going between different domains, whether it is HTTPS or not, and rel markers on a <a href=... such as "noreferrer".
I work in C# and so far I used WebRequest method to GET and POST data. I use Fiddler to check what is the browser doing and I got to a point where the data is retrieved from Ajax after posting some data.
I am not sure if I have to add to my project a javascript page or what and what code do I need in the javascript file and how to call it.
In essence, I have to post the data {"name":"ABCD"} to url www.example.com/Website.AJAX,Website.ashx.
Ajax is not so different from ordinary request, so you can just post it as usual. Most likely problem is how the backend treat that it is an ajax request (if it does at all).
As it looks like you are using the WebForms there on backend, you just need to add a special header most likely (X-Requested-With). Some frameworks add it, though it's not a real requirement of the ajax request.
All in all I would just post an ordinary request with WebRequest as you did before. If that does not work, you need to study the original request from web UI to see what is different. E.g. a special header or request Content-Type is JSON or something like that.
P.S. If you use JSON in the body it's better to explicitly set content type to application/json; charset=utf-8 unless there is something special with the server.
http://www.jfrog.com/confluence/display/RTF/Artifactory+REST+API#ArtifactoryRESTAPI-CreateorReplaceRepositoryConfiguration
I am using the Create or Replace Repository Configuration call. However I am getting a 406 Error: Not Acceptable. Other PUT calls are working but do not return JSON. I believe JSON is the source of the error but have not been able to resolve or prove this.
I have added the code as below
RestClient Client = new RestClient(uriString);
RestRequest Request = new RestRequest(requestType);
Request.AddHeader("Authorization", "Basic " + credentials);
Request.AddHeader("Accept", "application/json");
I've seen threads where adding the header to accept JSON resolves the error but this has not worked for me.
A 406 HTTP status means that if a web server detects that the data it wants to return is not acceptable to the client, it returns a header containing the 406 error code.
The client can define the characteristics of the data it will accept back from the web server by using the accept headers.
In this case you declare the you would like to accept application/json:
Request.AddHeader("Accept", "application/json");
however the REST API method you are invoking is returning text/plain.
You should change the code to accept text/plain:
Request.AddHeader("Accept", "text/plain");
Wanted to add this for for future users stuck like me. I was having the same issue and tried the request with Postman and saw that the Content-Type was "application/hal+json" I was trying it with application/json without luck.
So running a test in postman I was able to figure out what the server needed exactly.
client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/hal+json"));
I faced the same 406 Error: Not Acceptable when trying to get JSON on another site. In my case I could see correct JSON when typed url in my browser address field. But downloading it from the same url via my C# code have been producing 406 Error.
None of the answers in this topic solved my problem directly. But at least they pointed out to me that's the point is HTTP headers.
So I googled that page:
https://www.whatismybrowser.com/detect/what-http-headers-is-my-browser-sending
and added all browser headers to my code, and voila! It started to work.
In my case it was enough to fill some data in user-agent header.
First, the Accept header states what the client is ready to get back, not what the client sends.
The header that states what the client sends is Content-Type.
Also, this method does not accept application/json. As clearly stated in the docs, it accepts one of the following:
application/vnd.org.jfrog.artifactory.repositories.LocalRepositoryConfiguration+json
application/vnd.org.jfrog.artifactory.repositories.RemoteRepositoryConfiguration+json
application/vnd.org.jfrog.artifactory.repositories.VirtualRepositoryConfiguration+json