I need to check if url exists and can be reached. In order to do it I send Get request and handle the status:
var httpClient = new HttpClient();
var response = httpClient.GetAsync(new Uri(pageUrl));
isPageAccessible = response.Result.StatusCode == HttpStatusCode.OK;
However, the server uses NTLM for authentication. As I found it here, there are several steps (requests) before I get success OK status. For the first request I get 401 Unauthorized status and can't go to further steps.
All in all, how can I check url on the server with NTML upon completion of all requests?
If you are accessing an authenticated server, you should provide credentials. Credentials of running process for NTLM can be provided with HttpClient as below:
var handler = new HttpClientHandler {
Credentials = System.Net.CredentialCache.DefaultCredentials
};
var httpClient = new HttpClient(handler);
var response = httpClient.GetAsync(new Uri(pageUrl));
You're setting yourself up for failure since there are dozens of reasons why a request may not return a 200 OK response. One may be that the response has no content 204 No Content. Another may be that the request only accepts POST or PUT requests. Another, as you've discovered, may be that it has an authentication system in front of it 401 Not Authorized. Another may be just that the response is a redirect 301 Moved Permanently or 302 Found. Or it could be behind a proxy 305, 306, etc.
The only way you can determine if a URL really exists is to request that the other end prove it. Google does it, Facebook does it, Pinterest does it, etc. The way they do it is they ask the sender to set an MX record in their DNS or a meta tag on their index.html with a custom token they generate. If the token exists, then they're who they say they are.
Anything else is unreliable.
Related
I wanna send request to external API through the HttpClient from my own API in c#.
I am using dot net 5 and basic authentication.
Here is my code:
var client = new HttpClient
{
BaseAddress = new Uri(baseUrl)
};
HttpRequestMessage requestMessage = new HttpRequestMessage(HttpMethod.Put, "apiUrl");
client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
var param = JsonConvert.SerializeObject(new
{
param1="",
param2=""
});
requestMessage.Content = new StringContent(param, Encoding.UTF8, "application/json");
requestMessage.Headers.Authorization = new AuthenticationHeaderValue("Basic",
Convert.ToBase64String(Encoding.ASCII.GetBytes($"{user}:{pass}")));
HttpResponseMessage response = await client.SendAsync(requestMessage);
Usually, I send http request like this.
but now I have some problem.
After line HttpResponseMessage response = await client.SendAsync(requestMessage); authorization header removed from my request header and I got UnAuthorized http error.
I know that when redirection occurs, authorization header removed for security reason. Also I'm not sure about redirect in external API.
I add the HttpClientHandler with AllowAutoRedirect = false to my HttpClient
var handler = new HttpClientHandler()
{
AllowAutoRedirect = false,
};
var client = new HttpClient (handler)
{
BaseAddress = new Uri(baseUrl)
};
Now I got Redirect error 301(Permanently moved).
I decided to test the code in Postman. By default, when I call API in Postman, I got http error 405 method not allowed and error detail like this:
{
"detail": "Method "GET" not allowed."}
External API method is PUT. but here I got GET Error.
I tried many options in postman and finally I find the option in Postman:
When I toggle it on, external API work's properly.
Also I test it with Insomnia and it's work properly.
Does it related to my code or dot net 5 or what other thing in my code or it's related to external API?
If it's related to my code, How can I solve the error?
If error related to external API, why Postman and Insomnia response is ok?
External API has Core Policy for specific domain and I send request from other domain.
All I Know is that the CORS policy applied in browser. not in Postman, Insomnia or C# Code.
What about CORS? Does it related to CORS? if Yes, what shall I do?
I will be grateful for your answer.
Update
I detect WWW-Authenticate: JWT realm="api" in the response header.
What exactly is it? and what shall I do?
I find out the problem. It's really Ridiculous.
When I use URL like www.abc.com/api/something the request gets 301 error and postman sends another request like www.abc.com/api/something/. The difference is just / at the end of request.
I tried new URL in postman and first request got ok.
Also I tried URL in my C# code and again its ok.
But i could not understand why.
Thanks a lot dear #pharaz-fadaei
You are right about the removal of authorization headers after redirects. But keep in mind that this behavior is part of the design of the HttpClient in C#. Postman and Insomnia may have different mechanisms to send the authorization headers on each consecutive request that is caused by redirects. The option that you enabled in Postman will make it use the original HTTP method you specified (PUT) as the HTTP method to send further requests based on the redirect messages (Postman uses GET method by default on requests instructed by redirect messages).
The fact that you see a 301 shows that a redirection is required. You can check Location header value in response.Headers to see the real location and send your requests with the authorization headers to that endpoint directly. If I were you I wouldn't use the new location directly because the original endpoint is what you were given by the authors of the API. Instead I would programmatically send the request to the original endpoint and resend the request on 301 codes to the new Location (use PUT method due to the behavior of Postman) until you get the result. This answer can give you some ideas: https://stackoverflow.com/a/42566541/1539231
Since you see WWW-Authenticate: JWT realm="api" header, the external API is required a JWT token authentication, not basic authentication. I think first you might need to check external api's documentation.
I've hit a strange bit of behaviour and I'm pretty sure is related to my code rather than the RTC instance I'm working with.
I've got a web request setup and configured:
var cookies = new CookieContainer();
var request = (HttpWebRequest)WebRequest.Create(getCreationFactoryUri);
var xmlString = getRDF.ToString();
request.CookieContainer = cookies;
request.Accept = "application/rdf+xml";
request.Method = "POST";
request.ContentType = "application/rdf+xml";
request.Headers.Add("OSLC-Core-Version", "2.0");
request.Timeout = 40000;
request.KeepAlive = true;
byte[] bytes = Encoding.ASCII.GetBytes(xmlString);
request.ContentLength = bytes.Length;
Stream dataStream = request.GetRequestStream();
dataStream.Write(bytes, 0, bytes.Length);
dataStream.Close();
This is passed to another method wrote based on an RTC example using forms authentication for RTC.
Under the OSLC v2 spec, I'm using a creation factory URL to post to. I know the URL is fine because I've setup a call using RESTClient in Firefox. Added the headers that are needed (Content-Type: application/rdf+xml, Accept: application/rdf+xml, OSLC-Core-Version: 2.0) and used the generated XML that my code is trying to pass. My manual call works perfectly and the ticket is created.
In my logs I captured the response from RTC, which is a list of tickets rather than a response showing my ticket as being created. I can re-create this behavior by doing a GET on the creation factory URL I'm using to create an event ticket.
So although I know I'm sending a POST to the creation factory (I debugged to check that my web request method was 100% set to 'POST') RTC instead returns a list of tickets and I can only conclude somewhere my request is treated as a 'GET'.
As a test I changed my request to use PUT instead of POST. This isn't permitted for use on the creation factory URL and in testing it indeed throws an error. So I'm totally miffed as to why RTC isn't creating my ticket, but instead treating my request as a GET and returning a list of tickets.
Anyone have any ideas?
Thanks.
If the server using form authentication, as you state, then I expect what is happening is that the POST is resulting in an HTTP redirection to the authentication form. Even if your other code is handling that authentication (which it sounds like it is), the result of that authentication will be an HTTP redirection to the URL of the original request. However, that redirection is likely to result in a GET to that URL, not the original POST. (Also, I don't believe the redirection after authentication is 100% reliable, if your requests are multi-threaded).
The jazz.net information on form authentication says "After authentication succeeded, you always have to replay the original request at least once to get to the protected resource. More replays may be required if the first replay led to another set of redirections and the original request had a non-GET method."
So if your code received an authentication challenge, you will need to re-send the original POST.
I believe the reason why the RESTClient plug-in in your browser is working first time is that it is sending the cookies from your previous log-in to the RTC web UI in the browser. (I had this experience recently, and also found it very confusing).
Also, if you are not preserving cookies between requests to RTC in your client app, then you will meet an auth challenge for every request. If you preserve cookies between calls from your client app (how you do that will depend on your client library - I'm not familiar with the code in your examples) then my experience is that you won't receive an auth challenge for every request. (However, you still need to be able to handle an auth challenge on every request - including POSTs - otherwise it may fail intermittently if the session times out just before you send a POST).
I use both of these many times in my code and don't really know what the difference is , if a cookie is set shouldn't it be exactly the same in request and response? and is request going to the the most up to date , or response?
EDIT:
ok , I get the difference between a request and a response, but if I type
string a = HttpContext.Current.Request.Cookie["a"].Value;
it is most of the time the same as
string a = HttpContext.Current.Response.Cookie["a"].Value;
but I am wondering what is the difference between using the two.
As everyone says Request.Cookies are supposed to be cookies coming from client (browser) and Response.Cookies are cookies that will be send back to client (browser).
There is black magic well documented* code that copies values from Response cookies to Request.Cookies when you add cookies to Response. As result it looks like you have same cookies in both Request and Response. Note that these copied cookies did not come from the client... so beware of making wrong decisions.
Here is a link to discussion about the code: http://forums.asp.net/t/1279490.aspx. In particular, cookies added in the following way will show up in the Request.Cookies collection:
Response.Cookies.Add(HttpCookie("MyCookie", "MyValue"))
*The behavior of cookies getting copied from Response.Cookies is documented in the HttpResponse.Cookies article:
After you add a cookie by using the HttpResponse.Cookies collection, the cookie is immediately available in the HttpRequest.Cookies collection, even if the response has not been sent to the client.
The request cookie is what is send from the client to the server (thus what the browser provides). The response cookie are the cookies that you want to place in the browser. The next connection from the browser that accepted the cookie from the response object will provide the cookie in the request object.
The word Response is used in Asp.net to send data from the server to the client and the Request is used to get the data from the client ( in the form of cookies, query string ) etc.
Example:
Response.Write("will write the content on the form which will return to the client");
// Response.Cookies will send the cookie to the client browser.
Response.Cookies.Add(HttpCookie("MyCookie", "MyValue"))
//and Request.Cookies is used to get the cookie value which is already present in the clinet browswer
and as you mentioned
string a = HttpContext.Current.Request.Cookie["a"].Value;
// I think this will check the cookie which is present in the client browser [ If client has sent the cookie to the server ]
string a = HttpContext.Current.Response.Cookie["a"].Value;
// and this will see the only Response object. If the cookie present in the response object then it will return you otherwise not.
Depends on what context.
Request is the data that gets sent to the server with every http request. Response is the response after the request by the server to the client.
I´m getting a problem sending the authentication request with activated attribute exchange. It works with FF and Opera but IE seems to have a problem with it.
The error occurs within the request.RedirectToProvider(). The Url within the address bar shows the endpoints url.
Here is a code snippet where the authentication request is created and sent
using (OpenIdRelyingParty openid = this.createRelyingParty())
{
IAuthenticationRequest request = openid.CreateRequest(openid_identifier, realm);
;
var fetch = new FetchRequest();
fetch.Attributes.AddRequired(WellKnownAttributes.Contact.Email);
request.AddExtension(fetch);
// Send your visitor to their Provider for authentication.
request.RedirectToProvider();
}
I guess that inside RedirectToProvider() a POST is sent and short after a GET. By RFC a POST followed by a GET is not allowed. Unfortunately I don`t know how to validate it.
Does someone got the same problem?
On further investigation this seems to happen (based upon the providers logs):
The authentication request is sent via POST. The Provider answers with a redirect. This is how it happens that there is at first a POST and then a GET.
I dont think this is how the openId specification say to do it. Shouldnt there be a POST response if i request with a POST and a GET if I request with a GET?
Anyhow.. Is there a way to make the RedirectToProvider()-method sending the request via GET instead of POST?
I need to login to a website and perform an action. The website is REST based so I can easily login by doing this (the login info is included as a querystring on the URL, so I dont't need to set the credentials):
CookieContainer cookieJar = new CookieContainer();
HttpWebRequest firstRequest = (HttpWebRequest) WebRequest.Create(loginUrl);
firstRequest.CookieContainer = cookieJar;
firstRequest.KeepAlive = true;
firstRequest.Method = "POST";
HttpWebResponse firstResponse = (HttpWebResponse)firstRequest.GetResponse();
That works and logs me in. I get a cookie back to maintain the session and it's stored in the cookieJar shown above. Then I do a second request such as this:
HttpWebRequest secondRequest = (HttpWebRequest) WebRequest.Create(actionUrl);
secondRequest.Method = "POST";
secondRequest.KeepAlive = true;
secondRequest.CookieContainer = cookieJar;
WebResponse secondResponse = secondRequest.GetResponse();
And I ensure I assign the cookies to the new request. But for some reason this doesn't appear to work. I get back an error telling me "my session has timed out or expired", and this is done one right after the other so its not a timing issue.
I've used Fiddler to examine the HTTP headers but I'm finding that difficult since this is HTTPS. (I know i can decrypt it but doesn't seem to work well.)
I can take my URL's for this rest service and paste them into firefox and it all works fine, so it must be something I'm doing wrong and not the other end of the connection.
I'm not very familiar with HTTPS. Do I need to do something else to maintain my session? I thought the cookie would be it, but perhaps there is something else I need to maintain across the two requests?
Here are the headers returned when I send in the first request (except I changed the cookie to protect the innocent!):
X-DB-Content-length=19
Keep-Alive=timeout=15, max=50
Connection=Keep-Alive
Transfer-Encoding=chunked
Content-Type=text/html; charset=WINDOWS-1252
Date=Mon, 16 Nov 2009 15:26:34 GMT
Set-Cookie:MyCookie stuff goes here
Server=Oracle-Application-Server-10g
Any help would be appreciated, I'm running out of ideas.
I finally got it working after decrypting the HTTP traffic from my program.
The cookie I'm getting back doesn't list the Path variable. So .NET takes the current path and assigns that as the path on the cookie including the current page. ie: If it was at http://mysite/somepath/somepage.htm it would set the cookie path=/somepath/somepage.htm. This is a bug as it should be assigned to "/" which is what all web browsers do. (hope they fix this.)
After noticing this I grabbed the cookie and modified the path property and everything works fine now.
Anyone else with a problem like this check out Fiddler. .NET uses the windows certificate store so to decrypt http traffic from your program you will need to follow the instructions here: http://www.fiddler2.com/Fiddler/help/httpsdecryption.asp . You will also need to turn on decryption under the Options\HTTPS tab of Fiddler.
From MSDN:
When a user moves back and forth between secure and public areas, the ASP.NET-generated session cookie (or URL if you have enabled cookie-less session state) moves with them in plaintext, but the authentication cookie is never passed over unencrypted HTTP connections as long as the Secure cookie property is set.
So basically, the cookie can be passed over both HTTP and HTTPS if the 'Secure' property is set to 'false'.
see also how can I share an asp.net session between http and https