We Keep Coding

Why am I getting a 401 unauthorized when using HttpResponseMessage with a secure url?

I have this proprietary code I am working on for my job.
I am writing test cases for it because the code was changed and the test cases are now broken.
It is a C# web Api MVC .Net Framework app
I have a method that I enter a string url in
Then this code executes
HttpResponseMessage response = await Client.GetAsync(url).ConfigureAwait(true);
System.Uri uri = new System.Uri(url); // convert string to Uri
var cert = System.Net.ServicePointManager.FindServicePoint(uri).Certificate;
response.EnsureSuccessStatusCode();
when it gets to response.EnsureSuccessStatusCode() , it gives a 401 unauthorized and then throws an exception not allowing my test to pass
When I try the same thing with http://www.google.com which is not an https, then
it gives a 200. So something is going on with security stuff
What are the things I need to do to get a https to give a 200? Does it need username and password credentials or something or some other token of some sort?
Also, when I test it using Rest Client DHC with the secure https link that was failing above it gives me a 200. However, I had to refresh the bearer token for it to give a 200. If I used an old token it would give a 401.
Furthermoore, when I test a different link like https://www.facebook.com (which is not the one I want to test in my application, just troubleshooting) which is secure, it works giving me a 200 both in my application above and Rest Client DHC even with an old bearer token.

If you're trying to makes CORS requests to web API, you'll need to configure It to accept cross origins requests. If you dont configure your web api to accept cross origin requests, it'll throw these type of errors when calling It. And keep in mind that for web api access, https://www.domain.so and http://www.domain.so are completely diff clients.
Look at this link:
https://learn.microsoft.com/en-us/aspnet/web-api/overview/security/enabling-cross-origin-requests-in-web-api

Error 406 Not Acceptable JSON

http://www.jfrog.com/confluence/display/RTF/Artifactory+REST+API#ArtifactoryRESTAPI-CreateorReplaceRepositoryConfiguration
I am using the Create or Replace Repository Configuration call. However I am getting a 406 Error: Not Acceptable. Other PUT calls are working but do not return JSON. I believe JSON is the source of the error but have not been able to resolve or prove this.
I have added the code as below
RestClient Client = new RestClient(uriString);
RestRequest Request = new RestRequest(requestType);
Request.AddHeader("Authorization", "Basic " + credentials);
Request.AddHeader("Accept", "application/json");
I've seen threads where adding the header to accept JSON resolves the error but this has not worked for me.

A 406 HTTP status means that if a web server detects that the data it wants to return is not acceptable to the client, it returns a header containing the 406 error code.
The client can define the characteristics of the data it will accept back from the web server by using the accept headers.
In this case you declare the you would like to accept application/json:
Request.AddHeader("Accept", "application/json");
however the REST API method you are invoking is returning text/plain.
You should change the code to accept text/plain:
Request.AddHeader("Accept", "text/plain");

Wanted to add this for for future users stuck like me. I was having the same issue and tried the request with Postman and saw that the Content-Type was "application/hal+json" I was trying it with application/json without luck.
So running a test in postman I was able to figure out what the server needed exactly.
client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/hal+json"));

I faced the same 406 Error: Not Acceptable when trying to get JSON on another site. In my case I could see correct JSON when typed url in my browser address field. But downloading it from the same url via my C# code have been producing 406 Error.
None of the answers in this topic solved my problem directly. But at least they pointed out to me that's the point is HTTP headers.
So I googled that page:
https://www.whatismybrowser.com/detect/what-http-headers-is-my-browser-sending
and added all browser headers to my code, and voila! It started to work.
In my case it was enough to fill some data in user-agent header.

First, the Accept header states what the client is ready to get back, not what the client sends.
The header that states what the client sends is Content-Type.
Also, this method does not accept application/json. As clearly stated in the docs, it accepts one of the following:
application/vnd.org.jfrog.artifactory.repositories.LocalRepositoryConfiguration+json
application/vnd.org.jfrog.artifactory.repositories.RemoteRepositoryConfiguration+json
application/vnd.org.jfrog.artifactory.repositories.VirtualRepositoryConfiguration+json

Access YouTube with ASP.NET and YoutubeFisher but gets HTTP Error 403

I'm trying to use YoutubeFisher library with ASP.NET. I make an HttpWebRequest to grab html content, process the contest to extract the video links and display links on the web page. I managed to make it work on localhost. I can retrieve video links and download the video on the locahost. But when I push it to the Server, it works only if I send the request from the same Server. If that page is accessed by a client browser, the client can see the links properly, but when link is clicked the client gets the HTTP Error 403, everytime the client clicks on the link even though the link is correct.
My analysis is that when the Server makes HttpWebRequest to grab HTML contet, it sends HTTP header as well. The HTML content (links to the video file) that is sent back from YouTube server, I think, will reponse to only the request that matches that HTTP header, that is sent from the Server. So, when client clicks on the link it sends request to YouTube server with different HTTP header.
So, I'm thinking of getting the HTTP header from the client, then modify the Server HTTP header to include HTTP header info of the client before making HttpWebRequest. I'm not quite sure if this will work. As far as I know, HTTP heaer cannot be modified.
Below is the code that makes HttpWebRequest from YouTubeFisher library,
public static YouTubeService Create(string youTubeVideoUrl)
{
YouTubeService service = new YouTubeService();
service.videoUrl = youTubeVideoUrl;
service.GetVideoPageHtmlSource();
service.GetVideoTitle();
service.GetDownloadUrl();
return service;
}
private void GetVideoPageHtmlSource()
{
HttpWebRequest req = HttpWebRequest.Create(videoUrl) as HttpWebRequest;
HttpWebResponse resp = req.GetResponse() as HttpWebResponse;
videoPageHtmlSource = new StreamReader(resp.GetResponseStream(), Encoding.UTF8).ReadToEnd();
resp.Close();
}
Client browses the page but the links are there but give HTTP 403:
Browse the page the from the Server itself, everything works as expected:
How do I make HttpWebRequest on the behalf of the client then? Is my analysis of this problem correct?
Thank you for your input.

Use an http monitor such as Charles, Fiddler or even Firebug to find out what additional headers are being sent from the brower in the success case. I suspect you'll need to duplicate one or more of accept, user-agent or referer.

In the past I've just assumed that youtube has those links encoded so that they only work for the original request IP. If that were the case it would be far more difficult. I have no clue if this is the case or not, try forwarding all the header elements you can before going down this route...
The only possibility that comes to mind is that you'd have to use a javascript request to download the page to the client's browser, then upload that to your server for processing, or do the processing in javascript.
Or you could have the client download the video stream via your server, so your server would pass through the data. This would obviously use a ton of your bandwidth.

ServerProtocolViolation error

I am hitting up a server with the following code and am encountering a ServerProtocolViolation error:
// Prepare the webpage
HttpWebRequest request = (HttpWebRequest) WebRequest.Create(url + queryString);
// execute the request
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
Does anyone know how to work around this kind of error?

This error means that the webserver that you're sending the request to isn't conforming to the HTTP standard.
Other than fixing the server or rewriting HttpWebRequest to be more generous, there isn't much you can do.
What URL are you requesting, and what's the text of the exception?
EDIT: If you request the URL in Fiddler, you'll see that the server didn't return any headers. You should contact the owner of the server and complain.
As a workaround, if you run Fiddler while sending the request, Fiddler will fix the response and allow HttpWebResponse to parse it.

Just a quick note on another reason that I experienced:
If you configure your request to use a proxy server (i.e. through the HttpWebRequest.Proxy property), and you use a wrong proxy port, there might also be a chance to see that error.
In my case, I configured http://127.0.0.1/ as the proxy but had the actual proxy server running on http://127.0.0.1:808/ instead (i.e. port "808" instead of "80").
If this is the case for you, try using no proxy or of course, configure the correct proxy port.

How does HttpWebRequest differ (functional) from pasteing a URL into an address bar?

I'm narrowing in on an underlying problem related to two prior questions.
Basically, I've got a URL that when I fetch it manually (paste it into browser) works just fine, but when I run through some code (using the HttpWebRequest) has a different result.
The URL (example):
http://208.106.250.207:8192/announce?info_hash=-%CA8%C1%C9rDb%ADL%ED%B4%2A%15i%80Z%B8%F%C&peer_id=01234567890123456789&port=6881&uploaded=0&downloaded=0&left=0&compact=0&no_peer_id=0&event=started
The code:
String uri = BuildURI(); //Returns the above URL
HttpWebRequest req = (HttpWebRequest)HttpWebRequest.Create(uri);
req.Proxy = new WebProxy();
WebResponse resp = req.GetResponse();
Stream stream = resp.GetResponseStream();
... Parse the result (which is an error message from the server claiming the url is incorrect) ...
So, how can I GET from a server given a URL? I'm obviously doing something wrong here, but can't tell what.
Either a fix for my code, or an alternative approach that actually works would be fine. I'm not wed at all to the HttpWebRequest method.

I recommend you use Fiddler to trace both the "paste in web browser" call and the HttpWebRequest call.
Once traced you will be able to see any differences between them, whether they are differences in the request url, in the form headers, etc, etc.
It may actually be worth pasting the raw requests from both (obtained from Fiddler) here, if you can't see anything obvious.

Well, the only they might differ is in the HTTP headers that get transmitted. In particular the User-Agent.
Also, why are you using a WebProxy? That is not really necessary and it most likely is not used by your browser.
The rest of your code is fine.. Just make sure you set up the HTTP headers correctly. Check this link out:
I would suggest that you get yourself a copy of WireShark and examine the communication that happens between your browser and the server that you are trying to access. Doing so will be rather trivial using WireShark and it will show you the exact HTTP message that is being sent from the browser.
Then take a look at the communication that goes on between your C# application and the server (again using WireShark) and then compare the two to find out what exactly is different.
If the communication is a pure HTTP GET method (i.e. there is no HTTP message body involved), and the URL is correct then the only two things I could think of are:
make sure that your are send the right protocol (i.e. HTTP/1.0 or HTTP/1.1 or whatever it is that you should be sending)
make sure that you are sending all required HTTP headers correctly, and obviously that you are not sending any HTTP headers that you shouldn't be sending.

There could be something wrong with the URL. Instead of using a string, it's usually better to use an instance of System.Uri:
String url = BuildURI(); //Returns the above URL
Uri uri = new Uri(url);
HttpWebRequest req = (HttpWebRequest)HttpWebRequest.Create(url);
req.Proxy = new WebProxy();
using (WebResponse resp = req.GetResponse()) {
using (Stream stream = resp.GetResponseStream()) {
// whatever
}
}

I think you need to see exactly what's flowing to your server in the HTTP request. Does sound likely that the headers are interestingly different.
You can introduce a some kind of debugging proxy between your request and the server (for example RAD has such a capability in the box).