How to programmtically get a web page hosted in Sharepoint (returns Forbidden 403) - c#

I am on an internal network and I am authroised to view a web page hosted by Sharepoint - I can view the page in a browser perfectly fine.
I want to be able to 'get' the page programmatically, how can this be done?
I have tried using HttpClient GetAysnc with UseDefaultCredentials=true but it returns 403 Forbidden.
I suspect I am not appreciating the nuances of Sharepioint.
var httpClientHandler = new HttpClientHandler() {
UseDefaultCredentials = true,
UseProxy = true
};
using var httpClient = new HttpClient(httpClientHandler);
var getAsyncTask = Task.Run(() => httpClient.GetAsync(match.Value, HttpCompletionOption.ResponseContentRead, cancellationToken), cancellationToken);
getAsyncTask.Wait(cancellationToken);
var response = getAsyncTask.Result;
if (response.StatusCode == HttpStatusCode.OK)
{
var streamTask = Task.Run(() => response.Content.ReadAsStreamAsync(), cancellationToken);
streamTask.Wait(cancellationToken);
var stream = streamTask.Result;
}

Related

Get access token back from Onelogin using Authorization Code Flow with PKCe in step 2

I'm Trying to get the access token form OneLogin using the Authorization Code with PKCE. I'm able to go through step1 for PKCe and getting the authorization code back from OneLogin. But when i try to get the token using the authorization code sent by one login i keep getting 400 bad request error. I'm not sure what is wrong. I followed the info provided by oneLogin website to all required parameters in the request for Step 2. below the code i'm using. I will appreciate if some one can help on this
public async Task GetAccessToken(string redirecturl, string authCode) { HttpClientHandler clientHandler = new HttpClientHandler(); clientHandler.ServerCertificateCustomValidationCallback = (sender, cert, chain, sslPolicyErrors) => { return true; };
var client = new HttpClient(clientHandler);
var body = JsonConvert.SerializeObject(new
{
grant_type = "authorization_code",
code = authCode, //The code returned from OneLogin in step 1
client_id="XXXXXXXXXXXXXXXXXX386d707215718",
redirect_uri=redirecturl,//The redirect URL registered in onelogin account
code_verifier=GetCacheEntry(CodeKey)// The code verifier used in step one
});
var req = new HttpRequestMessage
{
Method = HttpMethod.Post,
RequestUri = new Uri("https://MySubdomain.onelogin.com/oidc/2/token"),
Content = new StringContent(body)
};
req.Content.Headers.ContentType= new MediaTypeHeaderValue(#"application/x-www-form-urlencoded");
var response = await client.SendAsync(req);
if (response.StatusCode == HttpStatusCode.OK)
{
var responseBody =await response.Content.ReadAsStringAsync();
var json = JsonConvert.DeserializeObject<OAuthTokenResponse>(responseBody);
memoryCache.Remove(CodeKey);
return Ok(json);
}
return BadRequest(response);
}

C# HttpClient and Windows Authentication: Cannot access a closed Stream

I'm using the native C# HTTP client with a handler to process Windows Authentication and I'm having ObjectDisposedException.
using (var httpClientHandler = new HttpClientHandler { Credentials = CredentialCache.DefaultNetworkCredentials })
{
bool disposeHandler = true; //Setting true or false does not fix the problem
using (var httpClient = new HttpClient(httpClientHandler, disposeHandler))
{
using (var content = new ByteArrayContent(Encoding.UTF8.GetBytes("Hello")))
{
// Commenting/uncommenting the line below does not fix the problem
// httpRequestMessage.Headers.Connection.Add("Keep-Alive");
using (var httpResponseMessage = await httpClient.PostAsync("http://SomeUrl", content)) // This line throws an ObjectDisposedException
{
}
}
}
}
Any idea?
After some new investigations, I think/fear there it is a Microsoft bug in HttpClientHandler (or HttpClient):
If instead of using the PostAsync method I use the SendAsync method, I can compose my request with more options and especially change the HTTP version from 1.1 (by default) to 1.0. Then in this case the real exception was revealed (and it was not anymore hidden/overridden by the ObjectDisposedException exception). For your information, my real exception was the following:
--> System.Net.WebException: The remote server returned an error: (401) Unauthorized.
--> System.ComponentModel.Win32Exception: The target principal name is incorrect
(Still for your information, this exception was due to a bad configuration of an SPN of a user in the Active Directory)
using (var httpClientHandler = new HttpClientHandler { Credentials = CredentialCache.DefaultNetworkCredentials })
{
bool disposeHandler = true; //Setting true or false does not fix the problem
using (var httpClient = new HttpClient(httpClientHandler, disposeHandler))
{
using (var content = new ByteArrayContent(Encoding.UTF8.GetBytes("Hello")))
{
// Commenting/uncommenting the line below does not fix the problem
// httpRequestMessage.Headers.Connection.Add("Keep-Alive");
var httpRequestMessage = new HttpRequestMessage(HttpMethod.Post, "http://SomeUrl")
{
Content = content,
Version = HttpVersion.Version10
};
using (var httpResponseMessage = await httpClient.SendAsync(httpRequestMessage)) // This line still throws but with the real exception (and not the ObjectDisposedException)
{
}
}
}
}
Beware: downgrading the HTTP version is just a hack to get the real exception message, HTTP 1.0 is getting very old.
Any experts analysis would be really appreciated. Hope this will help others.
Have you tried setting PreAuthenticate = true, it will add the authorization header with the AD/User token, you should get a 401 followed by a second request, with the header, suggest you view it in fiddler.
httpClientHandler.PreAuthenticate = true;
You're disposing of your HttpResponseMessage before it's had a chance to be evaluated. Try this instead.
HttpResponseMessage httpResponseMessage = null;
using (var content = new ByteArrayContent(Encoding.UTF8.GetBytes("Hello")))
{
var httpRequestMessage = new HttpRequestMessage(HttpMethod.Post, "http://SomeUrl")
{
Content = content,
Version = HttpVersion.Version10
};
using (var httpClient = new HttpClient(httpClientHandler, disposeHandler))
{
httpResponseMessage = await httpClient.SendAsync(httpRequestMessage);
}
}

How can I add a HTTP Header called "Content-Type" to an HttpClient request? [duplicate]

I need to add http headers to the HttpClient before I send a request to a web service. How do I do that for an individual request (as opposed to on the HttpClient to all future requests)? I'm not sure if this is even possible.
var client = new HttpClient();
var task =
client.GetAsync("http://www.someURI.com")
.ContinueWith((taskwithmsg) =>
{
var response = taskwithmsg.Result;
var jsonTask = response.Content.ReadAsAsync<JsonObject>();
jsonTask.Wait();
var jsonObject = jsonTask.Result;
});
task.Wait();
Create a HttpRequestMessage, set the Method to GET, set your headers and then use SendAsync instead of GetAsync.
var client = new HttpClient();
var request = new HttpRequestMessage() {
RequestUri = new Uri("http://www.someURI.com"),
Method = HttpMethod.Get,
};
request.Headers.Accept.Add(new MediaTypeWithQualityHeaderValue("text/plain"));
var task = client.SendAsync(request)
.ContinueWith((taskwithmsg) =>
{
var response = taskwithmsg.Result;
var jsonTask = response.Content.ReadAsAsync<JsonObject>();
jsonTask.Wait();
var jsonObject = jsonTask.Result;
});
task.Wait();
When it can be the same header for all requests or you dispose the client after each request you can use the DefaultRequestHeaders.Add option:
client.DefaultRequestHeaders.Add("apikey","xxxxxxxxx");
To set custom headers ON A REQUEST, build a request with the custom header before passing it to httpclient to send to http server.
eg:
HttpClient client = HttpClients.custom().build();
HttpUriRequest request = RequestBuilder.get()
.setUri(someURL)
.setHeader(HttpHeaders.CONTENT_TYPE, "application/json")
.build();
client.execute(request);
Default header is SET ON HTTPCLIENT to send on every request to the server.

C# HttpClient not displaying all cookies

Edit: I apologize for the confusion: I'm attempting to scrape a site that I did not write; I'm not writing an ASP app. I'm only attempting to scrape one.
After making a post request to a login page, I attempt to read the cookies from another page. I do not see all the cookies I expect, however. My code is as follows:
// Downloads login cookies for subsequent requests
public async Task<CookieContainer> loginCookies()
{
var cookies = new CookieContainer();
var handler = new HttpClientHandler {
UseCookies = true,
AllowAutoRedirect = true,
CookieContainer = cookies
};
var client = new HttpClient(handler);
var loginUri = new Uri("https://connect.example.edu/login.aspx");
var credentials = new Dictionary<string, string> {
{"example$txtUsername", this.Username},
{"example$txtPassword", this.Password}
};
var formCredentials = new FormUrlEncodedContent(credentials);
await client.PostAsync(loginUri, content: formCredentials);
var pointsUri = new Uri("https://info.example.edu/");
Console.WriteLine("COOKIES:");
foreach (Cookie cookie in cookies.GetCookies(pointsUri))
Console.WriteLine($"{cookie.Name} --> {cookie.Value}");
return cookies;
}
I believe the error is a result of loginUri and pointsUri having different subdomains. The info I need to scrape exists at the pointsUri page, but the login exists at loginUri.
A cookie that I'm missing in particular is ASP.NET_SessionID.

HttpClient and Impersonation

From my web service (A) usng impersonation i would like call a WebAPI service (B) using HttpClient. But the service B always gets the system user of service A even though i do impersonation there.
var baseUri = "http://service/api/"
var handler = new HttpClientHandler { UseDefaultCredentials = true };
var client = new HttpClient(handler) { BaseAddress = baseUri };
client.DefaultRequestHeaders.Accept.Clear();
client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
client.DefaultRequestHeaders.Add("ContentType", new List<string> { "application/json"});
var dataDto = new DataDto();
var json = JsonConvert.SerializeObject(dataDto );
var content = new StringContent(json);
content.Headers.ContentType = new MediaTypeHeaderValue("application/json");
var response = await client.PostAsync(SubUrl, content);
I know that Kerberos and SPN are set up correctly because it works using WebClient.
I think the problem is, that HttpClient.PostAsync creates a new task and therefore a new thread running under the credentials of the appPool of service A.
Does anyone know how i could force the task to run under the imperonated credentials?
I do not have access to the aspnet_config.config so the solution proveded here does not work for me.
Thanks a lot!
Tschuege

Categories