We Keep Coding

Get Data of online API

I want to download data of this website into a json file but as I am quite new to coding with C# I cant manage to get the data. I want to get Data of https://discosweb.esoc.esa.int/api/objects the authorization via token works but I dont know how I can send a request so the server gives me a json back and I cant find a solution online. I cant give you a screenshot of the API because you have to be logged in to see it. Plz ask me for detailed information if you can help me. Thank you realy for trying.
The code I want to run is here.
class Program
{
static HttpClient client = new HttpClient();
static void Main(string[] args)
{
client.BaseAddress = new Uri("https://discosweb.esoc.esa.int");
client.DefaultRequestHeaders.Accept.Clear();
client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/vnd.api+json"));
client.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("my_token");
var httpRequest = (HttpWebRequest)WebRequest.Create(client.BaseAddress);
var httpResponse = (HttpWebResponse)httpRequest.GetResponse();
using (var streamReader = new StreamReader(httpResponse.GetResponseStream()))
{
var streamReaderResult = streamReader.ReadToEnd();
}
Console.WriteLine("Status https://discosweb.esoc.esa.int : " + httpResponse.StatusCode);
}
}

Try this
var url = "https://discosweb.esoc.esa.int/api/objects";
var httpRequest = (HttpWebRequest)WebRequest.Create(url);
httpRequest.Method = "POST";
httpRequest.Headers["Authorization"] = "Basic XXXx";
httpRequest.ContentType = "";
httpRequest.Headers["Content-Length"] = "0";
var httpResponse = (HttpWebResponse)httpRequest.GetResponse();
using (var streamReader = new StreamReader(httpResponse.GetResponseStream()))
{
var result = streamReader.ReadToEnd();
}
Console.WriteLine(httpResponse.StatusCode);
Where XXXx is user:password in base64.

Here is a basic implementation for making that API call to get the JSON result. You will need to parse that JSON into something other than a string but I'll assume you can handle that part.
This uses System.Net.HttpClient which is the modern HTTP api provided by .NET. Its operations are async so hopefully your code is or can be written to properly await async operations.
//Someplace convenient, create a shared HttpClient to avoid
//creating and disposing for each request.
HttpClient client = new HttpClient();
string data = await GetObjects(client);
//Example implementation
public async Task<string> GetObjects(HttpClient client)
{
string url = "https://discosweb.esoc.esa.int/api/objects";
using (HttpRequestMessage msg = new HttpRequestMessage(HttpMethod.Get, url))
{
msg.Headers.Authorization =
new System.Net.Http.Headers.AuthenticationHeaderValue("Bearer", "your personal access token here");
using (var result = await client.SendAsync(msg))
{
string content = await result.Content.ReadAsStringAsync();
return content;
}
}
}

While I may be a month late, I've actually developed an SDK for this particular API.
So, if you use this SDK it's pretty simple to do what you want. You can essentially forget about handling anything HTTP related, my SDK abstracts all of that away.
For example, to fetch Sputnik's data (which has an ID of 1) you'd run.
HttpClient innerClient = new();
innerClient.BaseAddress = "https://discosweb.esoc.esa.int/api/"
innerClient.DefaultRequestHeaders.Authorization = new("bearer", yourApiKey);
DiscosClient client = new();
DiscosObject sputnik = await client.GetSingle<DiscosObject>("1");
If you're using ASP.NET, there's a set of DI extensions that can actually set it all up for you, so you can skip the first three lines.
If you do choose to use it, please let me know, as it would be nice knowing my SDK is getting some use. If you have any issues, please just reach out through the GitHub issues page and I'll try to help!

Threads with async Task functions C#/.NET / await not compiling

I have a project where I need multiple data for a list of projects. For each project I call an api to get me this information. The loop works, although it takes 4 to 5 minutes to finish(Which is A LOT).
The code used to look like this :
foreach (var project in projects)
{
string url = urlOneProject + project.name + secondPartUrl + "?authtoken=" + authToken;
HttpWebRequest request;
request = (HttpWebRequest)WebRequest.Create(url);
request.Accept = "application/json";
request.ContentType = "application/json";
var executions = new Execs();
var response = (HttpWebResponse)(await request.GetResponseAsync());
using (response)
{
using (var reader = new StreamReader(response.GetResponseStream()))
{
JavaScriptSerializer js = new JavaScriptSerializer();
var objText = reader.ReadToEnd();
executions = (Execs)js.Deserialize(objText, typeof(Execs));
}
}
execs.AddRange(executions.executions);
}
To improve performance I thought that using threads might be a good idea. So, I came up with something like this:
ManualResetEvent resetEvent = new ManualResetEvent(false);
int toProcess = projects.Count;
foreach (var project in projects)
{
new Thread(() =>
{
string url = urlOneProject + project.name + secondPartUrl + "?authtoken=" + authToken;
HttpWebRequest request;
request = (HttpWebRequest)WebRequest.Create(url);
request.Accept = "application/json";
request.ContentType = "application/json";
var executions = new Execs();
var response = (HttpWebResponse)(await request.GetResponseAsync());
using (response)
{
using (var reader = new StreamReader(response.GetResponseStream()))
{
JavaScriptSerializer js = new JavaScriptSerializer();
var objText = reader.ReadToEnd();
executions = (Execs)js.Deserialize(objText, typeof(Execs));
}
}
lock (execs)
{
execs.AddRange(executions.executions);
}
if (Interlocked.Decrement(ref toProcess) == 0)
resetEvent.Set();
}).Start();
}
The problem with this code is that the line:
var response = (HttpWebResponse)(await request.GetResponseAsync());
doesn't compile anymore from the moment I added Thread. And the error I get is
"The 'await' operator can only be used within an async lambda expression "
That wasn't a problem when I didn't use threads. The GetResponseAsync is an async function and the use of the await is compulsory. I tried deleting it (which wasn't logical I agree but I ran out of options) but the compiler tells me I need an await for an async function.
I don't quite understand what changes with the implementation of the Thread.
Didn't I use the threads mechanically correctly? What should I do to correct this or to implement what I want to do correctly?

You are mixing multiple paradigms of coding, viz async / await, and oldschool Thread starting and synchronization, which is likely to lead to trouble.
As per the above comments
The reason your code doesn't compile is because you are attempting to use await in otherwise synchronous code passed to the thread. You can qualify lambdas as async as well.
Task is a much safer paradigm than Thread, and TPL provides rich and expressive tools to assist in asynchrony and parallelism.
If you process each parallel task in isolation, without sharing any data (such as the collections that you are locking), but instead return the resultant data from each Task you can then use LINQ to collate the results in a thread-safe manner.
var myTasks = projects.Select(async project =>
{
var url = $"urlOneProject{project.name}{secondPartUrl}?authtoken={authToken}";
var request = (HttpWebRequest) WebRequest.Create(url);
request.Accept = "application/json";
request.ContentType = "application/json";
using (var response = (HttpWebResponse) (await request.GetResponseAsync()))
using (var reader = new StreamReader(response.GetResponseStream()))
{
var objText = await reader.ReadToEndAsync();
return JsonConvert.DeserializeObject<Execs>(objText);
}
});
var execs = (await Task.WhenAll(myTasks))
.SelectMany(result => result.executions);
Other notes
Don't use JavaScriptSerializer - even the MSDN docco says to use NewtonSoft Json
There's an async version of reader.ReadToEndAsync which I've included.
You can drop the locks and the ManualResetEvent - since each Task returns it's result, we'll leave it to Task.WhenAll to collate the data.
You can flatten the children of multiple executions with a SelectMany
Adjacent using clauses are stackable - it saves a bit of eyestrain on the indentation.

Download string from URL using a portable class library (PCL)

I'm trying to download a string from ANY webpage within my portable class library. I've created the most basic setup:
created a new PCL project
compatible with WP8 and WinRT as well as the compulsory components such as Silverlight
As WebClient is not compatible across these systems, it is not possible to use:
string data = new WebClient().DownloadString();
I've tried using this as well (uses this):
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
request.Method = HttpMethod.Get;
HttpWebResponse response = (HttpWebResponse)await request.GetResponseAsync();
string data = ""
using (var sr = new StreamReader(response.GetResponseStream()))
{
data = sr.ReadToEnd();
}
However, when I call the second set of code from an external C# application referencing the PCL, the debugger simply fails with NO warning or error message on:
request.GetResponseAsync();
Is there an easy way to download a string that I'm missing?
*also, why would the debugger simply exit with no explanation?
Edit:
Here is another method I have attempted - based on an answer already provided. Again, this method simply exits and force closes the debugger.
PCL Method:
public static async Task<string> DownloadString()
{
var url = "http://google.com";
var client = new HttpClient();
var data = await client.GetStringAsync(url);
return data;
}
Calling method:
private static async void Method()
{
string data = await PCLProject.Class1.DownloadString();
return data;
}

Install the NuGet packages:
Microsoft.Bcl.Async, which adds async/await support to PCLs.
Microsoft.Net.Http, which adds HttpClient support to PCLs.
Then you can do it the easy way:
var client = new HttpClient();
var data = await client.GetStringAsync(url);

This method worked for me, it returned the HTML source code from google.com:
public async void GetStringFromWebpage()
{
using (HttpClient wc = new HttpClient())
{
var data = await wc.GetStringAsync("http://google.com/");
Debug.WriteLine("string:" + data);
}
}

Get response from PostAsJsonAsync

I have this line of code
var response = new HttpClient().PostAsJsonAsync(posturi, model).Result;
The Called WebAPI controller returns a bool to make sure the object was saved, but how do I return that bool response?

Continue to get from content:
var httpClient = new HttpClient();
var response = httpClient.PostAsJsonAsync(posturi, model).Result;
bool returnValue = response.Content.ReadAsAsync<bool>().Result;
But, this is really naive approach for quick way to get result. PostAsJsonAsync and ReadAsAsync is not designed to do like this, they are designed to support async await programming, so your code should be:
var httpClient = new HttpClient();
var response = await httpClient.PostAsJsonAsync(posturi, model);
bool returnValue = await response.Content.ReadAsAsync<bool>();
Also, instead of using a flag to check whether an object is saved or not, you should make use of HTTP codes by returning 200 OK to determine that saving is successfully.

The accepted answer is technically correct but blocks the current thread on calls to .Result. If you are using .NET 4.5 or higher, you should avoid that in almost all situations. Instead, use the equivalent asynchronous (non-blocking) version:
var httpClient = new HttpClient();
var response = await httpClient.PostAsJsonAsync(posturi, model);
bool returnValue = await response.Content.ReadAsAsync<bool>();
Note that the method containing the above code needs to be marked async, and should itself be awaited.

Since its an Async operation don't immediately do .Result as its wrong
Instead you need to do it async by doing this:
var httpClient = new HttpClient()
var task = httpClient.PostAsJsonAsync(posturi, model)
.ContinueWith( x => x.Result.Content.ReadAsAsync<bool>().Result);
// 1. GETTING RESPONSE - NOT ASYNC WAY
task.Wait(); //THIS WILL HOLD THE THREAD AND IT WON'T BE ASYNC ANYMORE!
bool response = task.Result
// 2. GETTING RESPONSE - TASK ASYNC WAY (usually used in < .NET 4.5
task.ContinueWith( x => {
bool response = x.Result
});
// 3. GETTING RESPONSE - TASK ASYNC WAY (usually used in >= .NET 4.5
bool response = await task;
NOTE: I just wrote them in here, so I didnt actually test them but more or less that's what you want.
I hope it helps!

I used HttpStatusCode to check the result.
public HttpStatusCode PostStaffPositions(Foo foo)
{
string uri = myapiuri;
using (HttpClient httpClient = new HttpClient())
{
var response = httpClient.PostAsJsonAsync(uri, foo).Result;
return response.StatusCode;
}
}
And then in Controller check it like this:
HttpStatusCode update = staffrest.PostStaffPositions(foo);
if (update == HttpStatusCode.OK)
{
//Update Succeed
}
else
{
//Update Failed
}

If you call the generic version, it should give you back the bool:
var response = new HttpClient().PostAsJsonAsync<bool>(posturi, model).Result;
At least according to the docs.

It's July 2021 and I'm using .net 5 (namely the .net core 5).
I did not see any generic methods above in System.Net.Http. Now the code looks like this (tested):
using (var client = new HttpClient())
{
client.BaseAddress = new Uri("https://localhost:44330/api/Book/Add");
var response = client.PostAsJsonAsync(client.BaseAddress,
JsonSerializer.Serialize(_teamSummaries));
MessageBox.Show(#"Result is " + JsonSerializer.Serialize(response));
var returnValue = response.Result.Content.ReadAsStringAsync();
MessageBox.Show(#"Return value is " + returnValue.Result);
}
There are also ReadAsStringAsync, ReadAsByteArrayAsync, ReadAsStream, ReadFromJsonAsync, ReadFromJsonAsync<T> (this method returns Task<T>).
But from the text meaning "ReadFromJsonAsync", I think the T is not the bool mentioned above, but a class that contains the bool member. If you want to return something like book, give it a try.
On the other hands, since code on the server looks like this(.net 5):
[HttpPost]
[Route("Add")]
public async Task<ActionResult<IEnumerable<Book>>> Add(string value)
{
var all = await _dbCollection.FindAsync(Builders<Book>.Filter.Empty);
return Ok("Everything is ok.");
}
So, if we want to return true by bool, we should return Ok(...). If we want to return false by bool, we should return something else. There are more than 20 other types of results, which contains much more information rather than just "false".

Using HttpClient, how would I prevent automatic redirects and get original status code and forwading Url in the case of 301 [duplicate]

This question already has an answer here:
How does one configure HttpClient not to automatically redirect when it receives a 301 HTTP Status Code?
(1 answer)
Closed 4 years ago.
I have the following method that returns the Http status code of a given Url:
public static async void makeRequest(int row, string url)
{
string result;
Stopwatch sw = new Stopwatch(); sw.Start();
try
{
using (HttpClient client = new HttpClient())
{
HttpResponseMessage response = new HttpResponseMessage();
response = await client.GetAsync(url);
// dump contents of header
Console.WriteLine(response.Headers.ToString());
if (response.IsSuccessStatusCode)
{
result = ((int)response.StatusCode).ToString();
}
else
{
result = ((int)response.StatusCode).ToString();
}
}
}
catch (HttpRequestException hre)
{
result = "Server unreachable";
}
sw.Stop();
long time = sw.ElapsedTicks / (Stopwatch.Frequency / (1000L * 1000L));
requestComplete(row, url, result, time);
}
It works well for 200/404 etc, however in the case of 301 codes I believe the returned result is the already-redirected (200) result, rather than the actual 301 that should be returned and which would have a header containing where the redirect would be pointed to.
I have seen something like this in other .Net web requests classes and the technique there was to set some sort of allowAutoRedirect property to false. If this is along the right lines, can anyone tell me the correct alternative for the HttpClient class?
This post has info on the above allowAutoRedirect concept I mean
Else, how might I get this method to return 301s rather than 200s for Urls I know to be genuine 301s?

I have found that the way to do this is by creating an instance of HttpClientHandler and passing it in the constructor of HttpClient
public static async void makeRequest(int row, string url)
{
string result;
Stopwatch sw = new Stopwatch(); sw.Start();
// added here
HttpClientHandler httpClientHandler = new HttpClientHandler();
httpClientHandler.AllowAutoRedirect = false;
try
{
// passed in here
using (HttpClient client = new HttpClient(httpClientHandler))
{
}
See here for more info.