We Keep Coding

Get Data of online API

I want to download data of this website into a json file but as I am quite new to coding with C# I cant manage to get the data. I want to get Data of https://discosweb.esoc.esa.int/api/objects the authorization via token works but I dont know how I can send a request so the server gives me a json back and I cant find a solution online. I cant give you a screenshot of the API because you have to be logged in to see it. Plz ask me for detailed information if you can help me. Thank you realy for trying.
The code I want to run is here.
class Program
{
static HttpClient client = new HttpClient();
static void Main(string[] args)
{
client.BaseAddress = new Uri("https://discosweb.esoc.esa.int");
client.DefaultRequestHeaders.Accept.Clear();
client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/vnd.api+json"));
client.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("my_token");
var httpRequest = (HttpWebRequest)WebRequest.Create(client.BaseAddress);
var httpResponse = (HttpWebResponse)httpRequest.GetResponse();
using (var streamReader = new StreamReader(httpResponse.GetResponseStream()))
{
var streamReaderResult = streamReader.ReadToEnd();
}
Console.WriteLine("Status https://discosweb.esoc.esa.int : " + httpResponse.StatusCode);
}
}

Try this
var url = "https://discosweb.esoc.esa.int/api/objects";
var httpRequest = (HttpWebRequest)WebRequest.Create(url);
httpRequest.Method = "POST";
httpRequest.Headers["Authorization"] = "Basic XXXx";
httpRequest.ContentType = "";
httpRequest.Headers["Content-Length"] = "0";
var httpResponse = (HttpWebResponse)httpRequest.GetResponse();
using (var streamReader = new StreamReader(httpResponse.GetResponseStream()))
{
var result = streamReader.ReadToEnd();
}
Console.WriteLine(httpResponse.StatusCode);
Where XXXx is user:password in base64.

Here is a basic implementation for making that API call to get the JSON result. You will need to parse that JSON into something other than a string but I'll assume you can handle that part.
This uses System.Net.HttpClient which is the modern HTTP api provided by .NET. Its operations are async so hopefully your code is or can be written to properly await async operations.
//Someplace convenient, create a shared HttpClient to avoid
//creating and disposing for each request.
HttpClient client = new HttpClient();
string data = await GetObjects(client);
//Example implementation
public async Task<string> GetObjects(HttpClient client)
{
string url = "https://discosweb.esoc.esa.int/api/objects";
using (HttpRequestMessage msg = new HttpRequestMessage(HttpMethod.Get, url))
{
msg.Headers.Authorization =
new System.Net.Http.Headers.AuthenticationHeaderValue("Bearer", "your personal access token here");
using (var result = await client.SendAsync(msg))
{
string content = await result.Content.ReadAsStringAsync();
return content;
}
}
}

While I may be a month late, I've actually developed an SDK for this particular API.
So, if you use this SDK it's pretty simple to do what you want. You can essentially forget about handling anything HTTP related, my SDK abstracts all of that away.
For example, to fetch Sputnik's data (which has an ID of 1) you'd run.
HttpClient innerClient = new();
innerClient.BaseAddress = "https://discosweb.esoc.esa.int/api/"
innerClient.DefaultRequestHeaders.Authorization = new("bearer", yourApiKey);
DiscosClient client = new();
DiscosObject sputnik = await client.GetSingle<DiscosObject>("1");
If you're using ASP.NET, there's a set of DI extensions that can actually set it all up for you, so you can skip the first three lines.
If you do choose to use it, please let me know, as it would be nice knowing my SDK is getting some use. If you have any issues, please just reach out through the GitHub issues page and I'll try to help!

Is it wise to pass a Stream response from an API around in ASP.NET? [duplicate]

I'm working on a web service using ASP.NET MVC's new WebAPI that will serve up binary files, mostly .cab and .exe files.
The following controller method seems to work, meaning that it returns a file, but it's setting the content type to application/json:
public HttpResponseMessage<Stream> Post(string version, string environment, string filetype)
{
var path = #"C:\Temp\test.exe";
var stream = new FileStream(path, FileMode.Open);
return new HttpResponseMessage<Stream>(stream, new MediaTypeHeaderValue("application/octet-stream"));
}
Is there a better way to do this?

Try using a simple HttpResponseMessage with its Content property set to a StreamContent:
// using System.IO;
// using System.Net.Http;
// using System.Net.Http.Headers;
public HttpResponseMessage Post(string version, string environment,
string filetype)
{
var path = #"C:\Temp\test.exe";
HttpResponseMessage result = new HttpResponseMessage(HttpStatusCode.OK);
var stream = new FileStream(path, FileMode.Open, FileAccess.Read);
result.Content = new StreamContent(stream);
result.Content.Headers.ContentType =
new MediaTypeHeaderValue("application/octet-stream");
return result;
}
A few things to note about the stream used:
You must not call stream.Dispose(), since Web API still needs to be able to access it when it processes the controller method's result to send data back to the client. Therefore, do not use a using (var stream = …) block. Web API will dispose the stream for you.
Make sure that the stream has its current position set to 0 (i.e. the beginning of the stream's data). In the above example, this is a given since you've only just opened the file. However, in other scenarios (such as when you first write some binary data to a MemoryStream), make sure to stream.Seek(0, SeekOrigin.Begin); or set stream.Position = 0;
With file streams, explicitly specifying FileAccess.Read permission can help prevent access rights issues on web servers; IIS application pool accounts are often given only read / list / execute access rights to the wwwroot.

For Web API 2, you can implement IHttpActionResult. Here's mine:
using System;
using System.IO;
using System.Net;
using System.Net.Http;
using System.Net.Http.Headers;
using System.Threading;
using System.Threading.Tasks;
using System.Web;
using System.Web.Http;
class FileResult : IHttpActionResult
{
private readonly string _filePath;
private readonly string _contentType;
public FileResult(string filePath, string contentType = null)
{
if (filePath == null) throw new ArgumentNullException("filePath");
_filePath = filePath;
_contentType = contentType;
}
public Task<HttpResponseMessage> ExecuteAsync(CancellationToken cancellationToken)
{
var response = new HttpResponseMessage(HttpStatusCode.OK)
{
Content = new StreamContent(File.OpenRead(_filePath))
};
var contentType = _contentType ?? MimeMapping.GetMimeMapping(Path.GetExtension(_filePath));
response.Content.Headers.ContentType = new MediaTypeHeaderValue(contentType);
return Task.FromResult(response);
}
}
Then something like this in your controller:
[Route("Images/{*imagePath}")]
public IHttpActionResult GetImage(string imagePath)
{
var serverPath = Path.Combine(_rootPath, imagePath);
var fileInfo = new FileInfo(serverPath);
return !fileInfo.Exists
? (IHttpActionResult) NotFound()
: new FileResult(fileInfo.FullName);
}
And here's one way you can tell IIS to ignore requests with an extension so that the request will make it to the controller:
<!-- web.config -->
<system.webServer>
<modules runAllManagedModulesForAllRequests="true"/>

For those using .NET Core:
You can make use of the IActionResult interface in an API controller method, like so.
[HttpGet("GetReportData/{year}")]
public async Task<IActionResult> GetReportData(int year)
{
// Render Excel document in memory and return as Byte[]
Byte[] file = await this._reportDao.RenderReportAsExcel(year);
return File(file, "application/vnd.openxmlformats", "fileName.xlsx");
}
This example is simplified, but should get the point across. In .NET Core this process is so much simpler than in previous versions of .NET - i.e. no setting response type, content, headers, etc.
Also, of course the MIME type for the file and the extension will depend on individual needs.
Reference: SO Post Answer by #NKosi

While the suggested solution works fine, there is another way to return a byte array from the controller, with response stream properly formatted :
In the request, set header "Accept: application/octet-stream".
Server-side, add a media type formatter to support this mime type.
Unfortunately, WebApi does not include any formatter for "application/octet-stream". There is an implementation here on GitHub: BinaryMediaTypeFormatter (there are minor adaptations to make it work for webapi 2, method signatures changed).
You can add this formatter into your global config :
HttpConfiguration config;
// ...
config.Formatters.Add(new BinaryMediaTypeFormatter(false));
WebApi should now use BinaryMediaTypeFormatter if the request specifies the correct Accept header.
I prefer this solution because an action controller returning byte[] is more comfortable to test. Though, the other solution allows you more control if you want to return another content-type than "application/octet-stream" (for example "image/gif").

For anyone having the problem of the API being called more than once while downloading a fairly large file using the method in the accepted answer, please set response buffering to true
System.Web.HttpContext.Current.Response.Buffer = true;
This makes sure that the entire binary content is buffered on the server side before it is sent to the client. Otherwise you will see multiple request being sent to the controller and if you do not handle it properly, the file will become corrupt.

The overload that you're using sets the enumeration of serialization formatters. You need to specify the content type explicitly like:
httpResponseMessage.Content.Headers.ContentType = new MediaTypeHeaderValue("application/octet-stream");

You could try
httpResponseMessage.Content.Headers.Add("Content-Type", "application/octet-stream");

You can try the following code snippet
httpResponseMessage.Content.Headers.Add("Content-Type", "application/octet-stream");
Hope it will work for you.

Where is the PostAsJsonAsync method in ASP.NET Core?

I was looking for the PostAsJsonAsync() extension method in ASP.NET Core. Based on this article, it's available in the Microsoft.AspNet.WebApi.Client assembly.
I had thought Microsoft had changed all of the assembly names from Microsoft.AspNet to Microsoft.AspNetCore to be more specific to .NET Core, however, and yet I cannot find an Microsoft.AspNetCore.WebApi.Client assembly.
Where is the PostAsJsonAsync() extension method in ASP.NET Core?

It comes as part of the library Microsoft.AspNet.WebApi.Client
https://www.nuget.org/packages/Microsoft.AspNet.WebApi.Client/

I dont deserve any credit for this. Have a look #danroth27 answer in the following link.
https://github.com/aspnet/Docs/blob/master/aspnetcore/mvc/controllers/testing/sample/TestingControllersSample/tests/TestingControllersSample.Tests/IntegrationTests/HttpClientExtensions.cs
He uses an extension method. Code as below. (Copied from the above github link). I am using it on .Net Core 2.0.
using System.Net.Http;
using System.Net.Http.Headers;
using System.Threading.Tasks;
using Newtonsoft.Json;
namespace TestingControllersSample.Tests.IntegrationTests
{
public static class HttpClientExtensions
{
public static Task<HttpResponseMessage> PostAsJsonAsync<T>(
this HttpClient httpClient, string url, T data)
{
var dataAsString = JsonConvert.SerializeObject(data);
var content = new StringContent(dataAsString);
content.Headers.ContentType = new MediaTypeHeaderValue("application/json");
return httpClient.PostAsync(url, content);
}
public static async Task<T> ReadAsJsonAsync<T>(this HttpContent content)
{
var dataAsString = await content.ReadAsStringAsync();
return JsonConvert.DeserializeObject<T>(dataAsString);
}
}
}

As of .NET 5.0, this has been (re)introduced as an extension method off of HttpClient, via the System.Net.Http.Json namespace. See the HttpClientJsonExtensions class for details.
Demonstration
It works something like the following:
var httpClient = new HttpClient();
var url = "https://StackOverflow.com"
var data = new MyDto();
var source = new CancellationTokenSource();
var response = await httpClient.PostAsJsonAsync<MyDto>(url, data, source.Token);
And, of course, you'll need to reference some namespaces:
using System.Net.Http; //HttpClient, HttpResponseMessage
using System.Net.Http.Json; //HttpClientJsonExtensions
using System.Threading; //CancellationToken
using System.Threading.Tasks; //Task
Alternatively, if you're using the .NET 6 SDK's implicit using directives, three of these will be included for you, so you'll just need:
using System.Net.Http.Json; //HttpClientJsonExtensions
Background
This is based on the design document, which was previously referenced by #erandac—though the design has since changed, particularly for the PostAsJsonAsync() method.
Obviously, this doesn't solve the problem for anyone still using .NET Core, but with .NET 5.0 released, this is now the best option.

That is not part of the ASP.NET Core project. However you can proceed with:
HttpRequestMessage request = new HttpRequestMessage(HttpMethod.Post, "http://myurl/api");
string json = JsonConvert.SerializeObject(myObj);
request.Content = new StringContent(json, System.Text.Encoding.UTF8, "application/json");
HttpClient http = new HttpClient();
HttpResponseMessage response = await http.SendAsync(request);
if (response.IsSuccessStatusCode)
{
}
else
{
}

You Can Use This Extension for use PostAsJsonAsync method in ASP.NET core
using System.Net.Http;
using System.Net.Http.Headers;
using System.Threading.Tasks;
using Newtonsoft.Json;
public static class HttpClientExtensions
{
public static Task<HttpResponseMessage> PostAsJsonAsync<T>(this HttpClient httpClient, string url, T data)
{
var dataAsString = JsonConvert.SerializeObject(data);
var content = new StringContent(dataAsString);
content.Headers.ContentType = new MediaTypeHeaderValue("application/json");
return httpClient.PostAsync(url, content);
}
public static Task<HttpResponseMessage> PutAsJsonAsync<T>(this HttpClient httpClient, string url, T data)
{
var dataAsString = JsonConvert.SerializeObject(data);
var content = new StringContent(dataAsString);
content.Headers.ContentType = new MediaTypeHeaderValue("application/json");
return httpClient.PutAsync(url, content);
}
public static async Task<T> ReadAsJsonAsync<T>(this HttpContent content)
{
var dataAsString = await content.ReadAsStringAsync().ConfigureAwait(false);
return JsonConvert.DeserializeObject<T>(dataAsString);
}
}
see: httpclient-extensions

this is coming late but I think it may help someone down the line. So the *AsJsonAsync() methods are not part of the ASP.NET Core project. I created a package that gives you the functionality. You can get it on Nuget.
https://www.nuget.org/packages/AspNetCore.Http.Extensions
using AspNetCore.Http.Extensions;
...
HttpClient client = new HttpClient();
client.PostAsJsonAsync('url', payload);

You need to add Nuget package System.Net.Http.Formatting.Extension to your project.
Or you can use
client.PostAsync(uri, new StringContent(data, Encoding.UTF8, "application/json"));

To follow on from the answers above, I have a small addition that was required for me to get it to work.
Previously I was using a .NET Core 2.1 web app using the PostAsJsonAsync() method, and when I upgraded to .NET Core 3.1 it no longer worked.
I could not get the above answers to work, and it turned out to be because the text to be posted had to be surrounded by quotes, and any quotes within it had to be escaped. I made the following extension method, which solved my problem:
public static async Task<HttpResponseMessage> PostJsonAsync(this HttpClient client, string uri, string json)
{
//For some reason, not doing this will cause it to fail:
json = $"\"{json.Replace("\"", "\\\"")}\"";
return await client.PostAsync(uri, new StringContent(json, Encoding.UTF8, "application/json"));
}
Note that I am using the System.Text.Json.JsonSerializer as opposed to the Newtonsoft version.

The methodPostAsJsonAsync (along with other *Async methods of the
HttpClient class) is indeed available out of the box – without using
directives.
Your .csproj file should start with
<Project Sdk="Microsoft.NET.Sdk.Web">, and contain the lines
<ItemGroup>
<PackageReference Include="Microsoft.AspNetCore.App" />
</ItemGroup>
I have given a more elaborated answer to a similar question.
(The PackageReference is no longer needed in .NET Core 3.0.)

I used this in a standard 2.0 library
System.Net.Http.Json

make the extension method truly async:
public static async Task<HttpResponseMessage> PostAsJsonAsync<T>(
this HttpClient httpClient, string url, T data)
{
var dataAsString = JsonConvert.SerializeObject(data);
var content = new StringContent(dataAsString);
content.Headers.ContentType = new MediaTypeHeaderValue("application/json");
return await httpClient.PostAsync(url, content);
}

If you are trying to use PostJsonAsync, PutJsonAsync or any other json extension methods in Blazor you need to add a following statement
using Microsoft.AspNetCore.Components;

Dotnet core 3.x runtime itself going to have set of extension methods for HttpClient which uses System.Text.Json Serializer
https://github.com/dotnet/designs/blob/main/accepted/2020/json-http-extensions/json-http-extensions.md

If you are in 2021 and having .Net Core 3.1, make sure in your project file csproj, Microsoft.AspNetCore.App is upto date, the latest is 2.2.8. You can check and update the package as below:
<ItemGroup>
...
<PackageReference Include="Microsoft.AspNetCore.App" Version="2.2.8" />
...
</ItemGroup>
then restore your project from cli like this:
dotnet restore

using Microsoft.AspNetCore.Blazor.HttpClient

Download string from URL using a portable class library (PCL)

I'm trying to download a string from ANY webpage within my portable class library. I've created the most basic setup:
created a new PCL project
compatible with WP8 and WinRT as well as the compulsory components such as Silverlight
As WebClient is not compatible across these systems, it is not possible to use:
string data = new WebClient().DownloadString();
I've tried using this as well (uses this):
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
request.Method = HttpMethod.Get;
HttpWebResponse response = (HttpWebResponse)await request.GetResponseAsync();
string data = ""
using (var sr = new StreamReader(response.GetResponseStream()))
{
data = sr.ReadToEnd();
}
However, when I call the second set of code from an external C# application referencing the PCL, the debugger simply fails with NO warning or error message on:
request.GetResponseAsync();
Is there an easy way to download a string that I'm missing?
*also, why would the debugger simply exit with no explanation?
Edit:
Here is another method I have attempted - based on an answer already provided. Again, this method simply exits and force closes the debugger.
PCL Method:
public static async Task<string> DownloadString()
{
var url = "http://google.com";
var client = new HttpClient();
var data = await client.GetStringAsync(url);
return data;
}
Calling method:
private static async void Method()
{
string data = await PCLProject.Class1.DownloadString();
return data;
}

Install the NuGet packages:
Microsoft.Bcl.Async, which adds async/await support to PCLs.
Microsoft.Net.Http, which adds HttpClient support to PCLs.
Then you can do it the easy way:
var client = new HttpClient();
var data = await client.GetStringAsync(url);

This method worked for me, it returned the HTML source code from google.com:
public async void GetStringFromWebpage()
{
using (HttpClient wc = new HttpClient())
{
var data = await wc.GetStringAsync("http://google.com/");
Debug.WriteLine("string:" + data);
}
}

How can I download HTML source in C#

How can I get the HTML source for a given web address in C#?

You can download files with the WebClient class:
using System.Net;
using (WebClient client = new WebClient ()) // WebClient class inherits IDisposable
{
client.DownloadFile("http://yoursite.com/page.html", #"C:\localfile.html");
// Or you can get the file content without saving it
string htmlCode = client.DownloadString("http://yoursite.com/page.html");
}

Basically:
using System.Net;
using System.Net.Http; // in LINQPad, also add a reference to System.Net.Http.dll
WebRequest req = HttpWebRequest.Create("http://google.com");
req.Method = "GET";
string source;
using (StreamReader reader = new StreamReader(req.GetResponse().GetResponseStream()))
{
source = reader.ReadToEnd();
}
Console.WriteLine(source);

The newest, most recent, up to date answer
This post is really old (it's 7 years old when I answered it), so no one of the other answers used the new and recommended way, which is HttpClient class.
HttpClient is considered the new API and it should replace the old ones (WebClient and WebRequest)
string url = "page url";
HttpClient client = new HttpClient();
using (HttpResponseMessage response = client.GetAsync(url).Result)
{
using (HttpContent content = response.Content)
{
string pageContent = content.ReadAsStringAsync().Result;
}
}
for more information about how to use the HttpClient class (especially in async cases), you can refer this question
NOTE 1: If you want to use async/await
string url = "page url";
HttpClient client = new HttpClient(); // actually only one object should be created by Application
using (HttpResponseMessage response = await client.GetAsync(url))
{
using (HttpContent content = response.Content)
{
string pageContent = await content.ReadAsStringAsync();
}
}
NOTE 2: If use C# 8 features
string url = "page url";
HttpClient client = new HttpClient();
using HttpResponseMessage response = await client.GetAsync(url);
using HttpContent content = response.Content;
string pageContent = await content.ReadAsStringAsync();

You can get the HTML source with:
var html = new System.Net.WebClient().DownloadString(siteUrl)

#cms way is the more recent, suggested in MS website, but I had a hard problem to solve, with both method posted here, now I post the solution for all!
problem:
if you use an url like this: www.somesite.it/?p=1500 in some case you get an internal server error (500),
although in web browser this www.somesite.it/?p=1500 perfectly work.
solution:
you have to move out parameters, working code is:
using System.Net;
//...
using (WebClient client = new WebClient ())
{
client.QueryString.Add("p", "1500"); //add parameters
string htmlCode = client.DownloadString("www.somesite.it");
//...
}
here official documentation