We Keep Coding

C# - AWS EC2 Outgoing Traffic Does Not Initiate

we are using Cloudinary and AWS S3 Bucket as CDN for our application.
At my local development machine, the Cloudinary.NET and AWS SDK works as expected and can upload files. However when I publish the application to the EC2 instance, it never even initiates the HTTP(s) connection. I can see that through WireShark and Microsoft Network Monitor tools. No TCP requests are made, and the code never reaches the lines after the network calls (the SDKs' methods).
To test things out, I even tried implement the API calls using C#'s HttpClient to no avail. Somehow the calls to the HTTP protocol from C# are not processed at all with no exception. It acts like an infinite timeout.
Since I get no errors at all, I have no idea what I am supposed to do.
NOTE
The EC2 instance's Security Group allows ALL outgoing traffic by the way. And it also allows incoming traffic for Ephimeral ports (whatever that is).
Any directions are appreciated.
Here is the code snippet for HttpClient:
using (HttpClient c = new HttpClient())
{
var fileBytes = new byte[model.FileStream.Length];
_logger.LogInformation("Reading bytes from incoming file stream...");
await model.FileStream.ReadAsync(fileBytes, 0, fileBytes.Length);
model.FileStream.Close();
_logger.LogInformation("Read bytes from incoming file stream...");
MultipartFormDataContent form = new MultipartFormDataContent();
// Also tried including an actual file as ByteArrayContent
//form.Add(new ByteArrayContent(fileBytes, 0, fileBytes.Length), "file");
form.Add(new StringContent("SOME_PUBLICLY_ACCESSABLE_URL"), "file");
form.Add(new StringContent(_cloudinarySettings.ApiKey), "api_key");
form.Add(new StringContent(timestamp.ToString()), "timestamp");
form.Add(new StringContent(signature), "signature");
HttpResponseMessage response = await c.PostAsync(
$"https://api.cloudinary.com/v1_1/{_cloudinarySettings.Cloud}/image/upload",
form);
_logger.LogInformation("RESPONSE: {0}", await response.Content.ReadAsStringAsync());
return null;
}
For Cloudinary:
var uploadParams = new ImageUploadParams()
{
File = new FileDescription(fileName, model.FileStream),
PublicId = publicId,
Overwrite = true,
// TODO:
NotificationUrl = model.CallbackUrl
};
_logger.LogInformation("Trying to upload to CDN");
var uploadResult = await _cloudinary.UploadAsync(uploadParams);
_logger.LogInformation("Uploaded image to CDN");
var url = _cloudinary.Api.UrlImgUp
.Format(uploadResult.Format)
.Transform(new Transformation().FetchFormat("auto"))
.BuildUrl(uploadResult.PublicId);
AWS SDK:
var fileTransferUtility =
new TransferUtility(_amazonS3Client);
var objectId = $"{folder}/{fileName}";
var fileTransferUtilityRequest = new TransferUtilityUploadRequest
{
BucketName = _awsS3Settings.BucketName,
InputStream = model.FileStream,
StorageClass = S3StorageClass.Standard,
PartSize = 6291456, // 6 MB.
Key = objectId,
CannedACL = S3CannedACL.PublicRead
};
fileTransferUtilityRequest.Metadata.Add("X-FileExtension", Path.GetExtension(model.FileName));
fileTransferUtilityRequest.Metadata.Add("X-OriginalFileName", model.FileName);
fileTransferUtilityRequest.Metadata.Add("X-GeneratedFileId", fileId.ToString());
fileTransferUtilityRequest.Metadata.Add("X-GeneratedFileName", fileName);
await fileTransferUtility.UploadAsync(fileTransferUtilityRequest);
BONUS
I even included a simple GET call to google.com with no success again at the
using (HttpClient c = new HttpClient())
{
_logger.LogInformation("Sending request to google...");
c.Timeout = TimeSpan.FromSeconds(2);
HttpResponseMessage response = await c.GetAsync(
$"https://google.com");
_logger.LogInformation("RESPONSE: {0}", await response.Content.ReadAsStringAsync());
return null;
}

This is weird but it seems like the default HttpClient somehow uses system level default proxy or something?
The answer to the question Web request from HttpClient stuck is kind of correct. The solution I came up with was forking the CloudinaryDotNet repository, inject a new HttpClientHandler with Proxy explicitly set to null and UseProxy explicitly set to false. This behaviour is not supported because the library internally creates a new HttpClient once at ApiShared.Proxy.cs and it internally decides based on target framework whether to use the HttpClientHandler or not.
For reference, this is the code change that makes it work:
ApiShared.Proxy.cs Line 17
Remove
public HttpClient Client = new HttpClient();
Add
public HttpClient Client = new HttpClient(new HttpClientHandler
{
UseProxy = false,
Proxy = null
});
Obviously this will not work for everyone and is not an ideal solution. The ideal solution would probably involve digging deeper on how EC2 uses default proxies, maybe somehow including the proxy options the SDKs. But I'm posting anyway in case someone else has a similar problem.

How to diagnose a 401 error attempting to get an OAuth2 bearer token in c# .NET Core?

I have some limited skills in c++ and have recently moved in C# (asp.net) and azure Web services. As a PoC I'm trying to make REST calls into PayPal (which I'll need to be using professionally in 3 -6 months).
I've set up my personal PayPal account using the instructions here and I get a bearer token back using curl as described in the link. Awesome.
I'm now trying to do this from .NET Core C# and all I get is a 401 error. I've examined the request and it seems the same as the curl in terms of headers; the base64 encoded credentials I think I'm adding are the same as the ones in the verbose curl log (I examined the two base64 strings by eye) so it must be something I'm doing (or not doing) in the set up of the call. I'm looking for suggestions, pointers, or flat out laughter at the obvious mistake I've made.
I've set up what I believe to be a named client thus:
public IConfiguration Configuration { get; }
// This method gets called by the runtime. Use this method to add services to the container.
public void ConfigureServices(IServiceCollection services)
{
services.AddHttpClient("PayPal", c =>
{
c.BaseAddress = new Uri("https://api.sandbox.paypal.com/v1/");
c.DefaultRequestHeaders.Add("Accept", "application/json");
c.DefaultRequestHeaders.Add("Accept-Language", "en_US");
});
(with all the other stuff that comes free with VS under it omitted for brevity).
I attempt the call thus:
string clientCredString = CLIENTID + ":" + SECRET;
var clientCreds = System.Text.Encoding.UTF8.GetBytes(clientCredString);
var client = _clientFactory.CreateClient("PayPal");
client.DefaultRequestHeaders.Authorization = new System.Net.Http.Headers.AuthenticationHeaderValue("Basic", System.Convert.ToBase64String(clientCreds));
var messageBody = new Dictionary<string,string > ();
messageBody.Add("grant_type", "client_credientials");
var request = new HttpRequestMessage(HttpMethod.Get, "oauth2/token")
{
Content = new FormUrlEncodedContent(messageBody)
};
string token;
var response = await client.SendAsync(request);
if (response.IsSuccessStatusCode)
{
var json = await response.Content.ReadAsStringAsync();
token = JsonConvert.DeserializeObject<string>(json);
}
else
{
throw new ApplicationException("Well that failed");
}
and get a 401 code for my trouble.
Suggestions for troubleshooting, better methods of doing this and laughter at my foolishness all welcomed.

Update:
I read the documentation, a couple of items stand out to me:
Requires a verb of post.
Uses FormUrlEncodedContent for client credentials.
Basic auth requires username and password (Client Id & Secret)
I believe the syntax should be:
var client = new HttpClient();
using var request = new HttpRequestMessage(HttpMethod.Post, "...");
request.Content = new Dictionary<string, string>() { "grant_type", "client_credentials" };
request.Headers.Authorization = new AuthenticationHeaderValue("Basic", $"{Encoding.UTF8.GetBytes($"{id}:{secret}")}");
HttpResponseMEssage = response = await client.PostAsync(request);
response.EnsureSuccessStatusCode();
var content = await response.Content.ReadAsStringAsync();

For the benefit of future readers:
It was, as suggested, an encoding problem. The line:
var clientCreds = System.Text.Encoding.UTF8.GetBytes(clientCredString);
needed to be
var clientCreds = System.Text.Encoding.ASCII.GetBytes(clientCredString);
It should also be noted that this particular operation requires a POST not a GET as I was using, but once I started sending properly encoded requests the errors started to make a lot more sense.

.NET Core Httpclient works but .Net Framework 4.7.2 httpclient doesn't

Sorry for the awful title, I'm not really sure how to phrase my issue in a short title format.
I'm trying to communicate with an external API. I make a basic authentication request to that API and get an x-csrf-token and a session token from the api.
I then make another request to that API, now using the x-csrf-token as a header and attach the session token to the header as "cookie".
The team that maintains the API sent me an example project that handles all of the above, and it looks like this:
public static async Task<string> Send(string apiname, string value)
{
// Fetch the authorization tokens from SAP
HttpClient client = new HttpClient();
client.BaseAddress = new Uri(basePath);
client.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Basic", System.Convert.ToBase64String(System.Text.Encoding.UTF8.GetBytes(user + ":" + password)));
client.DefaultRequestHeaders.Add("x-csrf-token", "Fetch");
string csrfToken = "";
string sessionCookie = "";
HttpResponseMessage response = await client.GetAsync(string.Empty);
IEnumerable<string> values;
if (response.Headers.TryGetValues("x-csrf-token", out values))
{
csrfToken = values.FirstOrDefault();
}
if (response.Headers.TryGetValues("set-cookie", out values))
{
sessionCookie = values.Where(s => s.StartsWith("SAP_SESSION")).FirstOrDefault();
}
// Reinstantiate the HttpClient, adding the tokens we just got from SAP
client = new HttpClient();
client.DefaultRequestHeaders.Add("x-csrf-token", csrfToken);
client.DefaultRequestHeaders.Add("cookie", sessionCookie);
client.BaseAddress = new Uri(basePath);
client.DefaultRequestHeaders.Accept.Clear();
client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
// Have to parse the string this way otherwise it'll break the dates
JToken token;
using (var sr = new StringReader(value))
using (var jr = new JsonTextReader(sr) { DateParseHandling = DateParseHandling.None })
{
token = JToken.ReadFrom(jr);
}
HttpResponseMessage response2 = await client.PostAsJsonAsync(apiname, token);
string responseBody = await response2.Content.ReadAsStringAsync();
return responseBody;
}
This all works great as a .NET Core webAPI (and also as a .netcore console app).
Interestingly enough (in my opinion anyway), when I use the exact same code in a .net 4.7.2 project, it doesn't append the "cookie" header properly, and so I'm getting an unauthorized redirect back from the API.
To be absolutely sure that I didn't change any code, I started from scratch with a brand new .netcore 2.0 console app and a brand new .net 4.7.2 console app and copy-pasted the exact same code and installed the same nuget packages (Newtonsoft.JSON and Microsoft.WebApi.Client). I inspected my web traffic with fiddler (seen below) and you can see that in .netcore, the cookie appends properly and everything works, but in .net 4.7.2, the API returns a redirect to authenticate.

HttpClient will eat the custom cookie if you do not setUseCookies to false,
using (var handler = new HttpClientHandler { UseCookies = false })
using (client = new HttpClient(handler) { BaseAddress = new Uri(Path) }){
client.DefaultRequestHeaders.Add("cookie", cookieValue);
}
It will try to use the cookie container and at the same time ignore any custom cookie headers, very frustrating behavior if you ask me.

.Net Framework uses Cookie Container.
Also core, perhaps its a better implementation then what you are doing now and more supported.
Please see cookie container docs
Small example:
var cookieContainer = new CookieContainer();
this.handler = new HttpClientHandler
{
CookieContainer = cookieContainer,
UseCookies = true
};
client = new HttpClient(handler);

401 Unauthorized after release site

I have my web api which uses such code to get data from other api and do some operations with it :
var credential = new NetworkCredential("login", "password");
var myCache = new CredentialCache();
myCache.Add(uri, "Negotiate", credential);
var handler = new HttpClientHandler();
handler.AllowAutoRedirect = true;
handler.Credentials = myCache;
var httpClient = new HttpClient(handler);
var response = await httpClient.GetStringAsync(GetEmployeesUrl);
employees = JsonConvert.DeserializeObject<IEnumerable<EmployeeDto>>(response);
When I was debugging this code with IIS Express on my local host it's pretty good works and give me OK response from server. When I tried to deploy my api to real IIS this var response = await httpClient.GetStringAsync(GetEmployeesUrl);always returns 401 Unauthorized error.
Have someone some ideas what it can be? I have used webclient and webrequest they worked on my local machine but answer after deploy on server was the same.

Downgrade code of HttpClient .NET 4.5 to .NET 4.0

I have this code that is working fine in .NET 4.5.
var handler = new HttpClientHandler();
handler.UseDefaultCredentials = true;
handler.PreAuthenticate = true;
handler.ClientCertificateOptions = ClientCertificateOption.Automatic;
var client = new HttpClient(handler);
client.BaseAddress = new Uri("http://localhost:22678/");
client.DefaultRequestHeaders.Accept.Add(
new MediaTypeWithQualityHeaderValue("application/json"));
var loginBindingModel = new LoginBindingModel { Password = "test01", UserName = "test01" };
var response = await client.PostAsJsonAsync("api/Account/Login", loginBindingModel);
response.EnsureSuccessStatusCode(); // Throw on error code.
tokenModel = await response.Content.ReadAsAsync<TokenModel>();
Now I have to do the same thing in .NET 4.0.
But I am facing two problems I do not know how to resolve them.
In .NET 4.0. method client.PostAsJsonAsync does not exist.
The existing method is client.PostAsync and it needs HttpContext.
I do request within WPF client... Guys, I have no clue what I can do to archive the same functionality...
Please, help!

Suggest using the BCL / async / "Microsoft HTTP Client Libraries" helper projects to "supplement" .Net 4.0 with equivalent functionality to .Net 4.5 (Can find the latest versions in the NuGet package manager.)
See the following link for more info: http://www.nuget.org/packages/microsoft.bcl.async
(Note: you can get support for http client via same basic mechanism)

Just for someone who is looking for the same .NET 4.0 pure solution I found that code working fine. But It does not have ASYNC/AWAIT thing that should be implemented as well. Anyway it is working exactly the same way as my code in the question.
var webAddr = "http://localhost:22678/api/Account/Login";
var httpWebRequest = (HttpWebRequest)WebRequest.Create(webAddr);
httpWebRequest.ContentType = "application/json; charset=utf-8";
httpWebRequest.Method = "POST";
using (var streamWriter = new StreamWriter(httpWebRequest.GetRequestStream()))
{
var loginBindingModel = new WebAPILoginBindingModel { Password = "test01", UserName = "test01" };
var myJsonString = Newtonsoft.Json.JsonConvert.SerializeObject(loginBindingModel);
streamWriter.Write(myJsonString);
streamWriter.Flush();
}
var httpResponse = (HttpWebResponse)httpWebRequest.GetResponse();
using (var streamReader = new StreamReader(httpResponse.GetResponseStream()))
{
var result = streamReader.ReadToEnd();
}
If you want to use use ASYNC/AWAIT for .NET 4.0 you have to install Microsoft.BCL dll and use async versions of HttpClient as well.

Even after installing the following packages i got other errors.
Install-Package Microsoft.Net.Http
Install-Package System.Net.Http.Formatting.Extension
Turns out Install-Package System.Net.Http.Formatting.Extension installs version of system.net that does not support dot net 4. So I has to stick to PostAsync and create an httpContent object from my custom object.
So
PostAsJsonAsync("api/Account/Login", loginBindingModel)
Becomes (Note i used json.net here )
public HttpContent CreateHttpContent(object data)
{
return new StringContent(JsonConvert.SerializeObject(data), Encoding.UTF8, "application/json");
}
PostAsync("api/Account/Login",CreateHttpContent(loginBindingModel))