We Keep Coding

C# - AWS EC2 Outgoing Traffic Does Not Initiate

we are using Cloudinary and AWS S3 Bucket as CDN for our application.
At my local development machine, the Cloudinary.NET and AWS SDK works as expected and can upload files. However when I publish the application to the EC2 instance, it never even initiates the HTTP(s) connection. I can see that through WireShark and Microsoft Network Monitor tools. No TCP requests are made, and the code never reaches the lines after the network calls (the SDKs' methods).
To test things out, I even tried implement the API calls using C#'s HttpClient to no avail. Somehow the calls to the HTTP protocol from C# are not processed at all with no exception. It acts like an infinite timeout.
Since I get no errors at all, I have no idea what I am supposed to do.
NOTE
The EC2 instance's Security Group allows ALL outgoing traffic by the way. And it also allows incoming traffic for Ephimeral ports (whatever that is).
Any directions are appreciated.
Here is the code snippet for HttpClient:
using (HttpClient c = new HttpClient())
{
var fileBytes = new byte[model.FileStream.Length];
_logger.LogInformation("Reading bytes from incoming file stream...");
await model.FileStream.ReadAsync(fileBytes, 0, fileBytes.Length);
model.FileStream.Close();
_logger.LogInformation("Read bytes from incoming file stream...");
MultipartFormDataContent form = new MultipartFormDataContent();
// Also tried including an actual file as ByteArrayContent
//form.Add(new ByteArrayContent(fileBytes, 0, fileBytes.Length), "file");
form.Add(new StringContent("SOME_PUBLICLY_ACCESSABLE_URL"), "file");
form.Add(new StringContent(_cloudinarySettings.ApiKey), "api_key");
form.Add(new StringContent(timestamp.ToString()), "timestamp");
form.Add(new StringContent(signature), "signature");
HttpResponseMessage response = await c.PostAsync(
$"https://api.cloudinary.com/v1_1/{_cloudinarySettings.Cloud}/image/upload",
form);
_logger.LogInformation("RESPONSE: {0}", await response.Content.ReadAsStringAsync());
return null;
}
For Cloudinary:
var uploadParams = new ImageUploadParams()
{
File = new FileDescription(fileName, model.FileStream),
PublicId = publicId,
Overwrite = true,
// TODO:
NotificationUrl = model.CallbackUrl
};
_logger.LogInformation("Trying to upload to CDN");
var uploadResult = await _cloudinary.UploadAsync(uploadParams);
_logger.LogInformation("Uploaded image to CDN");
var url = _cloudinary.Api.UrlImgUp
.Format(uploadResult.Format)
.Transform(new Transformation().FetchFormat("auto"))
.BuildUrl(uploadResult.PublicId);
AWS SDK:
var fileTransferUtility =
new TransferUtility(_amazonS3Client);
var objectId = $"{folder}/{fileName}";
var fileTransferUtilityRequest = new TransferUtilityUploadRequest
{
BucketName = _awsS3Settings.BucketName,
InputStream = model.FileStream,
StorageClass = S3StorageClass.Standard,
PartSize = 6291456, // 6 MB.
Key = objectId,
CannedACL = S3CannedACL.PublicRead
};
fileTransferUtilityRequest.Metadata.Add("X-FileExtension", Path.GetExtension(model.FileName));
fileTransferUtilityRequest.Metadata.Add("X-OriginalFileName", model.FileName);
fileTransferUtilityRequest.Metadata.Add("X-GeneratedFileId", fileId.ToString());
fileTransferUtilityRequest.Metadata.Add("X-GeneratedFileName", fileName);
await fileTransferUtility.UploadAsync(fileTransferUtilityRequest);
BONUS
I even included a simple GET call to google.com with no success again at the
using (HttpClient c = new HttpClient())
{
_logger.LogInformation("Sending request to google...");
c.Timeout = TimeSpan.FromSeconds(2);
HttpResponseMessage response = await c.GetAsync(
$"https://google.com");
_logger.LogInformation("RESPONSE: {0}", await response.Content.ReadAsStringAsync());
return null;
}

This is weird but it seems like the default HttpClient somehow uses system level default proxy or something?
The answer to the question Web request from HttpClient stuck is kind of correct. The solution I came up with was forking the CloudinaryDotNet repository, inject a new HttpClientHandler with Proxy explicitly set to null and UseProxy explicitly set to false. This behaviour is not supported because the library internally creates a new HttpClient once at ApiShared.Proxy.cs and it internally decides based on target framework whether to use the HttpClientHandler or not.
For reference, this is the code change that makes it work:
ApiShared.Proxy.cs Line 17
Remove
public HttpClient Client = new HttpClient();
Add
public HttpClient Client = new HttpClient(new HttpClientHandler
{
UseProxy = false,
Proxy = null
});
Obviously this will not work for everyone and is not an ideal solution. The ideal solution would probably involve digging deeper on how EC2 uses default proxies, maybe somehow including the proxy options the SDKs. But I'm posting anyway in case someone else has a similar problem.

What is the role of my machine's Application pool in Windows service development when consuming third party rest services [duplicate]

I am working on Windows Service in visual studio 2017. In the rest api's call, getting exceptions while debugging code. Sometimes first 2 3 calls working after that getting exceptions.
System.Net.WebException: 'The remote server returned an error: (503)
Server Unavailable.'
The remote server returned an error: (429)
Unable to connect to the remote server
When calling same api's from Postman, getting response successfully.
This is my code
private void timer1_Tick(object sender, ElapsedEventArgs e)
{
WriteToFile("timer1_Tick method called..");
try
{
string jsonString = "";
string jsonstring2 = "";
string prodfetchurl = HOST;
var req = WebRequest.Create(prodfetchurl) as HttpWebRequest;
req.Method = "GET";
InitializeRequest(req);
req.Accept = MIME_TYPE;
//System.Threading.Thread.Sleep(5000);
var response = (HttpWebResponse)req.GetResponse();
WriteToFile("First service called...");
if (response.StatusCode == HttpStatusCode.OK)
{
Stream responseStream = response.GetResponseStream();
StreamReader responseReader = new StreamReader(responseStream);
jsonString = responseReader.ReadToEnd();
}
var deserialsseobj = JsonConvert.DeserializeObject<ProductList>(jsonString).Products.Where(i => i.Failed > 0).ToList();
foreach (var a in deserialsseobj)
{
var pid = a.ID;
string url = FailedDevicesUrl + pid.Value + "/failed";
var req2 = WebRequest.Create(url) as HttpWebRequest;
req2.Method = "GET";
InitializeRequest(req2);
req2.Timeout = 300000;
req2.Accept = MIME_TYPE;
var response1 = (HttpWebResponse)req2.GetResponse();
Stream responsestream2 = response1.GetResponseStream();
WriteToFile("Second service called...");
if (response1.StatusCode == HttpStatusCode.OK)
{
StreamReader responsereader1 = new StreamReader(responsestream2);
jsonstring2 = responsereader1.ReadToEnd();
}
var output = JsonConvert.DeserializeObject<List<FailedDeviceList>>(jsonstring2); // Will get List of the Failed devices
List<int> deviceids = new List<int>();
Reprocessdata reproc = new Reprocessdata();
Reprocessdata.DeviceId rprod = new Reprocessdata.DeviceId();
reproc.ForceFlag = true;
reproc.ProductID = pid.Value;
foreach (var dd in output)
{
rprod.ID = dd.DeviceId;
reproc.DeviceIds.Add(rprod);
}
// Reprocess the Product in Devices
var req3 = WebRequest.Create(ReprocessUrl) as HttpWebRequest;
req3.Method = "POST";
InitializeRequest(req3);
req3.Accept = MIME_TYPE;
req3.Timeout = 300000;
req3.ContentType = "application/json";
using (StreamWriter writer = new StreamWriter(req3.GetRequestStream()))
{
string json = new JavaScriptSerializer().Serialize(reproc);
writer.Write(json);
writer.Close();
}
System.Threading.Thread.Sleep(5000);
var response5 = (HttpWebResponse)req3.GetResponse();
WriteToFile("Third service called...");
if (response5.StatusCode == HttpStatusCode.OK)
{
string result;
using (StreamReader rdr = new StreamReader(response5.GetResponseStream()))
{
result = rdr.ReadToEnd();
}
}
}
response.Close();
}
catch (Exception ex)
{
WriteToFile("Simple Service Error on: {0} " + ex.Message + ex.StackTrace);
}
}
Methods used in above code
protected override void OnStart(string[] args)
{
base.OnStart(args);
timer1 = new System.Timers.Timer();
timer1.Interval = 60000; //every 1 min
timer1.Elapsed += new System.Timers.ElapsedEventHandler(timer1_Tick);
timer1.Enabled = true;
WriteToFile("Service has started..");
}
public void InitializeRequest(HttpWebRequest request)
{
request.Headers.Add("aw-tenant-code", API_TENANT_CODE);
request.Credentials = new NetworkCredential(USER_NAME, PASSWORD);
request.KeepAlive = false;
request.AddRange(1024);
}
When I contacted service provide they said everything fine from there side. Is this my code is buggy or windows service not reliable? How can I fix this issue?
Note: All APIS are working fine from Angular application using Visual Studio Code. It means my code is not working.
Edit1: Three below services I am using from this document of VMware.
private const string HOST = "https:host/api/mdm/products/search?";
private const string FailedDevicesUrl = "https:host/api/mdm/products/";
private const string ReprocessUrl = "https:host/api/mdm/products/reprocessProduct";

Response http code 429 indicates that you sending too many requests on target web service.
This means service you trying to send requests has a policies that blocks some requests by request-per-time limit.
Also I admit that external service can be manually configured to throw 403 code in specific cases that you can't know about. If that, this information can be explained in external service documentation... or not :)
What you can do with this?
Fit in limitations
You can make detailed research what limits target webservice has and set up your code to fit in this limitations. For example if service has limitation for receiving only one request per 10 minutes - you must set up your timer to send one request each 10 or more minutes. If documentation not provide such information - you can test it manually by finding some patterns with external service responses.
Use proxy
Every limitation policy based on information about requests senders. Usually this information consists of IP address of sender only. This means if you send 2 requests from two different IP addresses - limitation policy will perceive that like 2 different computers sending these requests. So you can find/buy/rent some proxy IP addresses and send requests through there on target web server.
How to connect through proxy in C# using WebRequest you can see in this answer.
Negotiate with external service provider
If you have possibility to communicate with external service developers or help center, you can ask their to reduce limitations for your IP address (if it static) or provide some mechanisms to avoid limitation policy for you. If for some reason they cannot provide this opportunity, at least you can ask detailed information about limitations.
Repetition mechanism
Some times 503 error code that is outer exception you received may be caused by service unavailable. It means that server can be under maintenance or temporary overloaded. So you can write repetition mechanism to make continious sending requests to server until it'll be accessible.
Polly library may help you with repetition mechanism creation

The inner error of that 503 is:
The remote server returned an error: (429)
HTTP 429 indicates too many requests. Maybe your upstream server can’t process all requests sent.
This can happen when you reached rate limiting / throttling value if you’re calling a third party API.
UPDATE
As per page 28 in the API docs, you could configure throttling when creating a new API. Check if the throttling is too small or maybe turn off the throttling and see if that could fix the error?

Getting web API path from swagger

I have swagger URL http://somehost/swagger/index.html
end methods there as shown on image:
As someone said to me http://somehost/api/Referral/GetReferralByNumber is API address which I can refer it by HTTP request.
static void Main(string[] args)
{
try
{
System.Net.WebClient client = new System.Net.WebClient();
string result = client.DownloadString("http://somehost/api/Referral/GetReferralByNumber");
}
catch (System.Net.WebException ex)
{
Console.WriteLine(ex);
}
Console.ReadKey();
}
this is code for testing API, but
System.Net.WebException: The remote server returned an error: (404)
Not Found exception
is thrown. any help?

Client.DownloadString() makes an GET request. Your action supports POST. Try to use HttpClient, it should be better for your case.

You are hitting a Get Request and for Get request there is no such endpoint.
You should try adding the HTTP option Post to the server.
Code:
private static readonly HttpClient client = new HttpClient();
HttpResponseMessage response = await client.PostAsJsonAsync(
"api/referral/GetReferralByNumber", data);
Where data is the data which should be posted to the server.

You should create an http client and use POST like this:
var method = HttpMethod.Post;
var endPoint = "http://somehost/api/Referral/GetReferralByNumber";
var request = new HttpRequestMessage(method, endPoint);
var client = new HttpClient();
var response = await httpClient.SendAsync(request);

httpclient.GetAsync: The underlying connection was closed: An unexpected error occurred on a send

I have a simple web api which gets a url and sort of generates a short number as short url. I have created a VS console application where I am calling the web api via the HTTPClient object. When I run the code the first time, it throws the following error:
Error message:
One or more errors occurred.
InnerException:
System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.WebException: The underlying connection was closed: An unexpected error occurred on a send. ---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host at System.Net.Sockets.Socket.EndReceive(IAsyncResult asyncResult) at System.Net.Sockets.NetworkStream.EndRead(IAsyncResult asyncResult) --- End of inner exception stack trace --- at System.Net.TlsStream.EndWrite(IAsyncResult asyncResult) at System.Net.ConnectStream.WriteHeadersCallback(IAsyncResult ar) --- End of inner exception stack trace --- at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult) at System.Net.Http.HttpClientHandler.GetResponseCallback(IAsyncResult ar) --- End of inner exception stack trace ---
This is the code that I am using to call the web api. Calling the web api via the browser works just fine.
static void Main(string[] args)
{
string url = "https://app1/api/AddUrl?longUrl=http://google.com";
var result = GetTinyUrl(url).Result;
Console.WriteLine(result.ShortUrl);
}
protected static async Task<UrlResponse> GetUrl(string baseUrl)
{
HttpClientHandler handler = new HttpClientHandler();
handler.UseDefaultCredentials = true;
var client = new HttpClient(handler);
var response = client.GetAsync(baseUrl).Result;
if (response.IsSuccessStatusCode)
{
HttpResponseMessage response1 = response;
response.EnsureSuccessStatusCode();
var resp = await response.Content.ReadAsStringAsync();
var result = JsonConvert.DeserializeObject<UrlResponse>(resp);
return result;
}
return null;
}

Just to say I spent hours looking at a similar problem when connecting to an azure service I'm writing.
Couldn't work out why postman worked fine but my code failed miserably. Similarly other URLs I tried in my code worked fine or gave expected failures.
In the end the reason for the failures was that I forgot lower the Minimum TLS Version in the TLS/SSL Settings blade!!
I think older versions of the HttpClient can only deal with TLS v1.0.

Mixing .Result blocking calls with async await can cause deadlocks.
If using async await then go async all the way, and avoid creating a new instance of the HttpClient for each call as this can lead to socked exhaustion.
static Lazy<HttpClient> http = new Lazy<HttpClient>(() => {
//Handle TLS protocols
System.Net.ServicePointManager.SecurityProtocol =
System.Net.SecurityProtocolType.Tls
| System.Net.SecurityProtocolType.Tls11
| System.Net.SecurityProtocolType.Tls12;
var handler = new HttpClientHandler() {
UseDefaultCredentials = true
};
var client = new HttpClient(handler);
return client;
});
protected static async Task<UrlResponse> GetUrl(string baseUrl) {
var client = http.Value;
var response = await client.GetAsync(baseUrl); //Remove .Result and just await
if (response.IsSuccessStatusCode) {
HttpResponseMessage response1 = response;
response.EnsureSuccessStatusCode();
var resp = await response.Content.ReadAsStringAsync();
var result = JsonConvert.DeserializeObject<UrlResponse>(resp);
return result;
}
return null;
}
Since this is a console application and there is only one thread then calling the async then you have to change how you call the async method in main
static void Main(string[] args) {
string url = "https://app1/api/AddUrl?longUrl=http://google.com";
var result = GetTinyUrl(url).GetAwaiter().GetResult(); //<--
Console.WriteLine(result.ShortUrl);
}
Reference Async/Await - Best Practices in Asynchronous Programming

Choosing a higher target framework in the solution properties fixed my problem.

I usually don't advise using HttpClient for web requests it sometimes likes to misbehave with various web APIs. I would highly encourage you to use HttpWebRequest instead. You can properly craft the HTTP request with full control and will produce a better overall result.
string uri = "https://api.foo.com/bar";
var httpWebRequest = (HttpWebRequest)WebRequest.Create(uri);
httpWebRequest.ContentType = "application/json";
httpWebRequest.Method = "GET"; // PUT, POST, DELETE, etc
//httpWebRequest.ContentLength = 0; // sometimes you need to send zero (depending on API)
HttpWebResponse httpResponse = (HttpWebResponse)await httpWebRequest.GetResponseAsync();
using (var streamReader = new StreamReader(httpResponse.GetResponseStream()))
{
var response = streamReader.ReadToEnd();
// this is your code here...
var result = JsonConvert.DeserializeObject<UrlResponse>(response);
return result;
}

PostAsync in UWP App not working (no content sent)

I'm a bit stuck, I am trying to create a UWP App that will post XML content to a web service. I can get this to work in a regular .net console app without an issue. Trying to re-create this using UWP is proving to be tricky. Using fiddler I've narrowed down that the web service end point isn't receiving my content. It looks like the headers are setup properly the content length is sent correctly but the actual content isn't sent. Here is the heart of the code, it crashes/throws an exception after:
HttpResponseMessage ResponseMessage = await request.PostAsync(requestUri, httpContent2).ContinueWith(
(postTask) => postTask.Result.EnsureSuccessStatusCode());
When I try to execute the PostASync, looking at fiddler, I'm getting:
HTTP/1.1 408 Request body incomplete
Date: Mon, 14 Nov 2016 15:38:53 GMT
Content-Type: text/html; charset=UTF-8
Connection: close
Cache-Control: no-cache, must-revalidate
Timestamp: 10:38:53.430
The request body did not contain the specified number of bytes. Got 0, expected 617
I'm positive that I am getting content to post correct (I read it from a file, I send it to debug window to verify and it's correct). I think it might have to do with HttpContent httpContent2 - In regular .NET I've never needed to use this but with PostAsync I need to use it.
Any thoughts would be appreciated, thank you!
public async void PostWebService()
{
string filePath = "Data\\postbody.txt";
string url = "https://outlook.office365.com/EWS/Exchange.asmx";
Uri requestUri = new Uri(url); //replace your Url
var myClientHandler = new HttpClientHandler();
myClientHandler.Credentials = new NetworkCredential("user#acme.com", "password");
HttpClient request = new HttpClient(myClientHandler);
string contents = await ReadFileContentsAsync(filePath);
Debug.WriteLine(contents);
HttpContent httpContent2 = new StringContent(contents, Encoding.UTF8, "text/xml");
string s = await httpContent2.ReadAsStringAsync();
Debug.WriteLine(s); //just checking to see if httpContent has the correct data
//HttpResponseMessage ResponseMessage = await request.PostAsync(requestUri, httpContent);
request.MaxResponseContentBufferSize = 65000;
HttpResponseMessage ResponseMessage = await request.PostAsync(requestUri, httpContent2).ContinueWith(
(postTask) => postTask.Result.EnsureSuccessStatusCode());
Debug.WriteLine(ResponseMessage.ToString());
}

Well it seems like I found the root cause to my problem. This is appears to be a known bug with System.Net.Http.HttpClient when using network authentication. See this article here
My initial mistake was that I wasn't catching an exceptions thrown by PostAsync. once I wrapped that inside a try/catch block I got the following exception thrown:
“This IRandomAccessStream does not support the GetInputStreamAt method because it requires cloning and this stream does not support cloning.”
The first paragraph of the article I linked to above states:
When you use the System.Net.Http.HttpClient class from a .NET
framework based Universal Windows Platform (UWP) app and send a
HTTP(s) PUT or POST request to a URI which requires Integrated Windows
Authentication – such as Negotiate/NTLM, an exception will be thrown.
The thrown exception will have an InnerException property set to the
message:
“This IRandomAccessStream does not support the GetInputStreamAt method
because it requires cloning and this stream does not support cloning.”
The problem happens because the request as well as the entity body of
the POST/PUT request needs to be resubmitted during the authentication
challenge. The above problem does not happen for HTTP verbs such as
GET which do not require an entity body.
This is a known issue in the RTM release of the Windows 10 SDK and we
are tracking a fix for this issue for a subsequent release.
The recommendation and work around that worked for me was to use the Windows.Web.Http.HttpClient instead of System.Net.Http.HttpClient
Using that recommendation, the following code worked for me:
string filePath = "Data\\postbody.txt";
string url = "https://outlook.office365.com/EWS/Exchange.asmx";
Uri requestUri = new Uri(url); //replace your Url
string contents = await ReadFileContentsAsync(filePath);
string search_str = txtSearch.Text;
Debug.WriteLine("Search query:" + search_str);
contents = contents.Replace("%SEARCH%", search_str);
Windows.Web.Http.Filters.HttpBaseProtocolFilter hbpf = new Windows.Web.Http.Filters.HttpBaseProtocolFilter();
Windows.Security.Credentials.PasswordCredential pcred = new Windows.Security.Credentials.PasswordCredential(url, "username#acme.com", "password");
hbpf.ServerCredential = pcred;
HttpClient request = new HttpClient(hbpf);
Windows.Web.Http.HttpRequestMessage hreqm = new Windows.Web.Http.HttpRequestMessage(Windows.Web.Http.HttpMethod.Post, new Uri(url));
Windows.Web.Http.HttpStringContent hstr = new Windows.Web.Http.HttpStringContent(contents, Windows.Storage.Streams.UnicodeEncoding.Utf8, "text/xml");
hreqm.Content = hstr;
// consume the HttpResponseMessage and the remainder of your code logic from here.
try
{
Windows.Web.Http.HttpResponseMessage hrespm = await request.SendRequestAsync(hreqm);
Debug.WriteLine(hrespm.Content);
String respcontent = await hrespm.Content.ReadAsStringAsync();
}
catch (Exception ex)
{
string e = ex.Message;
Debug.WriteLine(e);
}
Hopefully this is helpful to someone else hitting this issue.