We Keep Coding

Cloudant endpoint DNS cannot be resolved

We have for some time been using Cloudant NoSQL from the IBM Cloud and we have been extremely happy with the speed, simplicity and reliability. BUT a few weeks ago our front-end server which stores data in Cloudant database startet periodically to log exceptions: "The remote name could not be resolved: '[A unique ID]-bluemix.cloudant.com" at System.Net.HttpWebRequest.EndGetRequestStream.
I added a DNS lookup when the error occurs which logs: "This is usually a temporary error during hostname resolution and means that the local server did not receive a response from an authoritative server" at System.Net.Dns.GetAddrInfo(String name).
This relaxed error message indicates it is not harmful but for us it is.
We see the error for 1-3 minutes every 30-120 minutes on servers but not while debugging locally (this could be lack of patience and/or traffic).
Below is one method of seven which fails
using (HttpClientHandler handler = new HttpClientHandler())
{
handler.Credentials = new NetworkCredential(Configuration.Cloudant.ApiKey, Configuration.Cloudant.ApiPassword);
using (var client = new HttpClient(handler))
{
var uri = new Uri(Configuration.Cloudant.Url); //{https://[A unique ID]-bluemix.cloudant.com/[Our Product]/_find}
var stringContent = new StringContent(QueryFromResource(),
UnicodeEncoding.UTF8,
"application/json");
var task = TaskEx.Run(async () => await client.PostAsync(uri, stringContent));
task.Wait(); // <------ Exception here
if (task.Result.StatusCode == HttpStatusCode.OK)
{
// Handle response deleted
}
}
}
We have updated our .Net framework, experimented with DnsRefreshTimeout, refactored code, extended caching but we keep seeing the issue.
We also added a DNS lookup to Google when the error occurs and this is consistently successful.
Initially we thought load might we an issue but we see the issue even when there is no traffic.
Suggestions are much appreciated!

.Net HttpListener.GetContext() gives "503 Service Unavailable" to the client

I am trying to write a very very simple custom Http server in C#.
The code to achieve a connection is as simple as .Net offers it:
// get our IPv4 address
IPAddress[] localIPs = Dns.GetHostAddresses(Dns.GetHostName());
IPAddress theAddress = localIPs.Where(ip => ip.AddressFamily == AddressFamily.InterNetwork).FirstOrDefault();
// Create a http listener.
var listener = new HttpListener();
string myUrlPrefix = $"http://{theAddress.ToString()}:{port}/MyService/";
listener.Prefixes.Add(myUrlPrefix);
Console.WriteLine($"Trying to listen on {myUrlPrefix}");
listener.Start();
Console.WriteLine("Listening...");
HttpListenerContext context = null;
try
{
context = listener.GetContext();
}
catch(Exception ex)
{
Console.WriteLine(ex.Message);
}
HttpListenerRequest request = context.Request;
When running this, GetContext() blocks, never returns and does not throw an exception.
listener.Start() works -- the URL is properly reserved for the correct user with 'netsh http add urlacl...'., else Start() would throw. The URL is also only reserved once.
I can see with netstat -na that the port is being listened on.
I am trying to access this with either a browser or with the cygwin implementation of wget. Both give me "ERROR 503: Service Unavailable.".
Here is the wget output:
$ wget http://127.0.0.1:45987/MyService/
--2016-03-06 14:54:37-- http://127.0.0.1:45987/MyService/
Connecting to 127.0.0.1:45987... connected.
HTTP request sent, awaiting response... 503 Service Unavailable
2016-03-06 14:54:37 ERROR 503: Service Unavailable.
This means that the TCP connection gets established, yet HttpListener.GetContext() does not bother to service the request and give me the context, it also doesn't bother to at least throw an exception to tell me what's wrong. It just closes the connection. From the debug output, it doesn't even have a First-Chance exception inside that it would catch itself.
I have searched the Net and Stackoverflow up and down and don't find anything useful.
There are no errors in the Windows Event log.
This is Windows 10 Pro and Microsoft Visual Studio Community 2015
Version 14.0.23107.0 D14REL with Microsoft .NET Framework Version 4.6.01038.
It may be worth mentioning that I have also tested this with all firewalls off, still the same result.
Does anyone have any idea what could be going wrong in the GetContext() and how I could debug or solve this?
======
EDIT: I have enabled .Net Framework source stepping and have stepped into GetContext(). The blocking and refusal to service http requests happens in something called "UnsafeNclNativeMethods.HttpApi.HttpReceiveHttpRequest" which, I guess, might be the native HttpReceiveHttpRequest method documented at https://msdn.microsoft.com/en-us/library/windows/desktop/aa364495(v=vs.85).aspx . So it's the native API that is refusing my request.

I just had the same problem, and it was caused by a URL ACL.
From an elevated command prompt
netsh http show urlacl will show all URL ACLs
netsh http delete urlacl http://+:1234/ will delete the URL ACL with the URL http://+:1234/
netsh http add urlacl url=http://+:1234/ user=Everyone will add a URL ACL that any local user can listen on (normally this should use the network service account)
This URL needs to match the prefix in your HttpListener.
If you're only listening on localhost, you don't need a URL ACL.
Normally not having the correct the URL ACL results in a permissions error when starting the listener. I'm not sure what the specific situation is that causes it to get past this but then return 503 Service Unavailable on GetContext.

I haven't found a solution, but a workaround.
Interestingly, if I go a level deeper and use a plain TcpListener, I have no problem receiving the request as plain text.
// Create a Tcp listener.
mTcpListener = new TcpListener(theAddress, port);
Console.WriteLine($"Trying to listen on {theAddress}:{port}");
mTcpListener.Start();
Console.WriteLine("Listening...");
Socket socket = null;
try
{
socket = mTcpListener.AcceptSocket();
}
catch (Exception ex)
{
Console.WriteLine(ex.Message);
return;
}
// wait a little for the socket buffer to fill up
await Task.Delay(20);
int bytesAvailable = socket.Available;
var completeBuffer = new List<byte>();
while (bytesAvailable > 0)
{
byte[] buffer = new byte[bytesAvailable];
int bytesRead = socket.Receive(buffer);
completeBuffer.AddRange(buffer.Take(bytesRead));
bytesAvailable = socket.Available;
}
byte[] receivedBytes = completeBuffer.ToArray();
string receivedString = Encoding.ASCII.GetString(receivedBytes);
... works like a charm. The returned string is
GET /MyService/ HTTP/1.1
User-Agent: Wget/1.17.1 (cygwin)
Accept: */*
Accept-Encoding: identity
Host: 127.0.0.1:45987
Connection: Keep-Alive
So, as a workaround, I could implement my own http request parser and response generator... not that I like being forced to reinvent the wheel.

NameResolutionFailure Nightmare: Even ModernHttpClient is failing

I am constantly getting NameResolutionFailure errors when I make web requests (GET) in my MVVM Cross Android app.
I've tried out much of the advice provided in forums re this issue as it's a common one but I am unable to fix this.
My current attempt uses the NuGet package ModernHttpClient to perform web requests. The same error occurs - unable to resolve the domain to an IP - however the error message is slightly different from what was happening when I was using HttpWebRequest so I guess thats a slight improvement?
java.net.UnknownHostException: Unable to resolve host "jsonplaceholder.typicode.com": No address associated with hostname
Can you provide advice on why this is always failing? Maybe its my method thats not truely utilising ModernHttpClient?
The following code is part of my IRestService class located in the Core PCL and not in the Android project.
public async Task MakeRequest(WebHeaderCollection headers = null)
{
var handler = new NativeMessageHandler();
string requestUrl = "http://jsonplaceholder.typicode.com/posts/1";
try
{
using (var client = new HttpClient(handler))
{
//client.BaseAddress = new Uri(baseAddress);
client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
var response = await client.GetAsync(requestUrl);
if (response.IsSuccessStatusCode)
{
var stream = await response.Content.ReadAsStreamAsync();
var reader = new StreamReader(stream);
Mvx.Trace("Stream: {0}", reader.ReadToEnd());
}
}
}
catch (WebException ex)
{
Mvx.Trace("MakeRequest Error: '{0}'", ex.Message);
}
return;
}
PS: I have also attempted to use Cheesebarons ModernHttpClient MVVM Cross plugin but this is causing compile errors in release mode and there is no documentation about what methods and classes it has - maybe its not supported anymore?
PPS: And yes my manifest has internet permission (I check the options and the actual manifest file to confirm)

The ModernHttpClient plugin for MvvmCross is not needed anymore, so don't use it.
So since you have Internet permission set in the AndroidManifest the problem is something else. I've experienced on some Android devices, that the first call to some Internet resource fails with the same error you get. The way I've usually worked around that is to retry the call.
There are various ways to do so.
You can create your own HttpClientHandler which wraps the one coming from ModernHttpClient and create your own retry handling in there.
You can retry using a library such as Polly
I tend to do the latter. So if you add the Polly NuGet you can pretty quickly test out if this solves the problem for you:
var policy = Policy.WaitAndRetryAsync(
5,
retryAttempt => TimeSpan.FromSeconds(Math.Pow(2, retryAttempt)),
(ex, span) => {
Mvx.Trace("Retried because of {0}", ex);
}
);
Then retry your task like:
await policy.ExecuteAsync(() => MakeRequest(someHeaders)).ConfigureAwait(false);
Usually on second try the exception goes away.
I've seen this problem on Nexus 5, Nexus 7 and on an Samsung Galaxy SII, but not on other devices. It might also help if you toggle the WiFi on the device prior to debugging.
In any case, you app should have some kind of retry logic as Internet connections can be spotty at times.

Request from C# to Apache: The underlying connection was closed: An unexpected error occurred on a receive

I have two linux Apache servers running behind load balancer.
On both Apache server I have installed exactly the same installation of Yourls API for shortening URLs. I am sending request to that Yourls API to short URL.
Request to these servers are sent from C# production site, for shortening URLs.
It worked fine for months, and then suddenly on request I start receiving error:
The underlying connection was closed: An unexpected error occurred on
a receive.
Nothing is changed in meanwhile. Apache server doesn't have any security restriction or something.
This code for request:
public dynamic SendRequest(string action, NameValueCollection parameters = null)
{
var client = new NoKeepAliveWebClient();
string jsonResult = string.Empty;
dynamic result;
//we need to form url with requests
var data = new NameValueCollection();
data["format"] = "json";
data["action"] = action;
data["signature"] = this.key;
if (parameters != null)
{
foreach (string key in parameters)
{
data[key] = parameters[key];
}
}
try
{
byte[] byteResult = client.UploadValues(url, data);
jsonResult = Encoding.UTF8.GetString(byteResult);
dynamic tempResult = JObject.Parse(jsonResult);
result = tempResult;
}
catch (Exception ex)
{
logger.Error("Error SendRequest (Action: {0}) Json: {1} {2}",
action,
jsonResult,
ex.Message + ex.StackTrace);
throw;
}
return result;
}
I am not getting this error for all request, but I am getting it for most of them.
I don't know anything about C#. I am responsible for Apache server, and I cannot find anything on those servers what can block those requests, or cause errors like this.
So my question is:
Does this can be caused by Apache server or it is specific for C#. Or maybe it have something with load balancer.
I guess that question is kind dumb, but I need to know asap which side is causing the problem.
EDIT
By adding more logs on C# side I got detailed errors.
ErrorType: WebException ErrorMessage: The underlying connection was closed: An unexpected error occurred on a receive.
InnerExceptionType: IOException InnerExceptionMessage: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host.
EDIT
I concluded that the problem is with load balancer (hproxy)
I have one server on which is installed hproxy. Hproxy serve two aplications servers. When I am sending requests over hproxy I am getting those errors.
Now I stopped using hproxy and I am calling requests directly to one aplications servers and it works fine now, without errors. But I need it works with hproxy.
So any idea, what in hproxy can cause those errors.

await HttpClient.SendAsync not throwing exception when expected

I have a Windows Phone 8 app containing code as follows:
using (var client = new HttpClient())
{
var httpRequest = new HttpRequestMessage(method, uri);
try
{
var response = client.SendAsync(httpRequest);
var httpResponse = await response;
if (httpResponse.IsSuccessStatusCode)
{
var result = await httpResponse.Content.ReadAsStringAsync();
return result;
}
else
{
HandleError(httpResponse);
return null;
}
}
catch (Exception ex)
{
throw;
}
}
If the client successfully connects to the server, I will get the expected results, including all the appropriate HTTP status codes and reason phrases.
If the client is unable to contact the server (e.g. incorrect domain/IP/port in the URL), the awaited task completes after some delay, with a 404 status and no reason phrase. The task never throws an exception. The task (the response variable in the code snippet) has the status of "ran to completion". Nothing about the result is indicative of the actual problem - be it networking failure, unreachable server, etc.
How can I capture more meaningful errors in the case where the URL points to a non-existent or unreachable server, socket connection refused, etc.? Shouldn't SendAsync be throwing specific exceptions for such cases?
FWIW, the client code is built into a PCL library using VS 2013 update 3 and running in the Windows Phone emulator, using System.Net.Http 2.2.28 from NuGet

Windows Phone's implementation of the .NET HttpClient is different from Desktop's implementation (wininet vs. custom stack). This is one of the incompatibilities that you need to be aware of, if there is a semantic difference for you.
In my simple test I do get a ReasonPhrase of "Not Found" on the phone emulator. Also you can see that the Headers collection is empty whereas if the server was found and actually returned a real 404 then there would be headers.