We Keep Coding

About: get REST - returning a NotFound exception C#

Attempting to retrieve quota information from Google Drive via REST in C#. Here is a code snippet:
public async Task<string> HasQuota(string accessToken) {
var url = string.Concat(new string[] {
" https://www.googleapis.com/drive/v2/about",
"?access_token=" + accessToken,
"&fields=quotaBytesTotal%2CquotaBytesUsed"
});
// Create the request.
var request = WebRequest.Create(url) as HttpWebRequest;
request.Method = "GET";
// Create the response.
var response = await Task<WebResponse>.Factory.FromAsync(
request.BeginGetResponse,
request.EndGetResponse,
request
);
if (((HttpWebResponse)response).StatusDescription == "OK") {
using (Stream stream = response.GetResponseStream()) {
StreamReader reader = new StreamReader(stream, Encoding.UTF8);
return reader.ReadToEnd();
}
}
return null;
}
When I obtained the access_token I scoped the authentication request with https://www.googleapis.com/auth/drive.file and my test user granted permissions. So as far as I aware I have the code requirements to execute this call.
However I get an NotFound exception when I attempt to execute this method. The documentation says I can pass the access_token in the query string or add the Authorization header to the request i.e
request.Headers["Authorization"] = string.Format("Bearer {0}", accessToken);
Any ideas why I might be getting this exception?
Thanks

I hadn't enabled the Drive API option in the console...

HttpWebRequest.GetResponse() keeps getting timed out

i wrote a simple C# function to retrieve trade history from MtGox with following API call:
https://data.mtgox.com/api/1/BTCUSD/trades?since=<trade_id>
documented here: https://en.bitcoin.it/wiki/MtGox/API/HTTP/v1#Multi_currency_trades
here's the function:
string GetTradesOnline(Int64 tid)
{
Thread.Sleep(30000);
// communicate
string url = "https://data.mtgox.com/api/1/BTCUSD/trades?since=" + tid.ToString();
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
StreamReader reader = new StreamReader(response.GetResponseStream());
string json = reader.ReadToEnd();
reader.Close();
reader.Dispose();
response.Close();
return json;
}
i'm starting at tid=0 (trade id) to get the data (from the very beginning). for each request, i receive a response containing 1000 trade details. i always send the trade id from the previous response for the next request. it works fine for exactly 4 requests & responses. but after that, the following line throws a "System.Net.WebException", saying that "The operation has timed out":
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
here are the facts:
catching the exception and retying keeps causing the same exception
the default HttpWebRequest .Timeout and .ReadWriteTimeout are already high enough (over a minute)
changing HttpWebRequest.KeepAlive to false didn't solve anything either
it seems to always work in the browser even while the function is failing
it has no problems retrieveing the response from https://www.google.com
the amount of successful responses before the exceptions varies from day to day (but browser always works)
starting at the trade id that failed last time causes the exception immediately
calling this function from the main thread instead still caused the exception
running it on a different machine didn't work
running it from a different IP didn't work
increasing Thread.Sleep inbetween requests does not help
any ideas of what could be wrong?

I had the very same issue.
For me the fix was as simple as wrapping the HttpWebResponse code in using block.
using (HttpWebResponse response = (HttpWebResponse) request.GetResponse())
{
// Do your processings here....
}
Details: This issue usually happens when several requests are made to the same host, and WebResponse is not disposed properly. That is where using block will properly dispose the WebResponse object properly and thus solving the issue.

There are two kind of timeouts. Client timeout and server timeout. Have you tried doing something like this:
request.Timeout = Timeout.Infinite;
request.KeepAlive = true;
Try something like this...

I just had similar troubles calling a REST Service on a LINUX Server thru ssl. After trying many different configuration scenarios I found out that I had to send a UserAgent in the http head.
Here is my final method for calling the REST API.
private static string RunWebRequest(string url, string json)
{
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
// Header
request.ContentType = "application/json";
request.Method = "POST";
request.AllowAutoRedirect = false;
request.KeepAlive = false;
request.Timeout = 30000;
request.ReadWriteTimeout = 30000;
request.UserAgent = "test.net";
request.Accept = "application/json";
request.ProtocolVersion = HttpVersion.Version11;
request.Headers.Add("Accept-Language","de_DE");
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls;
ServicePointManager.ServerCertificateValidationCallback = delegate { return true; };
byte[] bytes = Encoding.UTF8.GetBytes(json);
request.ContentLength = bytes.Length;
using (var writer = request.GetRequestStream())
{
writer.Write(bytes, 0, bytes.Length);
writer.Flush();
writer.Close();
}
var httpResponse = (HttpWebResponse)request.GetResponse();
using (var streamReader = new StreamReader(httpResponse.GetResponseStream()))
{
var jsonReturn = streamReader.ReadToEnd();
return jsonReturn;
}
}

This is not a solution, but just an alternative:
These days i almost only use WebClient instead of HttpWebRequest. Especially WebClient.UploadString for POST and PUT and WebClient.DownloadString. These simply take and return strings. This way i don't have to deal with streams objects, except when i get a WebException. i can also set the content type with WebClient.Headers["Content-type"] if necessary. The using statement also makes life easier by calling Dispose for me.
Rarely for performance, i set System.Net.ServicePointManager.DefaultConnectionLimit high and instead use HttpClient with it's Async methods for simultaneous calls.
This is how i would do it now
string GetTradesOnline(Int64 tid)
{
using (var wc = new WebClient())
{
return wc.DownloadString("https://data.mtgox.com/api/1/BTCUSD/trades?since=" + tid.ToString());
}
}
2 more POST examples
// POST
string SubmitData(string data)
{
string response;
using (var wc = new WebClient())
{
wc.Headers["Content-type"] = "text/plain";
response = wc.UploadString("https://data.mtgox.com/api/1/BTCUSD/trades", "POST", data);
}
return response;
}
// POST: easily url encode multiple parameters
string SubmitForm(string project, string subject, string sender, string message)
{
// url encoded query
NameValueCollection query = HttpUtility.ParseQueryString(string.Empty);
query.Add("project", project);
query.Add("subject", subject);
// url encoded data
NameValueCollection data = HttpUtility.ParseQueryString(string.Empty);
data.Add("sender", sender);
data.Add("message", message);
string response;
using (var wc = new WebClient())
{
wc.Headers[HttpRequestHeader.ContentType] = "application/x-www-form-urlencoded";
response = wc.UploadString( "https://data.mtgox.com/api/1/BTCUSD/trades?"+query.ToString()
, WebRequestMethods.Http.Post
, data.ToString()
);
}
return response;
}
Error handling
try
{
Console.WriteLine(GetTradesOnline(0));
string data = File.ReadAllText(#"C:\mydata.txt");
Console.WriteLine(SubmitData(data));
Console.WriteLine(SubmitForm("The Big Project", "Progress", "John Smith", "almost done"));
}
catch (WebException ex)
{
string msg;
if (ex.Response != null)
{
// read response HTTP body
using (var sr = new StreamReader(ex.Response.GetResponseStream())) msg = sr.ReadToEnd();
}
else
{
msg = ex.Message;
}
Log(msg);
}

For what it's worth, I was experiencing the same issues with timeouts every time I used it, even though calls went through to the server I was calling. The problem in my case was that I had Expect set to application/json, when that wasn't what the server was returning.

Xml response file: Received in browser, not via C#

I'm trying to access the last.fm APIs via C#. As a first test I'm querying similar artists if that matters.
I get an XML response when I pass a correct artist name, i.e. "Nirvana". My problem is when I deliver an invalid name (i.e. "Nirvana23") I don't receive XML but an error code (403 or 400) and a WebException.
Interesting thing: If I enter the URL inside a browser (tested with Firefox and Chrome) I receive the XML file I want (containing a lastfm specific error message).
I tried both XmlReader and XDocument:
XDocument doc = XDocument.Load(requestUrl);
and HttpWebRequest:
string httpResponse = "";
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(requestUrl);
HttpWebResponse response = null;
StreamReader reader = null;
try
{
response = (HttpWebResponse)request.GetResponse();
reader = new StreamReader(response.GetResponseStream());
httpResponse = reader.ReadToEnd();
}
The URL is something like "http://ws.audioscrobbler.com/2.0/?method=artist.getsimilar&artist=Nirvana23" (and a specific key given by lastfm, but even without it - it should return XML). A link to give it a try: link (this is the error file I cannot access via C#).
What I also tried (without success): comparing the request by both the browser and my program with the help of WireShark. Then I added some headers to the request, but that didn't help either.

In .NET the WebRequest is converting HTTP error codes into exceptions, while your browser is just ignoring them since the response is not empty. If you catch the exception then the GetResponseStream method should still return the error XML that you are expecting.
Edit:
Try this:
string httpResponse = "";
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(requestUrl);
WebResponse response = null;
StreamReader reader = null;
try
{
response = request.GetResponse();
}
catch (WebException ex)
{
response = ex.Response;
}
reader = new StreamReader(response.GetResponseStream());
httpResponse = reader.ReadToEnd();

Why don't you catch the exception and then process that accordingly. If you want to display any custom error, you can do that also in your catch block.

Reading remote file [C#]

I am trying to read a remote file using HttpWebRequest in a C# Console Application. But for some reason the request is empty - it never finds the URL.
This is my code:
HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create("http://uo.neverlandsreborn.org:8000/botticus/status.ecl");
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
How come this is not possible?
The file only contains a string. Nothing more!

How are you reading the response data? Does it come back as successful but empty, or is there an error status?
If that doesn't help, try Wireshark, which will let you see what's happening at the network level.
Also, consider using WebClient instead of WebRequest - it does make it incredibly easy when you don't need to do anything sophisticated:
string url = "http://uo.neverlandsreborn.org:8000/botticus/status.ecl";
WebClient wc = new WebClient();
string data = wc.DownloadString(url);

You have to get the response stream and read the data out of that. Here's a function I wrote for one project that does just that:
private static string GetUrl(string url)
{
HttpWebRequest request = (HttpWebRequest)WebRequest.CreateDefault(new Uri(url));
using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
{
if (response.StatusCode != HttpStatusCode.OK)
throw new ServerException("Server returned an error code (" + ((int)response.StatusCode).ToString() +
") while trying to retrieve a new key: " + response.StatusDescription);
using (var sr = new StreamReader(response.GetResponseStream()))
{
return sr.ReadToEnd();
}
}
}

Getting connection closed error when trying to read response from FitNesse REST URI

Have you run into this problem? I run code remarkably similar to that from a this previous question, When in nUnitTest mode and the URI includes "/?test&format=xml" the nUnit test fails with and IOException, "Unable to read data from the transport connection: The connection is closed."
However the Fiddler trace that was running at the time shows the very xml I expected.
I've recreated the request headers exactly (almost) as they are sent when sent through the browser.
Finally, if I leave off the "/?test&format=xml" from the URI, I get the html I would have otherwise expected.
SOURCE CODE:
public virtual bool Run()
{
var request = CreateRequest();
var response = GetResponse(request);
var responseString = ReadResponse(response);
this.SetResults(responseString);
return this.IsSuccessful;
}
protected internal virtual HttpWebRequest CreateRequest()
{
var address = TestConfig.Address;
var request = (HttpWebRequest)WebRequest.Create(address);
request.Accept = "*/*";
request.UseDefaultCredentials = true;
request.CachePolicy = new RequestCachePolicy(RequestCacheLevel.NoCacheNoStore);
return request;
}
protected internal virtual HttpWebResponse GetResponse(HttpWebRequest request)
{
var response = (HttpWebResponse) request.GetResponse();
return response;
}
protected internal virtual string ReadResponse(HttpWebResponse response)
{
var stream = response.GetResponseStream();
var responseString = ReadResponse(stream);
stream.Close();
response.Close();
return responseString;
}
protected internal virtual string ReadResponse(Stream stream)
{
var reader = new StreamReader(stream);
var responseString = reader.ReadToEnd();
return responseString;
}

The error message "Unable to read data from the transport connection: The connection is closed." doesn't really tie up with the fact you're seeing Fiddler getting a html response body back.
Check the StatusCode of the HttpWebResponse (should be 200 if ok), also wrap the request in try/catch block (example from http://msdn.microsoft.com/en-us/library/system.net.httpwebresponse.statuscode(v=vs.80).aspx)
try
{
// Creates an HttpWebRequest for the specified URL.
HttpWebRequest myHttpWebRequest = (HttpWebRequest)WebRequest.Create(url);
// Sends the HttpWebRequest and waits for a response.
HttpWebResponse myHttpWebResponse = (HttpWebResponse)myHttpWebRequest.GetResponse();
if (myHttpWebResponse.StatusCode == HttpStatusCode.OK)
Console.WriteLine("\r\nResponse Status Code is OK and StatusDescription is: {0}",
myHttpWebResponse.StatusDescription);
// Releases the resources of the response.
myHttpWebResponse.Close();
}
catch(WebException e)
{
Console.WriteLine("\r\nWebException Raised. The following error occured : {0}",e.Status);
}
catch(Exception e)
{
Console.WriteLine("\nThe following Exception was raised : {0}",e.Message);
}
If you're creating and disposing of the HttpWebRequest object quickly you might be getting the socket going into a time_wait state as it is shutting down, then you can't re-open it again until it has completely closed. If this is the case, look into using another port or changing the time the connection lives for.