We have a program that has been running for years making API calls to a web server using HttpWebRequest and yesterday it started giving an error (something like "connection forcibly closed by remote host"). The request works just fine when made through a web browser so I would love to be able to see the difference in requests. With the Firefox developer console, I can see the raw request that is made through the browser (that works) and I need to compare that to the http request that is made from our program. It seems like it should be simple (and very useful) to stream the request out to a string or a file so I can look at it (but I have not had any luck finding how to do that).
Can you tell me how to modify the below code to store the request that HttpWebRequest would send to a file or a string (instead of a network stream)?
public string Post(string uri, string data, string contentType, string method = "POST")
{
byte[] dataBytes = Encoding.UTF8.GetBytes(data);
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(uri);
request.AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate;
request.ContentLength = dataBytes.Length;
request.ContentType = contentType;
request.Method = method;
using(Stream requestBody = request.GetRequestStream())
{
requestBody.Write(dataBytes, 0, dataBytes.Length);
}
using(HttpWebResponse response = (HttpWebResponse)request.GetResponse())
using(Stream stream = response.GetResponseStream())
using(StreamReader reader = new StreamReader(stream))
{
return reader.ReadToEnd();
}
}
You can't simply log the httprequest. Because you should also consider logging all the headers.
I suggest you to use some http sniffer to log the traffic(if you can't debug or modify your code)
In addition you can catch the exceptions by using WebException and get the raw error message from the server. Maybe it'll give you idea what the problem is.
catch (WebException ex)
{
using (var stream = ex.Response.GetResponseStream())
using (var reader = new StreamReader(stream))
{
var responseString = reader.ReadToEnd();
}
}
Related
I am current using the following to try and create an upload session with the onedrive api
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
request.Method = "POST";
request.Headers["Authorization"] = "Bearer " + nODUserTokenObj.Access_Token;
request.ContentType = "application/json";
string data =
#"{
""#microsoft.graph.conflictBehavior"": ""replace""
}";
byte[] postBytes = Encoding.ASCII.GetBytes(data);
request.ContentLength = postBytes.Length;
request.Accept = "application/json";
request.GetRequestStream().Write(postBytes, 0, postBytes.Length);
using (HttpWebResponse httpWebResponse = (HttpWebResponse)request.GetResponse())
{
using(StreamReader sr = new StreamReader(httpWebResponse.GetResponseStream()))
{
string theData = sr.ReadToEnd();
JObject test = JObject.Parse(theData);
}
}
Currently I am receiving an exception stating
The remote server returned an error: (409) Conflict.
However looking at the Microsoft documentation I should have a json object returned to me with more information.
What am I doing wrong that I am getting an exception thrown when it hits (HttpWebResponse)request.GetResponse() as opposed to receiving a json object with more error details?
I came across two solutions:
I initially resolved this by sending the request through Fiddler and reading the response there. This was fine for my situation as I was developing and just trying to figure out how the onedrive api functioned.
A solution going forward would be to follow something like this post suggests (How to get error information when HttpWebRequest.GetResponse() fails)
To summarize the post:
catch (WebException ex)
{
using (var stream = ex.Response.GetResponseStream())
using (var reader = new StreamReader(stream))
{
Console.WriteLine(reader.ReadToEnd());
}
}
catch (Exception ex)
{
// Something more serious happened
// like for example you don't have network access
// we cannot talk about a server exception here as
// the server probably was never reached
}
I've been working on the Walmart API but I keep getting either the 401 error or the 500 error when I run the code
public void post()
{
byte[] data = Encoding.ASCII.GetBytes(
$"username={user}&password={password}");
HttpWebRequest request = (HttpWebRequest)WebRequest.Create("https://marketplace.walmartapis.com/v2/feeds?feedType=item");
request.Method = "POST";
request.Accept = "application/xml;";
request.ContentLength = data.Length;
request.Headers.Add("WM_SVC.NAME", "Walmart Marketplace");
request.Headers.Add(authId);
request.Headers.Add("WM_CONSUMER.ID", user);
request.Headers.Add( time);
request.Headers.Add(CorId);
using (Stream stream = request.GetRequestStream ())
{
stream.Write(data , 0, data.Length);
}
string responseContent = null;
using (WebResponse response = request.GetResponse())
{
using (Stream stream = response.GetResponseStream())
{
using (StreamReader sr99 = new StreamReader(stream))
{
responseContent = sr99.ReadToEnd();
}
}
}
MessageBox.Show(responseContent);
}
where authID is a signature generated from a jar file provided by walmart
time is also generated from the jar file
CorID is a randomly generated number
and user is the user id.
here is the link that describes the header parameters. Did I miss something in my header?
https://developer.walmartapis.com/#getting-started
There are multiple problems with your request. First, you are submitting a feed, but sending it as an application/xml when it should be a multipart/form-data request. Beyond this, your headers aren't set up properly and there is currently a major problem with submitting multipart/form-data requests to Walmart using C#. I have not seen a post from anyone successfully sending a feed to Walmart via C#. I am currently using C# to execute a batch file that then fires a modified version of the Walmart Java SDK which is capable of sending the multipart/form-data requests.
The reponse below is for any request other than feeds. I would start with the example listed below to get familiar with how you need to set your headers up. This is going to work for the majority of Walmart interfacing, but if the request is a feed style request, you will either need to come up with a better solution to the multipart/form-data issue, use the Java SDK, or wait for the C# SDK. If someone reads this and has a better answer as to how to submit feeds via C# exclusively I would love to hear about it!
Here is an example of an application/xml request that works.
string timestamp = CurrentTimeMillis().ToString().Trim();
string query = #"orders/"+poID+"/acknowledge";
string request = v3BaseUrl + query; //Constructed URI
string stringToSign = consumerId + "\n" +
request.Trim() + "\n" +
"POST" + "\n" +
timestamp + "\n";
string signedString = signData(stringToSign); //Your signed string
HttpWebRequest webRequest = (HttpWebRequest)WebRequest.Create(request);
webRequest.Accept = "application/xml";
webRequest.ContentType = "application/xml";
webRequest.Method = "POST";
webRequest.Headers.Add("WM_SVC.NAME", "Walmart Marketplace");
webRequest.Headers.Add("WM_SEC.AUTH_SIGNATURE", signedString);
webRequest.Headers.Add("WM_CONSUMER.ID", consumerId);
webRequest.Headers.Add("WM_SEC.TIMESTAMP", timestamp.ToString().Trim());
webRequest.Headers.Add("WM_QOS.CORRELATION_ID", Guid.NewGuid().ToString());
webRequest.Headers.Add("WM_CONSUMER.CHANNEL.TYPE", channelType);
webRequest.ContentLength = 0;
webRequest.Timeout = Timeout.Infinite;
webRequest.KeepAlive = true;
using (HttpWebResponse response = (HttpWebResponse)webRequest.GetResponse())
{
if (response.StatusCode == HttpStatusCode.OK)
{
success = true;
}
}
i wrote a simple C# function to retrieve trade history from MtGox with following API call:
https://data.mtgox.com/api/1/BTCUSD/trades?since=<trade_id>
documented here: https://en.bitcoin.it/wiki/MtGox/API/HTTP/v1#Multi_currency_trades
here's the function:
string GetTradesOnline(Int64 tid)
{
Thread.Sleep(30000);
// communicate
string url = "https://data.mtgox.com/api/1/BTCUSD/trades?since=" + tid.ToString();
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
StreamReader reader = new StreamReader(response.GetResponseStream());
string json = reader.ReadToEnd();
reader.Close();
reader.Dispose();
response.Close();
return json;
}
i'm starting at tid=0 (trade id) to get the data (from the very beginning). for each request, i receive a response containing 1000 trade details. i always send the trade id from the previous response for the next request. it works fine for exactly 4 requests & responses. but after that, the following line throws a "System.Net.WebException", saying that "The operation has timed out":
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
here are the facts:
catching the exception and retying keeps causing the same exception
the default HttpWebRequest .Timeout and .ReadWriteTimeout are already high enough (over a minute)
changing HttpWebRequest.KeepAlive to false didn't solve anything either
it seems to always work in the browser even while the function is failing
it has no problems retrieveing the response from https://www.google.com
the amount of successful responses before the exceptions varies from day to day (but browser always works)
starting at the trade id that failed last time causes the exception immediately
calling this function from the main thread instead still caused the exception
running it on a different machine didn't work
running it from a different IP didn't work
increasing Thread.Sleep inbetween requests does not help
any ideas of what could be wrong?
I had the very same issue.
For me the fix was as simple as wrapping the HttpWebResponse code in using block.
using (HttpWebResponse response = (HttpWebResponse) request.GetResponse())
{
// Do your processings here....
}
Details: This issue usually happens when several requests are made to the same host, and WebResponse is not disposed properly. That is where using block will properly dispose the WebResponse object properly and thus solving the issue.
There are two kind of timeouts. Client timeout and server timeout. Have you tried doing something like this:
request.Timeout = Timeout.Infinite;
request.KeepAlive = true;
Try something like this...
I just had similar troubles calling a REST Service on a LINUX Server thru ssl. After trying many different configuration scenarios I found out that I had to send a UserAgent in the http head.
Here is my final method for calling the REST API.
private static string RunWebRequest(string url, string json)
{
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
// Header
request.ContentType = "application/json";
request.Method = "POST";
request.AllowAutoRedirect = false;
request.KeepAlive = false;
request.Timeout = 30000;
request.ReadWriteTimeout = 30000;
request.UserAgent = "test.net";
request.Accept = "application/json";
request.ProtocolVersion = HttpVersion.Version11;
request.Headers.Add("Accept-Language","de_DE");
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls;
ServicePointManager.ServerCertificateValidationCallback = delegate { return true; };
byte[] bytes = Encoding.UTF8.GetBytes(json);
request.ContentLength = bytes.Length;
using (var writer = request.GetRequestStream())
{
writer.Write(bytes, 0, bytes.Length);
writer.Flush();
writer.Close();
}
var httpResponse = (HttpWebResponse)request.GetResponse();
using (var streamReader = new StreamReader(httpResponse.GetResponseStream()))
{
var jsonReturn = streamReader.ReadToEnd();
return jsonReturn;
}
}
This is not a solution, but just an alternative:
These days i almost only use WebClient instead of HttpWebRequest. Especially WebClient.UploadString for POST and PUT and WebClient.DownloadString. These simply take and return strings. This way i don't have to deal with streams objects, except when i get a WebException. i can also set the content type with WebClient.Headers["Content-type"] if necessary. The using statement also makes life easier by calling Dispose for me.
Rarely for performance, i set System.Net.ServicePointManager.DefaultConnectionLimit high and instead use HttpClient with it's Async methods for simultaneous calls.
This is how i would do it now
string GetTradesOnline(Int64 tid)
{
using (var wc = new WebClient())
{
return wc.DownloadString("https://data.mtgox.com/api/1/BTCUSD/trades?since=" + tid.ToString());
}
}
2 more POST examples
// POST
string SubmitData(string data)
{
string response;
using (var wc = new WebClient())
{
wc.Headers["Content-type"] = "text/plain";
response = wc.UploadString("https://data.mtgox.com/api/1/BTCUSD/trades", "POST", data);
}
return response;
}
// POST: easily url encode multiple parameters
string SubmitForm(string project, string subject, string sender, string message)
{
// url encoded query
NameValueCollection query = HttpUtility.ParseQueryString(string.Empty);
query.Add("project", project);
query.Add("subject", subject);
// url encoded data
NameValueCollection data = HttpUtility.ParseQueryString(string.Empty);
data.Add("sender", sender);
data.Add("message", message);
string response;
using (var wc = new WebClient())
{
wc.Headers[HttpRequestHeader.ContentType] = "application/x-www-form-urlencoded";
response = wc.UploadString( "https://data.mtgox.com/api/1/BTCUSD/trades?"+query.ToString()
, WebRequestMethods.Http.Post
, data.ToString()
);
}
return response;
}
Error handling
try
{
Console.WriteLine(GetTradesOnline(0));
string data = File.ReadAllText(#"C:\mydata.txt");
Console.WriteLine(SubmitData(data));
Console.WriteLine(SubmitForm("The Big Project", "Progress", "John Smith", "almost done"));
}
catch (WebException ex)
{
string msg;
if (ex.Response != null)
{
// read response HTTP body
using (var sr = new StreamReader(ex.Response.GetResponseStream())) msg = sr.ReadToEnd();
}
else
{
msg = ex.Message;
}
Log(msg);
}
For what it's worth, I was experiencing the same issues with timeouts every time I used it, even though calls went through to the server I was calling. The problem in my case was that I had Expect set to application/json, when that wasn't what the server was returning.
This is driving me a bit nuts. I am trying to do something quite simple, and I have done it many times before. Just trying to call a REST API.
I am trying to call GetMessage with endpoint = "http://feed.linksynergy.com/productsearch?token=717f8c8511725ea26fd5c3651f32ab187d8db9f4b208be781c292585400e682d&keyword=DVD", and it keeps returning empty string. If I pass it any other valid URL, it will work. But if I just copy and paste the original URL into the web browser, it returns fine!
Can any smart developer tell me what's going on?
Code below. Thanks in advance.
James
public string GetMessage(string endPoint)
{
HttpWebRequest request = CreateWebRequest(endPoint);
using (var response = (HttpWebResponse)request.GetResponse())
{
var responseValue = string.Empty;
if (response.StatusCode != HttpStatusCode.OK)
{
string message = String.Format("POST failed. Received HTTP {0}", response.StatusCode);
throw new ApplicationException(message);
}
// grab the response
using (var responseStream = response.GetResponseStream())
{
using (var reader = new StreamReader(responseStream))
{
responseValue = reader.ReadToEnd();
}
}
return responseValue;
}
}
private HttpWebRequest CreateWebRequest(string endPoint)
{
var request = (HttpWebRequest)WebRequest.Create(endPoint);
request.Method = "GET";
request.ContentLength = 0;
request.ContentType = "text/xml";
return request;
}
Not sure why your setting ContentLength/ContentType - that is generally for HTTP POST, where there is a request body for which you write data to via a stream.
This is a HTTP GET, so there is no request body. (just URI w/ query string)
This should work:
using System;
using System.IO;
using System.Net;
using System.Text;
// Create the web request
HttpWebRequest request = WebRequest.Create("http://www.someapi.com/") as HttpWebRequest;
// Get response
using (HttpWebResponse response = request.GetResponse() as HttpWebResponse)
{
// Get the response stream
StreamReader reader = new StreamReader(response.GetResponseStream());
// Console application output
Console.WriteLine(reader.ReadToEnd());
}
EDIT
#Gabe is also quite right - try this on another computer, that is isn't behind any kind of firewall/proxy server.
My work PC was behind a proxy server, so in order to make REST-based HTTP calls, i needed to do this:
var proxyObject = new System.Net.WebProxy("http://myDomain:8080/", true);
System.Net.WebRequest req = System.Net.WebRequest.Create("http://www.someapi.com/");
req.Proxy = proxyObject;
proxyObject.Credentials = New System.Net.NetworkCredential("domain\username","password")
This may be a pathetically simple problem, but I cannot seem to format the post webrequest/response to get data from the Wikipedia API. I have posted my code below if anyone can help me see my problem.
string pgTitle = txtPageTitle.Text;
Uri address = new Uri("http://en.wikipedia.org/w/api.php");
HttpWebRequest request = WebRequest.Create(address) as HttpWebRequest;
request.Method = "POST";
request.ContentType = "application/x-www-form-urlencoded";
string action = "query";
string query = pgTitle;
StringBuilder data = new StringBuilder();
data.Append("action=" + HttpUtility.UrlEncode(action));
data.Append("&query=" + HttpUtility.UrlEncode(query));
byte[] byteData = UTF8Encoding.UTF8.GetBytes(data.ToString());
request.ContentLength = byteData.Length;
using (Stream postStream = request.GetRequestStream())
{
postStream.Write(byteData, 0, byteData.Length);
}
using (HttpWebResponse response = request.GetResponse() as HttpWebResponse)
{
// Get the response stream.
StreamReader reader = new StreamReader(response.GetResponseStream());
divWikiData.InnerText = reader.ReadToEnd();
}
You might want to try a GET request first because it's a little simpler (you will only need to POST for wikipedia login). For example, try to simulate this request:
http://en.wikipedia.org/w/api.php?action=query&prop=images&titles=Main%20Page
Here's the code:
HttpWebRequest myRequest =
(HttpWebRequest)WebRequest.Create("http://en.wikipedia.org/w/api.php?action=query&prop=images&titles=Main%20Page");
using (HttpWebResponse response = (HttpWebResponse)myRequest.GetResponse())
{
string ResponseText;
using (StreamReader reader = new StreamReader(response.GetResponseStream()))
{
ResponseText = reader.ReadToEnd();
}
}
Edit: The other problem he was experiencing on the POST request was, The exception is : The remote server returned an error: (417) Expectation failed. It can be solved by setting:
System.Net.ServicePointManager.Expect100Continue = false;
(This is from: HTTP POST Returns Error: 417 "Expectation Failed.")
I'm currently in the final stages of implementing an C# MediaWiki API which allows the easy scripting of most MediaWiki viewing and editing actions.
The main API is here: http://o2platform.googlecode.com/svn/trunk/O2%20-%20All%20Active%20Projects/O2_XRules_Database/_Rules/APIs/OwaspAPI.cs and here is an example of the API in use:
var wiki = new O2MediaWikiAPI("http://www.o2platform.com/api.php");
wiki.login(userName, password);
var page = "Test"; // "Main_Page";
wiki.editPage(page,"Test content2");
var rawWikiText = wiki.raw(page);
var htmlText = wiki.html(page);
return rawWikiText.line().line() + htmlText;
You seem to be pushing the input data on HTTP POST, but it seems you should use HTTP GET.
From the MediaWiki API docs:
The API takes its input through
parameters in the query string. Every
module (and every action=query
submodule) has its own set of
parameters, which is listed in the
documentation and in action=help, and
can be retrieved through
action=paraminfo.
http://www.mediawiki.org/wiki/API:Data_formats