I am using xml.net in web application
When I try load xml through an http internet url using:
xmlDoc.Load("http://....")
i get an error:"connected host has failed to respond"
Anyone knows the fix for this?
Thanks
Connected Host has failed to respond is because you've not go the uri correct or you're not allowed to access it, or it's not responding to you, or it's down. http doesn't really care what it transmits.
It probably means exactly what it says: the web server responsible for requests at the URL you specify isn't sending back responses. Something's going wrong on the web server, and if so, you can't do anything about someone's web server out there in the cloud not functioning properly.
You can, however, accept the fact that not every URL will work, and that you'll have to catch the Exception that the XmlDocument or XDocument is throwing. It's reasonable to expect that this scenario may occur. Thus, you need to programming defensively and by including the appropriate exception handling to handle such cases.
EDIT: So you can access it from outside the .NET framework eh? Perhaps try using an HTTP debugger, like Fiddler, and compare the request your XML document object makes to the request your browser makes. What header fields are different? Is there a header that the browser includes that the XML document object doesn't? Or are there different header values between the two, that may be causing the .NET request not to be responded to? Go figure.
If the page is accessible through a web browser but not through the load method it sounds as like the method isn't making a proper HTTP Request to the web server for the page wanted.
You can try using an HTTPWebRequest with a standard GET method to make a proper HTTP request for the webpage. You can then pass the response to the XMLDocument.Load method as a stream and it should then load up fine.
HTTPWebRequest Class MSDN.com
Try making a WebRequest to the url and set its UserAgent property to something like "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2; SV1; .NET CLR 1.1.4322)" . If it works load the text you get in the xmldoc.
I tried loading the xml using .Net HttpWebRequest and also tried setting the userAgent property.
But its still giving me the error message:
"Unable to connect to the remote server"
The xml is however accesible through the browser.
Here is the code:
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(URL);
request.UserAgent ="Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2; SV1; .NET CLR 1.1.4322)";
string result = string.Empty;
using (HttpWebResponse response = request.GetResponse() as HttpWebResponse)
{
// Get the response stream
StreamReader reader = new StreamReader(response.GetResponseStream());
// Read the whole contents and return as a string
result = reader.ReadToEnd();
}
Thanks.
Is there any proxy being used by your browser?
just try telnet to see if you are able to connect to the web server by an application other than the browser.
so if you are using a url like http://www.xmlserver.com/file.xml then try the following in command prompt:
telnet xmlserver.com 80
A big difference between your request and a browser request could be bridged with the following line:
request.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8";
Related
I'm trying to programmatically ping a website (through a console application) and return details of all the cookies being used by that site.
The following approach I'm using only captures those cookies managed through the header request and misses the ones set using Javascript:
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
request.CookieContainer = new CookieContainer();
request.UserAgent = "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0)";
request.Method = "GET";
response = (HttpWebResponse)request.GetResponse();
foreach (Cookie c in response.Cookies)
{
cookie.Add(c);
}
Can someone possibly provide suggestions to how this can be extended to include javascript configured cookies?
Thanks!
Well, not sure how much help this will be but...
You are asking if there is an easy way to get cookies that are created dynamically by client-side javascript and the answer is no, there isn't (unless I'm missing something).
Is there a harder way, maybe, like wrapping the .NET browser control, letting the javascript execute through automated web scripts and then scraping the DOM... Doesn't sound like a good idea to me though.
Any other thoughts welcome.
Just managed to achieve something close to what I wanted (still testing the solution, and it seems to be missing some tracking cookies from 3rd parties!). However what I did was use Selenium, and the Chrome Driver executable. What it does is open up an instance of chrome, navigates to the URL, and pulls back all the dynamically generated information, which can then be interrogated using C#. :)
I need to call a webpage, but have my webclient act like it doesn't support HTML4.0, but only HTML3.2.
Is it possible to do this? Perhaps with a different user-agent or some header I'm unaware of?
Thanks.
This is related to this problem:
SSRS 2008, Force HTML3.2
The WebClient Class implements HTTP. It contains nothing related to HTML.
If the website you're retrieving serves different content depending on the HTTP "User-Agent" header, you can set this header as follows:
WebClient client = new WebClient();
client.Headers.Add("user-agent",
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2; .NET CLR 1.0.3705;)");
Which value you need to specify depends, of course, on the website.
WebClient has no notion of what kind of HTML it is downloading. If the site you're accessing is doing some sort of sniffing, use HttpWebRequest and set the UserAgent property to some really old browser.
You can set the User-Agent header using WebClient as well, but you have to set the header directly as there's no associated property.
I am hitting up a server with the following code and am encountering a ServerProtocolViolation error:
// Prepare the webpage
HttpWebRequest request = (HttpWebRequest) WebRequest.Create(url + queryString);
// execute the request
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
Does anyone know how to work around this kind of error?
This error means that the webserver that you're sending the request to isn't conforming to the HTTP standard.
Other than fixing the server or rewriting HttpWebRequest to be more generous, there isn't much you can do.
What URL are you requesting, and what's the text of the exception?
EDIT: If you request the URL in Fiddler, you'll see that the server didn't return any headers. You should contact the owner of the server and complain.
As a workaround, if you run Fiddler while sending the request, Fiddler will fix the response and allow HttpWebResponse to parse it.
Just a quick note on another reason that I experienced:
If you configure your request to use a proxy server (i.e. through the HttpWebRequest.Proxy property), and you use a wrong proxy port, there might also be a chance to see that error.
In my case, I configured http://127.0.0.1/ as the proxy but had the actual proxy server running on http://127.0.0.1:808/ instead (i.e. port "808" instead of "80").
If this is the case for you, try using no proxy or of course, configure the correct proxy port.
I'm narrowing in on an underlying problem related to two prior questions.
Basically, I've got a URL that when I fetch it manually (paste it into browser) works just fine, but when I run through some code (using the HttpWebRequest) has a different result.
The URL (example):
http://208.106.250.207:8192/announce?info_hash=-%CA8%C1%C9rDb%ADL%ED%B4%2A%15i%80Z%B8%F%C&peer_id=01234567890123456789&port=6881&uploaded=0&downloaded=0&left=0&compact=0&no_peer_id=0&event=started
The code:
String uri = BuildURI(); //Returns the above URL
HttpWebRequest req = (HttpWebRequest)HttpWebRequest.Create(uri);
req.Proxy = new WebProxy();
WebResponse resp = req.GetResponse();
Stream stream = resp.GetResponseStream();
... Parse the result (which is an error message from the server claiming the url is incorrect) ...
So, how can I GET from a server given a URL? I'm obviously doing something wrong here, but can't tell what.
Either a fix for my code, or an alternative approach that actually works would be fine. I'm not wed at all to the HttpWebRequest method.
I recommend you use Fiddler to trace both the "paste in web browser" call and the HttpWebRequest call.
Once traced you will be able to see any differences between them, whether they are differences in the request url, in the form headers, etc, etc.
It may actually be worth pasting the raw requests from both (obtained from Fiddler) here, if you can't see anything obvious.
Well, the only they might differ is in the HTTP headers that get transmitted. In particular the User-Agent.
Also, why are you using a WebProxy? That is not really necessary and it most likely is not used by your browser.
The rest of your code is fine.. Just make sure you set up the HTTP headers correctly. Check this link out:
I would suggest that you get yourself a copy of WireShark and examine the communication that happens between your browser and the server that you are trying to access. Doing so will be rather trivial using WireShark and it will show you the exact HTTP message that is being sent from the browser.
Then take a look at the communication that goes on between your C# application and the server (again using WireShark) and then compare the two to find out what exactly is different.
If the communication is a pure HTTP GET method (i.e. there is no HTTP message body involved), and the URL is correct then the only two things I could think of are:
make sure that your are send the right protocol (i.e. HTTP/1.0 or HTTP/1.1 or whatever it is that you should be sending)
make sure that you are sending all required HTTP headers correctly, and obviously that you are not sending any HTTP headers that you shouldn't be sending.
There could be something wrong with the URL. Instead of using a string, it's usually better to use an instance of System.Uri:
String url = BuildURI(); //Returns the above URL
Uri uri = new Uri(url);
HttpWebRequest req = (HttpWebRequest)HttpWebRequest.Create(url);
req.Proxy = new WebProxy();
using (WebResponse resp = req.GetResponse()) {
using (Stream stream = resp.GetResponseStream()) {
// whatever
}
}
I think you need to see exactly what's flowing to your server in the HTTP request. Does sound likely that the headers are interestingly different.
You can introduce a some kind of debugging proxy between your request and the server (for example RAD has such a capability in the box).
I'm writing a small tool in C# which will need to send and receive data to/from a website using POST and json formatting. I've never done anything like this before in C# (or any language really) so I'm struggling to find some useful information to get me started.
I've found some information on the WebRequest class in C# (specifically from here) but before I start diving into it, I wondered if this was the right tool for the job.
I've found plenty of tools to convert data into the json format but not much else, so any information would be really helpful here in case I end up down a dead end.
WebRequest and more specifically the HttpWebRequest class is a good starting point for what you want to achieve. To create the request you will use the WebRequest.Create and cast the created request to an HttpWebRequest to actually use it. You will then create your post data and send it to the stream like:
HttpWebRequest req = (HttpWebRequest)
WebRequest.Create("http://mysite.com/index.php");
req.Method = "POST";
req.ContentType = "application/x-www-form-urlencoded";
string postData = "var=value1&var2=value2";
req.ContentLength = postData.Length;
StreamWriter stOut = new
StreamWriter(req.GetRequestStream(),
System.Text.Encoding.ASCII);
stOut.Write(postData);
stOut.Close();
Similarly you can read the response back by using the GetResponse method which will allow you to read the resultant response stream and do whatever else you need to do. You can find more info on the class at:
http://msdn.microsoft.com/en-us/library/system.net.httpwebrequest.aspx
WebClient is sometimes easier to use than WebRequest. You may want to take a look at it.
For JSON deserialization you are going to want to look at the JavaScriptSerializer class.
WebClient example:
using (WebClient client = new WebClient ())
{
//manipulate request headers (optional)
client.Headers.Add (HttpRequestHeader.UserAgent, "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2; .NET CLR 1.0.3705;)");
//execute request and read response as string to console
using (StreamReader reader = new StreamReader(client.OpenRead(targetUri)))
{
string s = reader.ReadToEnd ();
Console.WriteLine (s);
}
}
Marked as wiki in case someone wants to update the code
When it comes to POSTing data to a web site, System.Net.HttpWebRequest (the HTTP-specific implementation of WebRequest) is a perfectly decent solution. It supports SSL, async requests and a bunch of other goodies, and is well-documented on MSDN.
The payload can be anything: data in JSON format or whatever -- as long as you set the ContentType property to something the server expects and understands (most likely application/json, text/json or text/x-json), all will be fine.
One potential issue when using HttpWebRequest from a system service: since it uses the IE proxy and credential information, default behavior may be a bit strange when running as the LOCALSYSTEM user (or basically any account that doesn't log on interactively on a regular basis). Setting the Proxy and Authentication properties to Nothing (or, as you C# folks prefer to call it, null, I guess) should avoid that.
I have used WebRequest for interacting with websites. It is the right 'tool'
I can't comment on the JSON aspect of your question.
The currently highest rated answer is helpful, but it doesn't send or receive JSON.
Here is an example that uses JSON for both sending and receiving:
How to post json object in web service
And here is the StackOverflow question that helped me most to solve this problem:
Problems sending and receiving JSON between ASP.net web service and ASP.Net web client
And here is another related question:
json call with C#
To convert from instance object to json formatted string and vice-versa, try out Json.NET:
http://json.codeplex.com/
I am currently using it for a project and it's easy to learn and work with and offers some flexibility in terms of serializing and custom type converters. It also supports a LINQ syntax for querying json input.
in 3.5 there is a built-in jsonserializer. The webrequest is the right class your looking for.
A few examples:
Link
http://dev.aol.com/blog/markdeveloper/ShareFileWithNETFramework
Link