I'm having trouble getting Proxy Automatic Configuration (PAC) in IE options to work as expected using .Net WebRequest.
According to this article:
Proxy Detection
Take the Burden Off Users with Automatic Configuration in .NET
The system proxy should be set by default with to each WebRequest.
That's how the proxy.js pac file looks like:
function FindProxyForURL(url, host)
{
return "PROXY ProxyServerName:3118; DIRECT;";
}
I also took a look at this post: How should I set the default proxy to use default credentials?
Which suggests to add this in the app.config:
<system.net>
<defaultProxy useDefaultCredentials="true" />
</system.net>
Adding this did not help.
I created a small console application just to test this out.. here it is:
static void Main(string[] args)
{
HttpWebRequest request = null;
try
{
String resolvedAddress = WebRequest.DefaultWebProxy.GetProxy(new Uri("http://www.google.com")).ToString();
Console.WriteLine("Proxy for address is: " + resolvedAddress);
Uri m_URLToTest = new Uri("http://www.google.com");
request = WebRequest.Create(m_URLToTest) as HttpWebRequest;
request.Method = "GET";
request.KeepAlive = false;
request.Timeout = 5000;
request.Proxy = WebRequest.DefaultWebProxy;
WebResponse response = request.GetResponse();
StreamReader reader = new StreamReader(response.GetResponseStream());
string message = reader.ReadToEnd();
}
catch (Exception ex)
{
Console.Write("Exception");
}
}
The output:
Proxy for address is http://www.google.com
instead of Proxy for address is ProxyServerName:3118
It happens only when using auto configuration script...
Did I miss anything? Please help!
Found the solution!
It is really important that the mime type of the PAC file would be: [Content-type: application/x-ns-proxy-autoconfig]
Other mime types might not work.
Make sure using fiddler2 (with cache disabled) that the mime type is appropriate.
Some configurations might show Content-Type: text/plain which is bad.
Make sure you have checked Internet (Client & Server) and Private Networks (Client & Server) capabilities in Package.appxmanifest.
[Source]
Related
I am working on Windows Service in visual studio 2017. In the rest api's call, getting exceptions while debugging code. Sometimes first 2 3 calls working after that getting exceptions.
System.Net.WebException: 'The remote server returned an error: (503)
Server Unavailable.'
The remote server returned an error: (429)
Unable to connect to the remote server
When calling same api's from Postman, getting response successfully.
This is my code
private void timer1_Tick(object sender, ElapsedEventArgs e)
{
WriteToFile("timer1_Tick method called..");
try
{
string jsonString = "";
string jsonstring2 = "";
string prodfetchurl = HOST;
var req = WebRequest.Create(prodfetchurl) as HttpWebRequest;
req.Method = "GET";
InitializeRequest(req);
req.Accept = MIME_TYPE;
//System.Threading.Thread.Sleep(5000);
var response = (HttpWebResponse)req.GetResponse();
WriteToFile("First service called...");
if (response.StatusCode == HttpStatusCode.OK)
{
Stream responseStream = response.GetResponseStream();
StreamReader responseReader = new StreamReader(responseStream);
jsonString = responseReader.ReadToEnd();
}
var deserialsseobj = JsonConvert.DeserializeObject<ProductList>(jsonString).Products.Where(i => i.Failed > 0).ToList();
foreach (var a in deserialsseobj)
{
var pid = a.ID;
string url = FailedDevicesUrl + pid.Value + "/failed";
var req2 = WebRequest.Create(url) as HttpWebRequest;
req2.Method = "GET";
InitializeRequest(req2);
req2.Timeout = 300000;
req2.Accept = MIME_TYPE;
var response1 = (HttpWebResponse)req2.GetResponse();
Stream responsestream2 = response1.GetResponseStream();
WriteToFile("Second service called...");
if (response1.StatusCode == HttpStatusCode.OK)
{
StreamReader responsereader1 = new StreamReader(responsestream2);
jsonstring2 = responsereader1.ReadToEnd();
}
var output = JsonConvert.DeserializeObject<List<FailedDeviceList>>(jsonstring2); // Will get List of the Failed devices
List<int> deviceids = new List<int>();
Reprocessdata reproc = new Reprocessdata();
Reprocessdata.DeviceId rprod = new Reprocessdata.DeviceId();
reproc.ForceFlag = true;
reproc.ProductID = pid.Value;
foreach (var dd in output)
{
rprod.ID = dd.DeviceId;
reproc.DeviceIds.Add(rprod);
}
// Reprocess the Product in Devices
var req3 = WebRequest.Create(ReprocessUrl) as HttpWebRequest;
req3.Method = "POST";
InitializeRequest(req3);
req3.Accept = MIME_TYPE;
req3.Timeout = 300000;
req3.ContentType = "application/json";
using (StreamWriter writer = new StreamWriter(req3.GetRequestStream()))
{
string json = new JavaScriptSerializer().Serialize(reproc);
writer.Write(json);
writer.Close();
}
System.Threading.Thread.Sleep(5000);
var response5 = (HttpWebResponse)req3.GetResponse();
WriteToFile("Third service called...");
if (response5.StatusCode == HttpStatusCode.OK)
{
string result;
using (StreamReader rdr = new StreamReader(response5.GetResponseStream()))
{
result = rdr.ReadToEnd();
}
}
}
response.Close();
}
catch (Exception ex)
{
WriteToFile("Simple Service Error on: {0} " + ex.Message + ex.StackTrace);
}
}
Methods used in above code
protected override void OnStart(string[] args)
{
base.OnStart(args);
timer1 = new System.Timers.Timer();
timer1.Interval = 60000; //every 1 min
timer1.Elapsed += new System.Timers.ElapsedEventHandler(timer1_Tick);
timer1.Enabled = true;
WriteToFile("Service has started..");
}
public void InitializeRequest(HttpWebRequest request)
{
request.Headers.Add("aw-tenant-code", API_TENANT_CODE);
request.Credentials = new NetworkCredential(USER_NAME, PASSWORD);
request.KeepAlive = false;
request.AddRange(1024);
}
When I contacted service provide they said everything fine from there side. Is this my code is buggy or windows service not reliable? How can I fix this issue?
Note: All APIS are working fine from Angular application using Visual Studio Code. It means my code is not working.
Edit1: Three below services I am using from this document of VMware.
private const string HOST = "https:host/api/mdm/products/search?";
private const string FailedDevicesUrl = "https:host/api/mdm/products/";
private const string ReprocessUrl = "https:host/api/mdm/products/reprocessProduct";
Response http code 429 indicates that you sending too many requests on target web service.
This means service you trying to send requests has a policies that blocks some requests by request-per-time limit.
Also I admit that external service can be manually configured to throw 403 code in specific cases that you can't know about. If that, this information can be explained in external service documentation... or not :)
What you can do with this?
Fit in limitations
You can make detailed research what limits target webservice has and set up your code to fit in this limitations. For example if service has limitation for receiving only one request per 10 minutes - you must set up your timer to send one request each 10 or more minutes. If documentation not provide such information - you can test it manually by finding some patterns with external service responses.
Use proxy
Every limitation policy based on information about requests senders. Usually this information consists of IP address of sender only. This means if you send 2 requests from two different IP addresses - limitation policy will perceive that like 2 different computers sending these requests. So you can find/buy/rent some proxy IP addresses and send requests through there on target web server.
How to connect through proxy in C# using WebRequest you can see in this answer.
Negotiate with external service provider
If you have possibility to communicate with external service developers or help center, you can ask their to reduce limitations for your IP address (if it static) or provide some mechanisms to avoid limitation policy for you. If for some reason they cannot provide this opportunity, at least you can ask detailed information about limitations.
Repetition mechanism
Some times 503 error code that is outer exception you received may be caused by service unavailable. It means that server can be under maintenance or temporary overloaded. So you can write repetition mechanism to make continious sending requests to server until it'll be accessible.
Polly library may help you with repetition mechanism creation
The inner error of that 503 is:
The remote server returned an error: (429)
HTTP 429 indicates too many requests. Maybe your upstream server can’t process all requests sent.
This can happen when you reached rate limiting / throttling value if you’re calling a third party API.
UPDATE
As per page 28 in the API docs, you could configure throttling when creating a new API. Check if the throttling is too small or maybe turn off the throttling and see if that could fix the error?
I am using a web client class in my source code for downloading a string using http.
This was working fine. However, the clients in the company are all connected now to a proxy server. And the problem started from this.
When I have tested my application I don't think it can pass through the proxy server, as the exception that keeps getting thrown is "no response from xxx.xxx.xxx.xxx which is the proxy server IP address.
However, I can still navigate to the web site URL and it displays the string correctly in the browser when connecting through a proxy server, but not when I use my web client.
Is there something in the web client that I have to configure to allow me to access the url from behind a proxy server?
using (WebClient wc = new WebClient())
{
string strURL = "http://xxxxxxxxxxxxxxxxxxxxxxxx";
//Download only when the webclient is not busy.
if (!wc.IsBusy)
{
string rtn_msg = string.Empty;
try
{
rtn_msg = wc.DownloadString(new Uri(strURL));
return rtn_msg;
}
catch (WebException ex)
{
Console.Write(ex.Message);
return false;
}
catch (Exception ex)
{
Console.Write(ex.Message);
return false;
}
}
else
{
System.Windows.Forms.MessageBox.Show("Busy please try again");
return false;
}
}
My solution:
WebClient client = new WebClient();
WebProxy wp = new WebProxy(" proxy server url here");
client.Proxy = wp;
string str = client.DownloadString("http://www.google.com");
If you need to authenticate to the proxy, you need to set UseDefaultCredentials to false, and set the proxy Credentials.
WebProxy proxy = new WebProxy();
proxy.Address = new Uri("mywebproxyserver.com");
proxy.Credentials = new NetworkCredential("usernameHere", "pa****rdHere"); //These can be replaced by user input
proxy.UseDefaultCredentials = false;
proxy.BypassProxyOnLocal = false; //still use the proxy for local addresses
WebClient client = new WebClient();
client.Proxy = proxy;
string doc = client.DownloadString("http://www.google.com/");
If all you need is a simple proxy, you skip most of the lines above though. All you need is:
WebProxy proxy = new WebProxy("mywebproxyserver.com");
The answer proposed by Jonathan is proper, but requires that you specify the proxy credentials and url in the code. Usually, it is better to allow usage of the credentials as setup in the system by default (Users typically configure LAN Settings anyway in case they use a proxy)...
The below answer has been provided by Davide in earlier answer, but that requires modifying the app.config files. This solution is probably more useful since it does the same thing IN CODE.
In order to let the application use the default proxy settings as used in the user's system, one can use the following code:
IWebProxy wp = WebRequest.DefaultWebProxy;
wp.Credentials = CredentialCache.DefaultCredentials;
wc.Proxy = wp;
This will allow the application code to use the proxy (with logged-in credentials and default proxy url settings)... No headaches! :)
Hope this helps future viewers of this page to solve their problem!
I've encountered the same issue but using a webclient for downloading a file from the internet with a Winform application the solution was adding in the app.config:
<system.net>
<defaultProxy useDefaultCredentials="true" />
</system.net>
The same solution will work for an asp.net app inserting the same rows in web.config.
Hope it will help.
You need to configure the proxy in the WebClient object.
See the WebClient.Proxy property:
http://msdn.microsoft.com/en-us/library/system.net.webclient.proxy(VS.80).aspx
byte[] data;
using (WebClient client = new WebClient())
{
ICredentials cred;
cred = new NetworkCredential("xmen#test.com", "mybestpassword");
client.Proxy = new WebProxy("192.168.0.1",8000);
client.Credentials = cred;
string myurl="http://mytestsite.com/source.jpg";
data = client.DownloadData(myUrl);
}
File.WriteAllBytes(#"c:\images\target.jpg", data);
All previous answers have some merit, but the actual answer only needs ONE line:
wc.Proxy = new WebProxy("127.0.0.1", 8888);
where wc is the WebClient object, and 8888 is the port number of the proxy server located on the same machine.
I created RESTful webservice (WCF) where I check credentials on each request. One of my clients is Android app and everything seems to be great on server side. I get request and if it's got proper header - I process it, etc..
Now I created client app that uses this service. This is how I do GET:
// Create the web request
var request = WebRequest.Create(Context.ServiceURL + uri) as HttpWebRequest;
if (request != null)
{
request.ContentType = "application/json";
// Add authentication to request
request.Credentials = new NetworkCredential(Context.UserName, Context.Password);
// Get response
using (var response = request.GetResponse() as HttpWebResponse)
{
// Get the response stream
if (response != null)
{
var reader = new StreamReader(response.GetResponseStream());
// Console application output
var s = reader.ReadToEnd();
var serializer = new JavaScriptSerializer();
var returnValue = (T)serializer.Deserialize(s, typeof(T));
return returnValue;
}
}
}
So, this code get's my resource and deserializes it. As you see - I'm passing credentials in my call.
Then when debugging on server-side I noticed that I get 2 requests every time - one without authentication header and then server sends back response and second request comes bach with credentials. I think it's bad for my server - I'd rather don't make any roundtrips. How should I change client so it doesn't happen? See screenshot of Fiddler
EDIT:
This is JAVA code I use from Android - it doesn't do double-call:
MyHttpResponse response = new MyHttpResponse();
HttpClient client = mMyApplication.getHttpClient();
try
{
HttpGet request = new HttpGet(serviceURL + url);
request.setHeader(new BasicHeader(HTTP.CONTENT_TYPE, "application/json"));
request.addHeader("Authorization", "Basic " + Preferences.getAuthorizationTicket(mContext));
ResponseHandler<String> handler = new BasicResponseHandler();
response.Body = client.execute(request, handler);
response.Code = HttpURLConnection.HTTP_OK;
response.Message = "OK";
}
catch (HttpResponseException e)
{
response.Code = e.getStatusCode();
response.Message = e.getMessage();
LogData.InsertError(mContext, e);
}
The initial request doesn't ever specify the basic header for authentication. Additionally, since a realm is specified, you have to get that from the server. So you have to ask once: "hey, I need this stuff" and the server goes "who are you? the realm of answering is 'secure area'." (because realm means something here) Just because you added it here:
request.Credentials = new NetworkCredential(Context.UserName, Context.Password);
doesn't mean that it's going to be for sure attached everytime to the request.
Then you respond with the username/password (in this case you're doing BASIC so it's base64 encoded as name:password) and the server decodes it and says "ok, you're all clear, here's your data".
This is going to happen on a regular basis, and there's not a lot you can do about it. I would suggest that you also turn on HTTPS since the authentication is happening in plain text over the internet. (actually what you show seems to be over the intranet, but if you do go over the internet make it https).
Here's a link to Wikipedia that might help you further: http://en.wikipedia.org/wiki/Basic_access_authentication
Ok, I got it. I manually set HttpHeader instead of using request.Credentials
request.Headers.Add(HttpRequestHeader.Authorization, "Basic " + Convert.ToBase64String(Encoding.UTF8.GetBytes(Context.UserName + ":" + Context.Password)));
Now I see only single requests as expected..
As an option you can use PreAuthenticate property of HttpClientHandler. This would require a couple of lines more
var client = new HttpClient(new HttpClientHandler
{
Credentials = yourCredentials,
PreAuthenticate = true
});
With using this approach, only the first request is sent without credentials, but all the rest requests are OK.
For the project I'm working on, we have a desktop program that contacts an online server for a store. Because it's used in schools, getting the proxy setup right is tricky. What we've gone for is to allow users to specify proxy details to use if they want, otherwise it uses the ones from IE. We've also tried to bypass incorrect details being put in, so the code tries the user specified proxy, if that fails the default one, if that fails, then with credentials, if that fails then null.
The problem I'm having is that in places where the proxy settings need to be changed in succession (for example, if their registration fails because the proxy is wrong, they change one tiny thing and try again, takes seconds.) I end up with calls to a HttpRequests .GetResponse() timing out, causing the program to freeze for a good while. Sometimes if I leave a minute or two between the changes, it doesn't freeze, but not every time (Just tried again now after 10mins and it's timing out again).
I can't spot anything in the code that could cause this - though it looks a bit messy. I don't think it could be the server refusing the request unless it's generic server behaviour as I've tried this with requests to our server and others such as google.co.uk.
I'm posting the code in the hope that someone may be able to spot something that's wrong with it, or knows a much simpler way of doing what we're trying to.
The tests we run are without any proxy, so the first part is usually skipped. The first time ApplyProxy is run, it works fine and finishes everything in the first try block, the second, it can either timeout on the GetResponse in the first try block and then go through the rest of the code, or it can work there and timeout on the actual requests made for the registration.
Code:
void ApplyProxy()
{
Boolean ProxySuccess = true;
String WebRequestURI = #"http://www.google.co.uk";
if (UseProxy)
{
try
{
String ProxyUrl = (ProxyUri.ToLower().Contains("http://")) ?
ProxyUri :
"http://" + ProxyUri;
WebRequest.DefaultWebProxy = new WebProxy(ProxyUrl);
if (!string.IsNullOrEmpty(ProxyUsername) && !string.IsNullOrEmpty(ProxyPassword))
WebRequest.DefaultWebProxy.Credentials = new NetworkCredential(ProxyUsername, ProxyPassword);
HttpWebRequest request = HttpWebRequest.Create(WebRequestURI) as HttpWebRequest;
request.Method = "GET";
HttpWebResponse response = request.GetResponse() as HttpWebResponse;
}
catch
{
ProxySuccess = false;
}
}
if(!ProxySuccess || !UseProxy)
{
try
{
WebRequest.DefaultWebProxy = WebRequest.GetSystemWebProxy();
HttpWebRequest request = HttpWebRequest.Create(WebRequestURI) as HttpWebRequest;
request.Method = "GET";
HttpWebResponse response = request.GetResponse() as HttpWebResponse;
}
catch (Exception e)
{ //try with credentials
//make a new proxy from defaults
WebRequest.DefaultWebProxy = WebRequest.GetSystemWebProxy();
String newProxyURI = WebRequest.DefaultWebProxy.GetProxy(new Uri(WebRequestURI)).ToString();
if (newProxyURI == String.Empty)
{ //check we actually get a result
WebRequest.DefaultWebProxy = null;
return;
}
//continue
WebProxy NewProxy = new WebProxy(newProxyURI);
NewProxy.UseDefaultCredentials = true;
NewProxy.Credentials = CredentialCache.DefaultCredentials;
WebRequest.DefaultWebProxy = NewProxy;
try
{
HttpWebRequest request = HttpWebRequest.Create(WebRequestURI) as HttpWebRequest;
request.Method = "GET";
HttpWebResponse response = request.GetResponse() as HttpWebResponse;
}
catch
{
WebRequest.DefaultWebProxy = null;
}
}
}
}
Is it not just a case of needing to set the Timeout property of the HttpWebRequest? It could be that the connection is being made, but not serviced (wrong type of proxy server or stalled server, for example), in which case it may be that the request is waiting for the Timeout period before giving up — a shorter timeout may be preferrable here.
Seems to be a programming error on my behalf. The requests were left open and obviously either the program or the server doesn't like this. Simply closing the HttpWebRequests once done seems to remove this issue.
I am using a web client class in my source code for downloading a string using http.
This was working fine. However, the clients in the company are all connected now to a proxy server. And the problem started from this.
When I have tested my application I don't think it can pass through the proxy server, as the exception that keeps getting thrown is "no response from xxx.xxx.xxx.xxx which is the proxy server IP address.
However, I can still navigate to the web site URL and it displays the string correctly in the browser when connecting through a proxy server, but not when I use my web client.
Is there something in the web client that I have to configure to allow me to access the url from behind a proxy server?
using (WebClient wc = new WebClient())
{
string strURL = "http://xxxxxxxxxxxxxxxxxxxxxxxx";
//Download only when the webclient is not busy.
if (!wc.IsBusy)
{
string rtn_msg = string.Empty;
try
{
rtn_msg = wc.DownloadString(new Uri(strURL));
return rtn_msg;
}
catch (WebException ex)
{
Console.Write(ex.Message);
return false;
}
catch (Exception ex)
{
Console.Write(ex.Message);
return false;
}
}
else
{
System.Windows.Forms.MessageBox.Show("Busy please try again");
return false;
}
}
My solution:
WebClient client = new WebClient();
WebProxy wp = new WebProxy(" proxy server url here");
client.Proxy = wp;
string str = client.DownloadString("http://www.google.com");
If you need to authenticate to the proxy, you need to set UseDefaultCredentials to false, and set the proxy Credentials.
WebProxy proxy = new WebProxy();
proxy.Address = new Uri("mywebproxyserver.com");
proxy.Credentials = new NetworkCredential("usernameHere", "pa****rdHere"); //These can be replaced by user input
proxy.UseDefaultCredentials = false;
proxy.BypassProxyOnLocal = false; //still use the proxy for local addresses
WebClient client = new WebClient();
client.Proxy = proxy;
string doc = client.DownloadString("http://www.google.com/");
If all you need is a simple proxy, you skip most of the lines above though. All you need is:
WebProxy proxy = new WebProxy("mywebproxyserver.com");
The answer proposed by Jonathan is proper, but requires that you specify the proxy credentials and url in the code. Usually, it is better to allow usage of the credentials as setup in the system by default (Users typically configure LAN Settings anyway in case they use a proxy)...
The below answer has been provided by Davide in earlier answer, but that requires modifying the app.config files. This solution is probably more useful since it does the same thing IN CODE.
In order to let the application use the default proxy settings as used in the user's system, one can use the following code:
IWebProxy wp = WebRequest.DefaultWebProxy;
wp.Credentials = CredentialCache.DefaultCredentials;
wc.Proxy = wp;
This will allow the application code to use the proxy (with logged-in credentials and default proxy url settings)... No headaches! :)
Hope this helps future viewers of this page to solve their problem!
I've encountered the same issue but using a webclient for downloading a file from the internet with a Winform application the solution was adding in the app.config:
<system.net>
<defaultProxy useDefaultCredentials="true" />
</system.net>
The same solution will work for an asp.net app inserting the same rows in web.config.
Hope it will help.
You need to configure the proxy in the WebClient object.
See the WebClient.Proxy property:
http://msdn.microsoft.com/en-us/library/system.net.webclient.proxy(VS.80).aspx
byte[] data;
using (WebClient client = new WebClient())
{
ICredentials cred;
cred = new NetworkCredential("xmen#test.com", "mybestpassword");
client.Proxy = new WebProxy("192.168.0.1",8000);
client.Credentials = cred;
string myurl="http://mytestsite.com/source.jpg";
data = client.DownloadData(myUrl);
}
File.WriteAllBytes(#"c:\images\target.jpg", data);
All previous answers have some merit, but the actual answer only needs ONE line:
wc.Proxy = new WebProxy("127.0.0.1", 8888);
where wc is the WebClient object, and 8888 is the port number of the proxy server located on the same machine.