I am writing a simple C# console application, whose main job is, when given a set of URLs, to ping those URLs and report whether or not an HTTP 200 OK result was returned. The real life data set is in the area of 20,000 URLs to test (to verify that an en-masse edit did not ruin any of the pages).
Currently, the code that checks the response looks like this:
public UrlTestResult TestUrl(string url)
{
if (string.IsNullOrWhiteSpace(url))
{
throw new ArgumentNullException("url");
}
using (var client = new HttpClient())
{
try
{
Task<HttpResponseMessage> message = client.GetAsync(url);
if (message == null || message.Result == null)
{
return new FailedUrlTestResult(url, "No response was returned.");
}
if (message.Result.StatusCode == HttpStatusCode.OK)
{
return new SuccessfulUrlTestResult(url);
}
return new FailedUrlTestResult(url, "{0}: {1}".Format((int)message.Result.StatusCode, message.Result.ReasonPhrase));
}
catch (Exception ex)
{
return new FailedUrlTestResult(url, "An exception occurred: " + ex);
}
}
}
This code does work for smaller sets of data. Even if I iterate over the collection of URLs using Parallel.ForEach instead of a normal foreach, it behaves fine. After running for a few minutes or so, however, when parsing the 20,000 dataset, IIS Express (hosting localhost) will crash.
I'm guessing that my options are:
Run out of IIS and see if that works
Throttle the number of requests to give IIS Express a chance to breathe (trick here is how much to throttle)
Test the URLs in smaller chunks (similar to the second option)
What I am wondering is:
Is there a "cheaper" way to ping a URL and get its HTTP response back than HttpClient?
Are there any configuration options for IIS or IIS Express that I should be taking into consideration?
EDIT: I'm finding that IIS Express seems to simply be running out of memory. Pictured is the instance where the crash occurs:
Which means that IIS Express is holding on to memory that it obviously doesn't need to be (because once the request is over, I don't care about it anymore). Don't know if this'll help solve my problem any, though.
I simply changed to running localhost out of IIS instead of IIS Express. The memory usage was about the same, but it never crashed at any point for the ten minutes that the application was running. I also took Gabi's comment/suggestion and made HttpClient only be instantiated one time instead of once per test. The final code looks like this:
public sealed class UrlTester : IUrlTester
{
private readonly HttpClient httpClient = new HttpClient();
public UrlTestResult TestUrl(string url)
{
if (string.IsNullOrWhiteSpace(url))
{
throw new ArgumentNullException("url");
}
try
{
Task<HttpResponseMessage> message = httpClient.GetAsync(url);
if (message == null || message.Result == null)
{
return new FailedUrlTestResult(url, "No response was returned.");
}
if (message.Result.StatusCode == HttpStatusCode.OK)
{
return new SuccessfulUrlTestResult(url);
}
return new FailedUrlTestResult(url, "{0}: {1}".FormatCurrentCulture((int)message.Result.StatusCode, message.Result.ReasonPhrase));
}
catch (Exception ex)
{
return new FailedUrlTestResult(url, "An exception occurred: " + ex);
}
}
public void Dispose()
{
if (httpClient != null)
{
httpClient.Dispose();
}
}
}
And the caller to this class utilizes C#'s using statement to ensure that the HttpClient instance is properly disposed of.
Related
I am using the following script to download XML files from a external site, but when the function is called fast after each other (Fast switching of tables to show) the queue seems to slip up.
When the function is called in a normal manner it works just fine, but when the user starts to switch between tables at a faster pace, the data won't load. It does not give any exceptions besides on some rare occasions it will say that the queue is busy. All tough I can't seem to find what is causing this queue to slip.
public void PreObtainData(ref MonavisaRequestForm request, string dateAndTime, string fileDateAndTime)
{
if (!initialized)
initialize();
try
{
if (!request.webclient.IsBusy && requestQueue.Count == 0)
{
request.url = request.url.Replace("&", "%26");
request.url = request.url.Replace("+", "%2B");
Uri uri = new Uri(string.Format("http://localhost/login.php?username={0}&password={1}&request={2}", request.username, request.password, request.url));
request.webclient.DownloadFile(uri, #"Nioo Graph Data " + fileDateAndTime + ".xml");
}
else if (!request.webclient.IsBusy && requestQueue.Count > 0)
{
Uri uri = new Uri(string.Format("http://localhost/login.php?username={0}&password={1}&request={2}", requestQueue.Peek().username, requestQueue.Peek().password, requestQueue.Peek().url));
requestQueue.Peek().webclient.DownloadStringAsync(uri);
requestQueue.Dequeue();
}
else
{
requestQueue.Enqueue(request);
}
}
catch (System.Net.WebException ex)
{
//if (ex.Status != System.Net.WebExceptionStatus.ProtocolError)
{
throw ex;
}
}
}
Queues are not designed to be accessed from multiple threads, and any number of things can go wrong when you do so. You should use a ConcurrentQueue or a BlockingCollection (which uses a ConcurrentQueue), as it is specifically designed to be used from multiple threads.
using IPC over local TCP to communicate from Client to a Server thread. The connection itself doesn't seem to be throwing any errors, but every time I try to make one of the associated calls, I get this message:
System.Runtime.Remoting.RemotingException: Could not connect to an IPC Port: The System cannot Find the file specified
What I am attempting to figure out is WHY. Because this WAS working correctly, until I transitioned the projects in question (yes, both) from .NET 3.5 to .NET 4.0.
Listen Code
private void ThreadListen()
{
_listenerThread = new Thread(Listen) {Name = "Listener Thread", Priority = ThreadPriority.AboveNormal};
_listenerThread.Start();
}
private void Listen()
{
_listener = new Listener(this);
LifetimeServices.LeaseTime = TimeSpan.FromDays(365);
IDictionary props = new Hashtable();
props["port"] = 63726;
props["name"] = "AGENT";
TcpChannel channel = new TcpChannel(props, null, null);
ChannelServices.RegisterChannel(channel, false);
RemotingServices.Marshal(_listener, "Agent");
Logger.WriteLog(new LogMessage(MethodBase.GetCurrentMethod().Name, "Now Listening for commands..."));
LogEvent("Now Listening for commands...");
}
Selected Client Code
private void InitializeAgent()
{
try
{
_agentController =
(IAgent)RemotingServices.Connect(typeof(IAgent), IPC_URL);
//Note: IPC_URL was originally "ipc://AGENT/AGENT"
// It has been changed to read "tcp://localhost:63726/Agent"
SetAgentPid();
}
catch (Exception ex)
{
HandleError("Unable to initialize the connected agent.", 3850244, ex);
}
}
//This is the method that throws the error
public override void LoadTimer()
{
// first check to see if we have already set the agent process id and set it if not
if (_agentPid < 0)
{
SetAgentPid();
}
try
{
TryStart();
var tries = 0;
while (tries < RUNCHECK_TRYCOUNT)
{
try
{
_agentController.ReloadSettings();//<---Error occurs here
return;
} catch (RemotingException)
{
Thread.Sleep(2000);
tries++;
if (tries == RUNCHECK_TRYCOUNT)
throw;
}
}
}
catch (Exception ex)
{
HandleError("Unable to reload the timer for the connected agent.", 3850243, ex);
}
}
If you need to see something I haven't shown, please ask, I'm pretty much flying blind here.
Edit: I think the issue is the IPC_URL String. It is currently set to "ipc://AGENT/AGENT". The thing is, I have no idea where that came from, why it worked before, or what might be stopping it from working now.
Update
I was able to get the IPC Calls working correctly by changing the IPC_URL String, but I still lack understanding of why what I did worked. Or rather, why the original code stopped working and I needed to change it in the first place.
The string I am using now is "tcp://localhost:63726/Agent"
Can anyone tell me, not why the new string works, I know that...but Why did the original string work before and why did updating the project target to .NET 4.0 break it?
I have highload ASP .NET MVC2 website and WCF service that site uses. Early I created one proxy every time I need it and even didn't close it. Refer to my previous question (with my big thanks for SO user Richard Blewett) I found out that I should close this proxy. In other way it will succeed sessions limit.
Now, I'm creating proxy one time app starts and then just check it and recreate it if is needed. So, here is the code:
public static bool IsProxyValid(MyServ.MyService client) {
bool result = true;
if ((client == null) || (client.State != System.ServiceModel.CommunicationState.Opened) // || (client.InnerChannel.State != CommunicationState.Opened)
)
result = false;
return result;
}
public static AServ.AServClient GetClient(HttpContext http) {
if (!IsProxyValid((MyService)http.Application["client"]))
http.Application["client"] = new MyService();
return (MyService)http.Application["client"];
}
public static MyServ.MyService GetClient(HttpContextBase http)
{
if (!IsProxyValid((MyService)http.Application["client"]))
http.Application["client"] = new MyService();
return (MyService)http.Application["client"];
}
public ActionResult SelectDepartment(string departments)
{
try
{
MyService svc = CommonController.GetClient(this.HttpContext);
Department[] depsArray = svc.GetData(departments);
// .... I cut here ....
return View();
}
catch (Exception exc)
{
// log here
return ActionUnavailable();
}
}
So, what do you guys think about it? Should it work properly? Sometimes my app stucked. I think it is because client proxy state determines uncorrectly and app tries to use broken proxy.
POST EDIT
Also in TCP Monitor I see a lot of established connections from site to service. Why it creates a lot of connectiong insteads of using one global? Maybe some exception occured while invoking service method makes it faulted state?
Hope for your help guys!
I think you need to abort the channel if it gets faulted before creating a new one and
Make sure to close/ abort old client if you creating the new client, use something like this for that (this one is used with DI in singleton)
public class MyServiceClientInitializer : IMyServiceClientInitializer
{
[ThreadStatic]
private static MyServ.MyService _client;
public MyServ.MyService Client
{
get
{
if (_client == null
|| (_client.State != CommunicationState.Opened
&& _client.State != CommunicationState.Opening))
IntializeClient();
return _client;
}
}
private void IntializeClient()
{
if (_client != null)
{
if (_client.State == CommunicationState.Faulted)
{
_client.Abort();
}
else
{
_client.Close();
}
}
string url = //get url;
var binding = new WSHttpBinding();
var address = new EndpointAddress(url);
_client = new MyServ.MyService(binding, address);
}
}
I am writing test harness to test a HTTP Post. Test case would send 8 http request using UploadValuesAsync in webclient class in 10 seconds interval. It sleeps 10 seconds after every 8 request. I am recording start time and end time of each request. When I compute the average response time. I am getting around 800 ms. But when I run this test case synchronously using UploadValues method in web client I am getting average response time 250 milliseconds. Can you tell me why is difference between these two methods? I was expecting the less response time in Aync but I did not get that.
Here is code that sends 8 requests async
var count = 0;
foreach (var nameValueCollection in requestCollections)
{
count++;
NameValueCollection collection = nameValueCollection;
PostToURL(collection,uri);
if (count % 8 == 0)
{
Thread.Sleep(TimeSpan.FromSeconds(10));
count = 0;
}
}
UPDATED
Here is code that sends 8 requests SYNC
public void PostToURLSync(NameValueCollection collection,Uri uri)
{
var response = new ServiceResponse
{
Response = "Not Started",
Request = string.Join(";", collection.Cast<string>()
.Select(col => String.Concat(col, "=", collection[col])).ToArray()),
ApplicationId = collection["ApplicationId"]
};
try
{
using (var transportType2 = new DerivedWebClient())
{
transportType2.Expect100Continue = false;
transportType2.Timeout = TimeSpan.FromMilliseconds(2000);
response.StartTime = DateTime.Now;
var responeByte = transportType2.UploadValues(uri, "POST", collection);
response.EndTime = DateTime.Now;
response.Response = Encoding.Default.GetString(responeByte);
}
}
catch (Exception exception)
{
Console.WriteLine(exception.ToString());
}
response.ResponseInMs = (int)response.EndTime.Subtract(response.StartTime).TotalMilliseconds;
responses.Add(response);
Console.WriteLine(response.ResponseInMs);
}
Here is the code that post to the HTTP URI
public void PostToURL(NameValueCollection collection,Uri uri)
{
var response = new ServiceResponse
{
Response = "Not Started",
Request = string.Join(";", collection.Cast<string>()
.Select(col => String.Concat(col, "=", collection[col])).ToArray()),
ApplicationId = collection["ApplicationId"]
};
try
{
using (var transportType2 = new DerivedWebClient())
{
transportType2.Expect100Continue = false;
transportType2.Timeout = TimeSpan.FromMilliseconds(2000);
response.StartTime = DateTime.Now;
transportType2.UploadValuesCompleted += new UploadValuesCompletedEventHandler(transportType2_UploadValuesCompleted);
transportType2.UploadValuesAsync(uri, "POST", collection,response);
}
}
catch (Exception exception)
{
Console.WriteLine(exception.ToString());
}
}
Here is the upload completed event
private void transportType2_UploadValuesCompleted(object sender, UploadValuesCompletedEventArgs e)
{
var now = DateTime.Now;
var response = (ServiceResponse)e.UserState;
response.EndTime = now;
response.ResponseInMs = (int) response.EndTime.Subtract(response.StartTime).TotalMilliseconds;
Console.WriteLine(response.ResponseInMs);
if (e.Error != null)
{
response.Response = e.Error.ToString();
}
else
if (e.Result != null && e.Result.Length > 0)
{
string downloadedData = Encoding.Default.GetString(e.Result);
response.Response = downloadedData;
}
//Recording response in Global variable
responses.Add(response);
}
One problem you're probably running into is that .NET, by default, will throttle outgoing HTTP connections to the limit (2 concurrent connections per remote host) that are mandated by the relevant RFC. Assuming 2 concurrent connections and 250ms per request, that means the response time for your first 2 requests will be 250ms, the second 2 will be 500ms, the third 750ms, and the last 1000ms. This would yield a 625ms average response time, which is not far from the 800ms you're seeing.
To remove the throttling, increase ServicePointManager.DefaultConnectionLimit to the maximum number of concurrent connections you want to support, and you should see your average response time go down alot.
A secondary problem may be that the server itself is slower handling multiple concurrent connections than handing one request at a time. Even once you unblock the throttling problem above, I'd expect each of the async requests to, on average, execute somewhat slower than if the server was only executing one request at a time. How much slower depends on how well the server is optimized for concurrent requests.
A final problem may be caused by test methodology. For example, if your test client is simulating a browser session by storing cookies and re-sending cookies with each request, that may run into problems with some servers that will serialize requests from a single user. This is often a simplification for server apps so they won't have to deal with locking cross-requests state like session state. If you're running into this problem, make sure that each WebClient sends different cookies to simulate different users.
I'm not saying that you're running into all three of these problems-- you might be only running into 1 or 2-- but these are the most likley culprits for the problem you're seeing.
As Justin said, I tried ServicePointManager.DefaultConnectionLimit but that did not fix the issue. I could not able reproduce other problems suggested by Justin. I am not sure how to reproduce them in first place.
What I did, I ran the same piece of code in peer machine that runs perfectly response time that I expected. The difference between the two machines is operating systems. Mine is running on Windows Server 2003 and other machine is running on Windows Server 2008.
As it worked on the other machines, I suspect that it might be one of the problem specified by Justin or could be server settings on 2003 or something else. I did not spend much time after that to dig this issue. As this is a test harness that we had low priority on this issue. We left off with no time further.
As I have no glue on what exactly fixed it, I am not accepting any answer other than this. Becuase at very least I know that switching to server 2008 fixed this issue.
I am trying to build a c# console app that will monitor about 3000 urls (Just need to know that HEAD request returned 200, not necessarily content, etc.)
My attempt here was to build a routine the checks the web URLS, looping and creating threads each executing the routine. What's happening is if i run with <20 threads, it executes ok most of the time, but if i use >20 threads, some of the url's time out. I tried increasing the Timeout to 30 seconds, same occurs. The network I am running this on is more than capable of executing 50 HTTP HEAD requests (10MBIT connection at ISP), and both the CPU and network run very low when executing the routine.
When a timeout occurs, i test the same IP on a browser and it works fine, I tested this repeatedly and there was never a case during testing that a "timed out" url was actually timing out.
The reason i want to run >20 threads is that i want to perform this test every 5 minutes, with some of the URL's taking a full 10sec (or higher if the timeout is set higher), i want to make sure that its able to run through all URLs within 2-3 minutes.
Is there a better way to go about checking if a URL is available, or, should I be looking at the system/network for an issue.
MAIN
while (rdr.Read())
{
Thread t = new Thread(new ParameterizedThreadStart(check_web));
t.Start(rdr[0]);
}
static void check_web(object weburl)
{
bool isok;
isok = ConnectionAvailable(weburl.ToString());
}
public static bool ConnectionAvailable(string strServer)
{
try
{
strServer = "http://" + strServer;
HttpWebRequest reqFP = (HttpWebRequest)HttpWebRequest.Create(strServer);
reqFP.Timeout = 10000;
reqFP.Method = "HEAD";
HttpWebResponse rspFP = (HttpWebResponse)reqFP.GetResponse();
if (HttpStatusCode.OK == rspFP.StatusCode)
{
Console.WriteLine(strServer + " - OK");
rspFP.Close();
return true;
}
else
{
Console.WriteLine(strServer + " Server returned error..");
rspFP.Close();
return false;
}
}
catch (WebException x)
{
if (x.ToString().Contains("timed out"))
{
Console.WriteLine(strServer + " - Timed out");
}
else
{
Console.WriteLine(x.Message.ToString());
}
return false;
}
}
Just remember, you asked.
Very bad implementation.
Do not go creating threads like that. It does very little good to have more threads than processor cores. The extra threads will pretty much just compete with each other, especially since they're all running the same code.
You need to implement using blocks. If you throw an exception (and chances are you will), then you will be leaking resources.
What is the purpose in returning a bool? Do you check it somewhere? In any case, your error and exception processing are a mess.
When you get a non-200 response, you don't display the error code.
You're comparing against the Message property to decide if it's a timeout. Microsoft should put a space between the "time" and "out" just to spite you.
When it's not a timeout, you display only the Message property, not the entire exception, and the Message property is already a string and doesn't need you to call ToString() on it.
Next Batch of Changes
This isn't finished, I don't think, but try this one:
public static void Main()
{
// Don't mind the interpretation. I needed an excuse to define "rdr"
using (var conn = new SqlConnection())
{
conn.Open();
using (var cmd = new SqlCommand("SELECT Url FROM UrlsToCheck", conn))
{
using (var rdr = cmd.ExecuteReader())
{
while (rdr.Read())
{
// Use the thread pool. Please.
ThreadPool.QueueUserWorkItem(
delegate(object weburl)
{
// I invented a reason for you to return bool
if (!ConnectionAvailable(weburl.ToString()))
{
// Console would be getting pretty busy with all
// those threads
Debug.WriteLine(
String.Format(
"{0} was not available",
weburl));
}
},
rdr[0]);
}
}
}
}
}
public static bool ConnectionAvailable(string strServer)
{
try
{
strServer = "http://" + strServer;
var reqFp = (HttpWebRequest)WebRequest.Create(strServer);
reqFp.Timeout = 10000;
reqFp.Method = "HEAD";
// BTW, what's an "FP"?
using (var rspFp = (HttpWebResponse) reqFp.GetResponse()) // IDisposable
{
if (HttpStatusCode.OK == rspFp.StatusCode)
{
Debug.WriteLine(string.Format("{0} - OK", strServer));
return true; // Dispose called when using is exited
}
// Include the error because it's nice to know these things
Debug.WriteLine(String.Format(
"{0} Server returned error: {1}",
strServer, rspFp.StatusCode));
return false;
}
}
catch (WebException x)
{
// Don't tempt fate and don't let programs read human-readable messages
if (x.Status == WebExceptionStatus.Timeout)
{
Debug.WriteLine(string.Format("{0} - Timed out", strServer));
}
else
{
// The FULL exception, please
Debug.WriteLine(x.ToString());
}
return false;
}
}
Almost Done - Not Tested Late Night Code
public static void Main()
{
using (var conn = new SqlConnection())
{
conn.Open();
using (var cmd = new SqlCommand("", conn))
{
using (var rdr = cmd.ExecuteReader())
{
if (rdr == null)
{
return;
}
while (rdr.Read())
{
ThreadPool.QueueUserWorkItem(
CheckConnectionAvailable, rdr[0]);
}
}
}
}
}
private static void CheckConnectionAvailable(object weburl)
{
try
{
// If this works, it's a lot simpler
var strServer = new Uri("http://" + weburl);
using (var client = new WebClient())
{
client.UploadDataCompleted += ClientOnUploadDataCompleted;
client.UploadDataAsync(
strServer, "HEAD", new byte[] {}, strServer);
}
}
catch (WebException x)
{
Debug.WriteLine(x);
}
}
private static void ClientOnUploadDataCompleted(
object sender, UploadDataCompletedEventArgs args)
{
if (args.Error == null)
{
Debug.WriteLine(string.Format("{0} - OK", args.UserState));
}
else
{
Debug.WriteLine(string.Format("{0} - Error", args.Error));
}
}
Use ThreadPool class. Don't spawn hundreds of threads like this. Threads have such a huge overhead and what happens in your case is that your CPU will spend 99% time on context switching and 1% doing real work.
Don't use threads.
Asynch Call backs and queues. Why create a thread when the resource that they are all wanting is access to the outside world. Limit your threads to about 5, and then implement a class that uses a queue. split the code into two parts, the fetch and the process. One controls the flow of data while the other controls access to the outside world.
Use whatever language you like but you won't got wrong if you think that threads are for processing and number crunching and async call backs are for resource management.