I am trying to build a c# console app that will monitor about 3000 urls (Just need to know that HEAD request returned 200, not necessarily content, etc.)
My attempt here was to build a routine the checks the web URLS, looping and creating threads each executing the routine. What's happening is if i run with <20 threads, it executes ok most of the time, but if i use >20 threads, some of the url's time out. I tried increasing the Timeout to 30 seconds, same occurs. The network I am running this on is more than capable of executing 50 HTTP HEAD requests (10MBIT connection at ISP), and both the CPU and network run very low when executing the routine.
When a timeout occurs, i test the same IP on a browser and it works fine, I tested this repeatedly and there was never a case during testing that a "timed out" url was actually timing out.
The reason i want to run >20 threads is that i want to perform this test every 5 minutes, with some of the URL's taking a full 10sec (or higher if the timeout is set higher), i want to make sure that its able to run through all URLs within 2-3 minutes.
Is there a better way to go about checking if a URL is available, or, should I be looking at the system/network for an issue.
MAIN
while (rdr.Read())
{
Thread t = new Thread(new ParameterizedThreadStart(check_web));
t.Start(rdr[0]);
}
static void check_web(object weburl)
{
bool isok;
isok = ConnectionAvailable(weburl.ToString());
}
public static bool ConnectionAvailable(string strServer)
{
try
{
strServer = "http://" + strServer;
HttpWebRequest reqFP = (HttpWebRequest)HttpWebRequest.Create(strServer);
reqFP.Timeout = 10000;
reqFP.Method = "HEAD";
HttpWebResponse rspFP = (HttpWebResponse)reqFP.GetResponse();
if (HttpStatusCode.OK == rspFP.StatusCode)
{
Console.WriteLine(strServer + " - OK");
rspFP.Close();
return true;
}
else
{
Console.WriteLine(strServer + " Server returned error..");
rspFP.Close();
return false;
}
}
catch (WebException x)
{
if (x.ToString().Contains("timed out"))
{
Console.WriteLine(strServer + " - Timed out");
}
else
{
Console.WriteLine(x.Message.ToString());
}
return false;
}
}
Just remember, you asked.
Very bad implementation.
Do not go creating threads like that. It does very little good to have more threads than processor cores. The extra threads will pretty much just compete with each other, especially since they're all running the same code.
You need to implement using blocks. If you throw an exception (and chances are you will), then you will be leaking resources.
What is the purpose in returning a bool? Do you check it somewhere? In any case, your error and exception processing are a mess.
When you get a non-200 response, you don't display the error code.
You're comparing against the Message property to decide if it's a timeout. Microsoft should put a space between the "time" and "out" just to spite you.
When it's not a timeout, you display only the Message property, not the entire exception, and the Message property is already a string and doesn't need you to call ToString() on it.
Next Batch of Changes
This isn't finished, I don't think, but try this one:
public static void Main()
{
// Don't mind the interpretation. I needed an excuse to define "rdr"
using (var conn = new SqlConnection())
{
conn.Open();
using (var cmd = new SqlCommand("SELECT Url FROM UrlsToCheck", conn))
{
using (var rdr = cmd.ExecuteReader())
{
while (rdr.Read())
{
// Use the thread pool. Please.
ThreadPool.QueueUserWorkItem(
delegate(object weburl)
{
// I invented a reason for you to return bool
if (!ConnectionAvailable(weburl.ToString()))
{
// Console would be getting pretty busy with all
// those threads
Debug.WriteLine(
String.Format(
"{0} was not available",
weburl));
}
},
rdr[0]);
}
}
}
}
}
public static bool ConnectionAvailable(string strServer)
{
try
{
strServer = "http://" + strServer;
var reqFp = (HttpWebRequest)WebRequest.Create(strServer);
reqFp.Timeout = 10000;
reqFp.Method = "HEAD";
// BTW, what's an "FP"?
using (var rspFp = (HttpWebResponse) reqFp.GetResponse()) // IDisposable
{
if (HttpStatusCode.OK == rspFp.StatusCode)
{
Debug.WriteLine(string.Format("{0} - OK", strServer));
return true; // Dispose called when using is exited
}
// Include the error because it's nice to know these things
Debug.WriteLine(String.Format(
"{0} Server returned error: {1}",
strServer, rspFp.StatusCode));
return false;
}
}
catch (WebException x)
{
// Don't tempt fate and don't let programs read human-readable messages
if (x.Status == WebExceptionStatus.Timeout)
{
Debug.WriteLine(string.Format("{0} - Timed out", strServer));
}
else
{
// The FULL exception, please
Debug.WriteLine(x.ToString());
}
return false;
}
}
Almost Done - Not Tested Late Night Code
public static void Main()
{
using (var conn = new SqlConnection())
{
conn.Open();
using (var cmd = new SqlCommand("", conn))
{
using (var rdr = cmd.ExecuteReader())
{
if (rdr == null)
{
return;
}
while (rdr.Read())
{
ThreadPool.QueueUserWorkItem(
CheckConnectionAvailable, rdr[0]);
}
}
}
}
}
private static void CheckConnectionAvailable(object weburl)
{
try
{
// If this works, it's a lot simpler
var strServer = new Uri("http://" + weburl);
using (var client = new WebClient())
{
client.UploadDataCompleted += ClientOnUploadDataCompleted;
client.UploadDataAsync(
strServer, "HEAD", new byte[] {}, strServer);
}
}
catch (WebException x)
{
Debug.WriteLine(x);
}
}
private static void ClientOnUploadDataCompleted(
object sender, UploadDataCompletedEventArgs args)
{
if (args.Error == null)
{
Debug.WriteLine(string.Format("{0} - OK", args.UserState));
}
else
{
Debug.WriteLine(string.Format("{0} - Error", args.Error));
}
}
Use ThreadPool class. Don't spawn hundreds of threads like this. Threads have such a huge overhead and what happens in your case is that your CPU will spend 99% time on context switching and 1% doing real work.
Don't use threads.
Asynch Call backs and queues. Why create a thread when the resource that they are all wanting is access to the outside world. Limit your threads to about 5, and then implement a class that uses a queue. split the code into two parts, the fetch and the process. One controls the flow of data while the other controls access to the outside world.
Use whatever language you like but you won't got wrong if you think that threads are for processing and number crunching and async call backs are for resource management.
Related
I have a few functions in a Solidworks Addin which call on a VBA macro (Via the runMacro2 method) a co-worker has been working on for the last few weeks. In his code he calls a Solidworks function which, under certain, unknown conditions, hangs for a long period of time. How long seems to depend upon the size and quantity of bodies in the part. Considering at least one of the functions we want to run this from i automatic, this just wont do.
I have tried using the Thread.Join(int) method (shown below) but it doesnt work. I also tried modifying the code from this answer Close a MessageBox after several seconds with the same results. Is there anything I can do either in C# or VBA to handle a timeout for this without re-writing his entire macro?
public void runBB()
{
Stopwatch testStop = new Stopwatch();
Thread workerThread = new Thread(bbRun);
testStop.Start();
workerThread.Start();
if (!workerThread.Join(50))
{
workerThread.Abort();
testStop.Stop();
MessageBox.Show("Unable to generate Bounding Box after " + testStop.ElapsedMilliseconds/1000 + " seconds. Please enter data manually.", "Solidworks Derped Error.");
}
return;
}//Still uses Macro (2-5-16)
public static void bbRun()
{
iSwApp.RunMacro2(macroPath + "BOUNDING_BOX.swp", "test11", "main", 0, out runMacroError);
return;
}
I was getting this same exact issue with SOLIDWORKS hanging on an open of a file. Almost all reference on SO was that you should never do this, but in this scenario, you either have to close it or wait forever. In C# I created a callWithTimeout method:
private void callWithTimeout(Action action, int timeoutMilliseconds, String errorText) {
Thread threadToKill = null;
Action wrappedAction = () =>
{
threadToKill = Thread.CurrentThread;
action();
};
IAsyncResult result = wrappedAction.BeginInvoke(null, null);
if (result.AsyncWaitHandle.WaitOne(timeoutMilliseconds)) {
wrappedAction.EndInvoke(result);
} else {
threadToKill.Abort();
throw new TimeoutException(errorText);
}
}
Then the code that was hanging put in a block as such:
bool timedOut = false;
try {
callWithTimeout(delegate() {
// code that hangs here
}, 60000, "Operation timed out. SOLIDWORKS could not open the file. This file will be processed later.");
} catch (TimeoutException){
timedOut = true;
} finally {
if(timedOut) {
Process[] prs = Process.GetProcesses();
foreach (Process p in prs) {
if (p?.ProcessName.Equals("SLDWORKS") ?? false)
p?.Kill();
}
}
}
I'm making examples for my ZeroMQ CLR namespace, however I have a problem with PUB/SUB.
Why do I get only the first message? Sometimes I get no message, if I debug through the client (on PubSub_Client(arg);) I get some messages.
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading;
using System.Security.Cryptography;
using ZeroMQ;
namespace ZeroMQ.Test
{
static partial class Program
{
static string PubSub_FrontendAddress = "tcp://127.0.0.1:2772";
public static void Main(string[] args)
{
if (args == null || args.Length < 1)
{
// say here were some arguments...
args = new string[] { "World" };
}
// Setup the ZContext
context = ZContext.Create();
CancellationTokenSource cancellor0 = null;
{
// Create the "Server" cancellor and threads
cancellor0 = new CancellationTokenSource();
var serverThread = new Thread(PubSub_Server);
serverThread.Start(cancellor0.Token);
serverThread.Join(64);
}
{
Thread.Sleep(1000);
Console.WriteLine("Starting...");
// foreach arg we are the Client, asking the Server
foreach (string arg in args)
{
PubSub_Client(arg);
// Thread.Sleep(1000);
}
Console.WriteLine("Ended...");
}
if (cancellor0 != null)
{
// Cancel the Server
cancellor0.Cancel();
}
// we could have done here context.Terminate()
}
static void PubSub_Server(object cancelluS)
{
var cancellus = (CancellationToken)cancelluS;
using (var socket = ZSocket.Create(context, ZSocketType.SUB))
{
socket.Bind(PubSub_FrontendAddress);
socket.SubscribeAll();
/* var poller = ZPollItem.Create(socket, (ZSocket _socket, out ZMessage message, out ZError _error) =>
{
while (null == (message = _socket.ReceiveMessage(/* ZSocketFlags.DontWait, * out _error)))
{
if (_error == ZError.EAGAIN)
{
_error = ZError.None;
Thread.Sleep(1);
continue;
}
throw new ZException(_error);
}
return true;
}); /**/
while (!cancellus.IsCancellationRequested)
{
ZError error;
ZMessage request;
/* if (!poller.TryPollIn(out request, out error, TimeSpan.FromMilliseconds(512)))
{
if (error == ZError.EAGAIN)
{
error = ZError.None;
Thread.Sleep(1);
continue;
}
throw new ZException(error);
} /**/
if (null == (request = socket.ReceiveMessage(ZSocketFlags.DontWait, out error)))
{
if (error == ZError.EAGAIN)
{
error = ZError.None;
Thread.Sleep(1);
continue;
}
throw new ZException(error);
} /**/
foreach (ZFrame frame in request)
{
string strg = frame.ReadString();
Console.WriteLine("{0} said hello!", strg);
}
}
socket.Unbind(PubSub_FrontendAddress);
}
}
static void PubSub_Client(string name)
{
using (var socket = ZSocket.Create(context, ZSocketType.PUB))
{
using (var crypto = new RNGCryptoServiceProvider())
{
var identity = new byte[8];
crypto.GetBytes(identity);
socket.Identity = identity;
}
socket.Connect(PubSub_FrontendAddress);
using (var request = new ZMessage())
{
request.Add(new ZFrame(name));
socket.Send(request);
}
socket.Disconnect(PubSub_FrontendAddress);
}
}
}
}
I'm having trouble with your design which seems just wrong:
A single subscriber and multiple publishers is an odd choice. I trust you have a good reason for it, but you should have said what that is. When sending messages from multiple clients to a single server, it is normal to use DEALER/ROUTER sockets instead. PUB/SUB is intended for a small set of publishers to a large number of subscribers.
A client that connects, sends one message, then immediately disconnects, is another very unusual use case that I hope is just an example:
For one thing, you are open to linger problems whereby the message will get dropped on the disconnect it is isn't sent within the linger timeout. [I don't know what the default linger is for your language binding, so that may or may not be an issue, but you should at least check to ensure that it isn't.]
For another, as you've already found, there are issues around the time it takes to connect to a socket, which may lead to PUB messages getting dropped if they are sent before the socket has properly connected.
If you insist on using PUB/SUB in this manner, you will need an out of band protocol to synchronise the PUB and SUB threads before the pub messages are sent. There are examples of how to do this reliable pub/sub in the zeromq guide. This will involve a second set of sockets in the same threads to send the synchronisation messages; DEALER sockets don't drop messages which is why they are suitable for that purpose...
But, DEALER/ROUTER sockets would appear to be a better choice than PUB/SUB unless there is some design requirement that hasn't been disclosed.
Well... There was a comment by Martin Sustrik: "The problem is that connecting is asynchronous and takes certain amount of time."
Now there is Thread.Sleep(64) - and it works...:
static void PubSub_Client(string name)
{
using (var socket = ZSocket.Create(context, ZSocketType.PUB))
{
socket.Connect(PubSub_FrontendAddress);
Thread.Sleep(64);
using (var request = new ZMessage())
{
request.Add(new ZFrame(name));
socket.Send(request);
}
socket.Disconnect(PubSub_FrontendAddress);
}
}
Do you know any better way to get the connection established?
I am using the following script to download XML files from a external site, but when the function is called fast after each other (Fast switching of tables to show) the queue seems to slip up.
When the function is called in a normal manner it works just fine, but when the user starts to switch between tables at a faster pace, the data won't load. It does not give any exceptions besides on some rare occasions it will say that the queue is busy. All tough I can't seem to find what is causing this queue to slip.
public void PreObtainData(ref MonavisaRequestForm request, string dateAndTime, string fileDateAndTime)
{
if (!initialized)
initialize();
try
{
if (!request.webclient.IsBusy && requestQueue.Count == 0)
{
request.url = request.url.Replace("&", "%26");
request.url = request.url.Replace("+", "%2B");
Uri uri = new Uri(string.Format("http://localhost/login.php?username={0}&password={1}&request={2}", request.username, request.password, request.url));
request.webclient.DownloadFile(uri, #"Nioo Graph Data " + fileDateAndTime + ".xml");
}
else if (!request.webclient.IsBusy && requestQueue.Count > 0)
{
Uri uri = new Uri(string.Format("http://localhost/login.php?username={0}&password={1}&request={2}", requestQueue.Peek().username, requestQueue.Peek().password, requestQueue.Peek().url));
requestQueue.Peek().webclient.DownloadStringAsync(uri);
requestQueue.Dequeue();
}
else
{
requestQueue.Enqueue(request);
}
}
catch (System.Net.WebException ex)
{
//if (ex.Status != System.Net.WebExceptionStatus.ProtocolError)
{
throw ex;
}
}
}
Queues are not designed to be accessed from multiple threads, and any number of things can go wrong when you do so. You should use a ConcurrentQueue or a BlockingCollection (which uses a ConcurrentQueue), as it is specifically designed to be used from multiple threads.
I'm developing an application that manages devices in the network, at a certain point in the applicaiton, I must ping (actually it's not a ping, it's a SNMP get) all computers in the network to check if it's type is of my managed device.
My problem is that pinging all computers in the network is very slow (specially because most of them won't respond to my message and will simply timeout) and has to be done asynchronously.
I tried to use TLP to do this with the following code:
public static void FindDevices(Action<IPAddress> callback)
{
//Returns a list of all host names with a net view command
List<string> hosts = FindHosts();
foreach (string host in hosts)
{
Task.Run(() =>
{
CheckDevice(host, callback);
});
}
}
But it runs VERY slow, and when I paused execution I checked threads window and saw that it only had one thread pinging the network and was thus, running tasks synchronously.
When I use normal threads it runs a lot faster, but Tasks were supposed to be better, I'd like to know why aren't my Tasks optimizing parallelism.
**EDIT**
Comments asked for code on CheckDevice, so here it goes:
private static void CheckDevice(string host, Action<IPAddress> callback)
{
int commlength, miblength, datatype, datalength, datastart;
string output;
SNMP conn = new SNMP();
IPHostEntry ihe;
try
{
ihe = Dns.Resolve(host);
}
catch (Exception)
{
return;
}
// Send sysLocation SNMP request
byte[] response = conn.get("get", ihe.AddressList[0], "MyDevice", "1.3.6.1.2.1.1.6.0");
if (response[0] != 0xff)
{
// If response, get the community name and MIB lengths
commlength = Convert.ToInt16(response[6]);
miblength = Convert.ToInt16(response[23 + commlength]);
// Extract the MIB data from the SNMP response
datatype = Convert.ToInt16(response[24 + commlength + miblength]);
datalength = Convert.ToInt16(response[25 + commlength + miblength]);
datastart = 26 + commlength + miblength;
output = Encoding.ASCII.GetString(response, datastart, datalength);
if (output.StartsWith("MyDevice"))
{
callback(ihe.AddressList[0]);
}
}
}
Your issue is that you are iterating a none thread safe item the List.
If you replace it with a thread safe object like the ConcurrentBag you should find the threads will run in parallel.
I was a bit confused as to why this was only running one thread, I believe it is this line of code:
try
{
ihe = Dns.Resolve(host);
}
catch (Exception)
{
return;
}
I think this is throwing exceptions and returning; hence you only see one thread. This also ties into your observation that if you added a sleep it worked correctly.
Remember that when you pass a string your passing the reference to the string in memory, not the value. Anyway, the ConcurrentBag seems to resolve your issue. This answer might also be relevant
The following code creates a new thread acting first as a named pipe client for sending parameters and then as a server for retrieving results. After that it executes a function in another AppDomain acting as a named pipe server and after that as a client to send the results back.
public OrderPrice DoAction()
{
Task<OrderPrice> t = Task<OrderPrice>.Factory.StartNew(NamedPipeClient, parameters);
if (domain == null)
{
domain = AppDomain.CreateDomain(DOMAINNAME);
}
domain.DoCallBack(AppDomainCallback);
return t.Result;
}
static OrderPrice NamedPipeClient(object parameters) {
OrderPrice price = null;
using (NamedPipeClientStream stream = new NamedPipeClientStream(PIPE_TO)) {
stream.Connect();
SerializeToStream(stream, parameters);
}
using (NamedPipeServerStream stream = new NamedPipeServerStream(PIPE_BACK)) {
stream.WaitForConnection();
price = (OrderPrice)DeserializeFromStream(stream);
}
return price;
}
void AppDomainCallback() {
OrderPrice price = null;
using (NamedPipeServerStream stream = new NamedPipeServerStream(PIPE_TO)) {
stream.WaitForConnection();
List<object> parameters = (List<object>)DeserializeFromStream(stream);
if (mi != null)
price = (OrderPrice)mi.Invoke(action, parameters.ToArray());
}
using (NamedPipeClientStream stream = new NamedPipeClientStream(PIPE_BACK)) {
stream.Connect();
SerializeToStream(stream, price);
}
}
The code is called once per second on average and it worked fine for 7+ hours. But at some point "system.io.ioexception all pipe instances are busy" is thrown and they wont reconnect anymore after that. Browsing here it seems like it could be because of not properly disposing the pipe objects, but I guess thats all good since they are inside using statements.
Does anyone have any clue what could be wrong here? The code is in .NET 4.0 running on windows server 2008.
Sounds like it should be a mutex instead of a simple lock
Lock, mutex, semaphore... what's the difference?
as far as the occasional halting, it could be starvation or a deadlock.
This is good reading material for abstracts on what may be happening
http://en.wikipedia.org/wiki/Dining_philosophers_problem