I'm trying to find all the ftp servers accepting anonymous connections on a given set of ips.
Basically, I get the IPs I want to check, and then try a ListDirectory on each of them. If I have no exception, the ftp exists and is accessible.
I'm using an asynchronous method to verify an IP, which make things much faster. However, I then need to wait until all the async calls returned. To do this, I keep a counter on the number of async calls I have, the problem is this counter never gets to 0.
My code looks as follows:
to iterate over the IPs:
static int waitingOn;
public static IEnumerable<Uri> GetFtps()
{
var result = new LinkedList<Uri>();
waitingOn = 0;
IPNetwork ipn = IPNetwork.Parse("192.168.72.0/21");
IPAddressCollection ips = IPNetwork.ListIPAddress(ipn);
foreach( var ip in ips )
{
VerifyFtpAsync(ip, result);
}
while (waitingOn > 0)
{
Console.WriteLine(waitingOn);
System.Threading.Thread.Sleep(1000);
}
return result;
}
and to verify each IP:
public async static void VerifyFtpAsync( IPAddress ip, LinkedList<Uri> ftps )
{
++waitingOn;
try
{
Uri serverUri = new Uri("ftp://" + ip);
FtpWebRequest request = (FtpWebRequest)WebRequest.Create(serverUri);
request.Method = WebRequestMethods.Ftp.ListDirectoryDetails;
request.Timeout = 10000;
request.Credentials = new NetworkCredential("anonymous", "roim#search.com");
FtpWebResponse response = (FtpWebResponse) await request.GetResponseAsync();
// If we got this far, YAY!
ftps.AddLast(serverUri);
}
catch (WebException)
{
}
--waitingOn;
}
Replace
FtpWebResponse response = (FtpWebResponse) await request.GetResponseAsync();
with sync code
FtpWebResponse response = (FtpWebResponse) request.GetResponse();
FtpWebRequest.GetResponseAsync is not overriden for FTP specific and comes from the base WebRequest class, which doesn't seem to be able to handle this right.
First, you should never use async void unless you're writing an event handler.
Next, you do need to protect variables and collections against multithreaded access if your async methods may run in parallel (e.g., if this code is run in a console app). In your case, it sounds like you may want to use Task.WhenAll instead of a manual counter, and remove the shared collection.
public async static Task<Uri> VerifyFtpAsync(IPAddress ip)
{
try
{
...
return serverUri;
}
catch (WebException)
{
return null;
}
}
...
var ipTasks = ips.Select(ip => VerifyFtpAsync(ip));
var allResults = await Task.WhenAll(ipTasks);
var result = allResults.Where(url => url != null).ToArray();
Related
I have a WCF service which accepts requests and for each request makes an HTTPWebRequest call and returns the response. I use a BlockingCollection to store the requests as they come in and a separate thread processes (makes the webrequest) the items in the collection. Sometimes the Webrequest returns a threadabortexception. I catch it and do a Thread.ResetAbort. But the exception flows up and and it clears the BlockingCollection. I have added snippets of the code below. I need to find a way for the foreach loop to keep continuing even if i get a threadabort exception.
public static class YBProcessor
{
static PriorityQueue<QueuePriorityLevel, string> queue = new PriorityQueue<QueuePriorityLevel, string>();
static BlockingCollection<KeyValuePair<QueuePriorityLevel, string>> requests;
static YBProcessor()
{
requests = new BlockingCollection<KeyValuePair<QueuePriorityLevel, string>>(queue);
Task.Factory.StartNew(() => SendRequestToYB());
}
public static void AddCalcRequest(string cusip, double price, QueuePriorityLevel priority)
{
requests.Add(new KeyValuePair<QueuePriorityLevel, string>(priority, cusip + "-" + price.ToString()));
}
private static void SendRequestToYB()
{
// this is a separate thread that processes the requests as the come in.
foreach (var obj in requests.GetConsumingEnumerable())
{
try
{
var request = GetXML(obj.Value);
var response = YBClient.GetResponse(request);
//code to handle response
}
catch (ThreadAbortException ex)
{
Thread.ResetAbort();
}
catch (Exception ex)
{
}
}
}
}
// In YBClient The GetResponse Method (just the key parts. Code wont compile)
private static String GetResponse(String text)
{
for (iTry = 0; iTry < MAX_TRY; iTry++)
{
try
{
// Create and setup request
bytes = encoding.GetBytes(text);
request = (HttpWebRequest)WebRequest.Create(uri);
request.Method = "POST";
request.ContentType = "text/xml";
request.ContentLength = bytes.Length;
request.CookieContainer = cookieJar;
request.Timeout = 100 * 1000;
request.ReadWriteTimeout = 100 * 1000;
// Prepare and send data
postStream = request.GetRequestStream();
postStream.Write(bytes, 0, bytes.Length);
postStream.Close();
// Get response from server
response = (HttpWebResponse)request.GetResponse();
if (response.StatusCode == HttpStatusCode.OK)
{
reader = new StreamReader(response.GetResponseStream(), encoding);
xmlResponse = reader.ReadToEnd();
reader.Close();
break;
}
response.Close();
}
catch (ThreadAbortException ex)
{
Thread.ResetAbort();
break;
}
catch (Exception ex)
{
if (ex.GetBaseException() is ThreadAbortException)
{
Thread.ResetAbort();
break;
}
}
}
}
return xmlResponse;
}
If the service itself is calling the thread abort exception then it might mean the service is going down, in that scenario your thread cannot continue to function (imagine a app pool recycle), If you want to make sure your pending requests are not lost then you can do one of the following:
1) Start a new appdomain http://msdn.microsoft.com/en-us/library/ms173139(v=vs.90).aspx
This method will mitigate service shutting down, however will still not resolve the fundamental issue of making sure all your requests are processed as the "other" app domain can also go down.
2) The better solution will be to keep writing your requests in serialized form in a central DB or a file and get a worker to keep popping items out of the same, if you want simplicity, create seperate files for each requests and delete them once processed (assuming you wont get thousands of requests/sec), for a more scalable solutions you can use Redis database (http://redis.io/) and use its "list" (queue) functionality.
P.S. You might want to mark your thread (task) as long running, if you don't, it uses the thread pool which is not recommended for very long running tasks.
Task.Factory.StartNew(Run, TaskCreationOptions.LongRunning);
Am I misusing the HttpClient class incorrectly. I am trying to test the HTTP status of images and it seems to not be executing at all. I have a list of a complex object so I want to run test on all image urls to see which urls are broken by doing this:
var client = new HttpClient();
var tasks = ObjectViewModel.Select(a => a.UserUrl).Select(url =>
client.GetAsync(url).ContinueWith(t =>
{
var response = t.Result;
if (!response.IsSuccessStatusCode)
url = "/Content/Images/MissingPic.png";
}));
I was originally doing it this in a foreach loop like so:
foreach(var Model in ObjectViewModel)
{
Model.UserUrl= Model.UserUrl.GetHttpRequest() ? Model.UserUrl:
"/Content/Images/MissingImage.png";
//Model.state= Model.state.ValidName();// this line is something seperate
//Model.property= Model.state.propertyCheck();// this line is something seperate
}
public static bool GetHttpRequest(this string s)
{
HttpWebRequest webRequest = (HttpWebRequest)WebRequest
.Create(s);
webRequest.AllowAutoRedirect = false;
HttpStatusCode responseStatusCode;
try
{
HttpWebResponse response = (HttpWebResponse)webRequest.GetResponse();
responseStatusCode = response.StatusCode;
}
catch (WebException we)
{
responseStatusCode = ((HttpWebResponse)we.Response).StatusCode;
}
if (responseStatusCode.ToString() == "OK")
return true;
else
return false;
}
which works perfectly fine, but takes about 5 to 7 seconds to complete all items since they are all running separately which is very long for a request to respond to the UI.
Consider using AsParallel() while iterating over your enumeration, should speed it up considerably.
var UrlToReponseMap = new Dictionary<string, bool>();
ObjectViewModel.AsParallel().ForAll(x =>
{
UrlToReponseMap[x.UserUrl] = x.UserUrl.GetHttpRequest();
});
Linq is (generally) lazy. This means that Linq statements only represent a query. Work happens when you enumerate the query (materialize).
You never materialize your tasks query. To get it to actually run the code in your select statement, you need to materialize the query by causing it to be enumerated.
One way would be to simply call tasks.ToList()
I am using this code to implement Http Server:
public Server()
{
_httpListener = new HttpListener();
_httpListener.Prefixes.Add(Server.UriAddress);
StartServer();
}
public void StartServer()
{
_httpListener.Start();
while (_httpListener.IsListening)
ProcessRequest();
}
void ProcessRequest()
{
var result = _httpListener.BeginGetContext(ListenerCallback, _httpListener);
result.AsyncWaitHandle.WaitOne();
}
void ListenerCallback(IAsyncResult result)
{
HttpListenerContext context = _httpListener.EndGetContext(result);
HttpListenerRequest request = context.Request;
string url = request.RawUrl;
url = url.Substring(1, url.Length - 1);
HttpListenerResponse response = context.Response;
string responseString = url;
byte[] buffer = System.Text.Encoding.UTF8.GetBytes(responseString);
response.ContentLength64 = buffer.Length;
System.IO.Stream output = response.OutputStream;
output.Write(buffer, 0, buffer.Length);
output.Close();
}
And i have a problem that if i wrote this in the browser(It's an example and it's occur on every call):
http://localhost:8888/Hello%20World
the ListenerCallback method is called twice,any idea how to fix it?
An answer has been accepted already, but I think it may still be useful to other people subsequently.
By default, most browsers given a URL would make at least two calls. One call to the request URL and the other to favicon.ico.
So a check should be made in ListenerCallback like
HttpListenerContext context = _httpListener.EndGetContext(result);
HttpListenerRequest request = context.Request;
string url = request.RawUrl;
url = url.Substring(1);
if (url == "favicon.ico")
{
return;
}
//normal request handling code
I hope this helps someone.
If your website requires several calls to the server, it will be called several times. This happens when you hav pictures or anything else on you page.
Try to call the synchronous method _httpListener.GetContext() or synchronize your calls with a lock or Mutex.
I am not sure but I think I see what might be a problem with your code.
You are mixing two patterns of async handling
You release the main thread by waiting on the waithandle in the async result.
But I think that only signals that you can call endgetcontext, not that another thread can use the listener.
If you are using the callback pattern, you should release the main thread using a wait handle other than that provided in the asyncresult
Okay so basically I have a function that returns a string, but to get that string it uses webrequest which means while it's doing that webrequest the form is locking up unless I put it in a different thread.
But I can't figure out a way to capture the returned data in a thread since it's started using thread.start and that's a void.
Any help please?
Current code if it matters to anyone:
string CreateReqThread(string UrlReq)
{
System.Threading.Thread NewThread = new System.Threading.Thread(() => CreateReq(UrlReq));
string ReturnedData = "";
return ReturnedData;
}
string CreateReq(string url)
{
try
{
WebRequest SendReq = WebRequest.Create(url);
SendReq.Credentials = CredentialCache.DefaultCredentials;
SendReq.Proxy = WebRequest.DefaultWebProxy; //For closed port networks like colleges
SendReq.Proxy.Credentials = CredentialCache.DefaultCredentials;
SendReq.Timeout = 15000;
System.IO.StreamReader Reader = new System.IO.StreamReader(SendReq.GetResponse().GetResponseStream());
string Response = Reader.ReadToEnd();
Reader.Close();
return Response;
}
catch (WebException e)
{
EBox(e.Message, "Unknown Error While Connecting");
return null;
}
}
A common means of doing this is to use a Task<T> instead of a thread:
Task<string> CreateReqThread(string UrlReq)
{
return Task.Factory.StartNew() => CreateReq(UrlReq));
// In .NET 4.5, you can use (or better yet, reimplement using await/async directly)
// return Task.Run(() => CreateReq(UrlReq));
}
You can then call Task<T>.Result to get the returned value (later), when it's needed, or schedule a continuation on the task which will run when it completes.
This could look something like:
var request = CreateReqThread(theUri);
request.ContinueWith(t =>
{
// Shove results in a text box
this.textBox.Text = t.Result;
}, TaskScheduler.FromCurrentSynchronizationContext());
This also works perfectly with the new await/async support in C# 5.
I have a problem, I want to wait in the Main() until the Download() is finished. However, the file downloading/checking starts, at the same time the other lines start executing.
How can I use awaitor anything else to wait in the Main?
private void Main()
{
Download("http://webserver/file.xml");
//Do something here ONLY if the file exists!!
}
//This method invokes the URL validation
private void Download(downloadURL)
{
System.Uri targetUri = new System.Uri(downloadURL);
HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create(targetUri);
request.BeginGetResponse(new AsyncCallback(WebRequestCallBack), request);
}
//In this method the URL is being checked for its validity
void WebRequestCallBack(IAsyncResult result)
{
HttpWebRequest resultInfo = (HttpWebRequest)result.AsyncState;
HttpWebResponse response;
string statusCode;
try
{
response = (HttpWebResponse)resultInfo.EndGetResponse(result);
statusCode = response.StatusCode.ToString();
}
catch (WebException e)
{
statusCode = e.Message;
}
onCompletion(statusCode);
}
//This method does not help! I just added if it could be any useful
private void onCompletion(string status)
{
if (status == HttpStatusCode.OK.ToString())
MessageBox.Show("file exists");
else
MessageBox.Show("file does not exists");
}
What I need, in detail is...
Download a file from a given URL
Before downloading verify the URL
if (verfied) then
continue downloading and do other tasks
else
fail and stop the process, Don't download! and give a message that URL was broken (couldn't verified)!
I am trying to do the "Verification" part, checking if the URL is correct and waiting for response. I need some kind of STATUS of the verification process, in order to continue.
Should try:
var task = Task.Factory.FromAsync<WebResponse>(request.BeginGetResponse,
request.EndGetResponse, null);
var response = task.Result;
You can use a ManualResetEventSlim object. Initialize it to true when you instantiate it. At the end of OnComplete method call the Reset method on the ManualResetEventSlim object. In your main application, you just have to invoke the WaitOne method on the ManualResetEventSlim object.