We Keep Coding

How to create a globally available WebClient object

I have a download program in windows forms which downloads the html source code of a bunch of URLs and extract further URLs and downloads that stuff too. I am using
using(WebClient client = new WebClient())
{
// Do download stuff here...
}
but .NET Mem profiler shows a severe memory leak because a lot of object are garbage collected but not disposed OR disposed but not garbage collected. Sound weird to me but my program has a memory leak.
To get a feel of my program consider this code please:
private void PreprocessURLs(List<string> URLs)
{
using(WebClient client = new WebClient())
{
// Download first batch of html source code and put in a List<string> property
}
}
private void ProcessURLs(List<string> URLs)
{
using(WebClient client = new WebClient())
{
// Download more stuff and save them
}
}
and so on. This is not the actual program but what I want to show you is, how do I create a "globally" available WebClient object/instance so I do not create a bunch off client objects? I want to reuse it. I suspect this to be the cause (maybe among others) to cause the memory leak. Maybe I am wrong because I have never created a program with a task where it could have a memory leak.
EDIT: After suggestion of tlemster I created a static instance of WebClient but that causes an I/O error because concurrent I/O are not supported (DownloadFileAsync()).
EDIT1: Here is my download method which does the main work and blocking my UI thread by doing so. Thus I believe the mem leak comes from this function.
private void Download()
{
var stopwatch = new Stopwatch();
string bla;
string chapterName;
string bla1;
string chapterNumber;
List<Tuple<string, int>> temp = new List<Tuple<string, int>>();
// Contains all URLs from preprocessing
foreach (var chapter in Chapters)
{
bla = chapter.Item2;
chapterName = ReplaceSpecialChars(bla);
bla1 = chapter.Item3;
chapterNumber = ReplaceSpecialChars(bla1);
// Skip this chapter if it already exists based on chapter name
if (Directory.Exists(string.Format("{0}/{1} - {2}", chapter.Item4, chapterNumber, chapterName)))
{
continue;
}
else
{
Directory.CreateDirectory(string.Format("{0}/{1} - {2}", chapter.Item4, chapterNumber, chapterName));
}
// Process each chapter and extract other URLs
foreach (var item in GetPagesLink(chapter.Item1))
{
// Add the extracted URLs to a list for download further down
temp.Add(new Tuple<string, int>(GetImageLink(item.Item1), item.Item2));
}
stopwatch.Start();
// The download of the files I want after processing it two times
foreach (var img in temp)
{
// A static WebClient does not work because DownloadFileAsync() does not support concurrent I/O
// Atm should not matter because this whole thing is not async anyway...
using (WebClient webClient = new WebClient())
{
webClient.DownloadProgressChanged += new DownloadProgressChangedEventHandler(ProgressChanged);
webClient.DownloadFileAsync(new Uri(img.Item1), string.Format("{0}/{1} - {2}/{3}.jpg", chapter.Item4, chapterNumber, chapterName, img.Item2)); // TODO: Find image type and replace hardcoded jpg
System.Threading.Thread.Sleep(150);
}
}
stopwatch.Stop();
temp.Clear();
txtDebug.AppendText("Finished chapter " + chapter.Item3 + " : " + stopwatch.ElapsedMilliseconds + "\r\n");
stopwatch.Reset();
}
Chapters.Clear();
}

Store the created WebClient in a static field, which keeps the instance alive as long as your program is running or until you get rid of it manually.

C# How to stop a method if it takes longer than 2 seconds?

Following program will connect to the web and get html content of “msnbc.com” webpage and print out the result. If it takes longer than 2 seconds to get data from the webpage, I want my method to stop working and return. Can you please tell me how can I do this with an example?
public partial class Form1 : Form
{
public Form1()
{
InitializeComponent();
}
private void button1_Click(object sender, EventArgs e)
{
gethtml();
MessageBox.Show("End of program");
}
public void gethtml()
{
HttpWebRequest WebRequestObject = (HttpWebRequest)HttpWebRequest.Create("http://msnbc.com/");
WebResponse Response = WebRequestObject.GetResponse();
Stream WebStream = Response.GetResponseStream();
StreamReader Reader = new StreamReader(WebStream);
string webcontent = Reader.ReadToEnd();
MessageBox.Show(webcontent);
}
}

Two seconds is far too long to block the UI. You should only block the UI if you are planning on getting the result in, say fifty milliseconds or less.
Read this article on how to do a web request without blocking the UI:
http://www.developerfusion.com/code/4654/asynchronous-httpwebrequest/
Note that this will all be much easier in C# 5, which is in beta release at present. In C# 5 you can simply use the await operator to asynchronously await the result of the task. If you would like to see how this sort of thing will work in C# 5, see:
http://msdn.microsoft.com/en-us/async

Set the Timeout property of your WebRequest object. Documentation
MSDN Example:
// Create a new WebRequest Object to the mentioned URL.
WebRequest myWebRequest=WebRequest.Create("http://www.contoso.com");
Console.WriteLine("\nThe Timeout time of the request before setting is : {0} milliseconds",myWebRequest.Timeout);
// Set the 'Timeout' property in Milliseconds.
myWebRequest.Timeout=10000;
// This request will throw a WebException if it reaches the timeout limit before it is able to fetch the resource.
WebResponse myWebResponse=myWebRequest.GetResponse();

As stated above .Timeout
public void gethtml()
{
HttpWebRequest WebRequestObject = (HttpWebRequest)HttpWebRequest.Create("http://msnbc.com/");
WebRequestObject.Timeout = (System.Int32)TimeSpan.FromSeconds(2).TotalMilliseconds;
try
{
WebResponse Response = WebRequestObject.GetResponse();
Stream WebStream = Response.GetResponseStream();
StreamReader Reader = new StreamReader(WebStream);
string webcontent = Reader.ReadToEnd();
MessageBox.Show(webcontent);
}
catch (System.Net.WebException E)
{
MessageBox.Show("Fail");
}
}

You can use the TimeOut property on HttpWebRequest

Consider switching to asynchronous downloading of the content. You will stop blocking UI thread and will be able to handle multiple requests easily. You will be able to increase timeout significantly without impact on UI, and can decide upon receiving response if you still want to fetch data.

Socket connection issues

My answer:
After getting annoyed, I have found a solution. The problem was indeed C# either C#'s garbage collector or C#'s multithreading, it probably thought the object was no longer needed within THAT thread, and deleted it. The solution was found as follows:
I implemented the ClientThread into the Server class, passing the Client object as a parameters, this minor change made it work. Thank you for all your responses, if anyone in the future has this problem maybe it wasn't C#'s garbage collector. But C# mutithreading OR networking must be done within the same class. I kept my client class and just made the thread object run the function within the Server class.
If anyone can figure out what my problem was, feel free to comment so I can expand my little knowledge of C#'s memory management.
Thanks again to all the people who attempted to help me in this thread.
Original Question
I'm a C++ programmer so I'm used to managing memory myself, and I'm really not sure how to solve this problem.
For instance in C++:
while(true)
{
void* ptr = new char[1000];
}
This would be an obvious memory leaking program, so I need to go ahead and clean it up with:
delete ptr;
But there are cases when I want to create memory for use in a different thread and I DO NOT WANT IT DELETED AFTER THE LOOP.
while(true)
{
socket.Accept(new Client());
}
//////////Client Constructor////////////
Client()
{
clientThread.Start();
}
This snippet is basically what I want to do in C#, but my client connects then disconnects immediately, I'm assuming this is because at the end of the while loop my new Client() is being deleted by our favorite Garbage Collector.
So my question is, how do I get around this and make it NOT delete my object.
Many have replied saying various things about having other links to it in the code. I forgot to mention that I also save the new client in a list of clients located globally
List<Client> clients;
//inside loop
clients.Add(new Client(socket.Accept()));
Ok because I'm unsure if I'm missing more information here is the ACTUAL code snippet
// Server class
internal Socket socket { get; set; }
internal Thread thread { get; set; }
internal List<Client> clients { get; set; }
internal void Init()
{
socket = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp);
thread = new Thread(AcceptThread);
}
internal void Start(int port,List<Client> clients)
{
var ipep = new IPEndPoint(IPAddress.Any, port);
this.socket.Bind(ipep);
this.socket.Listen(10);
this.clients = clients;
this.thread.Start();
}
internal void End()
{
socket.Close();
thread.Abort();
}
internal void AcceptThread()
{
int ids = 0;
while (true)
{
Client client = new Client();
client.Init(socket.Accept());
client.clientid = ids++;
client.Start();
clients.Add(client);
}
}
// Client class
public class Client
{
.....
#region Base Code
internal void Init(Socket socket)
{
this.socket = socket;
this.status = new SocketStatus();
this.thread = new Thread(ClientThread);
this.stream = new Stream();
}
internal void Start()
{
thread.Start();
}
internal void Close()
{
socket.Close();
status = SocketStatus.Null;
thread.Abort();
}
internal void ClientThread()
{
try
{
while (true)
{
byte[] data = new byte[1];
int rec = socket.Receive(data);
if (rec == 0)
throw new Exception();
else
stream.write(data[0]);
}
}
catch(Exception e)
{
Close();
}
}
#endregion
}
I thank you for all your replies.

That's not how it works at all. If there exists any reference to the instance of Client you created, it is not garbage-collected. This doesn't just apply to your own code, either. Therefore, if GCing is indeed the source of your issue, you never could have accessed it in the first place!
If you weren't intending to access it, you can hold on to them anyway by putting them in a List. However, I believe that once you actually use them in the other thread you're talking about, your problems will go away.

I've been out of the c# game for a while but I don't see anything immediately wrong there. Garbage collection shouldn't kick in until objects are actually not referenced anymore. if your socket.Accept() doesn't keep a reference, perhaps you could do this manually:
var clients = new List<Client>();
while(true)
{
client = new Client();
clients.Add(client);
socket.Accept(client);
}
////////// Client Constructor ////////////
Client()
{
clientThread.Start();
}

From MSDN:
If no data is available for reading, the Receive method will block until data is
available, unless a time-out value was set by using
Socket.ReceiveTimeout. If the time-out value was exceeded, the Receive
call will throw a SocketException. If you are in non-blocking mode,
and there is no data available in the in the protocol stack buffer,
the Receive method will complete immediately and throw a
SocketException. You can use the Available property to determine if
data is available for reading. When Available is non-zero, retry the
receive operation.
If you are using a connection-oriented Socket, the Receive method will
read as much data as is available, up to the size of the buffer. If
the remote host shuts down the Socket connection with the Shutdown
method, and all available data has been received, the Receive method
will complete immediately and return zero bytes.
This appears to be the only way to get a 0 return value from the Receive method, and not get an exception, so it would appear that whatever is on the other end is closing the connection.

The garbage collector only deletes resources that aren't reachable through any reference in your program. As long as you still have a variable that refers to the object, it'll continue to exist.

Webclient's DownloadStringCompleted event handler never called

I'm trying to make an asynchronous HTTP GET request using Webclient, however, the registered callback never gets called. I've also tried with the sync one, and it worked fine. What am I doing wrong?
WebClient asyncWebRequest;
public AsyncWebRequest(Uri url)
{
asyncWebRequest = new WebClient();
url = new Uri("http://www.google.com/");
// string test = asyncWebRequest.DownloadString(url); // this works
asyncWebRequest.DownloadStringCompleted += new DownloadStringCompletedEventHandler(asyncWebRequest_DownloadStringCompleted);
asyncWebRequest.DownloadStringAsync(url);
}
void asyncWebRequest_DownloadStringCompleted(object sender, DownloadStringCompletedEventArgs e)
{
throw new NotImplementedException();
}

Maybe because you disposing the WebClient before it finished downloading. The code execution don't stop on asyncWebRequest.DownloadStringAsync(url); and you are disposing the WebClient object by closing the using statement.
try to dispose the WebClient on asyncWebRequest_DownloadStringCompleted.
results

The simpliest solution is to add Console.ReadKey() at the end of AsyncWebRequest(url) method. This way asyncWebRequest.DownloadStringAsync(url) will be able to retrieve data.

Calling a aspx page from windows service - Problem

I have a windows service that calls a page after a certain interval of time. The page in turn creates some reports.
The problem is that the service stops doing anything after 2-3 calls. as in it calls the page for 2-3 times and then does not do any work though it shows that the service is running...i am using timers in my service..
please can someone help me with a solution here
thank you
the code:(where t1 is my timer)
protected override void OnStart(string[] args)
{
GetRecords();
t1.Elapsed += new ElapsedEventHandler(OnElapsedTime);
t1.Interval = //SomeTimeInterval
t1.Enabled = true;
t1.Start();
}
private void OnElapsedTime(object source, ElapsedEventArgs e)
{
try
{
GetRecords();
}
catch (Exception ex)
{
EventLog.WriteEntry(ex.Message);
}
}
public void GetRecords()
{
try
{
string ConnectionString = //Connection string from web.config
WebRequest Request = HttpWebRequest.Create(ConnectionString);
Request.Timeout = 100000000;
HttpWebResponse Response = (HttpWebResponse)Request.GetResponse();
}
catch (Exception ex)
{
}
}

Well, what does the code look like? WebClient is the easiest way to query a page:
string result;
using (WebClient client = new WebClient()) {
result = client.DownloadString(address);
}
// do something with `result`
The timer code might also be glitchy if it is stalling...

It's possible that HttpWebRequest will restrict the number of concurrent HTTP requests to a specific page or server, as is generally proper HTTP client practice.
The fact that you're not properly disposing your objects most likely means you are maintaining 2 or 3 connections to a specific page, each with large timout value, and HttpWebRequest is queueing or ignoring your requests until the first few complete (die from a client or server timeout, most likely the server in this case).
Add a 'finally' clause and dispose of your objects properly!

I think you're missing something about disposing your objects like StreamReader, WebRequest, etc.. You should dispose your expensive objects after using them.

possibly the way you are requesting athe page is throwing an unnhandled exception which leaves the service in an inoperable state.
Yes, we need code.

Marc's advice worked for me, in the context of a service
Using WebClient worked reliably, where WebRequest timed out.
#jscharf explanation looks as good as any to me.