I have a download program in windows forms which downloads the html source code of a bunch of URLs and extract further URLs and downloads that stuff too. I am using
using(WebClient client = new WebClient())
{
// Do download stuff here...
}
but .NET Mem profiler shows a severe memory leak because a lot of object are garbage collected but not disposed OR disposed but not garbage collected. Sound weird to me but my program has a memory leak.
To get a feel of my program consider this code please:
private void PreprocessURLs(List<string> URLs)
{
using(WebClient client = new WebClient())
{
// Download first batch of html source code and put in a List<string> property
}
}
private void ProcessURLs(List<string> URLs)
{
using(WebClient client = new WebClient())
{
// Download more stuff and save them
}
}
and so on. This is not the actual program but what I want to show you is, how do I create a "globally" available WebClient object/instance so I do not create a bunch off client objects? I want to reuse it. I suspect this to be the cause (maybe among others) to cause the memory leak. Maybe I am wrong because I have never created a program with a task where it could have a memory leak.
EDIT: After suggestion of tlemster I created a static instance of WebClient but that causes an I/O error because concurrent I/O are not supported (DownloadFileAsync()).
EDIT1: Here is my download method which does the main work and blocking my UI thread by doing so. Thus I believe the mem leak comes from this function.
private void Download()
{
var stopwatch = new Stopwatch();
string bla;
string chapterName;
string bla1;
string chapterNumber;
List<Tuple<string, int>> temp = new List<Tuple<string, int>>();
// Contains all URLs from preprocessing
foreach (var chapter in Chapters)
{
bla = chapter.Item2;
chapterName = ReplaceSpecialChars(bla);
bla1 = chapter.Item3;
chapterNumber = ReplaceSpecialChars(bla1);
// Skip this chapter if it already exists based on chapter name
if (Directory.Exists(string.Format("{0}/{1} - {2}", chapter.Item4, chapterNumber, chapterName)))
{
continue;
}
else
{
Directory.CreateDirectory(string.Format("{0}/{1} - {2}", chapter.Item4, chapterNumber, chapterName));
}
// Process each chapter and extract other URLs
foreach (var item in GetPagesLink(chapter.Item1))
{
// Add the extracted URLs to a list for download further down
temp.Add(new Tuple<string, int>(GetImageLink(item.Item1), item.Item2));
}
stopwatch.Start();
// The download of the files I want after processing it two times
foreach (var img in temp)
{
// A static WebClient does not work because DownloadFileAsync() does not support concurrent I/O
// Atm should not matter because this whole thing is not async anyway...
using (WebClient webClient = new WebClient())
{
webClient.DownloadProgressChanged += new DownloadProgressChangedEventHandler(ProgressChanged);
webClient.DownloadFileAsync(new Uri(img.Item1), string.Format("{0}/{1} - {2}/{3}.jpg", chapter.Item4, chapterNumber, chapterName, img.Item2)); // TODO: Find image type and replace hardcoded jpg
System.Threading.Thread.Sleep(150);
}
}
stopwatch.Stop();
temp.Clear();
txtDebug.AppendText("Finished chapter " + chapter.Item3 + " : " + stopwatch.ElapsedMilliseconds + "\r\n");
stopwatch.Reset();
}
Chapters.Clear();
}
Store the created WebClient in a static field, which keeps the instance alive as long as your program is running or until you get rid of it manually.
Related
I am developing a game in which I need to retrieve data from a stream (that hasn't end).
I have a class called StreamingChannel which creates the streaming channel
public StreamingChannel (){
//stuff to set the stream
webResponse = (HttpWebResponse) webRequest.GetResponse();
responseStream = new StreamReader (webResponse.GetResponseStream (), encode);
}
and to read from it i have this method
public string Read(){
try{
string jsonText = responseStream.ReadLine();
return jsonText;
}catch(ObjectDisposedException){
return null;
}
}
I perform the reading every tot secs with an InvokeRepeating and I do that for the whole game.
It works great except that for the fact that my stream lasts for about a couple of minute. After that it throws an ObjectDisposedException.
At first I wanted to restore the connection but I didn't manage to do that without reinstantiate the whole connection. In this case the problem is that the game lags for about a seconds.
So how can I tell the StreamReader that has to leave open the channel?
ps I cannot use the constructor
public StreamReader(
Stream stream,
Encoding encoding,
bool detectEncodingFromByteOrderMarks,
int bufferSize,
bool leaveOpen)
because it has been introduced in the version 4.5 of the .NET Framework, and Unity doesn't support that.
A streaming API expects your code to pull data out of Stream pretty aggressively. You may not be able to wait for Unity to schedule your ReadLine method. I think a better model is to use a separate thread to pull data as fast as possible from the Stream and store it in a buffer. (I think this is possible in Unity.) Then you can pull the stream data out of your buffer in the standard Unity thread without worrying about the pull rate. A ConcurrentQueue would be a great buffer, but Unity doesn't support it, so I've used a locked List.
Using a separate thread also allows you to restart after failures without blocking the main game.
using System.Collections.Generic;
using System.Threading;
public class StreamingChannel
{
private readonly List<string> backgroundLinesList;
private readonly object listLock = new object();
private Thread streamReaderThread;
public StreamingChannel()
{
streamReaderThread = new Thread(this.ReadWebStream);
streamReaderThread.Start();
}
public List<string> Read()
{
if (!streamReaderThread.IsAlive)
{
streamReaderThread = new Thread(this.ReadWebStream);
streamReaderThread.Start();
}
List<string> lines = null;
lock (listLock)
{
if (backgroundLinesList != null)
{
lines = backgroundLinesList;
backgroundLinesList = null;
}
}
return lines;
}
private void ReadWebStream()
{
try
{
//stuff to set the stream
HttpWebRequest webRequest;
HttpWebResponse webResponse = (HttpWebResponse)webRequest.GetResponse();
StreamReader responseStream = new StreamReader(webResponse.GetResponseStream(), encode);
while (!responseStream.EndOfStream)
{
var line = responseStream.ReadLine()
lock (listLock)
{
if (backgroundLinesList == null)
{
backgroundLinesList = new List<string>();
}
backgroundLinesList.Add(line);
}
}
log.Debug("Stream closed");
}
catch (Exception e)
{
log.Debug("WebStream thread failure: " + e + " Stack: " + e.StackTrace);
}
}
}
Writing Stringbuilder to file asynchronously. This code takes control of a file, writes a stream to it and releases it. It deals with requests from asynchronous operations, which may come in at any time.
The FilePath is set per class instance (so the lock Object is per instance), but there is potential for conflict since these classes may share FilePaths. That sort of conflict, as well as all other types from outside the class instance, would be dealt with retries.
Is this code suitable for its purpose? Is there a better way to handle this that means less (or no) reliance on the catch and retry mechanic?
Also how do I avoid catching exceptions that have occurred for other reasons.
public string Filepath { get; set; }
private Object locker = new Object();
public async Task WriteToFile(StringBuilder text)
{
int timeOut = 100;
Stopwatch stopwatch = new Stopwatch();
stopwatch.Start();
while (true)
{
try
{
//Wait for resource to be free
lock (locker)
{
using (FileStream file = new FileStream(Filepath, FileMode.Append, FileAccess.Write, FileShare.Read))
using (StreamWriter writer = new StreamWriter(file, Encoding.Unicode))
{
writer.Write(text.ToString());
}
}
break;
}
catch
{
//File not available, conflict with other class instances or application
}
if (stopwatch.ElapsedMilliseconds > timeOut)
{
//Give up.
break;
}
//Wait and Retry
await Task.Delay(5);
}
stopwatch.Stop();
}
How you approach this is going to depend a lot on how frequently you're writing. If you're writing a relatively small amount of text fairly infrequently, then just use a static lock and be done with it. That might be your best bet in any case because the disk drive can only satisfy one request at a time. Assuming that all of your output files are on the same drive (perhaps not a fair assumption, but bear with me), there's not going to be much difference between locking at the application level and the lock that's done at the OS level.
So if you declare locker as:
static object locker = new object();
You'll be assured that there are no conflicts with other threads in your program.
If you want this thing to be bulletproof (or at least reasonably so), you can't get away from catching exceptions. Bad things can happen. You must handle exceptions in some way. What you do in the face of error is something else entirely. You'll probably want to retry a few times if the file is locked. If you get a bad path or filename error or disk full or any of a number of other errors, you probably want to kill the program. Again, that's up to you. But you can't avoid exception handling unless you're okay with the program crashing on error.
By the way, you can replace all of this code:
using (FileStream file = new FileStream(Filepath, FileMode.Append, FileAccess.Write, FileShare.Read))
using (StreamWriter writer = new StreamWriter(file, Encoding.Unicode))
{
writer.Write(text.ToString());
}
With a single call:
File.AppendAllText(Filepath, text.ToString());
Assuming you're using .NET 4.0 or later. See File.AppendAllText.
One other way you could handle this is to have the threads write their messages to a queue, and have a dedicated thread that services that queue. You'd have a BlockingCollection of messages and associated file paths. For example:
class LogMessage
{
public string Filepath { get; set; }
public string Text { get; set; }
}
BlockingCollection<LogMessage> _logMessages = new BlockingCollection<LogMessage>();
Your threads write data to that queue:
_logMessages.Add(new LogMessage("foo.log", "this is a test"));
You start a long-running background task that does nothing but service that queue:
foreach (var msg in _logMessages.GetConsumingEnumerable())
{
// of course you'll want your exception handling in here
File.AppendAllText(msg.Filepath, msg.Text);
}
Your potential risk here is that threads create messages too fast, causing the queue to grow without bound because the consumer can't keep up. Whether that's a real risk in your application is something only you can say. If you think it might be a risk, you can put a maximum size (number of entries) on the queue so that if the queue size exceeds that value, producers will wait until there is room in the queue before they can add.
You could also use ReaderWriterLock, it is considered to be more 'appropriate' way to control thread safety when dealing with read write operations...
To debug my web apps (when remote debug fails) I use following ('debug.txt' end up in \bin folder on the server):
public static class LoggingExtensions
{
static ReaderWriterLock locker = new ReaderWriterLock();
public static void WriteDebug(string text)
{
try
{
locker.AcquireWriterLock(int.MaxValue);
System.IO.File.AppendAllLines(Path.Combine(Path.GetDirectoryName(System.Reflection.Assembly.GetExecutingAssembly().GetName().CodeBase).Replace("file:\\", ""), "debug.txt"), new[] { text });
}
finally
{
locker.ReleaseWriterLock();
}
}
}
Hope this saves you some time.
I am working on a logging system for a web application which logs a sequence of events in a dictionary object before sending it to my logging object using Task.Factory.StartNew(() => iLogEventSave()). The logger seemed to work fine, but in some instances some events were not being saved properly so I used the lock() statement to correct the issue. This seemed to do the trick, but the application's performance has dramatically decreased by doing this. How can I have the UI/Page render without having to wait for the Tasks to finish their job?
Below is the code
private static readonly object Locker = new object();
public void iLogEventSave(object state)
{
XmlDocument doc = new XmlDocument();
IDictionary<string, string> EventDetails = (IDictionary<string, string>)state;
string logFile = "";
if(ConfigurationManager.AppSettings["Log_File_Path"].ToString() =="")
{
logFile = HttpRuntime.AppDomainAppPath + "Logs\\" + DateTime.Now.ToString("yyyy_MM_dd") + ".txt";
}
else
{
logFile = ConfigurationManager.AppSettings["Log_File_Path"].ToString() + DateTime.Now.ToString("yyyy_MM_dd") + ".txt";
}
lock (Locker)
{
if (File.Exists(logFile))
{
doc.Load(logFile);
}
else
{
var root = doc.CreateElement("Log");
doc.AppendChild(root);
}
var el = (XmlElement)doc.DocumentElement.AppendChild(doc.CreateElement("Event"));
foreach (KeyValuePair<string, string> item in EventDetails)
{
XmlElement Desc = doc.CreateElement("Details");
Desc.SetAttribute(item.Key.ToString(), item.Value);
el.AppendChild(Desc);
}
doc.Save(logFile);
}
}
If your log did not save several events while being executed asynchronously, you have an unhandled error that you did not address. Considering that you're using a file, I'm going to go out on a limb and say that it failed because two threads were competing for access to the same log file and the first thread to grab it locked the other one out. This is why your lock would now work, it prevents other threads from trying to grab the file.
But logging to a file means that you've effectively restricted yourself to one thread at a time and dealing with the entire file as it grows. You have to load more and more, append more and more, and locking the thread means that the more threads are waiting to log the events, the higher your overhead. All this could certainly add up to a decrease in performance.
May I recommend using a database table to log events? File I/O is very expensive, resource and time-wise. Databases have less overhead and far better throughput by comparison in these very scenarios.
Recently, I found that Windows Phone Background Transfer Service seems to have memory leak issue.
Every background transfer you add will take a memory space, which cannot be removed by GC forever.
I already read through http://msdn.microsoft.com/en-us/library/windowsphone/develop/hh202959(v=vs.105).aspx , and still have no idea where the memory leak comes from.
What I test is very simple, add a background transfer request to the BackgroundTransferService and when that request is complete, remove it from BackgroundTransferService and add another one. If I keep doing it, I'll see the memory grows even when GC.collect being called every sec. Please download the testing code in http://hdtp.synology.me/BTS.zip and you'll know what I'm saying. The following is the testing code summary.
private int _transferCount = 1000;
private void CreateTask()
{
if (--_transferCount < 0)
{
MessageBox.Show("End");
return;
}
// Get the URI of the file to be transferred from the Tag property
// of the button that was clicked.
//string transferFileName = ((Button)sender).Tag as string;
string transferFileName = "http://hdtp.synology.me/a.jpg";
Uri transferUri = new Uri(Uri.EscapeUriString(transferFileName + "?ranNum=" + _transferCount), UriKind.RelativeOrAbsolute);
// Create the new transfer request, passing in the URI of the file to
// be transferred.
BackgroundTransferRequest transferRequest = new BackgroundTransferRequest(transferUri);
// Set the transfer method. GET and POST are supported.
transferRequest.Method = "GET";
// Get the file name from the end of the transfer Uri and create a local Uri
// in the "transfers" directory in isolated storage.
string downloadFile = transferFileName.Substring(transferFileName.LastIndexOf("/") + 1);
Uri downloadUri = new Uri("shared/transfers/" + downloadFile, UriKind.RelativeOrAbsolute);
transferRequest.DownloadLocation = downloadUri;
// Pass custom data with the Tag property. This value cannot be more than 4000 characters.
// In this example, the friendly name for the file is passed.
transferRequest.Tag = downloadFile;
// Add the transfer request using the BackgroundTransferService. Do this in
// a try block in case an exception is thrown.
try
{
BackgroundTransferService.Add(transferRequest);
}
catch (InvalidOperationException ex)
{
// TBD - update when exceptions are finalized
MessageBox.Show("Unable to add background transfer request. " + ex.Message);
}
catch (Exception)
{
MessageBox.Show("Unable to add background transfer request.");
}
InitialTansferStatusCheck();
}
private void InitialTansferStatusCheck()
{
UpdateRequestsList();
foreach (var transfer in transferRequests)
{
transfer.TransferStatusChanged += new EventHandler<BackgroundTransferEventArgs>(transfer_TransferStatusChanged);
ProcessTransfer(transfer);
}
}
private void transfer_TransferStatusChanged(object sender, BackgroundTransferEventArgs e)
{
ProcessTransfer(e.Request);
}
private void UpdateRequestsList()
{
// The Requests property returns new references, so make sure that
// you dispose of the old references to avoid memory leaks.
if (transferRequests != null)
{
foreach (var request in transferRequests)
{
request.Dispose();
}
}
transferRequests = BackgroundTransferService.Requests;
}
private void ProcessTransfer(BackgroundTransferRequest transfer)
{
switch (transfer.TransferStatus)
{
case TransferStatus.Completed:
// If the status code of a completed transfer is 200 or 206, the
// transfer was successful
if (transfer.StatusCode == 200 || transfer.StatusCode == 206)
{
// Remove the transfer request in order to make room in the
// queue for more transfers. Transfers are not automatically
// removed by the system.
RemoveTransferRequest(transfer.RequestId);
// In this example, the downloaded file is moved into the root
// Isolated Storage directory
using (IsolatedStorageFile isoStore = IsolatedStorageFile.GetUserStoreForApplication())
{
string filename = transfer.Tag;
if (isoStore.FileExists(filename))
{
isoStore.DeleteFile(filename);
}
isoStore.MoveFile(transfer.DownloadLocation.OriginalString, filename);
}
CreateTask();
}
else
{
// This is where you can handle whatever error is indicated by the
// StatusCode and then remove the transfer from the queue.
RemoveTransferRequest(transfer.RequestId);
if (transfer.TransferError != null)
{
// Handle TransferError, if there is one.
}
}
break;
}
}
private void RemoveTransferRequest(string transferID)
{
// Use Find to retrieve the transfer request with the specified ID.
BackgroundTransferRequest transferToRemove = BackgroundTransferService.Find(transferID);
// try to remove the transfer from the background transfer service.
try
{
BackgroundTransferService.Remove(transferToRemove);
}
catch (Exception ex)
{
}
}
Another few questions, according to the documentation above, we'll get the new instance from BackgroundTransferService.Requests every time, but if I called GetHashCode(), I get the same hash code every time and the hash code is even the same with the one I newed and added into BackgroundTransferService. So is it because MS override the GetHashCode method of BackgroundTransferRequest? or I misunderstand something. But in the sample code above, I did not use BackgroundTransferService.Requests to get any instance, the memory still keep growing.
Please tell me what I do wrong or any workaround, thanks...
Sorry if the title is not clear or correct, dont know what title should i put. Please correct if wrong.
I have this code to download images from IP camera and it can download the images.The problem is how can i do the images downloading process at the same time for all cameras if i have two or more cameras?
private void GetImage()
{
string IP1 = "example.IPcam1.com:81/snapshot.cgi;
string IP2 = "example.IPcam2.com:81/snapshot.cgi;
.
.
.
string IPn = "example.IPcamn.com:81/snapshot.cgi";
for (int i = 0; i < 10; i++)
{
string ImagePath = Server.MapPath("~\\Videos\\liveRecording2\\") + string.Format("{0}", i, i + 1) + ".jpeg";
string sourceURL = ip;
WebRequest req = (WebRequest)WebRequest.Create(sourceURL);
req.Credentials = new NetworkCredential("user", "password");
WebResponse resp = req.GetResponse();
Stream stream = resp.GetResponseStream();
Bitmap bmp = (Bitmap)Bitmap.FromStream(stream);
bmp.Save(ImagePath);
}
}
You should not run long-running code like that from an ASP.NET application. They are meant to simply respond to requests.
You should place this code in a service (Windows Services are easy), and control the service through a WCF service running inside of it.
You're also going to get into trouble because you don't have your WebResponse and Stream in using blocks.
There are several methods that will depend on how you want to report feedback to the user. It all comes down to multi-threading.
Here is one example, using the ThreadPool. Note that this is missing a bunch of error checking throughout... It is here as an example of how to use the ThreadPool, not as a robust application:
private Dictionary<String, String> _cameras = new Dictionary<String, String> {
{ "http://example.IPcam1.com:81/snapshot.cgi", "/some/path/for/image1.jpg" },
{ "http://example.IPcam2.com:81/snapshot.cgi", "/some/other/path/image2.jpg" },
};
public void DoImageDownload() {
int finished = 0;
foreach (KeyValuePair<String, String> pair in _cameras) {
ThreadPool.QueueUserWorkItem(delegate {
BeginDownload(pair.Key, pair.Value);
finished++;
});
}
while (finished < _cameras.Count) {
Thread.Sleep(1000); // sleep 1 second
}
}
private void BeginDownload(String src, String dest) {
WebRequest req = (WebRequest) WebRequest.Create(src);
req.Credentials = new NetworkCredential("username", "password");
WebResponse resp = req.GetResponse();
Stream input = resp.GetResponseStream();
using (Stream output = File.Create(dest)) {
input.CopyTo(output);
}
}
This example simply takes the work you are doing in the for loop and off-loads it to the thread pool for processing. The DoImageDownload method will return very quickly, as it is not doing much actual work.
Depending on your use case, you may need a mechanism to wait for the images to finish downloading from the caller of DoImageDownload. A common approach would be the use of event callbacks at the end of BeginDownload to notify when the download is complete. I have put a simple while loop here that will wait until the images finish... Of course, this needs error checking in case images are missing or the delegate never returns.
Be sure to add your error checking throughout... Hopefully this gives you a place to start.