I have several large image files that I need to convert to different image formats. I am using the following code to do this:
using (Image img =new Bitmap(inputImageName))
{
img.Save(outputImageName, imageFormat);
}
It do the conversation, but also since the images are big, it generate outofmemory exception. I read several articles about how to overcome fragmentation of LOH, but I cannot use any of them in this case.
What Can I do?
The images are around 100MByte and it happens after opening 3 or 4 images.
The question you need to ask here is "Do I have to do this in .NET and / or C#"
While I can see why that to many folks a flexible language like C# may be the answer to performing many tasks, I also have to say that "When All you have is a hammer, every problem looks like a Nail"
If this is a one time conversion, and you only need to use them for one project, then my advice to you is to use a stand alone tool better suited for the Job.
There are tons out there ranging from paid applications like:
AcdSee Photo Manager
http://www.acdsee.com/
to free tools such as
Ifran View and it's image conversion functions
http://www.bleepingcomputer.com/forums/topic50519.html
If command line scripting is your game, then use a tool set such as ImageMagik:
http://www.imagemagick.org/script/index.php
Iamage Magik also has .NET bindings so it's functionality can be used from within .NET projects as well as many others if your project requires that you convert these on the fly in your program code.
http://www.imagemagick.org/script/api.php
For scenarios like this one there really is no reason to re-invent the wheel, this problem has been solved so many times before that it really is not a decision you should have to make.
Have a look at WPF image manipulation classes, which do not use GDI, according to this article. The following source code is a good starting point :
public static void Resize(string input, string output)
{
using (var inputStream = File.OpenRead(input))
{
var photoDecoder = BitmapDecoder.Create(inputStream, BitmapCreateOptions.PreservePixelFormat, BitmapCacheOption.None);
var frame = photoDecoder.Frames[0];
using (var ouputStream = File.Create(output))
{
var targetEncoder = new PngBitmapEncoder();
targetEncoder.Frames.Add(frame);
targetEncoder.Save(ouputStream);
}
}
}
It doesn't have anything to do with the Large Object Heap. The Bitmap class is a managed wrapper for GDI+ and GDI+ is a big chunk of unmanaged code. It allocates pixel data buffers in unmanaged memory. The amount of managed memory your code uses is very small.
A 100 megabyte image file doesn't say much about how much unmanaged memory is required. In all likelihood it is a compressed image format, like JPEG or PNG. Which does require a great deal more unmanaged memory after it is uncompressed. So you could easily end up needing hundreds of megabytes.
And that's a problem when you run your code on a 32-bit operating system or you've selected the x86 as your EXE's platform target setting (the default on VS2010 and up). Your program allocates virtual memory from the holes that are available between the chunks of code and data that are already loaded. GDI+ requires a contiguous chunk of memory to load the pixel data and that can be hard to come by when your program has lots of available virtual memory but it is spread among many holes. An additional problem is address space fragmentation, loading a DLL or allocating memory could cut the size of a large hole in two.
There's a simple fix available for this problem, run your code on a 64-bit operating system. It has gobs of virtual memory available, big holes. The odds that you can do anything about your current problem on a 32-bit operating system are slim to none. You have no direct control over the memory manager.
Related
I decided to benchmark reading an image in C#, and in C++, to decide which language to use in a project i'm thinking about making for myself.
I expected the benchmarks to be extremely close with C++ maybe pushing ahead slightly.
The C# code takes about 300ms each run (I ran each test 100 times), where the C++ code takes about 1.5ms.
So is my C# code wrong? Am I benchmarking it badly? Or is it really just this much slower?
Here's the c# code I used:
Stopwatch watch = new Stopwatch();
watch.Start();
Image image = Image.FromFile(imagePath);
watch.Stop();
Console.WriteLine("DEBUG: {0}", watch.ElapsedMilliseconds);
And the C++ code pretty much boiled down to this:
QueryPerformanceCounter(&start);
Image * img = Image::FromFile(imagePath);
QueryPerformanceCounter(&stop);
delete img;
return (stop.QuadPart - start.QuadPart) * 1000.0 / freq.QuadPart;
Regardless of which language, they need to end up in an Image object, as it provides the functionality i'm going to need.
=======================================================================
As xanatos pointed out in the comments, the Image.FromFile does do checking.
More specifically, this:
num = SafeNativeMethods.Gdip.GdipImageForceValidation(new HandleRef(null, zero));
if (num != 0)
{
SafeNativeMethods.Gdip.GdipDisposeImage(new HandleRef(null, zero));
throw SafeNativeMethods.Gdip.StatusException(num);
}
Using Image.FromStream() instead, you can avoid this.
What i'm wondering is, if you do avoid this and try to load an invalid image file it throws an OutOfMemory exception.
And in C++, you don't do checking like this. So how important is this checking? Can anyone give me a situation where it would be bad to avoid this?
Yes, your benchmark is flawed. The problem is that you forgot to actually do something with the bitmap. Like paint it.
GDI+ heavily optimizes the loading of an image. Very similar to the way .NET optimizes loading an assembly. It does the bare things necessary, it reads the header of the file to retrieve essential properties. Format, Width, Height, Dpi. Then it creates a memory-mapped file to create a mapping to the pixel data in the file. But doesn't actually read the pixel data.
Now the difference comes into play. System.Drawing.Image next actually reads the pixel data. That causes page faults, the operating system now reads the file and copies the pixel data into RAM. Highly desirable, if there's anything wrong with the file then you'll get an exception at the FromFile() call instead of some time later, typically when your program draws the image and is buried in framework code you didn't write. Your bench mark for the C# code times the creation of the mmf plus the reading of the pixel data.
The C++ program is always going to have to pay for reading the pixel data too. But you didn't measure that, you only measured the cost of creating the MMF.
Few Points which I know
There is a thing called as CLR. If the said c++ framework(It seems Qt) used The system calls which not depends on .net framework, then obviously it will run fast.
Regarding c# => It is possible to load that assembly before you call it in your code. If you do so,then You can find it's fastness in good scale.
If you use windows platform, then MS wont reduce the execution speed of its own language unless otherwise there is some necessity.
I was wondering if it is possible for C# to write objects to the pagefile.
I already know that a virtual machine for a .NET application is limited to allow one object to only use up 2 GB of ram and will run out of Memory long before that - even on 64-bit version of Windows.
However I need to be able to load a huge amount of strings (not 1 big one but many small ones) and was wondering if it is possible to load them and let them be written to the swap space until they are needed again (unfortunately it is not possible for me to prevent the loading of all the strings because it is forced by an application calling the code I am working on).
To prototype it I tried out to see if the following program will run out of memory as well (which it will), even though it does not try to allocate one huge string:
public static void Main()
{
ICollection<StringBuilder> builders = new LinkedList<StringBuilder>();
double used_ram = 2*1024*1024*1024L;
int blocksize = 12800000;
int blocks = (int) (used_ram/blocksize);
for (int i = 1; i < blocks ; i++)
{
StringBuilder sb = new StringBuilder(blocksize/2);
builders.Add(sb);
}
Console.ReadLine();
}
I was hoping that the strings would be written to the swap space and was wondering if there is any way I can force the application to do just that.
EDIT: Changed the use of swap file to pagefile (thanks for the clarifications).
Ok so to enhance my question: if the only limitation of the runtime is that ONE object can not allocate more than 2 gb of memory - why is the above code running out of memory - because the list goes over the 2 gb of memory by holding reference to all the string builders?
Short answer: no you can't.
Using the swap is not something applications do: the memory management system of the operating system is responsible of that.
If you need to load more than 2GB of data in your process in one time (and not retrieve data from disk per chunks as necessary), then you have a serious design problem.
EDIT:
As you're already using a 64-bit OS, make sure you're compiling your application for x64 platforms (or AnyCPU) and your application is not running as a 32-bit process using WOW64.
I think your question amount to:
Can a 32 bit .NET application address more than 2GB of memory.
Yes, it is possible to increase the amount of memory that applications can address by adjusting the split between the memory allocated to user applications and the operating system but I would not recommend it in most cases as it can cause subtle and not so subtle problems with O/S functions, such as networking. For details on the /3GB and /USERVA switches please see this article. This will allow your application to address 3GB (minus overhead) of memory rather than 2GB.
Tools at your disposal are:
Switching to 64 bit .NET.
Using memory mapped files.
Writing your own paging system.
Using some backing store e.g. file system, database.
You can utilize all of the windows memory management functions and allocate as much memory as the system will allow, but what you are doing screams for some type of flush to disk system. Even if you could get windows to allocate that large an object you performance would be terrible. There are many out of the box system (called databases) which allow you to put large amounts of data in them and access them it very short order.
You may be doing something wrong. I've written .net applications that use way over 2 GB of memory. If you are running 64 bit windows, there's no reason your application shouldn't be able to use much more memory.
I recommend you should write your own Cache-Class. For an example have a look here
I think what cames closes to your approach on the .Net application level would be a MemoryMappedFile.
But this would mean the application wouldn't try to get all the strings in one go.
I have written a small WinForm application in C#. The EXE that is made is 74 Kb and all together with the resources and all it sizes 179 Kb. But when I run it, it takes 9.1 MBs in memory according to Task Manager.
So my question is:
Why is it happening?
What can I do to reduce the size of this?
If the size could be reduced how much reduction is possible?
Firstly, using Task Manager to determine memory usage is fraught with peril. Have a read of this article to gain a clearer understanding of it.
What other libraries are you referencing?
What files are you loading, into streams or any other way?
What resources do you request from the Operating System via P/Invoke
How many instances of each form do you load?
How many variables do you create, and of what size?
For example:
// Will take up a lot of memory!
var x = new byte[int.MaxValue];
All that said, 9.1Mb isn't really a lot (less than 0.5% of memory on a machine spec'd with 2Gb of RAM) and more importantly, does it actually matter that your application's using 9.1Mb of RAM, or are you wasting your time investigating? Remember, your time's valuable. Would your end users rather the time was spent on something else? =)
Size of executable and memory usage are two completely separate notions. For example this simple program:
class Program
{
static void Main()
{
var b = new byte[int.MaxValue];
}
}
is only 4KB but it uses all of the available RAM on your computer and crash. This is to demonstrate you that you could have an extremely simple application but depending on what it is doing it could consume lots of memory. So what is your application doing?
The memory usage of a program is not 100% related to the size of it's binary or resources.
It depends on what your program does. For example if you create something like this:
List<int> list = new List<int>();
for (i=1; i<100000; i++) list.Add(i);
It will take as much memory as it need to store int's plus it's object overhead.
And it depends on what usings you've used.
You've tagged your post with winforms - I assume you've got a gui app. Gui's memory usage depend on used controls and their gui style (e.g. animations, hover effects ...)
And .NET has a garbage collector which will free unused memory during runtime.
I have an out of memory exception using C# when reading in a massive file
I need to change the code but for the time being can I increase the heap size (like I would in Java) as a shaort term fix?
.Net does that automatically.
Looks like you have reached the limit of the memory one .Net process can use for its objects (on 32 bit machine this is 2 standard or 3GB by using the /3GB boot switch. Credits to Leppie & Eric Lippert for the info).
Rethink your algorithm, or perhaps a change to a 64 bit machine might help.
No, this is not possible. This problem might occur because you're running on a 32-bit OS and memory is too fragmented. Try not to load the whole file into memory (for instance, by processing line by line) or, when you really need to load it completely, by loading it in multiple, smaller parts.
No you can't see my answer here: Is there any way to pre-allocate the heap in the .NET runtime, like -Xmx/-Xms in Java?
For reading large files it is usually preferable to stream them from disk, reading them in chunks and dealing with them a piece at a time instead of loading the whole thing up front.
As others have already pointed out, this is not possible. The .NET runtime handles heap allocations on behalf of the application.
In my experience .NET applications commonly suffer from OOM when there should be plenty of memory available (or at least, so it appears). The reason for this is usually the use of huge collections such as arrays, List (which uses an array to store its data) or similar.
The problem is these types will sometimes create peaks in memory use. If these peak requests cannot be honored an OOM exception is throw. E.g. when List needs to increase its capacity it does so by allocating a new array of double the current size and then it copies all the references/values from one array to the other. Similarly operations such as ToArray makes a new copy of the array. I've also seen similar problems on big LINQ operations.
Each array is stored as contiguous memory, so to avoid OOM the runtime must be able to obtain one big chunk of memory. As the address space of the process may be fragmented due to both DLL loading and general use for the heap, this is not always possible in which case an OOM exception is thrown.
What sort of file are you dealing with ?
You might be better off using a StreamReader and yield returning the ReadLine result, if it's textual.
Sure, you'll be keeping a file-pointer around, but the worst case scenario is massively reduced.
There are similar methods for Binary files, if you're uploading a file to SQL for example, you can read a byte[] and use the Sql Pointer mechanics to write the buffer to the end of a blob.
I have some C# code in an ASP.Net application that does this:
Bitmap bmp = new Bitmap(1184, 1900);
And occasionally it throws an exception "Parameter is not valid". Now i've been googling around and apparently GDI+ is infamous for throwing random exceptions, and lots of people have had this problem, but nobody has a solution to it! I've checked the system and it has plenty of both RAM and swap space.
Now in the past if i do an 'iisreset' then the problem goes away, but it comes back in a few days. But i'm not convinced i've caused a memory leak, because as i say above there is plenty of ram+swap free.
Anyone have any solutions?
Stop using GDI+ and start using the WPF Imaging classes (.NET 3.0). These are a major cleanup of the GDI+ classes and tuned for performance. Additionally, it sets up a "bitmap chain" that allows you to easily perform multiple actions on the bitmap in an efficient manner.
Find more by reading about BitmapSource
Here's an example of starting with a blank bitmap just waiting to receive some pixels:
using System.Windows.Media.Imaging;
class Program {
public static void Main(string[] args) {
var bmp = new WriteableBitmap(1184, 1900, 96.0, 96.0, PixelFormat.Bgr32, null);
}
}
For anyone who's interested, the solution i'm going to use is the Mono.Cairo libraries from the mono C# distribution instead of using system.drawing. If i simply drag the mono.cairo.dll, libcairo-2.dll, libpng13.dll and zlib1.dll files from the windows version of mono into the same folder as my executable, then i can develop in windows using visual studio 2005 and it all works nicely.
Update - i've done the above, and stress tested the application and it all seems to run smoothly now, and uses up to 200mb less ram to boot. Very happy.
Everything I've seen to date in my context is related to memory leaks / handle leaks. I recommend you get a fresh pair of eyes to investigate your code.
What actually happens is that the image is disposed at a random point in the future, even if you've created it on the previous line of code. This may be because of a memory/handle leak (cleaning some out of my code appears to improve but not completely resolve this problem).
Because this error happens after the application has been in use for a while, sometimes using lots of memory, sometimes not, I feel the garbage collector doesn't obey the rules because of some special tweaks related to services and that is why Microsoft washes their hands of this problem.
http://blog.lavablast.com/post/2007/11/The-Mysterious-Parameter-Is-Not-Valid-Exception.aspx
You not only need enough memory, it needs to be contiguous. Over time memory becomes fragmented and it becomes harder to find big blocks. There aren't a lot of good solutions to this, aside from building up images from smaller bitmaps.
new Bitmap(x, y) pretty much just needs to allocate memory -- assuming that your program isn't corrupted in some way (is there any unsafe code that could corrupt the heap), then I would start with this allocation failing. Needing a contiguous block is how a seemingly small allocation could fail. Fragmentation of the heap is something that is usually solved with a custom allocator -- I don't think this is a good idea in IIS (or possible).
To see what error you get on out of memory, try just allocation a gigantic Bitmap as a test -- see what error it throws.
One strategy I've seen is to pre-allocate some large blocks of memory (in your case Bitmaps) and treat them as a pool (get and return them to the pool). If you only need them for a short period of time, you might be able to get away with just keeping a few in memory and sharing them.
I just got a reply from microsoft support. Apparently if you look here:
http://msdn.microsoft.com/en-us/library/system.drawing.aspx
You can see it says "Classes within the System.Drawing namespace are not supported for use within a Windows or ASP.NET service. Attempting to use these classes from within one of these application types may produce unexpected problems, such as diminished service performance and run-time exceptions."
So they're basically washing their hands of the issue.
It appears that they're admitting that this section of the .Net framework is unreliable. I'm a bit disappointed.
Next up - can anyone recommend a similar library to open a gif file, superimpose some text, and save it again?
Classes within the System.Drawing namespace are not supported for use within a Windows or ASP.NET service
For a supported alternative, see Windows Imaging Components (msdn), a native library which ironically System.Drawing is based on.