I must admit that I never understood what are the streams are all about- I always thought it's an internet thing. But now I run into a code that used a stream to load a file localy and I wonder if there is advantage for using a stream over... well the way I always loaded files:
private void loadingfromStream()
{
DirectoryInfo dirInfo = new DirectoryInfo("c:/");
FileInfo[] fileInfoArr = dirInfo.GetFiles();
FileInfo fileInfo = fileInfoArr[0];
// creating a bitmap from a stream
FileStream fileStream = fileInfo.OpenRead();
Bitmap bitmap = new Bitmap(fileStream);
Image currentPicture = (Image)bitmap
}
vs.
private void loadingUsingImageClass
{
Image currentPicture = Image.FromFile(originalPath);
}
If you know your code will be loading the data from a file, use Image.FromFile - it's obviously rather simpler code, and it's just possible that there are optimizations within the framework when it's dealing with files.
Using a stream is more flexible, but unless you need that flexibility, go with the file solution.
If you want to deal with image files, of course the second solution is better. In your first section, you have Bitmap bitmap = new Bitmap(fileStream); you know that an image file is not always Bitmap, it also can be JPEG/PNG/TIFF and so on. While Image.FromFile is quite professional to deal with image files with different extensions.
Generally speaking, FileStream is common at file issues, while Image.FromFile is more particular at image files. It depends on what kind of files you are going to deal with.
Well, a file is often treated as a stream as well. That's why the primary class to open files is called FileStream. But there's a specific operating system feature that can make dealing with image files a lot more efficient. It is called 'memory mapped files', a feature that maps the content of a file directly to memory. There's some smoke and mirrors involved, but it essentially makes the file directly available without having to read it. The memory you need to store the file data doesn't take space in the paging file.
Very efficient, you'll get it for free when you use FromFile() or the Bitmap(string) constructor for an image in the .bmp format. Loading an image from a stream tends to require twice the amount of memory, always a problem with big images.
As an a addition to Jon´s answer:
As far as I see, the two methods don't do the same thing either. The first is given you the first image in "C:\" where the second just give you a image from a path. So the added complexity in the first is not just because it is using streams.
This would be equivalent:
using (var fs = File.OpenRead(path))
using (var img = Image.FromStream(fs))
{
//...
}
and in that case, it is certainly better to just do it with Image.FromFile as Jon explained.
Related
I'm trying to figure out if there is something seriously wrong with the following code. It reads the binary from the database, stores it as a picture and associates with an object of an Animal record.
For each row (record of an animal):
byte[] ba = (byte[])x.ItemArray[1]; //reading binary from a DB row
using (MemoryStream m=new MemoryStream(ba))
{
Image i = Image.FromStream(m); //exception thrown occassionally
c.Photo = i;
listOfAnimals.Add(c);
}
First of all, with 18 pictures loaded (the JPG files have 105 Mb in total), the running app uses 2 gb of memory. With no pictures loaded, it is only 500 Mb.
Often the exception gets raised in the marked point, the source of which is System Drawing.
Could anyone help me optimize the code or tell me what the problem is? I must have used some wrong functions...
According to Image.FromStream Method
OutOfMemoryException
The stream does not have a valid image format.
Remarks
You must keep the stream open for the lifetime of the Image.
The stream is reset to zero if this method is called successively with the same stream.
For more information see: Loading an image from a stream without keeping the stream open and Returning Image using Image.FromStream
Try the following:
Create a method to convert byte[] to image
ConvertByteArrayToImage
public static Image ConvertByteArrayToImage(byte[] buffer)
{
using (MemoryStream ms = new MemoryStream(buffer))
{
return Image.FromStream(ms);
}
}
Then:
byte[] ba = (byte[])x.ItemArray[1]; //reading binary from a DB row
c.Photo = ConvertByteArrayToImage(ba);
listOfAnimals.Add(c);
Checking the documentation, a possible reason for out of memory exceptions are that the stream is not a valid image. If this is the case it should fail reliably for a given image, so check if any particular source image is causing this issue.
Another possibility should be that you simply run out of memory. Jpeg typically gets a 10:1 compression level, so 105Mib of compressed data could use +1Gib of memory. I would recommend switching to x64 if at all possible, I see be little reason not to do so today.
There could also be a memory leak, the best way to investigate this would be with a memory profiler. This might be in just about any part of your code, so it is difficult to know without profiling.
You might also need to care about memory fragmentation. Large datablocks are stored in the large object heap, and this is not automatically defragmented. So after running a while you might still have memory available, just not in any continuous block. Again, switching to x64 would mostly solve this problem.
Also, as mjwills comments, please do not store large files in the database. I just spent several hours recovering a huge database, something that would have been much faster if images where stored as files instead.
I am working on a steganography software in C#, more precisely for video files. My approach is to append the extra information at the end of a video file. However, I must read the whole video file in memory first. I used the File.ReadAllBytes() function in C# to read a video file (video around 200MB) into a byte array. Then I create a new array with the video's bytes and my data's bytes. But, this sometimes causes an OutOfMemoryException. And when it doesn't it is very slow. Is there a more efficient way to append bytes to an existing file in C# which will solve this issue? Thank you.
Open the file with FileMode.Append
var stream = new FileStream(path, FileMode.Append)
FileMode Enumeration
FileMode.Append:
Opens the file if it exists and seeks to the end of the file, or
creates a new file. This requires FileIOPermissionAccess.Append
permission. FileMode.Append can be used only in conjunction with
FileAccess.Write. Trying to seek to a position before the end of the
file throws an IOException exception, and any attempt to read fails
and throws a NotSupportedException exception.
Sure, it's easy:
using (var stream = File.Open(path, FileMode.Append))
{
stream.Write(extraData);
}
No need to read the file first.
I wouldn't class this as steganography though - that would involve making subtle changes to the video frames such that it's still a valid video and looks the same to the human eye, but the extra data is encoded within those frames so it can be extracted later.
attempt this method, I am unsure if it will yield faster results, but logically it should.
: https://stackoverflow.com/a/6862460/2835725
There's a strange problem with DotNetZip that I can't seem to find a solution to.
I've searched for a few hours now and I just can't find anything on this, so here goes.
var ms = new MemoryStream();
using (var archive = new Ionic.Zip.ZipFile()) {
foreach (var file in files) {
// string byte[]
var entry = archive.AddEntry(file.Name, file.Data);
entry.ModifiedTime = DateTime.Now.AddYears(10); // Just for testing
}
archive.Save(ms);
}
return ms.GetBuffer();
I need to add the modified time, which is rather crucial, but right now I just have a dummy timestamp.
When I open the file with WinRAR, it says "Unexpected end of archive". Each individual file has checksum 00000000, and WinRAR says "The archive is either in unknown format or damaged". I can repair it, which brings it down 20% in size and makes everything OK. But that's not really useful..
When I make a breakpoint after adding all the entries, I can see in zip.Entries that all the entries have that same bad CRC, but all the data seems to be there.
So it shouldn't be the way I save the archive that's the problem.
I use my file collection elsewhere without problems, which adds to DotNetZip being weird. Well either that or I misunderstand something :)
GetBuffer is certainly wrong. It returns the internal buffer of the MemoryStream, which is often bigger than the actual content.
To return an array that only contains the actual content, use ToArray().
Or you could carefully handle the incompletely filled buffer in the consuming code. This would reduce GC pressure, since you don't need to allocate a whole new array for the return value.
If the zip-archive is large, I'd also consider saving to a file directly, instead of assembling the archive in-memory.
I am working with iTextSharp, and need to generate hundreds of thousands of RTF documents - the resulting files are between 5KB and 500KB.
I am listing 2 approaches below - the original approach wasn't necessarily slow, but I figured why write and retrieve to/from file to get the output string I need. I saw this other approach using MemoryStream, but it actually slowed things down. I essentially just need the outputted RTF content, so that I can run some filters on that RTF to clean up unnecessary formatting. The queries bringing back the data are very quick instant seeming . To generate a 1000 files (actually 2000 files are created in process) with original approach files takes about 15 minutes, the same with second approach takes about 25-30 minutes. The resulting files that I've run are averaging around 80KB.
Is there something wrong with the second approach? Seems like it should be faster than the first one, not slower.
Original approach:
RtfWriter2.GetInstance(doc, new FileStream(RTFFilePathName, FileMode.Create));
doc.Open();
//Add Tables and stuff here
doc.Close(); //It saves a file here to (RTFPathFileName)
StreamReader srRTF = new StreamReader(RTFFilePathName);
string rtfText = srRTF.ReadToEnd();
srRTF.Close();
//Do additional things with rtfText before writing to my final file
New approach, trying to speed it up but this is actually half as fast:
MemoryStream stream = new MemoryStream();
RtfWriter2.GetInstance(doc, stream);
doc.Open();
//Add Tables and stuff here
doc.Close();
string rtfText =
ASCIIEncoding.ASCII.GetString(stream.GetBuffer());
stream.Close();
//Do additional things with rtfText before writing to my final file
The second approach I am trying I found here:
iTextSharp - How to generate a RTF document in the ClipBoard instead of a file
How big your resulting stream is? MemoryStream performs a lot of memory copy operations while growing, so for large results it may take significantly longer to write data by small chunks compared with FileStream.
To verify if it is the problem set inital size of MemoryStream to some large value around resulting size and re-run the code.
To fix it you can pre-grow memory stream initially (if you know approximate output) or write your own stream that uses different scheme when growing. Also using temporary file might be good enough for your purposes as is.
Like Alexei said, its probably caused by fact, yo are creating MemoryStream every time, and every time it continously re-alocates memory as it grows. Try creating only 1 stream and reset it to begining before every write.
Also I think stream.GetBuffer() again returns new memory, so try using same StreamReader with your MemoryStream.
And it seems your code can be easily paralelised, so you can try run it using Paralel Extesions or using TreadPool.
And it seems little weird, you are writing your text as bytes in stream, then reading this stream as bytes and converting to text. Wouldnt it be possible to save your document directly as text?
A MemoryStream is not associated with a file, and has no concept of a filename. Basically, you can't do that.
You certainly can't cast between them; you can only cast upwards an downwards - not sideways; to visualise:
Stream
|
| |
FileStream MemoryStream
You can cast a MemoryStream to a Stream trivially, and a Stream to a MemoryStream via a type-check; but never a FileStream to a MemoryStream. That is like saying a dog is an animal, and an elephant is an animal, so we can cast a dog to an elephant.
You could subclass MemoryStream and add a Name property (that you supply a value for), but there would still be no commonality between a FileStream and a YourCustomMemoryStream, and FileStream doesn't implement a pre-existing interface to get a Name; so the caller would have to explicitly handle both separately, or use duck-typing (maybe via dynamic or reflection).
Another option (perhaps easier) might be: write your data to a temporary file; use a FileStream from there; then (later) delete the file.
I know this is old but there is a lot of misinformation in this thread.
It's all about buffer size. The internal buffers are significantly smaller with a memory stream vs a file stream. Smaller buffers cause more read\writes.
Just intilaize your memory stream with either a file stream or a byte array with a size of around 80k. Close the doc, set stream position to 0 and read to end the contents.
On a side note, get buffer will return the whole allocated buffer. So if you only wrote 1 byte and the buffer is 4k, you will have a lot of garbage in your string.
I want to know whether it is possible to programmatically change an image encoding on the fly without saving the image to a file?
Use Case: When the user copies a binary image from a source, would it be possible to change the image encoding from binary to base64?
You could change the encoding of an image without saving it to a file, but not without saving it to a variable in your code. The Clipboard class basically just has some Get and Set methods. The only way to change what's in the clipboard is to call one of the Get methods into a local variable, change whatever it is you just got, and then call one of the Set methods, passing in your changed object. This results in a changed clipboard object, but not without the intermediate step of "saving" it to a variable.
Clipboard does not expose any methods for directly manipulating the memory of the object in the clipboard. Even if such a method were exposed, changing the encoding of an image from binary to Base64 involves fundamentally changing all of the memory, so there wouldn't be much value to it.
Update: here's a method that will take an image from the clipboard, convert it to a base64 string, and put it back into the clipboard:
if (Clipboard.ContainsImage())
{
using (MemoryStream memory = new MemoryStream())
{
using (Image img = Clipboard.GetImage())
{
img.Save(memory, img.RawFormat);
}
string base64 = Convert.ToBase64String(memory.ToArray());
Clipboard.SetText(base64);
}
}
And you'll need these two using statements:
using System.IO;
using System.Windows.Forms;
It's untested (because it's past my bedtime), but it should work. It does involve the use of local variables, but this is unavoidable (as well as normal).
Using the new ClipBoard class in WPF
The below example reads a stream from a file but you could use any stream
var image = new BitmapImage();
image.BeginInit();
image.StreamSource = File.Open("image.png", FileMode.Open);
image.EndInit();
System.Windows.Clipboard.SetImage(image);
http://msdn.microsoft.com/en-us/library/system.windows.clipboard.setimage.aspx
I think that you might be asking how to intercept the copy operation and replace the logical contents of the clipboard with new contents, all from some background app, rather than how to replace the memory bytes allocated to the original copy operation.
If that's the intention, you should look for Win32 API calls to hook into the clipboard so that the background app can process the data copied before it is available for paste.
This article might get you going:
http://www.radsoftware.com.au/articles/clipboardmonitor.aspx