We Keep Coding

Reading pictures from DB using memorystreams and images often crashes with Out of Memory

I'm trying to figure out if there is something seriously wrong with the following code. It reads the binary from the database, stores it as a picture and associates with an object of an Animal record.
For each row (record of an animal):
byte[] ba = (byte[])x.ItemArray[1]; //reading binary from a DB row
using (MemoryStream m=new MemoryStream(ba))
{
Image i = Image.FromStream(m); //exception thrown occassionally
c.Photo = i;
listOfAnimals.Add(c);
}
First of all, with 18 pictures loaded (the JPG files have 105 Mb in total), the running app uses 2 gb of memory. With no pictures loaded, it is only 500 Mb.
Often the exception gets raised in the marked point, the source of which is System Drawing.
Could anyone help me optimize the code or tell me what the problem is? I must have used some wrong functions...

According to Image.FromStream Method
OutOfMemoryException
The stream does not have a valid image format.
Remarks
You must keep the stream open for the lifetime of the Image.
The stream is reset to zero if this method is called successively with the same stream.
For more information see: Loading an image from a stream without keeping the stream open and Returning Image using Image.FromStream
Try the following:
Create a method to convert byte[] to image
ConvertByteArrayToImage
public static Image ConvertByteArrayToImage(byte[] buffer)
{
using (MemoryStream ms = new MemoryStream(buffer))
{
return Image.FromStream(ms);
}
}
Then:
byte[] ba = (byte[])x.ItemArray[1]; //reading binary from a DB row
c.Photo = ConvertByteArrayToImage(ba);
listOfAnimals.Add(c);

Checking the documentation, a possible reason for out of memory exceptions are that the stream is not a valid image. If this is the case it should fail reliably for a given image, so check if any particular source image is causing this issue.
Another possibility should be that you simply run out of memory. Jpeg typically gets a 10:1 compression level, so 105Mib of compressed data could use +1Gib of memory. I would recommend switching to x64 if at all possible, I see be little reason not to do so today.
There could also be a memory leak, the best way to investigate this would be with a memory profiler. This might be in just about any part of your code, so it is difficult to know without profiling.
You might also need to care about memory fragmentation. Large datablocks are stored in the large object heap, and this is not automatically defragmented. So after running a while you might still have memory available, just not in any continuous block. Again, switching to x64 would mostly solve this problem.
Also, as mjwills comments, please do not store large files in the database. I just spent several hours recovering a huge database, something that would have been much faster if images where stored as files instead.

C# string multi-byte c++ , gif binary data convert to png or image

I'm seriously stuck in this problem.
this problem caused because i'm weak with C# concept.
all i want do is electronic equipment return gif format data. which is binary i believe.
so i want convert this data to image.
/// below is just send command to instrument that i want " Returns an image of the display in .gif format "
my6705B.WriteString("hcop:sdump:data?", true);
string image_format = my6705B.ReadString();
So i received gif data from instrument, manual said this is " Returns an image of the display in .gif format " ==> I believe this is binary format.
below link is what's in side in string image_format.
string image_format
http://i.stack.imgur.com/UcYqV.png
my goal is convert this string to image file. (png or jpg whatever)
so i convert this string variable to byte array.
below is my code after this command ....
//// this also couldn't work ~~~
System.Text.UnicodeEncoding encode = new System.Text.UnicodeEncoding();
byte[] byte_array22 = encode.GetBytes(image_format);
MemoryStream ms4 = new MemoryStream(byte_array22);
Image image = Image.FromStream(ms4); //// error point
image.Save(#"C:\Users\Administrator\Desktop\imageTest.png");
//// this also couldn't work ~~~
byte[] byte_array22 = Encoding.Unicode.GetBytes(image_format);
MemoryStream ms4 = new MemoryStream(byte_array22);
Image image = Image.FromStream(ms4, true, true); /// always error here,,,
image.Save(#"C:\Users\Administrator\Desktop\imageTest.png", System.Drawing.Imaging.ImageFormat.Png);
both code didn't work and error point is same. i commented error point.
and anyway string to byte array is work.
I'm pain with this problem several days.
but my vendor make this code with C++,, this is working .
let me share my vendor's code,.this is implemented C++.
char szReadBuffer[102400] = {'\0', };
char szReadBinary[102400] = {'\0', };
m_iStatus = viOpenDefaultRM(&m_vDefaultRM);
m_iStatus = viOpen(m_vDefaultRM, (LPSTR)(LPCTSTR)m_strVISA, VI_NULL, VI_NULL, (ViPSession)&m_iDevHandle);
m_iStatus = viSetAttribute(m_iDevHandle, VI_ATTR_TMO_VALUE, 15000);
m_iStatus = QueryGPIB("HCOPy:SDUMp:DATA?", szReadBuffer, sizeof(szReadBuffer));
//Store the results in a text file
CFile file;
file.Open("PICTURE.GIF", CFile::modeReadWrite | CFile::modeCreate | CFile::typeBinary);
memcpy(szReadBinary, &szReadBuffer[2], sizeof(szReadBuffer));
file.Write(szReadBinary, sizeof(szReadBinary));
file.Close();
i think important point is what they declare. they declare char[] .
and adviced me that this C++ code did use String MultiByte ? (just hear from him)
i have no exp with C++.
and if i follow this c++ code then working.
my goal is implement with C#. so need to follow C++ code.
please advice my problem.

It can be confusing sometimes to port C++ to C# if you're unfamiliar with one or the other (never mind both! :) ). One thing to keep in mind: there's no "byte" type in C++. Instead, binary data is stored in char[] arrays, just like C strings.
On the other hand, C# distinguishes between the two. So when you see a char[] in C++ that's being used to store binary data instead of character data, the C# equivalent is a byte[], not a char[] or System.String as it might be for other C++ usages of char[].
Your "my6705B" object appears to be some kind of abstraction of your hardware device. Presumably in addition to the WriteString() and ReadString() methods, there are methods that can be used to write and read binary data, using a byte[] type instead of characters or strings. Use that instead.
Let's assume the proper method is named "ReadBytes()". Then your code would look like this:
byte[] image_format = my6705B.ReadBytes();
MemoryStream ms4 = new MemoryStream(image_format);
Image image = Image.FromStream(ms4);
image.Save(#"C:\Users\Administrator\Desktop\imageTest.png");
Now, that may or may not be exactly what you need. You haven't provided enough information about the "my6705B" object. Many I/O APIs allow for partial reads of available data, so it's possible you would need to read from the device in a loop until you know (somehow) that you've received all of the available bytes for the image. Or maybe the type you're using for the "my6705B" object handles that all for you. I have no way to know…you'll have to figure that out yourself.
But hopefully the above gets you oriented enough wrt the C++ vs C# issues to get you a little further.

Open a file without loading it into memory at once

Hi I have a problem to solve for college and I have a hard time understanding the sentence of the problem.
This is the problem I have :
Reverse the order of bytes from a file without loading the entire file into memory at once.You have to solve this problem in C# , Java , PHP and Python.
Now there are two things that I do not understand here.
First I am not sure if bytes refer to the actual characters of the file , or to something else.The problem does not state if it is a text file or not.
Second I am not sure how to open a file without actualy loading into memory.
This is how I would normaly approach this problem , but I think if I do it this way the file gets loaded into memory:
string fileName = 'file.txt';
reader = new StreamReader(fileName);
string line;
while ((line = reader.ReadLine()) != null)
{
Console.WriteLine(line + "\n");
}
Also I am not sure how I would actualy reverse all the characters if I am reading it one line at the time.
EDIT Sorry for posting in multiple languages I do not want the solution for the problem I only want to clarify it so I can solve it myself.I assumed that because I have to solve it in four different languages the concept would apply to all 4 and it did not matter who answer

Open a FileStream and use the Seek method to go to the end of the file. From there, go backwards, reading one byte at a time. This will read in reverse order. So, until you reach the beginning of the file, loop:
read 1 byte
// do whatever you want with that byte...write to another file?
seek back 2 bytes
As to efficiency, you can read a buffer of, say, 1024 bytes in memory. That way, you don't issue Read operations for each byte of the file. Once you have the buffer filled, reverse it and you're good to go.

Write as building table in HttpHandler?

I have to export very large files as an "excel export". Since .NET can't export excel files, I went with simple html tables.
It works fine, but it's slow.
Is it possible to context.response.write each line as they're being created instead of building some super huge string and trying to export the whole thing once it's done?
I could care less what function is used to do this, but I hope you know what I mean. I don't want to build a string into memory and then try to send it all at once. I'd rather export as I build the table.
Is this possible?
Thanks in advance!

Yes, using context.Response.Write on each line is just fine. If the reason for not wanting to build a large string is server memory use, then you'll need to turn off response buffering like so:
context.Response.BufferOutput = false;
Otherwise, .NET will just buffer your writes in memory until the end, anyway.
If the reason is execution time, then you may be experiencing performance hits from multiple string concatenations. In that case, you could use the StringBuilder class to construct the table instead.
For example...
StringBuilder sb = new StringBuilder();
for each (<row in database>) {
sb.AppendLine(<current table row>);
}
context.Response.Write(sb.ToString());
More info on response buffering:
http://msdn.microsoft.com/en-us/library/system.web.httpresponse.bufferoutput.aspx
More info on the StringBuilder class:
http://msdn.microsoft.com/en-us/library/system.text.stringbuilder(v=vs.110).aspx

C# reading and writing from & to same file [duplicate]

This question already has answers here:
Closed 10 years ago.
Possible Duplicate:
How to both Read/Write File in C#
I want to write and read text to and from the same text file. Here is the code
TextWriter notesWriter = new StreamWriter("text.txt");
TextReader notesReader; = new StreamReader("text.txt");
Once the file is open for writing it is being locked and prevents to read from it.
So, some exception is thrown like,
The process cannot access the file 'text.txt' because it is being used by another process.
what is the workaround for this ? thank you.
EDIT:
Suppose, if want to change text in a textbox and save it in a file and read from it when required to set the text in the textbox.

Every time you finish writing you should Close the file, unless you need to write AND read at the same time which is impossible something you should not do because not so safe/hard/useless in most cases with standard text files used to store data.

Try something like
using(TextWriter notesWriter = new StreamWriter("text.txt"))
{
//do write-things here
}
after the closing-braked the Streamwriter will be disposed and you can read the file.

The workaround is not to do it. While technically this can be done, the way you want to do it (by accessing the file using stream semantics) is almost impossible to be correct as, even if you fix the file sharing, it would imply you're reading back the same stuff you wrote in an infinite loop.
You can use a paging based file access metaphor, which again is very unlikely what you want to do.
The most likely option is that you want to write into a different file, a (modified?) copy of the original file, and then swap the copy with the original file.

Sure you can read and write at the same time, but you only need one reference:
Stream l_fileStream = File.Open( "text.txt", FileMode.Open, FileAccess.ReadWrite );
http://msdn.microsoft.com/en-us/library/s67691sb.aspx
Now you can read/write to the file:
StreamReader l_reader = new StreamReader( l_fileStream );
StreamWriter l_writer = new StreamWriter( l_fileStream );
Why would you want to do this? I have no idea. Seems like you'd want to finish one operation before beginning the next, unless you want to get down and dirty in the actual byte array (like an advanced paging system), in which case you may not be quite at the experience level to pull such a thing off.

You don't need to read and write at the same time, considering your edits.
Open the application
Read the file. Put the file's content in the textbox. Close the file
Save the textbox content into the file. Close the file.
As you can see, you never need to read and write at the same time if you close the file in between your uses.