Has anyone had any issues when converting a BLOB (jpg) from SQL Server 2008 to a file on your file system? It's taking upward of 5-8 seconds for the blob to be retrieved then file created
ultimately I'm gonna store the filepaths and do quick lookups like that but for the initial hit per jpeg, it takes a long time to create the file and I have to assume there's a way to make it faster or I'm just doing it wrong or in an inefficient way in C#.
Here's how I'm getting it from the DB:
memberPhoto.PhotoBinary = (byte[])reader.GetValue(ordinals[(int)Enums.MemberPhotoColumn.PhotoBinary])
Here's how I'm creating the file off that:
using (FileStream file = File.Create(photoPhysicalFilepath))
{
BinaryWriter writer = new BinaryWriter(file);
writer.Write(photoBinary);
writer.Flush();
}
return relativePhotoFilePath;
photoBinary is the byte[] value I got from the datareader.
Not sure if this is efficient. I've seen people go this route but don't understand if this is an old way to do it and I've found a more syntactically elegant way of doing it or if my way is just a more elegant way I've found:
fs = new FileStream(FilePath, FileMode.Create);
fs.Write(imageData, 0, imageData.Length);
fs.Close();
MimeType = image.MIMEType;
where imageData is again a byte[]
So I'm wondering if anyone has done it my way or do you typically go with the second syntax...has there been any performance gains from the second vs. first on a large scaled website that anyone has encountered?
ultimately just trying to figure out why it's taking so long for the blob to file creation.
Related
Hope you're all doing well!
Lets say I'm downloading a file from an HTTP API endpoint and file size is quite large. API returns application/octet-stream i.e. HttpContent in my download method.
when I use
using (FileStream fs = new FileStrean(somepath, FileMode.Create))
{
// this operation takes a few seconds to write to disk
await httpContent.CopyToAsync(fs);
}
As soon as the using statement is executed - I see the file created on the file system at given path, although it is 0 KB at this point, but when CopyToAsync() finishes executing, file size is as expected.
Problem is there's another service running which is constantly polling the folder where above files are saved and often times 0 KB are picked up or sometimes even partial files (this seems to be the case when I use WriteAsync(bytes[]).
Is there a way to not save the file on file system until its ready to be saved...?
One weird work around I could think of was:
using (var memStream = new MemoryStream())
{
await httpContent.CopyToAsync(memStream);
using (FileStream file = new FileStream(destFilePath, FileMode.Create, FileAccess.Write))
{
memStream.Position = 0;
await memStream.CopyToAsync(file);
}
}
I copy the HttpContent over to a MemoryStream and then copy the memorystream over to FileStream... this seems to have worked but there's a cost to memory consumption...
Another work around I could think of was to first save the files into a secondary location and when operation is complete, Move the file over to Primary folder.
Thank you in Advance,
Johny
I ended up saving the file into a temporary folder and when the operation is complete, I move the downloaded file to my primary folder. Since Move is atomic I do not have this issue anymore.
Thank you for those who commented!
I have 22k text (rtf) files which I must append to one final one.
The code looks something like this:
using (TextWriter mainWriter = new StreamWriter(mainFileName))
{
foreach (string currentFile in filesToAppend)
{
using (TextReader currentFileRader = new StreamReader(currentFile))
{
string fileContent = currentFileRader.ReadToEnd();
mainWriter.Write(fileContent);
}
}
}
Clearly, this opens 22k times a stream to read from the files.
My questions are :
1) in general, is opening a stream a slow operation? Is reading from a stream a slow operation ?
2) is there any difference if I read the file as byte[] and append it as byte[] than using the file text?
3) any better ideas to merge 22k files ?
Thanks.
1) in general, is opening a stream a slow operation?
No, not at all. Opening a stream is blazing fast, it's only a matter of reserving a handle from the underlying Operating System.
2) is there any difference if I read the file as byte[] and append it
as byte[] than using the file text?
Sure, it might be a bit faster, rather than converting the bytes into strings using some encoding, but the improvement would be negligible (especially if you are dealing with really huge files) compared to what I suggest you in the next point.
3) any ways to achieve this better ? ( merge 22k files )
Yes, don't load the contents of every single file in memory, just read it in chunks and spit it to the output stream:
using (var output = File.OpenWrite(mainFileName))
{
foreach (string currentFile in filesToAppend)
{
using (var input = File.OpenRead(currentFile))
{
input.CopyTo(output);
}
}
}
The Stream.CopyTo method from the BCL will take care of the heavy lifting in my example.
Probably the best way to speed this up is to make sure that the output file is on a different physical disk drive than the input files.
Also, you can get some increase in speed by creating the output file with a large buffer. For example:
using (var fs = new FileStream(filename, FileMode.Create, FileAccess.Write, FileShare.None, BufferSize))
{
using (var mainWriter = new StreamWriter(fs))
{
// do your file copies here
}
}
That said, your primary bottleneck will be opening the files. That's especially true if those 22,000 files are all in the same directory. NTFS has some problems with large directories. You're better off splitting that one large directory into, say, 22 directories with 1,000 files each. Opening a file from a directory that contains tens of thousands of files is much slower than opening a file in a directory that has only a few hundred files.
What's slow about reading data from a file is the fact that you aren't moving around electrons which can propagate a signal at speeds that are...really fast. To read information in files you have to actually spin these metal disks around and use magnets to read data off of them. These disks are spinning at far slower than electrons can propagate signals through wires. Regardless of what mechanism you use in code to tell these disks to spin around, you're still going to have to wait for them to go a spinin' and that's going to take time.
Whether you treat the data as bytes or text isn't particularly relevant no.
I'm compressing a log file as data is written to it, something like:
using (var fs = new FileStream("Test.gz", FileMode.Create, FileAccess.Write, FileShare.None))
{
using (var compress = new GZipStream(fs, CompressionMode.Compress))
{
for (int i = 0; i < 1000000; i++)
{
// Clearly this isn't what is happening in production, just
// a simply example
byte[] message = RandomBytes();
compress.Write(message, 0, message.Length);
// Flush to disk (in production we will do this every x lines,
// or x milliseconds, whichever comes first)
if (i % 20 == 0)
{
compress.Flush();
}
}
}
}
What I want to ensure is that if the process crashes or is killed, the archive is still valid and readable. I had hoped that anything since the last flush would be safe, but instead I am just ending up with a corrupt archive.
Is there any way to ensure I end up with a readable archive after each flush?
Note: it isn't essential that we use GZipStream, if something else will give us the desired result.
An option is to let Windows handle the compression. Just enable compression on the folder where you're storing your log files. There are some performance considerations you should be aware of when copying the compressed files, and I don't know how well NT compression performs in comparision to GZipStream or other compression options. You'll probably want to compare compression ratios and CPU load.
There's also the option of opening a compressed file, if you don't want to enable compression on the entire folder. I haven't tried this, but you might want to look into it: http://social.msdn.microsoft.com/forums/en-US/netfxbcl/thread/1b63b4a4-b197-4286-8f3f-af2498e3afe5
Good news: GZip is a streaming format. Therefore corruption at the end of the stream cannot affect the beginning which was already written.
So even if your streaming writes are interrupted at an arbitrary point, most of the stream is still good. You can write yourself a little tool that reads from it and just stops at the first exception it sees.
If you want an error-free solution I'd recommend splitting the log into one file every x seconds (maybe x = 1 or 10?). Write into a file with extensions ".gz.tmp" and rename to ".gz" after the file was completely written and closed.
Yes, but it's more involved than just flushing. Take a look at gzlog.h and gzlog.c in the zlib distribution. It does exactly what you want, efficiently adding short log entries to a gzip file, and always leaving a valid gzip file behind. It also has protection against crashes or shutdowns during the process, still leaving a valid gzip file behind and not losing any log entries.
I recommend not using GZIPStream. It is buggy and does not provide the necessary functionality. Use DotNetZip instead as your interface to zlib.
In my project user can upload file up to 1GB. I want to copy that uploaded file stream data to second stream.
If I use like this
int i;
while ( ( i = fuVideo.FileContent.ReadByte() ) != -1 )
{
strm.WriteByte((byte)i);
}
then it is taking so much time.
If i try to do this by byte array then I need to add array size in long which is not valid.
If someone has better idea to do this then please let me know.
--
Hi Khepri thanks for your response. I tried Stream.Copy but it is taking so much time to copy one stream object to second.
I tried with 8.02Mb file and it took 3 to 4 minutes.
The code i have added is
Stream fs = fuVideo.FileContent; //fileInf.OpenRead();
Stream strm = ftp.GetRequestStream();
fs.CopyTo(strm);
If i am doing something wrong then please let me know.
Is this .NET 4.0?
If so Stream.CopyTo is probably your best bet.
If not, and to give credit where credit is due, see the answer in this SO thread. If you're not .NET 4.0 make sure to read the comments in that thread as there are some alternative solutions (Async stream reading/writing) that may be worth investigating if performance is at an absolute premium which may be your case.
EDIT:
Based off the update, are you trying to copy the file to another remote destination? (Just guessing based on GetRequestStream() [GetRequestStream()]. The time is going to be the actual transfer of the file content to the destination. So in this case when you do fs.CopyTo(strm) it has to move those bytes from the source stream to the remote server. That's where the time is coming from. You're literally doing a file upload of a huge file. CopyTo will block your processing until it completes.
I'd recommend looking at spinning this kind of processing off to another task or at the least look at the asynchronous option I listed. You can't really avoid this taking a large period of time. You're constrained by file size and available upload bandwidth.
I verified that when working locally CopyTo is sub-second. I tested with a half gig file and a quick Stopwatch class returned a processing time of 800 millisecondss.
If you are not .NET 4.0 use this
static void CopyTo(Stream fromStream, Stream destination, int bufferSize)
{
int num;
byte[] buffer = new byte[bufferSize];
while ((num = fromStream.Read(buffer, 0, buffer.Length)) != 0)
{
destination.Write(buffer, 0, num);
}
}
I want to save many pictures into a SQL Server database file. How can I compress pictures to store in database? What technologies can I use?
Please help me
After Edit:
How Can I Compress And Resize Pictures to Store in Sql Server DataBase?
Pictures don't compress much if they already in jpg and most other formats. You could try compressing them further before sending to SQL Server but it probably isn't worth the effort, unless you have BMPs.
I suggest you look at FILESTREAM to store them in SQL Server (assuming SQL Server 2008) which is more efficient for this kind of data.
You could save them in JPEG and include the file. For doing it, you can use Image.Save method. And if you want to set the quality you can use Encoder.Compression.
You should also make sure that the resolution of the images is not too much.
You can store pictures in sql database by convert images to byte array.You can do it with below code :
byte[] ReadFile(string sPath)
{
byte[] data = null;
FileInfo fInfo = new FileInfo(sPath);
long numBytes = fInfo.Length;
FileStream fStream = new FileStream(sPath, FileMode.Open, FileAccess.Read);
BinaryReader br = new BinaryReader(fStream);
data = br.ReadBytes((int)numBytes);
return data;
}