I have a C# / SQL Server project. and database is reachable from different places (no lan between that 3 places) and data in database is important so I am taking recovery or my database every hour for last 30 days.
Documents which I want to save are kind of fax, excel, word, pdf type data and not formatted. So its impossible to get data inside them.
Problem is how can I store documents in SQL Server I don't want to enlarge its size so much because of increasing backup size.
So what is the efficient solution?
It seems like your main issue is the size of your backup. If you are doing a full backup every hour then you could save space by doing a differential backup instead.
There is no need to backup everything if it hasn't all changed, so you would only need to backup the new data that hadn't been in the last backup.
This would save you a lot of space and time and is generally better practice.
I would suggest you consider implementing a backup rotation scheme. You can find more information on this here:
http://en.wikipedia.org/wiki/Backup_rotation_scheme
I would also suggest you save the file in the filestream data type field in order to reduce the performance impact of having large pages in the mdf file.
If you want to store something it's going to take place. You have multiple choises:
Store only file path in SQL and store files seperatly on server and have seperate backup process for them
Compress files before putting them to sql server, it will save you some place especialy with plain text formats, though it won't help with allready compressed formats(.png, office .docx, .xlsx and so on)
Use FILESTREAM and differential backups (Example)
Similar question: Store Files in SQL Server or keep them on the File Server?
If you worries about backups size - save documents in filesystem and in DB store only patches.
If you worries about backups consistency - store documents inside the DB
Related
I have a application where user can save customer location one by one. But if customer has many locations then entering it one by one will take lot of time. So i need a mechanism where user can do a bulk import, meaning they can save all the locations in csv or excel file, and all I need to do is read all the row one by one and save it to the database without needing to save the actual file or upload file to the server.
I am looking for a solution where I can read csv file in memory and usually file size will be less than 5MB.
Currently its working when I do that from my own machine (because I have hard coded the file path) but it will not work when server and client is on different machine.
This is what i use to parse the csv
try
{
using (TextFieldParser parser = new TextFieldParser(#"C:\Users\temp\docs\Location.csv"))
{
parser.TextFieldType = FieldType.Delimited;
parser.SetDelimiters(",");
parser.ReadLine();
var row= new ImportLocationModel();
while (!parser.EndOfData)
{
row=ImportLocationModel.FromFields(parser.ReadFields();
clientFindProviderApi.InsertUpdateCustomerListLocation1(list); // saving location in database.
}
return true;
}
}
Getting data to a server without Uploading it is impossible. You can maybe prevent the need to store the Data on the Disk. Just cache into in a MemoryStream, wich will then be collected like all other unused managed Memory.
However it is unclear if that is what you relly need/want. It sounds more like you need/should use a Distributed Database. Once you have a mirror/caching DB in the Client, synching it with the Master on the Server (or vice versa) becomse a DMBS thing.
You should not be asuming anything about the Filesize. "The Developers of Foolproof Systems underestimate the ingenuity of the fools." Asume that some user will create a file/data to synch in the 100 MiB Size. Also asume that your client might be a Mobile Phone wich has a abysmal internet connection at that moment. So you should move the Synch to a Asynchronous task one way or the other.
I allow user to upload one excel file at a time. After the upload, the code will validate then save portions of the excel file to an existing database (sql server).
Should I save the excel file to the db, then pull it out and process it, or is it better to save to a temp directory on the web server, process/save the data, then delete the file?
Edit: If I end up saving the file to the DB, is it better to go the binary route (save to byte array, then save) or can I use a regular string array or file stream?
In my personal opinion its best to do this approach.
The answer will be based on your requirement, since you said you will process whats inside excel then this is my suggestion.
Save the file in a file/web server
Save the path of the file saved in SQL Server
Here is why:
To process the file from your code will be easy no need for additional process to access the file and no special coding needed.
You can physically access the file if needed.
Putting files on the database will increase its load, huge possibility of slowing your whole application down.
Database will be bigger so back ups will be slower
Talking about costs, DB strorage is more expensive than file system storage
I know how to store images to db (convert them to byte[] and then save it) and also for retrieving (select byte[] from db and use image methods to create image from byte[]). I'm cool so far, but how can I save/retrieve a PDF to database? What about .doc , .mp3 , .exe and say .ppt files?
Is there a general way to save and retrieve files to and from sql server? The worst part is retrieving, let's imagine we found a way to save any file to sql server, now how can we rebuild the file from db? We don't know what the file extension was before saving?
Well, Generally speaking, it's considered bad practice to save actual files to the database.
a part of the reason is the problems you mentioned in your post, and an even bigger part is that saving files directly to the database has a large overhead (such as translating an image to a byte array and back).
the easy (and recommended in most cases) way to handle files and databases is to save the files directly to the file system, and keep the path in the database along with other file-related data such as the user id that uploaded the file.
this way you don't need to worry about braking and rebuilding the files, you just send them to the server and back to the user as is.
Keep in mind it's not recommended to keep the full path of the file, only a relative path.
What I normally do is save all files from the users either on the serer itself or on the users's computer (in a desktop application). in any of these cases, there is a dedicated folder with only read/write permissions (NEVER let a user save a file into a directory with execute permissions!), and keep the path of this directory either on a 'General Params' table in the database or in the configuration file of the website / application / webservice.
Well, the file attributes (name, extension, author, etc) are usually kept in relational way, in table inside SQL Server. The file itself should be kept in SQL Server database, exactly how depends on version od SQL Server and size of file. Use FILESTREAM or FILETABLES feature for larger, or VARBINARY(MAX) for smaller files.
It doesn't matter whether its an image, or doc or pdf -- if you car read into a FileStream, you can save it to database.
Advantages of storing files in a database is simplified management, backups, security, integrity. With FILESTREAM and FILETABLES feature, accessing a file is almost the same as if it were on a file system, using the SqlFileStream object from .NET.
See more here:
http://technet.microsoft.com/en-us/library/gg471497.aspx
And here:
http://technet.microsoft.com/en-us/library/ff929144.aspx
Background information:
This application is .NET 4/C# Windows
Forms using SQLite as it's backend.
There is only one user using the
database and in no way does it
interact through a network.
My software needs to save images associated to a Project record. Should I save the image as binary information in the database itself; or should I save the path to the picture on the file system and use that to retrieve it.
My concerns when saving as path is that someone might change the filename of a picture and that would essentially break my applications use.
Can anyone give some suggestions?interact through a network.
"It depends". If there are a lot of images, then all that BLOB weight may make backups increasingly painful (and indeed, may preclude some database implementations that only support limited sizes). But it works, and works well. The file system is fine as long as you only store the path relative to some unknown root, i.e. you store "foo/blah/blip.png", which is combined with configuration data to get the full path - then you can relocate the path easily. File systems have simpler backup options in some cases, but you need to marry the file-system and database backups.
In general, it is better to store them on the filesystem, with a path stored in the DB.
However, Microsoft published a white paper some time ago with research showing that files up to 150K can benefit from being put inside the DB (of course, this only pertains to SQL Server).
The question has been asked here many many times before:
Exact Duplicate: User Images: Database or filesystem storage?
Exact Duplicate: Storing images in database: Yea or nay?
Exact Duplicate: Should I store my images in the database or folders?
Exact Duplicate: Would you store binary data in database or folders?
Exact Duplicate: Store pictures as files or or the database for a web app?
Exact Duplicate: Storing a small number of images: blob or fs?
Exact Duplicate: store image in filesystem or database?
First of all have you checked the SQLite limits? If this is of no concern for you application, I would still chose the FS for storage needs simply due to overhead from getting large BLOBS from DB vs. reading a file from FS. You can mark the files as read only and hidden to lessen the chance of them being renamed... You can also store the file hash (like MD5) of a file in the DB so you can have secondary lookup option in case someone does rename the file (of course, they could move it as well in which case this would not help much)...
I have been playing around with creation of pdf documents for a project that I'm working on. I would like to store the generated pdf document in a SQL database and then later be able to retrieve this pdf as well.
What are some suggestions for doing this? Can the document be stored in the database without physically creating the document on the server?
This is again going to bring up the debate for/against storing things on the file system or within sql server itself.
It really depends on your needs, the size that you're expected, etc. Here are some references, each with more references.
Storing a file in a database as opposed to the file system?
store image in database or in a system file?
What are some suggestions for doing
this? Can the document be stored in
the database without physically
creating the document on the server?
Sure just create the pdf as a byte stream (byte[]) and store it in the database. Depending on what you use to create it, you don't have to write it to the file system.
Actually, on the argument about where to store it. If you have SQL server 2008, you want to use that. It will store the file on the file system, but you can access it through the database like you would with any other data. You get the best of both worlds.
Keep in mind that SQL Server 2008 now has the FILESTREAM data type. You can write the data to the file system, yet still store it in a column.
Save the PDF as a byte[] then you can use itextsharp to created the PDF when ready for viewing.
You can use a table like this to store files in SQL Server.
CREATE TABLE [Documents]
(
[FileName] nvarchar(1000),
[FileContent] varbinary(max)
)
You have 2 ways to do that:
Store in FileServer and store the Filename in the database.
Encode file and store in database.
I recomend that you use the second..
why I choose that answer?? for security.
One of a lot of reasons:
A little Example:
If you do the Firt(store the file in the fileserver...) you are
crating a folder on your database.. so you server will be vulnerable
for attacks or for virus..
If you do the second. the file will be encode and store in database
and you dont need to be worried about attacks or machine infections..
I think that this are 1 simple reason about why never use the first WAY!!!!!