I am working on a table that has a LONGBLOB column and I need to SELECT/INSERT data.
At the moment the code to upload a file to the DB is the following:
using (connection = new MySqlConnection(connectionString))
{
string query = "INSERT INTO files(name, uploader, bin) VALUES (#fileName, #uploader, #bin)";
using (command = connection.CreateCommand())
{
command.CommandText = query;
command.Parameters.AddWithValue("#fileName", Path.GetFileName(filePath));
command.Parameters.AddWithValue("#uploader", "John Doe");
command.Parameters.AddWithValue("#bin", File.ReadAllBytes(filePath));
connection.Open();
command.ExecuteNonQuery();
connection.Close();
}
}
The problem is that the file is loaded as a whole in RAM, is there a way to stream data instead?
The code works, is just a matter of understanding if can be optimized
P.S. I am aware that storing big files directly into the database is bad practice, but this is legacy stuff.
In the case of a JPG that will be rendered in a web page, it is better to have it in a file on the server, then put the URL of that file in the database. When the page is rendered, it will fetch multiple images asynchronously -- making the page load seem faster to the end-user. And it avoids the need for discussing your Question.
If the BLOB is something else, please describe it and its usage.
It may be better to chunk it into multiple pieces, especially on a busy system that includes Replication.
Related
Problem:
There is a huge blob that is streamed from a server to a client application.
If there is an exception (connection lost), while writing the stream to the database, all so far received data is lost and the database field remains empty afterwards.
Question: Is there a way to change that behavior, so that the database holds all so far received data and just stores it, even if there is an exception? I want to resume the download later.
Important note: I don't want to buffer the whole received data in memory, I just want to directly write the received data to the database, to avoid memory peaks. I'm using HttpClient with HttpCompletionOption.ResponseHeadersRead in order to achieve that. If there is no connection issue, everything works as expected.
Workaround so far: I can write the received data in temporary file first, and then after a connection issue, I just store what I received to the database. That will work, but is there a better approach?
How the stream is written currently:
string connectionString = "connection String";
using (SqlConnection conn = new SqlConnection(connectionString)) {
conn.Open();
using (SqlCommand cmd = new SqlCommand("update TableX set Data=#data where id = #id")) {
cmd.Parameters.AddWithValue("#Id", ParameterDirection.Input);
cmd.Parameters.AddWithValue("#Data", streamToStore);
cmd.Connection = conn;
int rows = cmd.ExecuteNonQuery();
}
}
I am making a program which requires getting the local directory of an image from a MySQL database table. For the moment, I have made the directory of the image I want to retrieve equal to C:\Users\User\Pictures\PictureName.PNG in the MySQL table. The method I have written up is able to retrieve this data from the database through a Select statement and set the PictureBox image as a new Bitmap with the path retrieved from the Selectstatement.
I couldn't find the information I was looking for online as most of the answers relate to using the BLOB field in MySQL to store images, but my question is if I were to put my database on a server and with the pictures on there too, would my method scale to accommodate for those changes or would I have to add other functionality in order for it to work on a server as well?
Here is my code below:
public void setImage() {
string path = sql.RetrieveImage(SelectQuery.RetrievePicture());
PictureBox1.Image = new Bitmap(path);
}
public string RetrieveImage(Query query) {
string path = "";
// OpenDatabaseConnection() will open up a connection to the database
// through connection and if successful, will return true.
if (this.OpenDatabaseConnection()) {
// query.ToSql() returns a string format of the select statement
// required to retrieve the local image directory
using (MySqlCommand cmd = new MySqlCommand(query.ToSql(), connection)) {
MySqlDataReader dataReader = cmd.ExecuteReader();
dataReader.Read();
path = dataReader[0] + "";
dataReader.Close();
}
this.CloseDatabaseConnection();
}
return path;
}
Storing images in a DB is a bad idea. Storing paths to image files is a much better way to go, IMO.
This SO question has a few links to really good supporting information.
If you must go that route, however, then yes, the BLOB field (or a VarBinary(MAX) in SQLServer) is the way to do it. You can retrieve those images as Byte[] data.
I'm running into a peculiar issue when trying to process large excel files (300mb+) using a data reader.
The following code illustrates the way I open the excel file and iterate over the rows in sheet 'largesheet$':
const string inputFilePath = #"C:\largefile.xlsx";
const string connectionString =
"Provider=Microsoft.ACE.OLEDB.12.0;Extended Properties=\"Excel 12.0;IMEX=1;HDR=YES;\";Data Source=" +
inputFilePath;
// Initialize connection
using (var connection = new OleDbConnection(connectionString))
{
// Open connection
connection.Open();
// Configure command
var command = new OleDbCommand("largesheet$", connection) {CommandType = CommandType.TableDirect};
// Execute reader
var reader = command.ExecuteReader(); // <-- Completely loads file/sheet into memory
// Iterate results
while (reader.HasRows)
{
// Read single row
reader.Read();
// ...
}
// Close connection
connection.Close();
}
In my understanding this should open the excel file and load each row when needed by using the reader.Read() statement.
However, it appears that the ExecuteReader() statement does more than returning an OleDbDataReader instance. Using breakpoints I noticed that that one statement takes 30s+, and the Windows Resource Monitor indicates a steady increase of allocated memory during the execution of that statement.
Specifying the CommandBehavior parameter (e.g. SequentialAccess) of the ExecuteReader() method has no effect.
What am I doing wrong here? Are there alternative ways of processing large (excel) files?
Note: the IMEX & HDR extended properties of the connection string are intentional.
Edit: After some rational thinking I assume it is not possible to process an excel file without buffering it one way or another. Since excel files are basically a glorified collection of compressed XML files it is not possible to process a worksheet without decompressing it (and keeping it in ram or temporarily saving to disk).
The only alternative I can think of is using Microsoft.Office.Interop.Excel. Not sure how OpenXML handles it though.
From MSDN: "All rows and columns of the named table or tables will be returned when you call one of the Execute methods of a Command object." (under the Remarks section). So this would appear to be the default behavior of ExecuteReader().
ExecuteReader(CommandBehavior) may give you more options, particularly when CommandBehavior is set to SequentialAccess, though you would need to handle reading at the byte level.
I have one question and I'm hoping it's an easy one for you.
(Windows Forms Application, C#, Framework 3.5, SQL Server 2008 R2)
I don't know how to open Excel (it is loaded in byte[] type) via OleDB.
So, what have I done:
I have uploaded Excel (.xls(x)) file via form, and saved it in a database as varbinary(max). Now I need to read that Excel file via oleDB. I've managed to load that file from database and saved it into byte[] variable. How can I open byte[] in oleDB?
When I uploaded file for the first time (before saving it to the database), I've opened it via OleDB with just passing file path. How can I access Excel's data when it is already stored in memory as byte[]?
If you want to read using OleDB, then you have to write the bytes to disk. For example, you could do this:
var filename = System.IO.Path.GetTempFileName();
// Assuming that fileBytes is a byte[] containing what you read from your database
System.IO.File.WriteAllBytes(filename, fileBytes);
var connection = #"Provider=Microsoft.ACE.OLEDB.12.0;Data Source=" + filename + ";Extended Properties=\"Excel 12.0;HDR=YES\"";
// Do your work on excel
using (System.Data.OleDb.OleDbConnection conn = new System.Data.OleDb.OleDbConnection(connection))
{
conn.Open();
using (var cmd = conn.CreateCommand())
{
cmd.CommandText = "SELECT * FROM [Sheet1$]";
using (var rdr = cmd.ExecuteReader())
{
while (rdr.Read())
{
System.Diagnostics.Debug.WriteLine(rdr["ColumnName"]);
}
}
}
conn.Close();
}
//Cleanup
System.IO.File.Delete(filename);
If you don't want to write the file to disk, you could look into using a third party library that can read excel files from a memory stream. Tools like SpreadsheetGear or Aspose are commercial tools that can accomplish this.
I don't know anything about what library you're using to deal with your Excel data, but the first thing that comes to mind is that it almost certainly has a LoadFromFile method. It might not be called that, but that's what it does. See if it also has a LoadFromStream method, and if so, take your byte[] data and load it into a MemoryStream, and load the XLS from there.
emp.upload_file = Path.GetFileName(file.FileName);
emp as your table object created and file is HttpPostedFileBase file
I have a DBF file and a index file.
I want to read index file and search records satisfy some condition.
(for example: search records which its StudentName begin with "A" by using Student.DBF and StudentName.idx)
How do I do this programmatically?
It would be easiest to query via OleDB Connection
using System.Data.OleDb;
using System.Data;
OleDbConnection oConn = new OleDbConnection("Provider=VFPOLEDB.1;Data Source=C:\\PathToYourDataDirectory");
OleDbCommand oCmd = new OleDbCommand();
oCmd.Connection = oConn;
oCmd.Connection.Open();
oCmd.CommandText = "select * from SomeTable where LEFT(StudentName,1) = 'A'";
// Create an OleDBAdapter to pull data down
// based on the pre-built SQL command and parameters
OleDbDataAdapter oDA = new OleDbDataAdapter(oCmd);
DataTable YourResults
oDA.Fill(YourResults);
oConn.Close();
// then you can scan through the records to get whatever
String EachField = "";
foreach( DataRow oRec in YourResults.Rows )
{
EachField = oRec["StudentName"];
// but now, you have ALL fields in the table record available for you
}
I dont have the code off the top of my head, but if you do not want to use ODBC, then you should look into reading ESRI shape files, they consist of 3 parts (or more) a .DBF (what you are looking for), a PRJ file and a .SHP file. It could take some work, but you should be able to dig out the code. You should take a look at Sharpmap on codeplex. It's not a simple task to read a dbf w/o ODBC but it can be done, and there is a lot of code out there for doing this. You have to deal with big-endian vs little-endian values, and a range of file versions as well.
if you go here you will find code to read a dbf file. specifically, you would be interested in the public void ReadAttributes( Stream stream ) method.