Problem:
There is a huge blob that is streamed from a server to a client application.
If there is an exception (connection lost), while writing the stream to the database, all so far received data is lost and the database field remains empty afterwards.
Question: Is there a way to change that behavior, so that the database holds all so far received data and just stores it, even if there is an exception? I want to resume the download later.
Important note: I don't want to buffer the whole received data in memory, I just want to directly write the received data to the database, to avoid memory peaks. I'm using HttpClient with HttpCompletionOption.ResponseHeadersRead in order to achieve that. If there is no connection issue, everything works as expected.
Workaround so far: I can write the received data in temporary file first, and then after a connection issue, I just store what I received to the database. That will work, but is there a better approach?
How the stream is written currently:
string connectionString = "connection String";
using (SqlConnection conn = new SqlConnection(connectionString)) {
conn.Open();
using (SqlCommand cmd = new SqlCommand("update TableX set Data=#data where id = #id")) {
cmd.Parameters.AddWithValue("#Id", ParameterDirection.Input);
cmd.Parameters.AddWithValue("#Data", streamToStore);
cmd.Connection = conn;
int rows = cmd.ExecuteNonQuery();
}
}
Related
I am working on a table that has a LONGBLOB column and I need to SELECT/INSERT data.
At the moment the code to upload a file to the DB is the following:
using (connection = new MySqlConnection(connectionString))
{
string query = "INSERT INTO files(name, uploader, bin) VALUES (#fileName, #uploader, #bin)";
using (command = connection.CreateCommand())
{
command.CommandText = query;
command.Parameters.AddWithValue("#fileName", Path.GetFileName(filePath));
command.Parameters.AddWithValue("#uploader", "John Doe");
command.Parameters.AddWithValue("#bin", File.ReadAllBytes(filePath));
connection.Open();
command.ExecuteNonQuery();
connection.Close();
}
}
The problem is that the file is loaded as a whole in RAM, is there a way to stream data instead?
The code works, is just a matter of understanding if can be optimized
P.S. I am aware that storing big files directly into the database is bad practice, but this is legacy stuff.
In the case of a JPG that will be rendered in a web page, it is better to have it in a file on the server, then put the URL of that file in the database. When the page is rendered, it will fetch multiple images asynchronously -- making the page load seem faster to the end-user. And it avoids the need for discussing your Question.
If the BLOB is something else, please describe it and its usage.
It may be better to chunk it into multiple pieces, especially on a busy system that includes Replication.
From a third party I am receiving a SQLite '.db3' database file. I wish to read the data stored within the database in an Azure function, but have no requirement to modify the database.
In my Azure function I have also uploaded a text file alongside the .db3 file, I can read the contents of this text file just fine, meaning that I have the URI and read privileges on the folder.
When I try to read some data from the database I get the error 'database is locked'.
Code to read the database is below. This works locally.
List<object> obs = new List<object>();
using(var con = new SQLiteConnection($"URI=file:{fullUri};mode=ReadOnly"))
{
con.Open();
using(var cmd = new SQLiteCommand(sql, con))
{
using(SQLiteDataReader rdr = cmd.ExecuteReader())
{
while(rdr.Read())
{
obs.Add(rdr.GetValues());
}
}
}
con.Close();
}
return obs;
How do I read the database?
I've had this problem before, reading a local file ... more than once. These are the steps I took depending on the scenario:
If your Azure Function is deployed using zipdeploy then under the "Configuration" > "Application Settings", change "WEBSITE_RUN_FROM_PACKAGE" from 1 to 0.
If the file you are trying to reference is saved under wwwroot on the Azure Function I would try put it somewhere else e.g. d:\local.
I am making a program which requires getting the local directory of an image from a MySQL database table. For the moment, I have made the directory of the image I want to retrieve equal to C:\Users\User\Pictures\PictureName.PNG in the MySQL table. The method I have written up is able to retrieve this data from the database through a Select statement and set the PictureBox image as a new Bitmap with the path retrieved from the Selectstatement.
I couldn't find the information I was looking for online as most of the answers relate to using the BLOB field in MySQL to store images, but my question is if I were to put my database on a server and with the pictures on there too, would my method scale to accommodate for those changes or would I have to add other functionality in order for it to work on a server as well?
Here is my code below:
public void setImage() {
string path = sql.RetrieveImage(SelectQuery.RetrievePicture());
PictureBox1.Image = new Bitmap(path);
}
public string RetrieveImage(Query query) {
string path = "";
// OpenDatabaseConnection() will open up a connection to the database
// through connection and if successful, will return true.
if (this.OpenDatabaseConnection()) {
// query.ToSql() returns a string format of the select statement
// required to retrieve the local image directory
using (MySqlCommand cmd = new MySqlCommand(query.ToSql(), connection)) {
MySqlDataReader dataReader = cmd.ExecuteReader();
dataReader.Read();
path = dataReader[0] + "";
dataReader.Close();
}
this.CloseDatabaseConnection();
}
return path;
}
Storing images in a DB is a bad idea. Storing paths to image files is a much better way to go, IMO.
This SO question has a few links to really good supporting information.
If you must go that route, however, then yes, the BLOB field (or a VarBinary(MAX) in SQLServer) is the way to do it. You can retrieve those images as Byte[] data.
I'm running into a peculiar issue when trying to process large excel files (300mb+) using a data reader.
The following code illustrates the way I open the excel file and iterate over the rows in sheet 'largesheet$':
const string inputFilePath = #"C:\largefile.xlsx";
const string connectionString =
"Provider=Microsoft.ACE.OLEDB.12.0;Extended Properties=\"Excel 12.0;IMEX=1;HDR=YES;\";Data Source=" +
inputFilePath;
// Initialize connection
using (var connection = new OleDbConnection(connectionString))
{
// Open connection
connection.Open();
// Configure command
var command = new OleDbCommand("largesheet$", connection) {CommandType = CommandType.TableDirect};
// Execute reader
var reader = command.ExecuteReader(); // <-- Completely loads file/sheet into memory
// Iterate results
while (reader.HasRows)
{
// Read single row
reader.Read();
// ...
}
// Close connection
connection.Close();
}
In my understanding this should open the excel file and load each row when needed by using the reader.Read() statement.
However, it appears that the ExecuteReader() statement does more than returning an OleDbDataReader instance. Using breakpoints I noticed that that one statement takes 30s+, and the Windows Resource Monitor indicates a steady increase of allocated memory during the execution of that statement.
Specifying the CommandBehavior parameter (e.g. SequentialAccess) of the ExecuteReader() method has no effect.
What am I doing wrong here? Are there alternative ways of processing large (excel) files?
Note: the IMEX & HDR extended properties of the connection string are intentional.
Edit: After some rational thinking I assume it is not possible to process an excel file without buffering it one way or another. Since excel files are basically a glorified collection of compressed XML files it is not possible to process a worksheet without decompressing it (and keeping it in ram or temporarily saving to disk).
The only alternative I can think of is using Microsoft.Office.Interop.Excel. Not sure how OpenXML handles it though.
From MSDN: "All rows and columns of the named table or tables will be returned when you call one of the Execute methods of a Command object." (under the Remarks section). So this would appear to be the default behavior of ExecuteReader().
ExecuteReader(CommandBehavior) may give you more options, particularly when CommandBehavior is set to SequentialAccess, though you would need to handle reading at the byte level.
What is a faster way to export large (25,000 rows) SQL Server CE database into XML file?
Here is what I currently use:
using (SqlCeConnection cn = new SqlCeConnection(strConnection))
{
if (cn.State == ConnectionState.Closed)
cn.Open();
using (SqlCeCommand cmd = new SqlCeCommand(strCommand, cn))
{
SqlCeDataAdapter da = new SqlCeDataAdapter(cmd);
DataSet ds = new DataSet();
da.Fill(ds, "item");
StreamWriter xmlDoc = new StreamWriter("Output.xml");
ds.WriteXml(xmlDoc);
xmlDoc.Close();
}
}
It takes about 60 seconds inside emulator (Windows Mobile).
Also.. I am using Compact Framework 3.5 with SQL Server CE 3.5.
Current performance:
60 seconds entire code
~20 seconds for everything without ds.WriteXml(xmlDoc);, leaving ~40 seconds for ds.WriteXml(xmlDoc);.
If the effort is worth it to you, the fastest method is probably to roll your own XML generation. The library implementation of XmlWriter() is loops within loops, with considerable generality.
Declare your own output stream specifying the buffer size to something reasonably large (but still sane). I have tested StringBuilder() before and append() to it millions of times with good performance. Other output options may not be as fast, but I would hope the StreamWriter gives good performance when buffersize if appropriate and it force you to build everything in memory.
Don't use da.Fill() -- replace with a SqlCeDataReader.
For each row. Generate the XML for a single row using logic that is coded as close as possible to the metal. I.e., Precalculate column index values, etc. instead of using column names within this loop, use hard-coded type conversions as needed. Don't loop through each column, put each column
Also, test by having the database generating XML output. Although I don't expect this to be the fastest, it is easy to try, and if it turns out to be fast you would never have discovered it without trying.
Maybe something like this is more lightweight:
private void ExportDataTable(SqlCeCommand cmd, SqlCeConnection conn)
{
cmd.Connection = conn;
System.Data.DataTable table = new System.Data.DataTable();
table.Locale = CultureInfo.InvariantCulture;
table.Load(cmd.ExecuteReader());
StreamWriter xmlDoc = new StreamWriter("Output.xml");
table.WriteXml(xmlDoc);
}