We Keep Coding

SQLite.Net Won't Create In Win IoT Library

I have been struggling to find a way of persisting an SQLite database on a Pi under Win IoT which can be accessed by different background applications (not concurrently).
I thought I had the answer when I discovered Libraries (Music, Pictures, Videos - but perversely not Documents, without more work). I can create a text file in one app and write it to the Pictures library's default folder. I can then read the text file with another app. File.Exists returns true. Bingo (I thought)!
However, SQLite will not create a database in the folder or open an existing database that I copy to the folder. SQLite.Net.SQLiteConnection returns an SQLite exception: "Could not open database file: C:\Data\Users\DefaultAccount\Pictures\MyDb.db (CannotOpen)" - no further clues.
The folder appears to grant full permissions. Does anyone have any ideas, please?
Creating and Writing a text file:
using System;
using Windows.ApplicationModel.Background;
using System.IO;
using System.Diagnostics;
//*** NOTE: Pictures Library checked in Package.appxmanifest 'Capabilities'
namespace LibraryTest
{
public sealed class StartupTask : IBackgroundTask
{
private BackgroundTaskDeferral Deferral;
public async void Run (IBackgroundTaskInstance taskInstance)
{
Deferral = taskInstance.GetDeferral ();
var myPictures = await Windows.Storage.StorageLibrary.GetLibraryAsync
(Windows.Storage.KnownLibraryId.Pictures);
string path = myPictures.SaveFolder.Path;
Debug.WriteLine ($"'Pictures' Folder: {path}");
string newFilePath = Path.Combine (path, "TestTextFile.txt");
Debug.WriteLine ($"New File Path: {newFilePath}");
try {
using ( var stream = File.OpenWrite (newFilePath) ) {
using ( var writer = new StreamWriter (stream) ) {
writer.Write ("This is some test text.");
}
}
Debug.WriteLine ($"File created OK");
}
catch (Exception ex) { Debug.WriteLine ($"Exception: {ex.Message}"); }
}
}
}
Produced:
'Pictures' Folder: C:\Data\Users\DefaultAccount\Pictures
New File Path: C:\Data\Users\DefaultAccount\Pictures\TestTextFile.txt
File created OK
Reading:
using System;
using Windows.ApplicationModel.Background;
using System.IO;
using System.Diagnostics;
//*** NOTE: Pictures Library checked in Package.appxmanifest 'Capabilities'
namespace ReadLibraryTest
{
public sealed class StartupTask : IBackgroundTask
{
private BackgroundTaskDeferral Deferral;
public async void Run (IBackgroundTaskInstance taskInstance)
{
Deferral = taskInstance.GetDeferral ();
var myPictures = await Windows.Storage.StorageLibrary.GetLibraryAsync
(Windows.Storage.KnownLibraryId.Pictures);
string path = myPictures.SaveFolder.Path;
Debug.WriteLine ($"'Pictures' Folder: {path}");
string newFilePath = Path.Combine (path, "TestTextFile.txt");
Debug.WriteLine ($"New File Path: {newFilePath}");
try {
using ( var stream = File.OpenRead (newFilePath) ) {
using ( var reader = new StreamReader (stream) ) {
string fileContents = reader.ReadLine ();
Debug.WriteLine ($"First line of file: '{fileContents}'");
}
}
Debug.WriteLine ($"File read OK");
}
catch ( Exception ex ) { Debug.WriteLine ($"Exception: {ex.Message}"); }
}
}
}
Produced:
'Pictures' Folder: C:\Data\Users\DefaultAccount\Pictures
New File Path: C:\Data\Users\DefaultAccount\Pictures\TestTextFile.txt
First line of file: 'This is some test text.'
File read OK

However, SQLite will not create a database in the folder or open an
existing database that I copy to the folder.
SQLite.Net.SQLiteConnection returns an SQLite exception: "Could not
open database file: C:\Data\Users\DefaultAccount\Pictures\MyDb.db
(CannotOpen)" - no further clues.
Yes, I reproduced this issue. It seems this folder does not work with SQLite file operations but I don't know where the problem is.
As a workaround, you can use PublisherCacheFolder. I create the .db file and write data in one background app. And read the data from another background app. It works.
Contact class:
public sealed class Contact
{
public int Id { get; set; }
public string Name { get; set; }
}
Create and write file:
StorageFolder sharedFonts = Windows.Storage.ApplicationData.Current.GetPublisherCacheFolder("test");
var sqlpath = System.IO.Path.Combine(sharedFonts.Path, "MyDb.db");
using (SQLite.Net.SQLiteConnection conn = new SQLite.Net.SQLiteConnection(new SQLite.Net.Platform.WinRT.SQLitePlatformWinRT(), sqlpath))
{
conn.CreateTable<Contact>();
for (var i = 0; i < 100; i++)
{
Contact contact = new Contact()
{
Id = i,
Name = "A"
};
conn.Insert(contact);
}
}
Read file:
StorageFolder sharedFonts = Windows.Storage.ApplicationData.Current.GetPublisherCacheFolder("test");
var sqlpath = System.IO.Path.Combine(sharedFonts.Path, "MyDb.db");
using (SQLite.Net.SQLiteConnection conn = new SQLite.Net.SQLiteConnection(new SQLite.Net.Platform.WinRT.SQLitePlatformWinRT(), sqlpath))
{
var query = conn.Table<Contact>().Where(v => v.Name.Equals("A"));
foreach (var stock in query)
Debug.WriteLine("contact: " + stock.Id);
}
To use this publisher folder you need add the following lines in Package.appxmanifest:
<Extensions>
<Extension Category="windows.publisherCacheFolders">
<PublisherCacheFolders>
<Folder Name="test" />
</PublisherCacheFolders>
</Extension>
</Extensions>

Thanks, Rita. Worked very well. For the benefit of anyone reading, I am using the async version of SqlLite and create the connection as follows:
const string FileName = "MyFile.db";
string DbDir;
string DbPath;
Constructor:
DbDir = Windows.Storage.ApplicationData.Current.GetPublisherCacheFolder("test").Path;
DbPath = Path.Combine (DbDir, DbFileName);
public SQLite.Net.Async.SQLiteAsyncConnection GetConnectionAsync ()
{
var connectionFactory = new Func<SQLite.Net.SQLiteConnectionWithLock>(()=>
new SQLite.Net.SQLiteConnectionWithLock(new SQLitePlatformWinRT(),
new SQLite.Net.SQLiteConnectionString(DbPath, storeDateTimeAsTicks: false)));
var asyncConnection = new SQLiteAsyncConnection(connectionFactory);
return asyncConnection;
}
Then, for instance, read a table of type Parms:
public async Task<Parms> ReadParmsAsync ()
{
var db = GetConnectionAsync ();
var query = db.Table<Parms> ().Where (p => p.Id == 1);
return await query.FirstOrDefaultAsync ();
}
My concern about the SQLite async connection is that it is not IDisposable. Therefore, will the 'factory' eventually run out of steam (memory, handles)? But I guess that is a subject for another thread.

How to Check if a database exists in SQLite?

I am developing a windows phone 8 app using sqlite and am trying to check if the database exists and if it doesnt exist,it should be created. but i keep getting the error message "System.windows.shapes.path does not contain a definition for combine". Is there another way to do it or how can i improve it?
public static string DB_PATH = Path.Combine(Path.Combine(ApplicationData.Current.LocalFolder.Path, "ContactsManager.sqlite"));//DataBase Name
public App()
{
if (!CheckFileExists("ContactsManager.sqlite").Result)
{
using (var db = new SQLiteConnection(DB_PATH))
{
db.CreateTable<Contacts>();
}
}
}
private async Task<bool> CheckFileExists(string fileName)
{
try
{
var store = await Windows.Storage.ApplicationData.Current.LocalFolder.GetFileAsync(fileName);
return true;
}
catch
{
}
return false;
}

Do you really need to check if the database exist? I don't know about windows phone, but in Windows, as soon as you try to add a table into a SQLite database, if the database doesn't exist, it creates it. If you are worried about the table existing already, you can use:
CREATE TABLE IF NOT EXISTS tableName(...)
(I tried to ask it as comment but I don't have the reputation)

Why do you have a Path.Combine in a Path.Combine? If Path.Combine is not available with one or two parameter, why not simple concat two strings?
You have it 2x: public static string DB_PATH = Path.Combine(Path.Combine(ApplicationData.Current.LocalFolder.Path, "ContactsManager.sqlite"));

you can check by this:
public async Task<bool> isFilePresent(string fileName)
{
return System.IO.File.Exists(string.Format(#"{0}\{1}", ApplicationData.Current.LocalFolder.Path, fileName);
}

#Panagiotis Kanavos's comment was right, you have resolved the Path class using wrong namespace!
Remove
using System.Windows.Shapes; // for silverlite
using Windows.UI.Xaml.Shapes; // for winrt
and add
using System.IO;

Submit C# MapReduce Job Windows Azure HDInsight - Response status code does not indicate success: 500 (Server Error)

I'm trying to submit a MapReduce job to HDInsight cluster. In my job I didn't write reduce portion because I don't want to reduce anything. All I want to do is to parse the each filename and append the values to every line in the file. So that I will have all the data needed inside the file.
My code is
using Microsoft.Hadoop.MapReduce;
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
namespace GetMetaDataFromFileName
{
class Program
{
static void Main(string[] args)
{
var hadoop = connectAzure();
//Temp Workaround to Env Variables
Environment.SetEnvironmentVariable("HADOOP_HOME", #"c:\hadoop");
Environment.SetEnvironmentVariable("Java_HOME", #"c:\hadoop\jvm");
var result = hadoop.MapReduceJob.ExecuteJob<MetaDataGetterJob>();
}
static IHadoop connectAzure()
{
//TODO: Update credentials and other information
return Hadoop.Connect(
new Uri("https://sampleclustername.azurehdinsight.net//"),
"admin",
"Hadoop",
"password",
"blobstoragename.blob.core.windows.net", //Storage Account that Log files exists
"AccessKeySample", //Storage Account Access Key
"logs", //Container Name
true
);
}
//Hadoop Mapper
public class MetaDataGetter : MapperBase
{
public override void Map(string inputLine, MapperContext context)
{
try
{
//Get the meta data from name of the file
string[] _fileMetaData = context.InputFilename.Split('_');
string _PublicIP = _fileMetaData[0].Trim();
string _PhysicalAdapterMAC = _fileMetaData[1].Trim();
string _BootID = _fileMetaData[2].Trim();
string _ServerUploadTime = _fileMetaData[3].Trim();
string _LogType = _fileMetaData[4].Trim();
string _MachineUpTime = _fileMetaData[5].Trim();
//Generate CSV portion
string _RowHeader = string.Format("{0},{1},{2},{3},{4},{5},", _PublicIP, _PhysicalAdapterMAC, _BootID, _ServerUploadTime, _LogType, _MachineUpTime);
//TODO: Append _RowHeader to every row in the file.
context.EmitLine(_RowHeader + inputLine);
}
catch(ArgumentException ex)
{
return;
}
}
}
//Hadoop Job Definition
public class MetaDataGetterJob : HadoopJob<MetaDataGetter>
{
public override HadoopJobConfiguration Configure(ExecutorContext context)
{
//Initiate the job config
HadoopJobConfiguration config = new HadoopJobConfiguration();
config.InputPath = "asv://logs#sample.blob.core.windows.net/Input";
config.OutputFolder = "asv://logs#sample.blob.core.windows.net/Output";
config.DeleteOutputFolder = true;
return config;
}
}
}
}
Usually what do you thing the reason of 500 (Server Error) ? Am I suppling to wrong credentials ? Actually I didn't really understand the difference between Username and HadoopUser parameters in Hadoop.Connect method ?
Thank you,

I had approximately same issue in the past (was unable to submit hive job to the cluster with BadGateway response). I have contacted the support team and in my case the problem was in memory leakage at the head node, what means that the problem was not at client's side and it seems to be inherited hadoop problem.
I've solved that stuff by redeploying the cluster.
Have you tried to submit other jobs (simple ones)? If so, than I suggest to have a contact with azure support team or just redeploy the cluster if it's not painful for you.

.NET Application for XML and SQL Server Integration

As a database developer with a very small amount of programming experience, I'm currently attempting to develop a C# .NET console application to import XML files into a SQL Server database. Once the import is done, I need to create a 'Response' file to be passed back to another application.
In doing my own research, I've come across the SQLBULKLOAD class. In fact, I've found some example code online that shows (at least partially) exactly what I'm trying to do:
using System;
using System.IO;
using System.Collections;
using SQLXMLBULKLOADLib;
using System.Data.OleDb;
using System.Diagnostics;
namespace SQLXmlExample
{
class Program
{
[STAThread]
static void Main(string[] args)
{
string schema = "C:\\ImportSample\\MappingFile.xml";
string datafile = "C:\\ImportSample\\DataFile.xml";
string connectionString = #"provider=SQLOLEDB;data source=localhost;database=SqlXmlDemo;Integrated Security=SSPI;";
for (int i = 0; i < args.Length; i++)
{
switch (args[i].ToLower())
{
case "-schema":
schema = args[i + 1];
break;
case "-datafile":
datafile = args[i + 1];
break;
}
}
if (schema == string.Empty || datafile == string.Empty)
{
Console.WriteLine("Missing Schema or Data File. Format: SqlXmlExample -datafile [filename] -schema [filename]");
return;
}
Load(datafile, schema, connectionString);
}
static public void Load(string XMLFilename, string XMLMappingFilename, string ConnectionString)
{
SQLXMLBULKLOADLib.SQLXMLBulkLoad loader = new SQLXMLBULKLOADLib.SQLXMLBulkLoad();
loader.CheckConstraints = true;
loader.XMLFragment = true;
loader.SchemaGen = true;
loader.SGDropTables = false;
loader.Transaction = false;
loader.ConnectionString = ConnectionString;
loader.Execute("C:\\ImportSample\\MappingFile.xml", "C:\\ImportSample\\DataFile.xml");
}
}
}
With the code above, I'm able to import an .XML file into a SQL Server instance. However, my problem is generating the 'Response' .XML file to provide information about the bulk loading operation (i.e. how many records were inserted, if the insert was successful).
As it stands now, I'm thinking that I may be using the wrong class for what I'm trying to accomplish. Am I using the correct class, or should I be using a different one?
If possible, could anyone point me in the direction of some more material to assist me? Any help would be greatly appreciated.

How about using the ErrorLogFile?
This is where the bulk loader stores all its errors and messages, simply try one out, make it fail, see what the format is and then you can then load, parse and generate xml?

How do I automatically delete temp files in C#?

What's a good way to ensure that a temp file is deleted if my application closes or crashes? Ideally, I would like to obtain a temp file, use it, and then forget about it.
Right now, I keep a list of my temp files and delete them with an EventHandler that's triggered on Application.ApplicationExit.
Is there a better way?

Nothing is guaranteed if the process is killed prematurely, however, I use "using" to do this..
using System;
using System.IO;
sealed class TempFile : IDisposable
{
string path;
public TempFile() : this(System.IO.Path.GetTempFileName()) { }
public TempFile(string path)
{
if (string.IsNullOrEmpty(path)) throw new ArgumentNullException("path");
this.path = path;
}
public string Path
{
get
{
if (path == null) throw new ObjectDisposedException(GetType().Name);
return path;
}
}
~TempFile() { Dispose(false); }
public void Dispose() { Dispose(true); }
private void Dispose(bool disposing)
{
if (disposing)
{
GC.SuppressFinalize(this);
}
if (path != null)
{
try { File.Delete(path); }
catch { } // best effort
path = null;
}
}
}
static class Program
{
static void Main()
{
string path;
using (var tmp = new TempFile())
{
path = tmp.Path;
Console.WriteLine(File.Exists(path));
}
Console.WriteLine(File.Exists(path));
}
}
Now when the TempFile is disposed or garbage-collected the file is deleted (if possible). You could obviously use this as tightly-scoped as you like, or in a collection somewhere.

Consider using the FileOptions.DeleteOnClose flag:
using (FileStream fs = new FileStream(Path.GetTempFileName(),
FileMode.OpenOrCreate, FileAccess.ReadWrite, FileShare.None,
4096, FileOptions.RandomAccess | FileOptions.DeleteOnClose))
{
// temp file exists
}
// temp file is gone

You could P/Invoke CreateFile and pass the FILE_FLAG_DELETE_ON_CLOSE flag. This tells Windows to delete the file once all handles are closed. See also: Win32 CreateFile docs.

I would use the .NET TempFileCollection class, as it's built-in, available in old versions of .NET, and implements the IDisposable interface and thus cleans up after itself if used e.g. in conjunction with the "using" keyword.
Here's an example that extracts text from an embedded resource (added via the projects property pages -> Resources tab as described here: How to embed a text file in a .NET assembly?, then set to "EmbeddedResource" in the embedded file's property settings).
// Extracts the contents of the embedded file, writes them to a temp file, executes it, and cleans up automatically on exit.
private void ExtractAndRunMyScript()
{
string vbsFilePath;
// By default, TempFileCollection cleans up after itself.
using (var tempFiles = new System.CodeDom.Compiler.TempFileCollection())
{
vbsFilePath= tempFiles.AddExtension("vbs");
// Using IntelliSense will display the name, but it's the file name
// minus its extension.
System.IO.File.WriteAllText(vbsFilePath, global::Instrumentation.Properties.Resources.MyEmbeddedFileNameWithoutExtension);
RunMyScript(vbsFilePath);
}
System.Diagnostics.Debug.Assert(!File.Exists(vbsFilePath), #"Temp file """ + vbsFilePath+ #""" has not been deleted.");
}

I use a more reliable solution:
using System.IO;
using System.Reflection;
namespace Helpers
{
public static partial class TemporaryFiles
{
private const string UserFilesListFilenamePrefix = ".used-temporary-files.txt";
static private readonly object UsedFilesListLock = new object();
private static string GetUsedFilesListFilename()
{
return Assembly.GetEntryAssembly().Location + UserFilesListFilenamePrefix;
}
private static void AddToUsedFilesList(string filename)
{
lock (UsedFilesListLock)
{
using (var writer = File.AppendText(GetUsedFilesListFilename()))
writer.WriteLine(filename);
}
}
public static string UseNew()
{
var filename = Path.GetTempFileName();
AddToUsedFilesList(filename);
return filename;
}
public static void DeleteAllPreviouslyUsed()
{
lock (UsedFilesListLock)
{
var usedFilesListFilename = GetUsedFilesListFilename();
if (!File.Exists(usedFilesListFilename))
return;
using (var listFile = File.Open(usedFilesListFilename, FileMode.Open))
{
using (var reader = new StreamReader(listFile))
{
string tempFileToDelete;
while ((tempFileToDelete = reader.ReadLine()) != null)
{
if (File.Exists(tempFileToDelete))
File.Delete(tempFileToDelete);
}
}
}
// Clean up
using (File.Open(usedFilesListFilename, FileMode.Truncate)) { }
}
}
}
}
Every time you need temporary file use:
var tempFile = TemporaryFiles.UseNew();
To be sure all temporary files are deleted after application closes or crashes put
TemporaryFiles.DeleteAllPreviouslyUsed();
at start of the application.

It's nice to see that you want to be responsible, but if the files aren't huge (>50MB), you would be in line with everyone (MS included) in leaving them in the temp directory. Disk space is abundant.
As csl posted, GetTempPath is the way to go. Users who are short on space will be able to run disk cleanup and your files (along with everyone else's) will be cleaned up.

I'm not primarily a C# programmer, but in C++ I'd use RAII for this. There are some hints on using RAII-like behaviour in C# online, but most seem to use the finalizer — which is not deterministic.
I think there are some Windows SDK functions to create temporary files, but don't know if they are automatically deleted on program termination. There is the GetTempPath function, but files there are only deleted when you log out or restart, IIRC.
P.S. The C# destructor documentation says you can and should release resources there, which I find a bit odd. If so, you could simply delete the temp file in the destructor, but again, this might not be completely deterministic.

You could launch a thread on startup that will delete files that exist when they "shouldn't" to recover from your crash.

If you're building a Windows Forms Application, you can use this code:
private void Form1_FormClosing(object sender, FormClosingEventArgs e)
{
File.Delete("temp.data");
}