As a database developer with a very small amount of programming experience, I'm currently attempting to develop a C# .NET console application to import XML files into a SQL Server database. Once the import is done, I need to create a 'Response' file to be passed back to another application.
In doing my own research, I've come across the SQLBULKLOAD class. In fact, I've found some example code online that shows (at least partially) exactly what I'm trying to do:
using System;
using System.IO;
using System.Collections;
using SQLXMLBULKLOADLib;
using System.Data.OleDb;
using System.Diagnostics;
namespace SQLXmlExample
{
class Program
{
[STAThread]
static void Main(string[] args)
{
string schema = "C:\\ImportSample\\MappingFile.xml";
string datafile = "C:\\ImportSample\\DataFile.xml";
string connectionString = #"provider=SQLOLEDB;data source=localhost;database=SqlXmlDemo;Integrated Security=SSPI;";
for (int i = 0; i < args.Length; i++)
{
switch (args[i].ToLower())
{
case "-schema":
schema = args[i + 1];
break;
case "-datafile":
datafile = args[i + 1];
break;
}
}
if (schema == string.Empty || datafile == string.Empty)
{
Console.WriteLine("Missing Schema or Data File. Format: SqlXmlExample -datafile [filename] -schema [filename]");
return;
}
Load(datafile, schema, connectionString);
}
static public void Load(string XMLFilename, string XMLMappingFilename, string ConnectionString)
{
SQLXMLBULKLOADLib.SQLXMLBulkLoad loader = new SQLXMLBULKLOADLib.SQLXMLBulkLoad();
loader.CheckConstraints = true;
loader.XMLFragment = true;
loader.SchemaGen = true;
loader.SGDropTables = false;
loader.Transaction = false;
loader.ConnectionString = ConnectionString;
loader.Execute("C:\\ImportSample\\MappingFile.xml", "C:\\ImportSample\\DataFile.xml");
}
}
}
With the code above, I'm able to import an .XML file into a SQL Server instance. However, my problem is generating the 'Response' .XML file to provide information about the bulk loading operation (i.e. how many records were inserted, if the insert was successful).
As it stands now, I'm thinking that I may be using the wrong class for what I'm trying to accomplish. Am I using the correct class, or should I be using a different one?
If possible, could anyone point me in the direction of some more material to assist me? Any help would be greatly appreciated.
How about using the ErrorLogFile?
This is where the bulk loader stores all its errors and messages, simply try one out, make it fail, see what the format is and then you can then load, parse and generate xml?
Related
I've got a Windows 10 UWP application written in C#. I'm using SQLite to store my data locally. The issue I'm experiencing is that the file is never saved and/or retrieved using this code. It should work, but I can't find out what's wrong.
dbExists always evaluates to false, so what am I missing here?
private SQLiteConnection localConn;
private string dbPath = Path.Combine(Windows.Storage.ApplicationData.Current.LocalFolder.Path, "myDatabase.db");
public async void DBInit()
{
bool dbExists = false;
try
{
var store = await ApplicationData.Current.LocalFolder.GetFileAsync(dbPath);
dbExists = true;
}
catch { dbExists = false; }
if (!dbExists)
{
using (localConn = new SQLiteConnection(new SQLitePlatformWinRT(), dbPath))
{
// Create table
localConn.CreateTable<MyTable>();
}
}
else // CURRENTLY NOT FIRING!!
{}
}
Please consider using below code to create and access database file:
StorageFile notesFile = await storageFolder.CreateFileAsync(dbPath, CreationCollisionOption.OpenIfExists);
This will create new file if it does not exists and retrieve it when it is already created.
Please check my blog article to see more about UWP Data Storage:
https://mobileprogrammerblog.wordpress.com/2016/05/23/universal-windows-10-apps-data-storage/
I think you're missing this important piece of code:
SQLiteConnection.CreateFile("mydatabase.sqlite");
Do that first, then create a connection instance referencing the (now) created file.
Also, I'd suggest that you name the db with the .sqlite extension, so that the rest of the team and incoming devs, when then look at the db file artifact, can immediately tell that this is an sqlite database.
EDIT:
The method is a static method. So you would use it like this...
using System.Data.SQLite;
namespace sqlite_sample
{
class Program
{
static void Main(string[] args)
{
SQLiteConnection.CreateFile("sample.db");
}
}
}
The following will not work:
var conn = SQLiteConnection(...);
conn.CreateFile(dbPath); //<-- static methods can't be invoked at the instance level...
I'm trying to submit a MapReduce job to HDInsight cluster. In my job I didn't write reduce portion because I don't want to reduce anything. All I want to do is to parse the each filename and append the values to every line in the file. So that I will have all the data needed inside the file.
My code is
using Microsoft.Hadoop.MapReduce;
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
namespace GetMetaDataFromFileName
{
class Program
{
static void Main(string[] args)
{
var hadoop = connectAzure();
//Temp Workaround to Env Variables
Environment.SetEnvironmentVariable("HADOOP_HOME", #"c:\hadoop");
Environment.SetEnvironmentVariable("Java_HOME", #"c:\hadoop\jvm");
var result = hadoop.MapReduceJob.ExecuteJob<MetaDataGetterJob>();
}
static IHadoop connectAzure()
{
//TODO: Update credentials and other information
return Hadoop.Connect(
new Uri("https://sampleclustername.azurehdinsight.net//"),
"admin",
"Hadoop",
"password",
"blobstoragename.blob.core.windows.net", //Storage Account that Log files exists
"AccessKeySample", //Storage Account Access Key
"logs", //Container Name
true
);
}
//Hadoop Mapper
public class MetaDataGetter : MapperBase
{
public override void Map(string inputLine, MapperContext context)
{
try
{
//Get the meta data from name of the file
string[] _fileMetaData = context.InputFilename.Split('_');
string _PublicIP = _fileMetaData[0].Trim();
string _PhysicalAdapterMAC = _fileMetaData[1].Trim();
string _BootID = _fileMetaData[2].Trim();
string _ServerUploadTime = _fileMetaData[3].Trim();
string _LogType = _fileMetaData[4].Trim();
string _MachineUpTime = _fileMetaData[5].Trim();
//Generate CSV portion
string _RowHeader = string.Format("{0},{1},{2},{3},{4},{5},", _PublicIP, _PhysicalAdapterMAC, _BootID, _ServerUploadTime, _LogType, _MachineUpTime);
//TODO: Append _RowHeader to every row in the file.
context.EmitLine(_RowHeader + inputLine);
}
catch(ArgumentException ex)
{
return;
}
}
}
//Hadoop Job Definition
public class MetaDataGetterJob : HadoopJob<MetaDataGetter>
{
public override HadoopJobConfiguration Configure(ExecutorContext context)
{
//Initiate the job config
HadoopJobConfiguration config = new HadoopJobConfiguration();
config.InputPath = "asv://logs#sample.blob.core.windows.net/Input";
config.OutputFolder = "asv://logs#sample.blob.core.windows.net/Output";
config.DeleteOutputFolder = true;
return config;
}
}
}
}
Usually what do you thing the reason of 500 (Server Error) ? Am I suppling to wrong credentials ? Actually I didn't really understand the difference between Username and HadoopUser parameters in Hadoop.Connect method ?
Thank you,
I had approximately same issue in the past (was unable to submit hive job to the cluster with BadGateway response). I have contacted the support team and in my case the problem was in memory leakage at the head node, what means that the problem was not at client's side and it seems to be inherited hadoop problem.
I've solved that stuff by redeploying the cluster.
Have you tried to submit other jobs (simple ones)? If so, than I suggest to have a contact with azure support team or just redeploy the cluster if it's not painful for you.
I found this answer https://stackoverflow.com/a/14336292/1537195 which gave a good way to detect password protection for DOC and XLS files.
//Flagged with password
if (bytes.Skip(0x20c).Take(1).ToArray()[0] == 0x2f) return true; //XLS 2003
if (bytes.Skip(0x214).Take(1).ToArray()[0] == 0x2f) return true; //XLS 2005
if (bytes.Skip(0x20B).Take(1).ToArray()[0] == 0x13) return true; //DOC 2005
However it does not seem to cover all XLS files and I am also looking for a way to detect PPT files in the same manner. Does anyway know which bytes to look at for these file types?
I saved a PowerPoint presentation as .ppt and .pptx with and without a password required for opening them, opened them in 7-Zip and came to the tentative conclusion that
.pptx files without a password always use a standard .zip file format
.ppt files are CompoundDocuments
.pptx files with a password also CompoundDocuments
All passworded CompoundDocuments contain an entry named *Encrypt*
To get this code running, you need to installed the NuGet package OpenMcdf. This is the first C# library that I could find for reading CompoundDocuments.
using OpenMcdf;
using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
namespace _22916194
{
//http://stackoverflow.com/questions/22916194/detecing-password-protected-ppt-and-xls-documents
class Program
{
static void Main(string[] args)
{
foreach (var file in args.Where(File.Exists))
{
switch (Path.GetExtension(file))
{
case ".ppt":
case ".pptx":
Console.WriteLine($"* {file} " + (HasPassword(file) ? "is " : "isn't ") + "passworded");
Console.WriteLine();
break;
default:
Console.WriteLine($" * Unknown file type: {file}");
break;
}
}
Console.ReadLine();
}
private static bool HasPassword(string file)
{
try
{
using (var compoundFile = new CompoundFile(file))
{
var entryNames = new List<string>();
compoundFile.RootStorage.VisitEntries(e => entryNames.Add(e.Name), false);
//As far as I can see, only passworded files contain an entry with a name containing Encrypt
foreach (var entryName in entryNames)
{
if (entryName.Contains("Encrypt"))
return true;
}
compoundFile.Close();
}
}
catch (CFFileFormatException) {
//This is probably a .zip file (=unprotected .pptx)
return false;
}
return false;
}
}
}
You should be able to extend this code to handle other Office formats. The conclusions at the top should hold true, except that you need to look for some other data in the CompoundDocument than a filename containing *Encrypt* (I had a quick look at .doc files and it didn't seem to work exactly the same).
The full Code can be found here: http://home.htw-berlin.de/~s0531210/eb/DataBaseTest.zip
It is a simple Project with testing Entity Framework.
I have a DLL that allows access to a SQL Server Compact database. This access happens by Enttiy Framework 5.0.
A second project is a console application that accesses this DLL. When calling a class from the DLL to store sample data into the database, the exception is "Error underlying provider Open."
This exception occurs when calling: db.SaveChanges ();
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
namespace DatabaseLibrary
{
public class SLD
{
public SLD()
{
}
public void enterData()
{
using (var db = new SLDDatabaseModelEntitiesContext())
{
for (int i = 0; i < 10; i++)
{
SLDEntity entrysfoo = new SLDEntity();
entrysfoo.Flip = i;
entrysfoo.Slidename = "bla" + i;
db.SLDEntity.Add(entrysfoo);
}
db.SaveChanges(); //DAtanbank speichern
}
}
public SLDEntity getFromDataBase(string wsiname)
{
using (var db = new SLDDatabaseModelEntitiesContext())
{
foreach (var item in db.SLDEntity)
{
if (item.Slidename.Equals(wsiname))
{
return item;
}
}
}
return new SLDEntity();
}
}
}
i hope you guys can help me. I have no clue where the problem is. I searched the internet and i found something about persmission iusses, but the connectionstring is reqiredpermissin=false.
thanks the tip with the connection string worked. The Database was not there where the App.config file from the consoleapp pointed at. but I do not understand why the connection string in the dll, which one also has a App.config File is ignored. If I want, that the connection string is available only in the DLL, Do I have to set the connection string in the DLL manually via a command?
I am making a program updater / launcher that can be used for any program.
I have a config file on the client and a config file on a http server. I get version numbers from both of them and compare them and if they are are not = then update the client.
I have everything working except for when the update starts. What I need is say if someone downloaded my application and do not use if for say a month and in between that time I have 5 or so updates.
The problem is how to I get my program to download the first update , install it and then download the next update untill they have all been downloaded?
I am new to programming and this is the only kind of app I can think of to work on to learn.
Thanks
My settings.conf on http server XML File.
<Table>
<Product>
<Product_id>1</Product_id>
<Product_name>Infected</Product_name>
<Product_version>1.0.0.1</Product_version>
<Product_Url>http://localhost/update/v1.0.0.1.exe</Product_Url>
<Product_id>2</Product_id>
<Product_name>Infected</Product_name>
<Product_version>1.0.0.2</Product_version>
<Product_Url>http://localhost/update/v1.0.0.2.exe</Product_Url>
<Product_id>3</Product_id>
<Product_name>Infected</Product_name>
<Product_version>1.0.0.3</Product_version>
<Product_Url>http://localhost/update/v1.0.0.3.exe</Product_Url>
<Product_id>4</Product_id>
<Product_name>Infected</Product_name>
<Product_version>1.0.0.4</Product_version>
<Product_Url>http://localhost/update/v1.0.0.4.exe</Product_Url>
<Product_id>5</Product_id>
<Product_name>Infected</Product_name>
<Product_version>1.0.0.5</Product_version>
<Product_Url>http://localhost/update/v1.0.0.5.exe</Product_Url>
</Product>
</Table>
My Client Config XML file.
<Table>
<Product>
<Product_id>1</Product_id>
<Product_name>Infected</Product_name>
<Product_version>1.0.0.0</Product_version>
<Product_Url>http://localhost/update/v1.0.0.1.exe</Product_Url>
</Product>
</Table>
My C# Form.
using System;
using System.Collections.Generic;
using System.ComponentModel;
using System.Data;
using System.Drawing;
using System.Linq;
using System.Text;
using System.Windows.Forms;
using System.Xml;
using System.IO;
using System.Net;
using System.Diagnostics;
using System.Runtime.Remoting;
namespace Launcher
{
public partial class Form1 : Form
{
public Form1()
{
InitializeComponent();
}
public string localversion { get; set; }
public string remoteversion { get; set; }
public string UpdateURL { get; set; }
private void Form1_Load(object sender, EventArgs e)
{
webBrowser1.Navigate("http://www.kceoc.com/");
webBrowser2.Navigate("http://www.kceoc.com/");
button1.Enabled = false; // Disable the launch button untill all updates are completed.
GetLocalXMLFile(); //Run first xml function to start everything off.
}
private void GetLocalXMLFile()
{
try //Start error checking.
{
using (XmlTextReader localxml = new XmlTextReader("settings.conf")) //Load xml file in same folder as launcher.exe
{
while (localxml.Read()) // Start reading the settings.conf file
{
switch (localxml.NodeType) //Get the Node that we will use.
{
case XmlNodeType.Text:
label1.Text = localxml.Value; //Change the text of label1 to value of Node.
string localversion = localxml.Value; // Store Node Value in string localversion for latter use.
GetRemoteXMLFile(localversion, remoteversion); //Everything went ok and got a value from Node so pass this all to our next function witch is get remote xml.
break;
}
}
}
}
catch (FileNotFoundException)
{
label1.Text = "Local Config not found. Reinstall the application"; // Catch error incase file is not there.
}
}
private void GetRemoteXMLFile(string localversion, string remoteversion)
{
try //Start error checking
{
using (XmlTextReader remotexml = new XmlTextReader("http://localhost/update/settings.conf")) //Load up remote xml on web server
{
while (remotexml.Read()) //Start reading xml file from server.
{
switch (remotexml.NodeType)
{
case XmlNodeType.Text:
label2.Text = remotexml.Value; // Change value of label2 to remote xml node value
remoteversion = remotexml.Value; // Set the remoteversion string to remotexml.value
CompareXMLFileVersions(localversion, remoteversion); // Everything went ok so send localversion string and remoteversion string to compare function.
break;
}
}
}
}
catch (FileNotFoundException)
{
label1.Text = "Remote config not found. Maby website id down?"; // Catch error incase file is not there.
}
}
private void CompareXMLFileVersions(string localversion, string remoteversion)
{
label1.Text = localversion; // Just so we can see the value in the lables to konw if they have value or not.
label2.Text = remoteversion; // Just so we can see the value in the lables to konw if they have value or not.
if (localversion == remoteversion) // Comparing the values of localversion and remoteversion and if they have same value then
{ // change label3 to You have latest version.
label3.Text = "You have the latest version";
}
else
{
label3.Text = "There is a new version. Starting update process here"; // If localversion and remoteversion are diffrent then let user know the files are out of date and start the updating process..
GetListOfUpdates(remoteversion); // Starting the updating process function..
}
}
private void GetListOfUpdates(string remoteversion)
{
//WebClient webClient = new WebClient();
//webClient.DownloadFileCompleted += new AsyncCompletedEventHandler(Completed);
//webClient.DownloadProgressChanged += new DownloadProgressChangedEventHandler(ProgressChanged);
//webClient.DownloadFileAsync(new Uri(remoteversion), #"v1.0.0.1.exe");
string url = "http://localhost/update/v1.0.0.1.exe";
WebClient downloader = new WebClient();
downloader.DownloadFileCompleted += new AsyncCompletedEventHandler(downloader_DownloadFileCompleted);
downloader.DownloadProgressChanged += new DownloadProgressChangedEventHandler(downloader_DownloadProgressChanged);
downloader.DownloadFileAsync(new Uri(url), "temp.exe");
}
void downloader_DownloadProgressChanged(object sender, DownloadProgressChangedEventArgs e)
{
label1.Text = e.BytesReceived + " " + e.ProgressPercentage;
}
void downloader_DownloadFileCompleted(object sender, AsyncCompletedEventArgs e)
{
//if (e.Error != null)
// MessageBox.Show(e.Error.Message);
//else
// MessageBox.Show("Completed!!!");
}
}
}
Thanks
Welcome to SO!
I am new to programming and this is the only kind of app I can think of to work on to learn. Thanks
Depending on how new you are, I'd really recommend you start with something a little easier. Otherwise, the first thing I'd recommend you do is to actually draw a flowchart. Your logic looks a little off, and it looks as if you're huffing it trying to design this system as you write it, which is something you never want to be doing.
There are a lot of solutions for this that provide a better, more reliable systems then anything you can make yourself, but I can understand the educational value of this sort of project. I made my own 'auto-update/launcher' recently for just that reason, and it works reasonably well, albeit on a free webserver with myself an some friends as the only users.
Here's the flowchart I made for it:
Large:http://i.imgur.com/qS1U8.png
This is actually the second iteration of my little project, with the first being less then overwhelming and somewhat disastrous in uncommon circumstances, but it's a good learning experiance. This one also has goofy command files that I can define things like showing messages to the user during an update, which is nice.
If you don't mind looking at terrible and messy code, you can look through the code repo here, although it's not documented and a few part's aren't actually used but haven't been removed from source control. An example application that uses it is here (source, also messy).
Sorry for what looks like a shameless self-plug, but I can't really answer your question directly and hope that you might be able to make use of some of this as an indication of how you should go about doing this, since it's actually a pretty fun project.