I'm new to snowflake database. I need to read the data from multiple tables and do data mapping like sql tables.
I'm trying to get the data from table using the below code:
`
using (IDbConnection snowflakedb = new SnowflakeDbConnection())
{
snowflakedb.ConnectionString = ConfigurationManager.ConnectionStrings["SnowFlakeParserConnection"].ConnectionString;
snowflakedb.Open();
var cmd = snowflakedb.CreateCommand();
cmd.CommandText = "select * from EMAILPARSER_TABLE;";
var reader = cmd.ExecuteReader();
dynamic accountList;
while (reader.Read())
{
Console.WriteLine(reader.GetString(0));
employeeList = reader.GetString(0);
}
snowflakedb.Close();
}
But its giving only single value- value of first row first column .How to read the entire row of the table? Later I also need to merge the data of two tables.
Also, When do we use snowflake ODBC driver connection:
var defaultConnection = _connectionFactory.GetOdbcConnection;
var snowflakeConnection = new OdbcConnection(defaultConnection.ConnectionString);
As #stuartd says, you are getting one column because that is all your code is requesting. You may want to try something like this code from the Github repo
// All types except TIME fail conversion when calling GetTimeSpan
for (int i = 0; i < 12; i++)
{
try
{
((SnowflakeDbDataReader)reader).GetTimeSpan(i);
Assert.Fail("Data should not be converted to TIME");
}
catch (SnowflakeDbException e)
{
Assert.AreEqual(270003, e.ErrorCode);
}
}
Also see this Stack Overflow answer for another approach using a data set.
As for when you should use the ODBC connection, consider using it as you would make that decision for any other database. One thing that has been noted before by other Snowflake developers is that the ODBC driver seems to be more strongly supported than the .Net driver.
Related
Disclaimer: I have no prior experience with querying databases from c# code (so go easy on me)
I am trying to insert data from my SQL Server database into my listbox. Right now I am trying this in the form of an array. I first connect to the database, and then insert the "state" from the database into the index of the array. I want all 50 states to be put into my array and then this information to be put into my listbox. Right now, my data is being inserted but when I view it in the list box it shows System.Data.SqlClient.SqlCommand.
public string connString = "Not displaying this for security reasons, it is set up correctly though."; //Setting up the connection to my DB
public frmState()
{
InitializeComponent();
this.FormClosed += new System.Windows.Forms.FormClosedEventHandler(this.frmState_FormClosed);
using (SqlConnection dbConn = new SqlConnection(connString))
{
dbConn.Open();
string cmdString = "select State_Name from [State]";
SqlCommand cmd = new SqlCommand(cmdString, dbConn);
SqlDataReader reader = cmd.ExecuteReader();
try
{
while (reader.Read())
{
string[] stateList = new string[50];
for (int i = 1; i <= 50; i++)
{
stateList[i - 1] = cmd.ToString();
}
for (int i = 0; i < stateList.Length; i++)
{
lbStates.Items.Add(stateList[i].ToString());
}
}
}
finally
{
reader.Close();
}
}
}
Also, I am aware that as of right now I will be showing the same state 50 times. I am trying to figure out how to insert one state at a time. Is this an efficient way of doing this? Also, any tips on working with databases in c#? I am on Visual Studio 2017 and Microsoft SQL Server 2016.
The problem comes from where you did:
stateList[i - 1] = cmd.ToString();
It's wrong because you are converting an SqlCommand object to string and putting it inside an array of type of string to retrieve data from your SqlCommand.
Changing the above line as below will fix your problem:
tateList[i - 1] = reader.GetString(0);
any tips on working with databases in c#?
for a beginner with C# and SQL, I suggest you to keep learning basic database access tools of ADO.net like using SqlDataReader, SqlDataReader, SqlDataAdapter, ... . but to have professional and of course secure application witch also needs to be simple; you have to move toward using ORM tool (witch are medium to access database securely) like "Entity Framework", linq, ... witch will make talking to database much more convenient.
Complementary:
I suggest you to reading this tutorial about how to use SqlDataReader.
I've been tasked with creating a backup of the data in our "default schema" database dbo to the same database using a new schema called dbobackup.
I honestly do not understand what this means as far as a database goes. Apparently, it is like having a database backup inside the existing database. I guess there is some advantage to doing that.
Anyway, I can't seem to find anywhere online that will allow me to do this.
I have found a few posts on here about copying the schema without data, but I need the data too.
Backup SQL Schema Only?
How do I check to see if a schema exists, delete it if it does, and then create a schema that accepts data in the current database?
Once I have the new schema created, can I dump data in there with a simple command like this?
SELECT * INTO [dbobackup].Table1 FROM [dbo].Table1;
That line only backs up one table, though. If I need to do this to 245 tables for this particular customer, I'd need a script.
We have several customers, too, and their databases are not structured identically.
Could I do something along these lines?
I was thinking about creating a small console program to walk through the tables.
How would I modify something like the code below to do what I want?
public static void Backup(string sqlConnection)
{
using (var conn = new SqlConnection(sqlConnection))
{
conn.Open();
var tables = new List<String>();
var sqlSelectTables = "SELECT TableName FROM [dbo];";
using (var cmd = new SqlCommand(sqlSelectTables, conn))
{
using (var r = cmd.ExecuteReader())
{
while (r.Read())
{
var item = String.Format("{0}", r["TableName"]).Trim();
tables.Add(item);
}
}
}
var fmtSelectInto = "SELECT * INTO [dbobackup].{0} FROM [dbo].{0}; ";
using (var cmd = new SqlCommand(null, conn))
{
foreach (var item in tables)
{
cmd.CommandText = String.Format(fmtSelectInto, item);
cmd.ExecuteNonQuery();
}
}
}
}
SQL Server already has this built in. If you open SQL Server Management Studio and right click on the database you want to back up, then select all tasks then backup, you will get an option to back up your database into an existing database.
This is the important part and why you should use the built in functionality: You must copy the data from one DB to the other DB in the correct order or you'll get foreign key errors all over the place. If you have a lot of data tables with a lot of relationships, this will really be hard to nail down on your own. You could write code to make a complete graph of all of the dependencies and then figure out what order to copy the table data (which is essentially what SQL Server already does).
Additionally, there are third-party programs available to do this type of backup as well (see: Google).
This is sort of a "work in progress" approach I got started with that looks promising:
public static void CopyTable(
string databaseName, // i.e. Northwind
string tableName, // i.e. Employees
string schema1, // i.e. dbo
string schema2, // i.e. dboarchive
SqlConnection sqlConn)
{
var conn = new Microsoft.SqlServer.Management.Common.ServerConnection(sqlConn);
var server = new Microsoft.SqlServer.Management.Smo.Server(conn);
var db = new Microsoft.SqlServer.Management.Smo.Database(server, databaseName);
db.Tables.Refresh();
for (var itemId = 0; itemId < db.Tables.Count; itemId++)
{
var table = db.Tables.ItemById(itemId);
if (table.Name == tableName)
{
table.Schema = String.Format("{0}", DatabaseSchema.dboarchive);
table.Create();
}
}
}
The only issue I am currently running into is that my db variable always comes back with Tables.Count == 0.
If I get a chance to fix this, I will update.
For now, I've been told to remove this piece of code and check my code in.
Please read the comments of the answer for a more complete understanding of what the problem is/was
First, I read through a lot of the other SO questions related to this and still can't get this to work with a basic setup. Here is the related question I have already read:
Passing query parameters in Dapper using OleDb
EDIT: The troubleshooting below is somewhat misleading. The only thing that was going wrong was the query syntax from the Github example was not valid using the ProgressDB OpenEdge driver.
The problem with that question's answer and with the example given in the documented Git examples is that a true ODBC object is not being used, but rather an OleDbConnection object. This causes problems with the scenario where I am trying to use Dapper. Some background and restrictions to my scenario:
I cannot change the DB technology, we are connecting to an Progress DB. The connection string to connect to the DB: connectionString="PROVIDER=MSDASQL;DRIVER={Progress OpenEdge 10.2A Driver};HOST=...;PORT=...;DB=mfgsys;UID=...;PWD=...;DIL=READ UNCOMMITTED" Notice the Provider: MSDASQL
According to MSDN, https://msdn.microsoft.com/en-us/library/a6cd7c08%28v=vs.110%29.aspx - "The .NET Framework Data Provider for OLE DB does not work with the OLE DB provider for ODBC (MSDASQL). To access an ODBC data source using ADO.NET, use the .NET Framework Data Provider for ODBC."
When I attempt to use the OdbcConnection object with Dapper I get the following error: "System.Data.Odbc.OdbcException : ERROR [HY000] [DataDirect][ODBC Progress OpenEdge Wire Protocol driver][OPENEDGE]Syntax error in SQL statement at or about "= ?, Age = ?" (10713)"
I am using the exact same query syntax as the other SO question:
var row = _odbcConn.Query("select Id = ?, Age = ?", new DynamicParameters(new{foo = 12, bar = 23}) {RemoveUnused = false}).Single();
I also removed the DynamicParameters object and attempted with a dynamic object with same result:
var row = _odbcConn.Query("select Id = ?, Age = ?", new{foo = 12, bar = 23}).Single();
Is there a way to accomplish this simple query using an OdbcConnection object? Or does this really have more to do with the specific Progress driver we are using and as such precludes using Dapper?
Edit
Including working ADO.Net code per requests below, the Build.FromReader<EmployeeDataModel>(reader) just loops through the reader and maps the columns with hard coding and is confirmed to work:
public class EmployeeRepository : IEmployeeRepository
{
private readonly OdbcConnection _sqlConn = new OdbcConnection();
public EmployeeRepository() : this(ConfigurationManager.ConnectionStrings["TCI_Epicor"].ConnectionString) { }
public EmployeeRepository(string connString)
{
_sqlConn.ConnectionString = connString;
}
public EmployeeDataModel GetById(string id)
{
try
{
_sqlConn.Open();
using (OdbcCommand command = new OdbcCommand())
{
command.Connection = _sqlConn;
command.CommandType = CommandType.Text;
command.CommandText = GetEmployeeDataQuery();
command.Parameters.Add("empID", OdbcType.NVarChar);
command.Parameters["empID"].Value = id;
var reader = command.ExecuteReader();
return Build.FromReader<EmployeeDataModel>(reader);
}
}
catch
{
return new EmployeeDataModel();
}
finally
{
_sqlConn.Close();
}
}
private string GetEmployeeDataQuery()
{
var sb = new StringBuilder();
sb.AppendLine("SELECT EmpID as 'EmployeeID',");
sb.AppendLine(" FirstName + ' ' + LastName as 'EmployeeName'");
sb.AppendLine(" FROM MFGSYS.PUB.EmpBasic");
sb.AppendLine(" WHERE EmpID = ?");
return sb.ToString();
}
}
If the problem is using anonymous (?) parameters, then:
var row = _odbcConn.Query(
"select Id = ?foo?, Age = ?bar?", new { foo = 12, bar = 23 }
).Single();
Dapper will rewrite that as per your original query, but will know which parameter to put where.
However, if the problem is that the ODBC provider does not support parameters: I can't help much with that :( If you can show how to do it in working ADO.NET code, I can probably show you how to do it easier via dapper.
It has been a long time since I have used .NET, but thankfully have almost finished writing a tool to compare an sqlite and mysql database. I am running into an issue though when trying to write a function for my wrapper that will handle SELECT calls as I cannot entirely figure out the Data Reader.
My understanding is that each iteration of a loop on the reader is the next row, and GetString(x) returns the column value for "x" as the index. All the examples I found though went into it knowing the row/column names they needed. How can I do this with a "SELECT * FROM" call and save the column names/values for later access? Microsoft seems to have a "FieldCount" function but I am coming up empty on the MySQL Connector.
Thanks!
public void Query(string query, string tableName)
{
//Open connection
if (this.OpenConnection() == true)
{
//Create Command
MySqlCommand cmd = new MySqlCommand(query, connection);
MySqlDataReader dataReader = cmd.ExecuteReader();
//Read the data and store them in the list
while (dataReader.Read())
{
int count = Count(tableName);
for (int x = 0; x < count; x++)
{
Console.WriteLine(dataReader.GetString(count));
}
}
//close Data Reader
dataReader.Close();
//close Connection
this.CloseConnection();
}
}
You can use DbDataReader.GetName to get the name of a column given its ordinal x.
use the "mysql connector" to access data in MySql it is more simple then to write the queries by your self:
http://dev.mysql.com/downloads/connector/net/
Then use the EntityFramework to access the data through this connector. Also you can automaticly generate *.edmx model from your existing DB in mysql, this will let you to fastly access and work with the Data in your database. Here is information about adding *.edmx model from existing DB:
http://msdn.microsoft.com/en-us/library/vstudio/cc716703(v=vs.100).aspx
The query which will select all data from the table, for example - "Products" will look like that:
List products = dbContext.Products.Where(e=>e.ProductId!=-1).ToList();
this will return the whole list of products in your data base in table Products.
then you can work with products as you want. for example taking the "Name" column for the first product in 'products' will look like that:
String firstProductName = products[0].name;
I hope it helps.
I have a DataSet populated from Excel Sheet. I wanted to use SQLBulk Copy to Insert Records in Lead_Hdr table where LeadId is PK.
I am having following error while executing the code below:
The given ColumnMapping does not match up with any column in the
source or destination
string ConStr=ConfigurationManager.ConnectionStrings["ConStr"].ToString();
using (SqlBulkCopy s = new SqlBulkCopy(ConStr,SqlBulkCopyOptions.KeepIdentity))
{
if (MySql.State==ConnectionState.Closed)
{
MySql.Open();
}
s.DestinationTableName = "PCRM_Lead_Hdr";
s.NotifyAfter = 10000;
#region Comment
s.ColumnMappings.Clear();
#region ColumnMapping
s.ColumnMappings.Add("ClientID", "ClientID");
s.ColumnMappings.Add("LeadID", "LeadID");
s.ColumnMappings.Add("Company_Name", "Company_Name");
s.ColumnMappings.Add("Website", "Website");
s.ColumnMappings.Add("EmployeeCount", "EmployeeCount");
s.ColumnMappings.Add("Revenue", "Revenue");
s.ColumnMappings.Add("Address", "Address");
s.ColumnMappings.Add("City", "City");
s.ColumnMappings.Add("State", "State");
s.ColumnMappings.Add("ZipCode", "ZipCode");
s.ColumnMappings.Add("CountryId", "CountryId");
s.ColumnMappings.Add("Phone", "Phone");
s.ColumnMappings.Add("Fax", "Fax");
s.ColumnMappings.Add("TimeZone", "TimeZone");
s.ColumnMappings.Add("SicNo", "SicNo");
s.ColumnMappings.Add("SicDesc", "SicDesc");
s.ColumnMappings.Add("SourceID", "SourceID");
s.ColumnMappings.Add("ResearchAnalysis", "ResearchAnalysis");
s.ColumnMappings.Add("BasketID", "BasketID");
s.ColumnMappings.Add("PipeLineStatusId", "PipeLineStatusId");
s.ColumnMappings.Add("SurveyId", "SurveyId");
s.ColumnMappings.Add("NextCallDate", "NextCallDate");
s.ColumnMappings.Add("CurrentRecStatus", "CurrentRecStatus");
s.ColumnMappings.Add("AssignedUserId", "AssignedUserId");
s.ColumnMappings.Add("AssignedDate", "AssignedDate");
s.ColumnMappings.Add("ToValueAmt", "ToValueAmt");
s.ColumnMappings.Add("Remove", "Remove");
s.ColumnMappings.Add("Release", "Release");
s.ColumnMappings.Add("Insert_Date", "Insert_Date");
s.ColumnMappings.Add("Insert_By", "Insert_By");
s.ColumnMappings.Add("Updated_Date", "Updated_Date");
s.ColumnMappings.Add("Updated_By", "Updated_By");
#endregion
#endregion
s.WriteToServer(sourceTable);
s.Close();
MySql.Close();
}
I've encountered the same problem while copying data from access to SQLSERVER 2005 and i found that the column mappings are case sensitive on both data sources regardless of the databases sensitivity.
Well, is it right? Do the column names exist on both sides?
To be honest, I've never bothered with mappings. I like to keep things simple - I tend to have a staging table that looks like the input on the server, then I SqlBulkCopy into the staging table, and finally run a stored procedure to move the table from the staging table into the actual table; advantages:
no issues with live data corruption if the import fails at any point
I can put a transaction just around the SPROC
I can have the bcp work without logging, safe in the knowledge that the SPROC will be logged
it is simple ;-p (no messing with mappings)
As a final thought - if you are dealing with bulk data, you can get better throughput using IDataReader (since this is a streaming API, where-as DataTable is a buffered API). For example, I tend to hook CSV imports up using CsvReader as the source for a SqlBulkCopy. Alternatively, I have written shims around XmlReader to present each first-level element as a row in an IDataReader - very fast.
The answer by Marc would be my recomendation (on using staging table). This ensures that if your source doesn't change, you'll have fewer issues importing in the future.
However, in my experience, you can check the following issues:
Column names match in source and table
That the column types match
If you think you did this and still no success. You can try the following.
1 - Allow nulls in all columns in your table
2 - comment out all column mappings
3 - rerun adding one column at a time until you find where your issue is
That should bring out the bug
One of the reason is that :SqlBukCOpy is case sensitive . Follow steps:
In that Case first you have to find your column in Source Table by
using "Contain" method in C#.
Once your Destination column matched with source column get index of
that column and give its column name in SqlBukCOpy .
For Example:`
//Get Column from Source table
string sourceTableQuery = "Select top 1 * from sourceTable";
DataTable dtSource=SQLHelper.SqlHelper.ExecuteDataset(transaction, CommandType.Text, sourceTableQuery).Tables[0];// i use sql helper for executing query you can use corde sw
for (int i = 0; i < destinationTable.Columns.Count; i++)
{ //check if destination Column Exists in Source table
if (dtSource.Columns.Contains(destinationTable.Columns[i].ToString()))//contain method is not case sensitive
{
int sourceColumnIndex = dtSource.Columns.IndexOf(destinationTable.Columns[i].ToString());//Once column matched get its index
bulkCopy.ColumnMappings.Add(dtSource.Columns[sourceColumnIndex].ToString(), dtSource.Columns[sourceColumnIndex].ToString());//give coluns name of source table rather then destination table so that it would avoid case sensitivity
}
}
bulkCopy.WriteToServer(destinationTable);
bulkCopy.Close();
I would go with the staging idea, however here is my approach to handling the case sensitive nature. Happy to be critiqued on my linq
using (SqlConnection connection = new SqlConnection(conn_str))
{
connection.Open();
using (SqlBulkCopy bulkCopy = new SqlBulkCopy(connection))
{
bulkCopy.DestinationTableName = string.Format("[{0}].[{1}].[{2}]", targetDatabase, targetSchema, targetTable);
var targetColumsAvailable = GetSchema(conn_str, targetTable).ToArray();
foreach (var column in dt.Columns)
{
if (targetColumsAvailable.Select(x => x.ToUpper()).Contains(column.ToString().ToUpper()))
{
var tc = targetColumsAvailable.Single(x => String.Equals(x, column.ToString(), StringComparison.CurrentCultureIgnoreCase));
bulkCopy.ColumnMappings.Add(column.ToString(), tc);
}
}
// Write from the source to the destination.
bulkCopy.WriteToServer(dt);
bulkCopy.Close();
}
}
and the helper method
private static IEnumerable<string> GetSchema(string connectionString, string tableName)
{
using (SqlConnection connection = new SqlConnection(connectionString))
using (SqlCommand command = connection.CreateCommand())
{
command.CommandText = "sp_Columns";
command.CommandType = CommandType.StoredProcedure;
command.Parameters.Add("#table_name", SqlDbType.NVarChar, 384).Value = tableName;
connection.Open();
using (var reader = command.ExecuteReader())
{
while (reader.Read())
{
yield return (string)reader["column_name"];
}
}
}
}
What I have found is that the columns in the table and the columns in the input must at least match. You can have more columns in the table and the input will still load. If you have less you'll receive the error.
Thought a long time about answering...
Even if column names are case equally, if the data type differs
you get the same error. So check column names and their data type.
P.S.: staging tables are definitive the way to import.