I have an application that reads from an SQL Server CE 4.0 database file.
The user has the option on startup to choose a database. Each database has the same schema, but different data.
Given that I want to ensure that they dont use an invalid database (or point the app at a word file or something), Is it possible to validate the schema of a selected database?
In the past I have used ADO.net to check that each column in each table exists but this seems dreadfully silly when entity framework is there. surely there must be something in EF that performs this, but I cant find it.
I am looking for an answer more sophsticated than "Run a query and if it fails then the database is invalid" as there could be many other reasons why such a query would fail
There is no functionality in EF to do this.
You can use my SQL CE Scripting API, available via NuGet http://www.nuget.org/packages/ErikEJ.SqlCeScripting/
First use
DataSet GetSchemaDataSet(GetAllTableNames());
And save it and add to your app.
Then use
DataSet GetSchemaDataSet(GetAllTableNames());
on the loaded database
Then compare the two DataSets
DataSet dsDifferences = new Dataset();
dsOriginal.Merge(dsChanged);
dsDifferences = dsOriginal.GetChanges();
(If dsDifferences has tables with rows, then there were differences)
My library also has a method DetermineVersion(string fileName) to check if the file appears to be a valid SQLCE file.
Related
I have almost the exact same issue as the scenario (linked) below, but unfortunately i'm unable to recreate the solutions succesfully.
I have a c# application using SQL Bulk Import with a datareader and writetoserver, where it's the SQLDatReader or an OracleDataReader, and i need to add columns to the result set.
I can not do it on the source sql statement.
I can not load a data table first and modify it (as it's 100's of gb's of data, almost a terabyte).
How to add columns to DataReader
can anyone provide a working example an help "push" me over this problem?
I temporarily found a solution of using SQL Server Integration Services (SSIS), but what i found while watching it run is it downloads all the data to a dts_buffer, than does the column modifications and then pumps the data into sql server, try doing that with a couple 100gb of data and it is not a good performing thing, if you can even get your infrastructure to build you a 24 core VM with 128gb of memory).
I finally have a small working example, the codeproject (jdweng) was helpful.
I will pose followup, i've tested with sql server (sqldatareader), need to do a test with oracle data reader.
One of the cases i was trying was converting a oracle unique id (stored as a string) to sql server as a uniqueidentifier. I want convert that on the fly, there is no way to adjust the source oracle statement (ADyson) to return a compatible datatype to sql server. Altering a 1tb table afterwards from varchar(40) to uniqueidentifier is painful, but if i could just change as part of the bulk insert, it'd be quick.
and i think now i will be able to.
I'm looking for good ideas how to copy the whole Schema from a SqlAzure Database to another SQL Azure Database. (Tables, Data)
Goal:
I'd like to use a "template" Database seperated with Schemas. Each schema represent another kind of "template" data. On creating a new database (ef6, c#) I want to add the selected Template-Data...
What I tried:
=> Creating bacpac and try to import this.
Works great, but you can only import to new database, not to an existing database.
=> Creating scripts
Works but painfull.
Anyone can give me an advise ?
Have you tried using the schema compare feature in the SQL Server Data Tools available for Visual Studio? It will allow you to compare the complete schema of your template database and your target database. Once it has done the diff, you should use Group By > Schema to organize the results which will then enable you to selectively apply the changes for one or more specific Schema(s) to the target database. For more info on SQL Server Data Tools see https://msdn.microsoft.com/en-us/mt186501
I have a VERY large (50 million+ records) dataset that I am importing from an old Interbase database into a new SQL Server database.
My current approach is:
acquire csv files from the Interbase database (done, used a program called "FBExport" I found somewhere online)
The schema of the old database doesn't match the new one (not under my control), so now I need to mass edit certain fields in order for them to work in the new database. This is the area I need help with
after editing to the correct schema, I am using SqlBulkCopy to copy the newly edited data set into the SQL Server database.
Part 3 works very quickly, diagnostics shows that importing 10,000 records at once is done almost instantly.
My current (slow) approach to part 2 is I just read the csv file line by line, and lookup the relevant information (ex. the csv file has an ID that is XXX########, whereas the new database has a separate column for each XXX and ########. ex2. the csv file references a model via a string, but the new database references via an ID in the model table) and then insert a new row into my local table, and then SqlBulkCopy after my local table gets large.
My question is: What would be the "best" approach (perfomance wise) for this data-editing step? I figure there is very likely a linq-type approach to this, would that perform better, and how would I go about doing that if it would?
If step #3’s importing is very quick, I would be tempted to create a temporary database whose schema exactly matches the old database and import the records into it. Then I’d look at adding additional columns to the temporary table where you need to split the XXX######## into XXX and ########. You could then use SQL to split the source column into the two separate ones. You could likewise use SQL to do whatever ID based lookups and updates you need to ensure the record relationships continue to be correct.
Once the data has been massaged into a format which is acceptable, you can insert the records into the final tables using IDENTITY_INSERT ON, excluding all legacy columns/information.
In my mind, the primary advantage of doing it within the temporary SQL DB is that at any time you can write queries to ensure that record relationships using the old key(s) are still correctly related to records using the new database’s auto generated keys.
This is of coursed based on me being more comfortable doing data transformations/validation in SQL than in C#.
As stated in the title, how can I programmatically create a SQL Server CE 4.0 from a remote SQL Server?
I want my application to allow users to delete the .sdf and create a new one based on the new remote database schema when there is a database schema update. And then download relevant data for offline use.
I already read up about the SqlCeEngine part, but I am not good at SQL Server CE queries - seem to give many syntax errors when trying out in Management Studio.
I also tried Microsoft Sync Framework Snapshot synchronization but it feels too bulky and the Local Cache Database modifies my database schema and generates a lot of junk I do not need. Maybe a lower level solution like querying information.schema or something may work better?
Checkout DMO. Using managed code, you can enumerate objects like tables, columns on the sql server side.
http://msdn.microsoft.com/en-us/library/aa174487(v=sql.80).aspx
Here's a tutorial to get you started:
http://www.codeproject.com/KB/database/SMO_Tutorial_1.aspx
Concerning the data, one option is the bcp utility
http://msdn.microsoft.com/en-us/library/aa337544.aspx
Those are good starting points if you want to extract and create a new database. For mirroring/sync, probably not a good path. If it's read only data on the client and you jst want to update the local data, then you can just extract again and throw away the old 'data cache'
You can use my scripting API and command line tools to do this: http://exportsqlce.codeplex.com - see for example this blog post: http://erikej.blogspot.com/2010/02/how-to-use-exportsqlce-to-migrate-from.html
This may be a more up to date way using SQL only:
SELECT * FROM INFORMATION_SCHEMA.COLUMNS
I am trying to replace a DTS access exporter package with a exe we can call from our stored procedures (using xp_cmdshell).
We are in the middle of a transition between SQL 2000 and SQL 2005, and for the moment if we can not use DTS OR SSIS that would be the best options.
I believe I have the following options:
Using a SQL data reader to read SQL records, and using ADO.net to insert the read records into Access.
I have implemented this and it is WAY too slow. This is not a option
Setting up Linked tables in access, then getting access to pull the data out of sql.
If anyone has any experience in doing this I would be grateful for some code examples or pointing out some resources?
If there are any other options for transferring large amounts of data from SQL into a Access database that would be awesome, but performance is a big issue as we can be dealing with up to 1mil records per table.
Have you tried this?
Why not creating a linked table in Access, and pulling data from Sql Server instead of pushing from Sql to Access ?
I've done plenty of cases where I start with an Access database, attach to SQL Server, create a Create Table or Insert Querydef, and write some code to execute the querydef, possibly with arguments. But there are a lot of assumptions I would need to make about your problem and your familiarity with Access to go into more detail. How far can you get with that description?
I have ended up using Access interop, thanks to le dorfier for pointing me in the direction of the import function which seems to be the simplest way..
I now have something along these lines:
Access.ApplicationClass app = new Access.ApplicationClass();
Access.DoCmd doCmd = null;
app.NewCurrentDatabase(_args.Single("o"));
doCmd = app.DoCmd;
//Create a view on the server temporarily with the query I want to export
doCmd.TransferDatabase(Access.AcDataTransferType.acImport,
"ODBC Database",
string.Format("ODBC;DRIVER=SQL Server;Trusted_Connection=Yes;SERVER={0};Database={1}", _args.Single("s"), _args.Single("d")),
Microsoft.Office.Interop.Access.AcObjectType.acTable,
viewName,
exportDetails[0], false, false);
//Drop view on server
//Releasing com objects and exiting properly.
Have you looked at bcp? It's a command line utility that's supposed work well for importing and exporting large amounts of data. I've never tried to make it play nice with Access, but it's a great lightweight alternative to DTS and/or SSIS.
Like others have said, the easiest way I know to get data into an Access mdb is to set things up in Access to begin with. Roughly speaking:
Create linked tables to the SQL data you want to export. (in Access: File --> get ecternal data --> link tables) This just gives you a connection to sql server.
Create a local table that represents teh schema of the data you want to export. (on the tables tab, click the "new" button and follow your nose).
Create an Update query that selects data from the linked tables (SQL Server) and appends rows to the local table (access mdb).
On the macros tab, create a new macro that executes the query you just created above (I can't recall the exact "action" to use, but it's something like OpenQuery or RunQuery); name the macro "autoexec", which will cause it to automatically run when the mdb is opened.
Use a script (or whatever) to copy and open the mdb when appropriate; the autoexec macro will kick things off and the query will copy data from SQL server to the mdb.