I am looking for an idea or some direction. I have to transfer data from one database to another both are structurally and schema wise same.
Its a complete database with maybe 70 tables and having relationship between tables at different levels. Even though i ' m going to mess up the identity when i move across database but as of now i am ok with it.
Idea which i thought was to load required data from all table into an XML and then create connection to second database and push data from this XML its kind of repeated and not best way at all. So looking for direction.
Can i use entity framework for this somehow??
I cannot use SSIS for this it has to be C# Sorry.
You can create a linked server as stated in the comments to your question. You seemed to indicate that you know how to do this, but in case not, you can do this from SQL Server Management Studio by drilling down to "Server Objects > Linked Servers" beneath your source database on the source database server, then right-click, "New Linked Server", etc.
Then you would use a statement like this, for example, from your C# code:
insert into DestServer.DBName.dbo.TableName
select * from SourceServer.DBName.dbo.TableName
Assuming you are connected to 'SourceServer' and that 'SourceServer' maintains a linked server object pointing to 'DestServer'. Note: you don't actually need to use the fully-qualified name for the table on 'SourceServer', but I've put it there for clarity. I.E. you could also do this:
insert into DestServer.DBName.dbo.TableName
select * from TableName
Don't forget to setup the permissions properly in your linked server object so that your query can write to the table in the destination server. You can do this any number of ways, and often (because I work in a small environment where it's maintained by just me and a couple other folks) I just use the "sa" login:
Yes, can use linked servers in .NET.
You just use the 4 part name.
What you can do in TSQL in SSMS you can do in a .NET SQLcommand.
My experience is that you get better performance connecting to the server you are writing to.
Related
I have a conceptual question.
I have two databases which have the same structure. One database has already contained a lot of data. These data should be transferred to the other database via Select and Insert.
How can I do this data migration with the highest performance?
My first approach was to sort all the tables in a list where the tables which contain foreign keys will be stored behind the referenced tables. But with this solution it will be impossible to start parallel processing.
The second idea was to create a custom type which contains the tablename and the tablenames of the referenced tables and a bool flag which stores whether the data in the table have been copied. This type is stored for each table in a list. Then I start a new thread that checks before copying whether the referenced tables have already been created for each table. If not, I execute Thread.Sleep() after which I will check it again.
Is there a well performing approach to this problem?
Any suggestions will be helpful.
EDIT:
The old database is a SQL Base database.
The new database is a ms sql server database.
You may use either:
1) SQL Server Replication
Or
2) SQL Server Merge statement
You may use SQL Server Linked Servers to connect different database platforms (e.g. sql server, mysql, db2, ...)
Best advice : Stick with SQLBase.
Second best advice , use the SQLBase UNLOAD command via SQLTalk. This will write all DDL statements required to recreate the database else where - including Triggers , Stored Procs, Indexes etc. plus all the Data to load , if you use the right options, to an external file . This file can optionally then be edited programmatically if need be to be in a sql server format ( not much difference ) . There are many options to the UNLOAD command which can't be written here in detail , but here's a link to the syntax.
Note that SQLBase v12 has been released in recent months and performance has increased tenfold. With the right tuning and indexes etc . it will outstrip Sql Server in terms of efficiency and performance. On a 100Gb database our response times have gone from 50 seconds to 3 seconds with no additional work .
I have two databases. One of them belongs to a CRM software and is the source.
The other one will be the destination used by a tool I'm developing.
The destination will contain a table ADDRESSES with a subset of the columns of a table of the same name in the source database.
What is the best (most efficient) way to copy the data between those databases (btw: they're on different SQL Server instances if that's important).
I could write a loop which does INSERT into the destination for each row obtained from the source but I don't think that this is efficient.
My thoughts and information:
The data won't be altered on its way from source to destination
It will be altered on its way back
I don't have the complete structure of the source but I know which fields I need and that they're warranted to be in the source (hence, access to the rows obtained from source isn't possible using the index of columns)
I can't use LINQ.
Anything leading me in the right direction here is appreciated.
Edit:
I really need a C# way to copy the data. I also need to know how to merge the copied rows back to the source. Is it really necessary (or even best practise) to do this row after row?
Why write code to do this?
The single fastest and easiest way is just to use SQL Server's bcp.exe utility (bcp: Bulk Copy Program).
Export the data from the source server.
Zip it or tar it if it needs it.
FTP it over to where it needs to go, if you need to move it to another box.
Import it into the destination server.
You can accomplish the same thing via SQL Server Management Studio in a number of different ways. Once you've defined the task, it can be saved and it can be scheduled.
You can use SQL Server's Powershell objects to do this as well.
If you're set on doing it in C#:
write your select query to get the data you want from the source server.
execute that and populate a temp file with the output.
execute SQL Server's bulk insert statement against the destination server to insert the data.
Note: For any of these techniques, you'll need to deal with identity columns if the target table has them. You'll also need to deal with key collisions. It is sometimes easier to bulk load the data into a perma-temp table first, and then apply the prerequisite transforms and manipulations to get it to where it needs to go.
According to your comment on Jwrit's answer, you want two way syncs.
If so, you might want to look into Microsoft Sync Framework.
We use it to sync 200+ tables on Premise SQL to SQL Azure and SQL Azure to SQL Azure.
You can use purely C#. However, it might offer a lot more than you want, or it might be over kill for a small project.
I'm just saying so that you can have different option for your project.
If these databases exist on two servers you can setup a link between the servers by executing sp_addlinkedserver there are instructions for setting this up here. This may come in handy if you plan on regularly "Sharing" data.
http://msdn.microsoft.com/en-us/library/ff772782.aspx
Once the servers are linked a simple select statement can copy the rows from one table to another
INSERT INTO db1.tblA( Field1,Field2,Field2 )
SELECT Field1,Field2,Field2 FROM db2.tblB
If the Databases are on the same instance you only need to execute similar SQL to the above
If this is one time - the best bet is normally SSIS (SQL server integration services), unless there are complex data transformations - you can quickly and easily do column mappings and have it done (reliably) in 15 mins flat......
what is the best way For Transfer Data From One DataBase (sql server 2008) To Another db(sql server 2008) With Different Schema?
is there a program for doing that ?
thanks 4 your future asnwer
best regards
Depending on how complex your needs are, you may want to look into using SSIS.
I would use SSIS (which comes with sql server) to do this. This will not be an easy or quick task however. Mapping from one schema to another can be very time consuming and you will almost certainly find data that doesn't fit from one schema to the other that you have to deal with. For example if the field is varchar in the first schema and datetime in the second, you will almost certainly have to exclude bad values (like "ASAP") which can be a problem especially if the field is a required field. In fact you will have to deal with those types of mismathces as well - if it wasn't required and now is, what value do you put in there for the records where it currently is null. What do you do when field sizes don't match up, you may have to truncate data. If you have different lookup values from one to the other, you need a conversion table. You may need a conversion table to relate old and new id fields so that you can properly load child tables. This is a very complex task.
Link the source server to the dest server.
Then you can, from the dest server, executes your very own queries such as
INSERT INTO Destination_Table (col1, col2, col3)
SELECT colA, colB, colC
FROM [SRC_MACHINE\SRC_SERVER].Source_Database.dbo.Source_Table
If you're comfortable with writing queries, you can include all your data transformation (if you need any) in the nested SELECT, and play with JOINs and stuff to make the selected data structure fit the destination schema.
Doable even if both servers are running SQL Server Express (which is not shipped with SSIS).
Documentation on Linking Servers
I am planning for monitoring application which will use SQL Server 2005/08 database. It will monitor old server(s) which are using SQL Server 2000 database. Tools task is to pick the data from the servers do some statistics and insert the data back into tools own database for reporting purpose.
I am using .NET(C#) as development environment.
What I would really like to do is implement stored procedures which will pick the monitoring data from other servers and insert it into the monitoring tools databases table.
Now I would like to know whether its possible to write stored procedure like this ? Or I would have to pick the data from servers in program using one connection and then insert it into tools database using the other connection?
Please let me know if in case you can foresee any complication that I should be aware of.
Also some informative/helpful pointers(links, books) will be helpful.
Thanks all for valuable inputs.
You can do it using stored procedures (assuming both servers can see each other of course), by using Linked Servers.
Once setup, you can query across servers by fully qualifying the object name:
SELECT * FROM [AnotherDbServer].[DatabaseName].[Schema].[TableName]
AdaTheDev's answer is right on, but here are some additional pointers.
Make sure you give the table name an alias if you call out specific columns. Otherwise you will get an error for basically having too many qualifications (4 is the max). Also, if you are linked to an instanced server, make sure you use brackets [ ] in the server name. For example:
SELECT FiscalCalendar.Week, FiscalCalendar.Year
FROM [LinkedServer\Instance].MyDatabase.dbo.FiscalCalendar FiscalCalendar
i know this should be db 101, but its just not as clear as it can be for me. I am using SQL2005 express and i want to copy data from databaseA to databaseB. DatabaseB already contains existing data - it may even contain data with pk's that have changed or dont exist - for example:
DataBase A
pk1 = peaches
pk2 = apples
DataBase B
pk1 = peaches
pk2 = oranges
pk3 = apples
now both A & B have related tables that are tied to the pk.
Historically i would write an app that selects the data from A and copies it to B via its Insert/Update procs (using .NET), but obv this is very cumbersome, tho some advantages are i could have a dropdown that allows you to select A to copy - and B to copy to if it exists, or say add new - for instance:
(dropdown - select source)
Peaches
Apples
(dropdown - select target)
-new-
peaches
oranges
apples
I was going to use SSIS, but the target db has only SQL express so this is not available nor do i have time to learn it (ive used dts much in the past, but not this newer tool). In SQL2000 i would just use DTS, but with the more complicated schema now im not even sure i would trust that.
Any suggestions or should i just continue to write my custom apps to migrate data from A to B?
Im looking forward to everyones suggestions - i would love to continue down a path that i feel is the best way to do this :-)
if the servers are on the same network add a linked server (look up in transact sql books online). Then you can run queries across the two servers.
SQL Server 2005 Express sucks in that it does not include SSIS. However you can also use BCP (bulk copy, look this up in the transact sql books online as well) to copy your data to different tables on the target server. It is probably in your path so from dos you can type bcp /? to get a list of options. Once the data is in a table on your target server you can run queries against it.
You do need to create a mapping between keys. I don't know how from peaches/apples you get peaches/oranges/apples. No doubt there is some business logic. Once you link the servers or get the data on one server. You can either use business logic to map old primary keys to new keys. Or a mapping table (oldkey, newkey) or (oldkey1, oldkey2, newkey1, newkey2, newkey3) that you manually fill out.
sql 2008 has an interesting feature called Change Data Capture that might be interesting down the road for you
http://blog.benhall.me.uk/2007/06/sql-server-2008-change-data-capture-cdc.html
Not real familiar with the limitations of SQL Server Express, but could you backup? And then import as another database?
You might look into merge replication - http://msdn.microsoft.com/en-us/library/ms165713(SQL.90).aspx
Is it always a one way push? A->B?
As a note, if you could get SSIS, you should take a little time to learn it. It's better than DTS imho.