I need to import the data form .csv file into the database table (MS SQL Server 2005). SQL BULK INSERT seems like a good option, but the problem is that my DB server is not on the same box as my WEB server. This question describes the same issue, however i don't have any control over my DB server, and can't share any folders on it.
I need a way to import my .csv programatically (C#), any ideas?
EDIT: this is a part of a website, where user can populate the table with .csv contents, and this would happen on a weekly basis, if not more often
You have several options:
SSIS
DTS
custom application
Any of these approaches ought to get the job done. If it is just scratch work it might be best to write a throwaway app in your favorite language just to get the data in. If it needs to be a longer-living solution you may want to look into SSIS or DTS as they are made for this type of situation.
Try Rhino-ETL, its an open source ETL engine written in C# that can even use BOO for simple ETL scripts so you don't need to compile it all the time.
The code can be found here:
https://github.com/hibernating-rhinos/rhino-etl
The guy who wrote it:
http://www.ayende.com/blog
The group lists have some discussions about it, I actually added bulk insert for boo scripts a while ago.
http://groups.google.com/group/rhino-tools-dev
http://groups.google.com/group/rhino-tools-dev/browse_thread/thread/2ecc765c1872df19/d640cd259ed493f1
If you download the code there are several samples, also check the google groups list if you need more help.
i ended up using CSV Reader. I saw a reference to it in one of the #Jon Skeet's answers, can't find it again to put the link to it
How big are your datasets? Unless they are very large you can get away with parameterized insert statements. You may want to load to a staging table first for peace of mind or performance reasons.
Related
I have a huge amount of data in excel files, with at least 20 columns each file.
I am working with .net (c#), my task is to import rows that met the conditions to insert data into SQL database, for an example, I need to insert only rows with current year (or selected year), also I have column name is 'Full Employee Name' I need to check it if it exists in table Resource Human.
Also other condition is to check if the column name is the same in the SQL table.
I am succeeding to do it with code, but at least 200 lines to do all the possible checks. I read about SSIS (integration service, BI tool), and it looks that can help me to do my task.
My question how doing it? I am stacking with this new concept.
I think that choosing the best approach is based on your needs:
If you are looking to create automated jobs and to perform data import from excel to SQL periodically, i think it is better to go SSIS
If you are trying to create a small tool that convert an excel file to SQL table, then working with .NET is fine
If you are looking to loop over Excel files with different structure, then you should use .NET or you have to convert files to .csv then use SSIS.
Also you can refer to the following Microsoft documentation for more options in importing Excel files to SQL: (SQL queries, Linked servers, OPENROWSET ...)
Import data from Excel to SQL Server or Azure SQL Database
If you've already got a working .net solution, and 200 lines of code doesn't sound that bad to me, I wouldn't bother looking into SSIS to replace it.
In the past I was able to use SQL select statements on a flat file from VBA from within Excel.
How can I do something similar in C# using VS 2013?
As an overview, I'm creating a desktop app that needs to search a static table of ~100k rows and 6 columns and I'm thinking it doesn't warrant incorporating a real db.
Any pointers or other ideas out there?
Fred
LINQ is made for this. It's a layer between 'something' (text file, database, XML etc.) and the LINQ language. Take a look here.
Linq To Text Files
Microsoft has a great tool for querying different filetypes: It's called log parser .
I personally use it for parsing IIS logs ( *.csv)
Have fun ;-)
Ok I`m new to C# and new to Programing. I have taken on a small task(goal) for writing my own program for work.
Basically this program will take two databases. One data base will be update weekly (lets call this doctor list) and the other data base will be updated as needed (lets call this employee).
Basically I want to make my program compare the two databases and look for any matching employees on this list doctor list.
I can figure out the code for the searching and basic stuff.. But I have no clue where to begin with databases.
I'm ok with SQL but my problem is that my doctor list comes in a dbf file. I've looked up converting to access and sql but this program needs to be easy to use for our hr department.
Is there away to write the conversion? (again new to this)
What kind of options do I have with just working with programing it to read off an excel sheet?
If I do go the access or sql route do the computers this program run off of need to have access or sql installed?
I`m not looking for someone to write me the code.. just point me in a good directions and answer some questions...
To convert the database to SQL you'll need a third party application.
I'd check out: http://www.whitetown.com/dbf2sql/ because it includes the DLLs for use in your application as well when you buy the site license. If you're receiving a new database each time and need to transform it, that would be the way to go.
Your client should be the one accessing the database, which should (hopefully) be stored on a server. If it's a local database (so not multiple user based) then I'd recommend going SQLite as it's probably the best lightweight standalone database out there. You can also get a good ORM for SQL or SQLite (such as DrivenDB) to be able to access your data.
If you're going to stay in dbf format then you'll need to know if it's a VisualFox Pro, Dbase, etc. type of database and get the appropriate DLLs for accessing it.
I find myself in possession of an Excel Spreadsheet containing about 3,000 rows of data that represent either additions or changes to data that I need to make to an SQL Table. As you can imagine that's a bit too much to handle manually. For a number of reasons beyond my control, I can't simply use an SSIS package or other simpler method to get these changes into the database. The only option I have is to create SQL scripts that will make the changes represented in the spreadsheet to MS SQL 2005.
I have absolutely no experience with Office automation or VSTO. I've tried looking online, but most of the tutorials I've seen seem a bit confusing to me.
So, my thought is that I'd use .NET and VSTO to iterate through the rows of data (or use LINQ, whatever makes sense) and determine if the item involved is an insert or an update item. There is color highlighting in the sheet to show the delta, so I suppose I could use that or I could look up some key data to establish if the entry exists. Once I establish what I'm dealing with, I could call methods that generate a SQL statement that will either insert or update the data. Inserts would be extremely easy, and I could use the delta highlights to determine which fields need to be updated for the update items.
I would be fine with either outputting the SQL to a file, or even adding the test of the SQL for a given row in the final cell of that row.
Any direction to some sample code, examples, how-tos or whatever would lead me in the right direction would be most appreciated. I'm not picky. If there's some tool I'm unaware of or a way to use an existing tool that I haven't thought of to accomplish the basic mission of generating SQL to accomplish the task, then I'm all for it.
If you need any other information feel free to ask.
Cheers,
Steve
I suggest before trying VSTO, keep things simple and get some experience how to solve such a problem with Excel VBA. IMHO that is the easiest way of learning the Excel object model, especially because you have the macro recorder at hand. You can re-use this knowledge later when you think you have to switch to C#, VSTO or Automation or (better !) Excel DNA.
For Excel VBA, there are lots of tutorials out there, here is one:
http://www.excel-vba.com/excel-vba-contents.htm
If you need to know how to execute arbitrary SQL commands like INSERT or UPDATE within a VBA program, look into this SO post:
Excel VBA to SQL Server without SSIS
Here is another SO post showing how to get data from an SQL server into an Excel spreadsheet:
Accessing SQL Database in Excel-VBA
I am just beginning to write an application. Part of what it needs to do is to run queries on a database of nutritional information. What I have is the USDA's SR21 Datasets in the form of flat delimited ASCII files.
What I need is advice. I am looking for the best way to import this data into the app and have it easily and quickly queryable at run time. I'll be using it for all the standard things. Populating controls dynamically, Datagrids, calculations, etc. I will also need to do user specific persistent data storage as well. This will not be a commercial app, so hopefully that opens up the possibilities. I am fine with .Net Framework 3.5 so Linq is a possibility when accessing the data (just don't know if it would be the best solution or not). So, what are some suggestions for persistent storage in this scenario? What sort of gotchas should I be watching for? Links to examples are always appreciated of course.
It looks pretty small, so I'd work out an appropriate object model, load the whole lot into memory, and then use LINQ to Objects.
I'm not quite sure what you're asking about in terms of "persistent storage" - aren't you just reading the data? Don't you already have that in the text files? I'm not sure why you'd want to introduce anything else.
I would import the flat files into SQL Server and access via standard ADO.NET functionality. Not only is DB access always better (more robust and powerful) than file I/O as far as data querying and manipulation goes, but you can also take advantage of SQL Server's caching capabilities, especially since this nutritional data won't be changing too often.
If you need to download updated flat files periodically, then look into developing a service that polls for these files and imports into SQL Server automatically.
EDIT: I refer to SQL Server, but feel free to use any DBMS.
My temptation would be to import the data into SQL Server (Express if you aren't looking to deploy the app) as it's a familiar source for me. Alternatively you can probably create an ODBC data source using the text file handler to get you a database-like connection.
I agree that you would benefit from a database, especially for rapid querying, and even more so if you are saving user changes to the data. In order to load the flat file data into a SQL Server (including Express), you can use SSIS.
Use Linq or text data to list method
1.create a list.
2.Read the text file line by line (or all lines).
3.process the line - get required data and attach to the list.
4.process the list for any further use.
the persistence storage will be files and List is volatile.