I'm new to C# and WPF. I will need to write a program to collect and display RS232 data and save it as a CSV file. Do I need database or XML? Is there any relevant tutorial anyone could recommend?
Around the storage..
Do you want to sort / filter / search the data?
Then I would store it in a SQL Database and export it later on to CSV (using a simple StringWriter).
If you don't want to access the data, just collect it and put it into CSV I would simply write it to CSV as the data comes in.
For reading CSV I would recommend using a LINQ to CSV implementation.
E.g. http://www.codeproject.com/KB/linq/LINQtoCSV.aspx
or http://www.thinqlinq.com/Post.aspx/Title/LINQ-to-CSV-using-DynamicObject
Don't know much about RS232, but I gather this is binary data coming off a serial port?
I would question the whole idea of outputing it as a CSV file - text based formats are not ideal for this.
XML is a better choice - you could use Base64 to sort out the encoding issues, and you could structure your document sensibly to meet your purposes.
Another idea, depending on what you're trying to do, might be to use a file based Database. I don't mean MS Access either (Spit!) - look up SQLite.
If someone is really forcing you down a CSV route for binary data, then I suggest you and they need to have a talk about whether this really is, when it comes down to it, really such a great idea.
Related
I have a C# Windows Forms project which uses a SQL Server Compact (.SDF) database for only retrieving data. My application does not update the database. Thus the database is static.
I recently read somewhere that for such kind of static work it is best to use XML or JSON as they reduce the I/O time which is spent on connecting to the database, retrieving,and closing the database. Is this true?
If So, is there a way by which I can directly convert my database contents to that of a JSON file? It has 7 tables(relations) now and total of 850 rows(tuples) of data. The data is in Kannada language and not English (if this makes any diff).
Yes, with 850 read-only records that you don't join, reading from a file rather than a database is way faster.
What you want to do is save your data in files (one per table will come in handy) and read them once at program start into a list of classes that look like your respective table structure. You can then operate on those lists a lot faster than on a database. You can use JSON as a file format, or XML or CSV if your data is simple, or anything else you can come up with.
How you do this in detail is way to broad as a question for this site. Read a few turorials and ask a more detailed question once you have any problems with it.
I want to use a GTFS feed in Google Maps, but I don't know how to. I want to display the buses available from a route. Just so you know, I'm planning on implementing the Google Map I make in a Visual C# application.
This is a very general question, so my answer will necessarily be general as well. If you can provide more detail about what you're trying to accomplish I'll try to offer more specific help.
At a high level, the steps for working with a GTFS feed are:
Parse the data. From the GTFS feed's URL you'll obtain a ZIP file containing a set of CSV files. The format of these files is specified in Google's GTFS reference, and most languages already have a CSV-parsing library available that can be used to read in the data. Additionally, for some languages there are GTFS-parsing libraries available that will return data from these files as objects; it looks like there's one available for C#, gtfsengine, you might want to check out.
Load the data. You'll need to store the data somewhere, at least temporarily, to be able to work with it. This could simply be a data structure in memory (particularly if you've written your own parsing code) but since larger feeds can take some time to read you'll probably want to look at using a relational database or some other kind of storage you can serialize to disk. In the application I'm developing, a separate process parses and loads GTFS data into a relational database in one pass.
Query the data. Obviously how you do this will depend on the method you use for storing the data and the purpose of your application. If you're using a relational database, you will typically have one table per GTFS entity (or CSV file) on which you can construct indices and against which you can execute SQL queries. If you're working with objects in memory, you might construct a hash-table index in memory as well and query that to find the data you need.
I'm making a small game using XNA, but it is cumbersome to effectively type-in the stats to all the entities in the game.
I was thinking that it would be much simpler to save the required information in a separate file with a table-format and use that instead.
I have looked into reading Excel tables with C# but it seems to be overly complex.
Is there any other table-format file types that let me easily edit the contents of the file and also read the contents using C# without too much hassle??
Basically, is there any other simple alternative to Excel? I just need the simplest table files to save some text in.
CSV is probably the simplest format to store table data, Excel can save data in it. There is no built in classes to read data from CSV as far as I know.
You may also consider XML or JSON to store data if you want some more structured data. Both have built in classes to serialize objects to/from.
If you are comfortable using Excel try exporting to a .CSV (Comma Seperate Value) file. The literal string will look like below.
row1col1,row1col2\nrow2col1,row2col2\nrow3col1,row3col2
The format is incredibly simple. Each row is on a separate line (separated by "\n") and the columns within a line are separated by commas. Very easy to parse just iterate though the lines and split on the commas.
while ((row = tr.ReadLine()) != null)
{
row.split(",")[0] //first column
row.split(",")[1] //second column
row.split(",")[2] //ect...
}
This may be overkill, but SQLlite might be worth looking into if you want expand-ability and maintainability. It is an easy setup and learning SQL will be useful in many applications.
This is a good tutorial to get you started:
http://www.dreamincode.net/forums/topic/157830-using-sqlite-with-c%23/
I understand if this isn't exactly what you were looking for, but I wanted to give you a broader range of options. If you are going for absolute simplicity go with CSV or XML like Alexei said.
Edit: If necessary there is a C# SQLlite version for managed environments(XBOX,WP7) http://forums.create.msdn.com/forums/p/47127/282261.aspx
I have lot of Forex Tick Data to be saved. My question is what is the best way?
Here is am example: I collect only 1 month data from the EURUSD pair. It is originally in CSV file which is 136MB large and has 2465671 rows. I use a library written by : http://www.codeproject.com/Articles/9258/A-Fast-CSV-Reader and it took around 30 seconds to read all the ticks and saved it in 2465671 objects. first of all, whether it is fast enough?
Secondly, is there any way better than CSV? For example, the binary file which might be faster and whether you have any recommendation about any database which is best? I tried the db4o but it is not very impressive. I think here are some overhead to save data as properties of object and when we have to save 2465671 objects in Yap file of db4o.
I've thought about this before, and if I was collecting this data, I would break up the process:
collect data from the feed, form a line (I'd use fixed width), and append it to a text file.
I would create a new text file every minute and name it something like rawdata.yymmddhhmm.txt
Then I would have another process working in the background reading these files and pushing then into a database via a parameterized insert query.
I would probably use text over a binary file because I know that would append without any problems, but I'd also look into opening a binary file for append as well. This might actually be a bit better.
Also, you want to open the file in append mode since that's the fastest way to write to a file. This will obviously need to be super fast.
Perhaps look at this product:
http://kx.com/kdb+.php
it seems to made for that purpose.
One way to save data space (and hopefully time) is to save numbers as numbers and not as text, which is what CSV does.
You can perhaps make an object out of each row, and the make the reading and writing each object a serialization problem, which there is good support for in C#.
Kx's kdb database would be a great of-the-shelf package if you had a few million to spare. However you could easily write your own column-orientated database to store and analyse high-frequency data for optimal performance.
I save terabytes as compressed binary files (GZIP) that I dynamically uncompress using C#/.NETs built-in gzip compression/decompression readers.
HDF5 is widely used for big data, including by some financial firms. Unlike KDB it's free to use, and there are plenty of libraries to go on top of it, such as the .NET wrapper
This SO question might help you get started.
HDF5 homepage
I'm writing a program in C# that will save lots of data points and then later make a graph. What is the best way to save these points?
Can I just use a really long array or should I use a text file or excel file or something like that?
Additional information: It probably wont be more than a couple thousand. And it would be good if I could access it from a windows mobile app. Basically a user will be able to save times that something happens at, and then the app will use the data to find a cross correlation.
If it's millions or even thousands of records, I would probably look at using a database. You can get SQL Server 2008 Express for free, or use MySQL, or something like that.
If you go that route, LINQ to SQL makes database access a piece of cake in .NET. Entity Framework is also available, but LINQ to SQL probably has a quicker time-to-implement.
If you use a text file or excel file, etc. You'll still need to load them back into memory to plot the graph.
So if you're collecting data over a long period of time, or you want to plot the graph some time in the future, write them to a plain text file. When you're ready to plot the graph, load the file up and plot the graph.
If the data collection is within a short period of time, don't bother writing to a file - it'll just add steps to the process for nothing.
A really easy way of doing this would be to serialize your object list into a BinaryWriter or XMLWriter, which automatically format your data into a readable and writable format so that, when your program needs to load the data, all you have to do is deserialize it (1 line of code).
Alternatively, if you have very many records, I suggest trying to use a database. It's quite easy to interface C# with SQL Server (there's a free version called Express Edition) or MySQL, and storing and retrieving huge amounts of data is not a pain. This would be the most efficient way to accomplish your task.
Depending on how much data you have and whether you want to accomplish something like this with 1 line of code (serialization) or interface with a seperate product (the database approach), you can choose either one of the above. Of course, if you wanted to, you could just manually write the contents of your data to a text file or CSV file, as you suggested, but, from personal experience, I recommend the methods I explained above.
It probably wont be more than a couple thousand. And it would be good if I could access it from a windows mobile app. Basically a user will be able to save times that something happens at, and then the app will use the data to find a cross correlation.
Is there any need for interoperability with other processes? If so, time to swat-up on file formats.
However, from the sound of it, you're asking on a matter of "style", with no real requirement to open the file anywhere but your own app. I'd suggest using a BinaryWriter for the task.
If debugging is an issue, a human-readable format might be preferable, but would be considerably larger than the binary equivalent.
Probably the quickest way to do it would be using binary serialization.