I work with a data logging system in car racing. I am developing an application that aids in the analysis of this logged data and have found some of the query functionality with, datasets, datatables and LINQ to be very useful, i.e. Minimums, Averages, etc.
Currently, I am extracting all data from its native format to a data table and post-processing that data. I am also currently working with data where all channels are logged at the same rate, i.e. 50 Hz (50 samples per second). I would like to start writing this logged data to a database so it is somewhat platform independent, and the extraction process doesn't have to happen everytime I want to analyze a dataset.
Which leads me to the main question... Does anyone have a recommendation for the best way to store data that is related by time, but logged at different rates? I have approximately 200 channels that are logged and the rates vary from 1 Hz to 500 Hz.
Some of the methods I have thought of so far are:
creating a datatable for all data at 500 Hz using Double.NaN for values that are between actual logged samples
creating separate tables for each logging frquency, i.e. one table for 1 Hz, another for 10 Hz, and another for 500 Hz.
creating a separate table for each channel with a relationship to a time table. Each time step would then be indexed, and the table of data for each channel would not be dependent on a fixed time frequency
I think I'm leaning towards the index time stamp with a separate table for each channel, but I wanted to find out if anyone has advice on a best practice.
For the record, the datasets can range from 10 Mb to 200-300 Mb depending on the duration of the time that the car is on track.
I would like to have a single data store that houses an entire season, or at least an entire race event, so that is something I am considering as well.
Thanks very much for any advice!
Can you create a table something like:
Channel, Timestamp, Measurement
?
The database structure doesn't need to depend on the frequency; the frequency can be determined by the amount of time between timestamps.
This gives you more flexibility as you can write one piece of code to handle the calculations on all the channels, just give it a channel name.
Related
Running a C# .net app that receives data every 30 seconds from 100 clients and then stores data in a database. The data is for two parameters for each client. I need to determine the total for each parameter, for each client per hour and make decisions based on the results. The decisions algorithm would be making decisions for the last hour worth of data in a sliding window fashion. My initial thinking is to keep a dictionary of those 100 clients with key being client IP, and value being a running total. However 1) if my app restarts half way through the hour or at minute 59, I lose all those warm running totals. 2) if more clients start sending data, dictionary will consume more memory, 3) if in the future the 2 parameters become 100, the dictionary grows even bigger 4) making the running total value always reflect one-hour worth of recent data is not straightforward.
Is there any different approaches I should consider? best-practice? design patterns?
Very broad, but I will try to define structure:
Identify each client with sequential 8-byte integer, UID. Not GUID, not even sequential GUID. 4-byte integer is an option but I would stick with 8-byte. Seed from 100,000.
Identify each call from user with sequential 8-byte integer, CID. Not GUID, not even sequential GUID. 4-byte integer is an option but I would stick with 8-byte. I would take number of microseconds from 1970-01-01T00:00:00 as CID.
Store all data in archive database table REPORT_ARCHIVE, UID+CID being the complex PK. Cluster table on CID hash, make it chunky (1 file per year/quater of recordings).
Store last N records (N depends on your time window, should be your config value) in operational database table REPORT_OPER, UID+CID being the complex PK. Cluster on UID hash (8-16 files).
Pipeline all you incoming calls in memory structure like queue. Async processing agents should grab records out of queue. Grab by chunks, save into DB using DB chunking (SQL server and Oracle support that). Save to REPORT_OPER table, set trigger on INSERT to push data from REPORT_OPER to REPORT_ARCHIVE.
Run all your work queries against REPORT_OPER (summing up, etc), your analytics may run over REPORT_ARCHIVE.
For something like SUM of last X reports I would cache SUM in memory in ConcurrentDictionary using UID as key. IMPORTANT: cache on request call (admin asks for totals), not on insert call (user calls in on 30 sec interval). For that you need to agree on SLA - what is the accepted latency on reporting totals. If client wants near-real-time - negotiate frequency of calls to calculate cache hit/miss.
Good luck.
I have a small table(23 rows, 2 int columns), just a basic user-activity monitor. The first column represents user id. The second column holds a value that should be unique to every user, but I must alert the users if two values are the same. I'm using an Azure Sql database to hold this table, and Linq to Sql in C# to run the query.
The problem: Microsoft will bill me based on data transferred out of their data-centers. I would like have all of my users to be aware of the current state of this table at all times, second by second, and keep data-transfer under 5 GB per month. I'm thinking along the lines of a Linq-To-Sql expression such as
UserActivity.Where(x => x.Val == myVal).Count() > 1;
But this would download the table to the client, which cannot happen. Should I be implementing a Linq solution? Or would SqlDataReader download less metadata from the server? Am I taking the right approach by using a database at all? Gimme thoughts!
If it is data transfer you are worried about you need to do your processing on the server and return only the results. A SQLDataReader solution can return a smaller, already processed set of data to minimise the traffic.
A couple thoughts here:
First, I strongly encourage you to profile the SQL generated by your LINQ-to-SQL queries. There are several tools available for this, here's one at random (I have no particular preference or affiliation):
LINQ Profiler from Devart
Your prior experience with LINQ query inefficiency notwithstanding, the LINQ sample you quote in your question isn't particularly complex so I would expect you could make it or similar work efficiently, given a good feedback mechanism like the tool above or similar.
Second, you don't explicitly mention whether your query client is running in Azure or outside, but I gather from your concern about data egress costs that its running outside Azure. So the data egress costs are going to be query results using the TDS protocol (low-level protocol for SQL Server), which is pretty efficient. Some quick back-of-the-napkin math shows that you should be fine to stay below your monthly 5 GB limit:
23 users
10 hours/day
30 days/month (less if only weekdays)
3600 requests/hour/user
32 bits of raw data per response
= about 95 MB of raw response data per month
Even if you assume 10x overhead of TDS for header metadata, etc. (and if my math is right :-) ) then you've still got plenty of room underneath 5 GB. The point isn't that you should stop thinking about it and assume it's fine... but don't assume it isn't fine, either. In fact, don't assume anything. Test, and measure, and make an informed choice. I suspect you'll find a way to stay well under 5 GB without much trouble, even with LINQ.
One other thought... perhaps you could consider running your query inside Azure, and weigh the cost of that vs. the cost of data egress under the "query running outside Azure" scenario? This could (for example) take the form of a small Azure Web Job that runs the query every second and notifies the 23 users if the count goes above 1.
Azure Web Jobs
In essence, you wouldn't notify them if the condition is false, only when it's true. As for the notification mechanism, there are various cloud-friendly options:
Azure mobile push notifications
SMS messaging
SignalR notifications
The key here is to determine whether its more cost-effective and in line with any bigger-picture technology or business goals to have each user issue the query continuously, or to use some separate process in Azure to notify users asynchronously if the "trigger condition" is met.
Best of luck!
I am writing an application that will perform analytics on logs for the purpose of graphical display.
Each line of data will be analyzed and counters for different tracked metrics will be updated.
For instance, the following line:
[01:15:45] WARNING Application1 Error1 Message Text Goes Here
Would translate to the following updated metrics:
+1 Log received during Hour 01
+1 Log received during Minute 15
+1 Log received during Second 45
+1 WARNING severity received
+1 Application1 application received
+1 Error1 error received
Depending on the underlying data architecture that single line could end up being 6 INSERT/UPDATE statements. As the number of metrics increases so does the load on the database. What if I wanted to track 30 other things about the above line? That would be 30 statements, and depending on the database size, UPDATEs could take a while.
The easiest way I can think to store this data is simply as objects during the application execution, except I'm now constrained by memory limits. In addition, when the application restarts it would have to parse the entire data set over again.
Are there other database-like technologies out there for managing data of this type? The only thing I can think that makes this data "special" is the fact that there will be a LARGE number of small changes. Since this tool will be single-threaded there is no immediate concern for the data to be transaction-ally sound.
Is there a term for this type of data or solution that would help search for a solution? Surely someone has come across this type of need before.
As you said, use Custom Objects and whenever you reach 30 lines serialize it to disk via XML or Binary serialization then free memory, so in this case you will have only 30 lines to work at a time. at the end of each day or when you are finished processing lines, create a thread or process to deserialize the data and BULK insert them into database which will only require one DB hit to insert many rows.
The requirement: I have multiple databases (Oracle / SQL Server) etc. From database I need to get large/huge amount of data into a c# program and compare the data with one and other. Each data file from a dataset will have a key (not 100% unique, might have duplicates as well), using that key I can compare other dataset files/databases.
Each database will return approx around 1.5 million rows. I have 5 different databases from which I will be getting data. i.e 7.5 million rows will loaded into my program.
What is the best way to load the data into the program (currently each SQL takes 5 minutes on the database side). Load into CSV and then read in C#? Any other ideas?
I am planning to load data into HashSet in c#, is that good option?
DB 1:
Account Amount
1234 1
9999 66
DB 2:
Account Amount
1234 2
9999 66
DB 3:
Account Amount
1234 1
9999 66
DB 4:
Account Amount
1234 10
9999 66
After comparing the output looks like
Account DB1 Amt DB1 Amt DB3 Amt DB4 Amt Match?
1234 1 2 1 10 No
9999 66 66 66 66 Yes
With respect, this is not a huge problem. It's a medium sized problem, in which you must process 7.5 megarows. In your example these rows seem to be relatively short. If you have access to a computer with more than two GB of RAM, you can probably do this whole job in RAM fairly easily. A typical 2011-era laptop can do that. Almost any Win x64 laptop can do it in RAM.
You asked whether you should draw your data directly from the database systems or from CSVs. If you're planning to use this system in production, you should stick to working with the database systems. That avoids the possibility of working with stale data by mistake.
From your question it looks like the Account values in your various systems match each other exactly, without a lot of monkey-business about fuzzy matching. That is, it seems that an account is called "1234" in several databases, and not "1234" in one of them, "1234-001" in another, and "A1234-2014" in a third. That is very good news. It means you can use such things as HashSets to handle them in memory.
You probably should set your system up so it can process either all data or an arbitrary subset of Account values. For example, you might allow a subset to be specified as '1000' - '1999'. This will come in very handy for testing, because you'll be able to do short runs with just a few thousand accounts. This should mean you can get everything working with short-running subset queries. When you're satisfied that everything is working well, you can start a production run and go home for the night.
Notice that you might also, if this is a one-off job, simply install some DBMS (MySQL or PostgreSQL would be good open source choices) on your personal machine, load the various extracts from various database systems into tables in it, and do JOINs on them.
Finally, if you are inheriting data of unknown quality, Google lets you download a very helpful data-inspection and cleaning tool called OpenRefine.
I have a demanding project and I need your starting guidelines on this!
I need to have a database with approximately 2.000.000 records with markers lat,lng. These markers are moving objects and update their positions every 10 seconds. If the received marker does not exist in the database it needs to be inserted.
I need somehow the end user to have a realtime data in the web request e.g (www.example.com/getmarkers?minlat=x&maxlat=x&minlng=x&maxlng=x&zoom=x) for the specified zoom and eliminate the markers that overlap each other.
The main server app will receive the update commands via TCP and UDP protocol on multiple ports
Can I use C sharp and a memory datatable to do all these updates every second? Also can the end user hit this datatable so everything stays in memory to be faster? What do you think about performance and what is your opinion for develop a project like this? Real time data is what I need
I prefer to user C#, SQL Server 2008
Thanks a lot
I’d start of by making estimates based on following data with the goal of estimation number of requests per minute or second.
Average number of moving markers at any time. If you have 200 vehicles to track how many do you expect to be moving simultaneously? Does time of the day matter? If it does make sure you make calculations based on the peak hours.
How many simultaneous requests from users do you expect? If you have 800 users are they going to be using the application throughout the whole day or only several times a day or once a week?
Once you get the data multiply it by at least 3. This will accommodate for all false assumptions you may have made in the calculations and allow for future growth.
Once you get the final number it will be a lot easier to decide whether you need only one two 6-core CPU server, four 12 core CPU server or a mini data center with in memory databases and other advanced stuff