I'm trying to develop a unity game app that focus on spelling the word.
I have words database word_db and a separate players information database Player_db. In the registration of new player I'm getting id of the word from word_db in random order and insert it into player_db. But it seems to need too much time to complete the task. Is there any way to make it faster or is it advisable to merge two databases?
The word_db contains thousands of words and its definition.
The main issue with SQLite with Unity3D is you're likely retaining a separate instance of SQLite than the ones native to the OS (unless you are writing your own native implementations). As such your writes/reads aren't as balanced and more over the time it takes to do both at the same time essentially queues (which depending on your read/write lock implementations as well).
We found SQLite to be a pig on devices and given we're not inclined to write iOS/Android/Microsoft flavoured specific versions we instead opted for SiaqoDB instead (which uses LightingDB). There's some minor heavy lifting when it comes to setup but in the end, we found that partitioning and/or hot swapping Databases based on user credentials to be more efficient (1000's time faster).
We store around 300million+ records in some cases just in one "table".
Related
First, my game is being made in Unity3D, utilizing a sqlite database.
Second, my database is used to store thousands of random star systems and their accompanying data (planets, stars, location etc etc)
What I want players to be able to do is to type in a systems "address" (for instance if there are 5 star systems inthe DB;
Sys001
Sys0012
Sys0013
Sys004
Sys050
and the user searches for 001, then the search results will return;
Sys001, Sys0012, Sys0013
ignoring Sys004 and Sys050.
This will happen instantly/near instantly (extremely quickly) so if the user then adds another character (IE searches for 0013) then the results will update in real-time instantly.
My question is....to achieve this, which would be better to use? Should I stick to just using my Database (IE using SQL functions to get this data) or would it be faster/better/easier/more ideal to import all database data into Lists, and then search the list (ie List[i].nameVariable.Contains("001") etc)
If it's being accessed a lot and is used often, then yes, you can load it into memory in a collection and use that. If not, create an index on the address and search on the column. Again, if the DB is accessed often then open a connection at game start, reuse that connection and close at the end. If not, open it and close after use in those rate instances you need to access it - this just helps free up resources a bit.
I'm building a C# application where the end users won't have any sql database manager. I need a way of storing around 3 million objects (with roughly 10-20 string fields) so that the program can import the info. I've tried binary formatting but im getting out of memory exceptions due to the large amount of data. Is there any other way to store the objects in a speedy and size efficient way?
Thank You
This is what databases are made for. It doesn't matter that the end user won't have a database system already. There are a number of light-weight options available that you can deploy with your app. Here are three:
If you are Windows-only, then the Access database engine is built into the operating system. Just build a database with only table definitions (no records), and your app will be able to use it.
There is an implementation of sqlite completely in managed code, allowing you to compile it directly into your app.
Sql Server Compact Edition is just two *.dll files that are easy to deploy with the app.
None of these options require you to have anything running in the background on the target system.
You could also consider HDF5. This is designed for large-volume, high IO, distributed and high-performance computations. It is very flexible and imposes no install restriction on your application (can be completely self contained).
Hope this helps!
I am working on re-engineering/upgrade of a tool. The database communication is in C++(unmanaged ADO) and connects to SQL server 2005.
I had a few queries regarding archiving and backup/restore techniques.
Generally archiving is different than backup/restore . can someone provide any link which explains me that .Presently the solution uses bcp tool for archival.I see lot of dependency on table names in the code. what are the things i have to consider in choosing the design(considering i have to take up the backup/archival on a button click, database size of 100mb at max)
Will moving the entire communication to .net will be of any help? considering lot of ORM tools. also all the bussiness logic and UI is in C#
What s the best method to verify the archival data ?
PS: the questionmight be too high level, but i did not get any proper link to understand this. It will be really helpful if someone can answer. I can provide more details!
Thanks in advance!
At 100 MB, I would say you should probably not spend too much time on archiving, and just use traditional backup strategies. The size of your database is so small that archiving would be quite an elaborate operation with very little gain, as the archiving process would typically only be relevant in the case of huge databases.
Generally speaking, a backup in database terms is a way to provide recoverability in case of a disaster (accidental data deletion, server crash, etc). Archiving mostly means you partition your data.
A possible goal with archiving is to keep specific data available for querying, but without the ability to alter it. When dealing with high volume databases, this is an excellent way to increase performance, as read-only data can be indexed much more densely than "hot" data. It also allows you to move the read-only data to an isolated RAID partition that is optimized for READ operations, and will not have to bother with the typical RDBMS IO. Also, by removing the non-active data from the regular database means the size of the data contained in your tables will decrease, which should boost performance of the overall system.
Archiving is typically done for legal reasons. The data in question might not be important for the business anymore, but the IRS or banking rules require it to be available for a certain amount of time.
Using SQL Server, you can archive your data using partitioning strategies. This normally involves figuring out the criteria based on which you will split the data. An example of this could be a date (i.e. data older than 3 years will be moved to the archive-part of the database). In case of huge systems, it might also make sense to split data based on geographical criteria (I.e. Americas on one server, Europe on another).
To answer your questions:
1) See the explanation written above
2) It really depends on what the goal of upgrading is. Moving it to .NET will get the code to be managed, but how important is that for the business?
3) If you do decide to partition, verifying it works could include issuing a query on the original database for data that contains both values before and after the threshold you will be using for partitioning, then splitting the data, and re-issuing the query afterwards to verify it still returns the same record-set. If you configure the system to use an automatic sliding window, you could also keep an eye on the system to ensure that data will automatically be moved to the archive partition.
Again, if the 100MB is not a typo, I would think your database is too small to really benefit from archiving. If your goal is to speed things up, put the system on a server that is able to load the whole database into RAM, or use SSD drives.
If you need to establish a data archive for legal or administrative reasons, give horizontal table partitioning a look. It's a pretty straight-forward process that is mostly handled by SQL Server automatically.
Hope this helps you out!
I'm working on a little experimental utility to use within our company that indexes notes stored in our custom CRM software for full-text searching. These notes are stored in a Btrieve database (a file called NOTES.DAT). It's possible to connect to the database and retrieve the notes for indexing by using Pervasive's ADO.NET provider. However, the indexer currently loops through each note and re-indexes it every 5 minutes. This seems grossly inefficient.
Unfortunately, there's no way for our CRM software to signal to the indexing service that a note has been changed, because it's possible for the database to exist on a remote machine (and the developers aren't going to write a procedure to communicate with my service over a network, since it's just a hobby project for now).
Rather than give up, I'd like to take this opportunity to learn a little more about raw Btrieve databases. So, here's my plan...
The NOTES.DAT file has to be shared, since our CRM software uses the Btrieve API rather than the ODBC driver (which means client installations have to be able to see the file itself on the network). I would like to monitor this file (using something like FileSystemWatcher?) and then determine the bytes that were changed. Using that information, I'll try to calculate the record at that position and get its primary key. Then the indexer will update only that record using Pervasive's ADO.NET provider.
The problem (besides the fact that I don't quite know the structure of Btrieve files yet or if determining the primary key from the raw data is possible) is that I don't know how to determine the start and end range of bytes that were changed in NOTES.DAT.
I could diff two versions, but that would mean storing a copy of NOTES.DAT somewhere (and it can be quite large, hence the reason for a full-text indexing service).
What's the most efficient way to do this?
Thanks!
EDIT: It's possible for more than one note to be added, edited, or deleted in one transaction, so if possible, the method needs to be able to determine multiple separate byte ranges.
If your NOTES.DAT file is stored on an NTFS partition, then you should be able to perform one of the following:
use the USN journal to identify changes to your file (preferred)
use the volume shadow copy service to track changes to your file by taking periodic snapshots through VSS (very fast), and then either:
diffing versions N and N-1 (probably not as slow as reindexing, but still slow), or
delving deeper and attempting to do diff the $Mft to determine which blocks changed at which offsets for the file(s) of interest (much more complex, but also much faster - yet still not as fast, reliable and simple as using the USN journal)
Using the USN journal should be your preferred method. You can use the FSUTIL utility to create and truncate the USN journal.
I am working on a Sometimes Connected CRUD application that will be primarily used by teams(2-4) of Social Workers and Nurses to track patient information in the form of a plan. The application is a revisualization of a ASP.Net app that was created before my time. There are approx 200 tables across 4 databases. The Web App version relied heavily on SP's but since this version is a winform app that will be pointing to a local db I see no reason to continue with SP's. Also of note, I had planned to use Merge Replication to handle the Sync'ing portion and there seems to be some issues with those two together.
I am trying to understand what approach to use for the DAL. I originally had planned to use LINQ to SQL but I have read tidbits that state it doesn't work in a Sometimes Connected setting. I have therefore been trying to read and experiment with numerous solutions; SubSonic, NHibernate, Entity Framework. This is a relatively simple application and due to a "looming" verion 3 redesign this effort can be borderline "throwaway." The emphasis here is on getting a desktop version up and running ASAP.
What i am asking here is for anyone with any experience using any of these technology's(or one I didn't list) to lend me your hard earned wisdom. What is my best approach, in your opinion, for me to pursue. Any other insights on creating this kind of App? I am really struggling with the DAL portion of this program.
Thank you!
If the stored procedures do what you want them to, I would have to say I'm dubious that you will get benefits by throwing them away and reimplementing them. Moreover, it shouldn't matter if you use stored procedures or LINQ to SQL style data access when it comes time to replicate your data back to the master database, so worrying about which DAL you use seems to be a red herring.
The tricky part about sometimes connected applications is coming up with a good conflict resolution system. My suggestions:
Always use RowGuids as your primary keys to tables. Merge replication works best if you always have new records uniquely keyed.
Realize that merge replication can only do so much: it is great for bringing new data in disparate systems together. It can even figure out one sided updates. It can't magically determine that your new record and my new record are actually the same nor can it really deal with changes on both sides without human intervention or priority rules.
Because of this, you will need "matching" rules to resolve records that are claiming to be new, but actually aren't. Note that this is a fuzzy step: rarely can you rely on a unique key to actually be entered exactly the same on both sides and without error. This means giving weighted matches where many of your indicators are the same or similar.
The user interface for resolving conflicts and matching up "new" records with the original needs to be easy to operate. I use something that looks similar to the classic three way merge that many source control systems use: Record A, Record B, Merged Record. They can default the Merged Record to A or B by clicking a header button, and can select each field by clicking against them as well. Finally, Merged Records fields are open for edit, because sometimes you need to take parts of the address (say) from A and B.
None of this should affect your data access layer in the slightest: this is all either lower level (merge replication, provided by the database itself) or higher level (conflict resolution, provided by your business rules for resolution) than your DAL.
If you can install a db system locally, go for something you feel familiar with. The greatest problem I think will be the syncing and merging part. You must think of several possibilities: Changed something that someone else deleted on the server. Who does decide?
Never used the Sync framework myself, just read an article. But this may give you a solid foundation to built on. But each way you go with data access, the solution to the businesslogic will probably have a much wider impact...
There is a sample app called issueVision Microsoft put out back in 2004.
http://windowsclient.net/downloads/folders/starterkits/entry1268.aspx
Found link on old thread in joelonsoftware.com. http://discuss.joelonsoftware.com/default.asp?joel.3.25830.10
Other ideas...
What about mobile broadband? A couple 3G cellular cards will work tomorrow and your app will need no changes sans large pages/graphics.
Excel spreadsheet used in the field. DTS or SSIS to import data into application. While a "better" solution is created.
Good luck!
If by SP's you mean stored procedures... I'm not sure I understand your reasoning from trying to move away from them. Considering that they're fast, proven, and already written for you (ie. tested).
Surely, if you're making an app that will mimic the original, there are definite merits to keeping as much of the original (working) codebase as possible - the least of which is speed.
I'd try installing a local copy of the db, and then pushing all affected records since the last connected period to the master db when it does get connected.