We Keep Coding

How to generate dynamic multi-dimensional list or array and insert data accordingly based on a dynamic SQL Server table in c#?

I want to create a dynamic 2-dimensional array (or any other structure) based on a dynamic database table in C# or T-SQL, which means the data source (which is a database table) is dynamic too.
EDIT:
Table structure:
For example:
If User1 meets the condition of Admin and Group1, it will be inserted into (Admin, Group1). And the users are constantly added in with different user type and group. So, every cellular can have as many as users.
And the problem is I don't know how many user types and groups there are, because new user types and new groups are added constantly too.
For now, I think I need to parse every data to find if it meets the existing conditions. If yes, insert it into the specific condition; if not, create a new condition and insert data into it.
But I don't have any idea about how to implement it? Do you have any ideas or algorithms?
Thanks very much for any suggestion or information.

I've solved this statically with Tuple, but not dynamically. I think I should re-fresh my tuple list periodically. But at least, it works now!
Any suggestions are welcome! Thanks!

fastest way to search huge list of big texts

I have a windows application written in C# that needs to load load 250,000 rows from database and provide a "search as you type" feature which means as soon as user types something in a text box, the application needs to search all 250,000 records (which are btw, single column with 1000 characters each row) using like search and display the found records.
The approach I followed was:
1- The application loads all the records into a typed List<EmployeeData>
while (objSQLReader.Read())
{
lstEmployees.Add(new EmployeesData(
Convert.ToInt32(objSQLReader.GetString(0)),
objSQLReader.GetString(1),
objSQLReader.GetString(2)));
}
2- In TextChanged event, Using LINQ, I search (with combination of Regular Expression) and attach the IEnumerable<EmployeesData> to a ListView which is in Virtual Mode.
String strPattern = "(?=.*wood*)(?=.*james*)";
IEnumerable<EmployeesData> lstFoundItems = from objEmployee in lstEmployees
where Regex.IsMatch(Employee.SearchStr, strPattern, RegexOptions.IgnoreCase)
select objEmployee;
lstFoundEmployees = lstFoundItems;
3- RetrieveVirtualItem event is handled to display items in ListView to display the item.
e.Item = new ListViewItem(new String[] {
lstFoundEmployees.ElementAt(e.ItemIndex).DateProjectTaskClient,
e.ItemIndex.ToString() });
Though the lstEmployees is loaded relatively fast (1.5 seconds) for loading the list from SQL Server, to search on TextChanged, it takes more than 7 minutes to search using LINQ. Searching thru SQL Server directly by performing a LIKE search takes less than 7 seconds.
What am I doing wrong here? How can I make this search faster (not more 2 seconds)? This is a requirement from my client. So, any help is highly appreciated. Please Help...

Does the database column that stores the text data have an index on it? If so, something similar to the trie structure that Nicholas described is already in use. Indexes in SQL Server are implemented using B+ trees, which have a an average search time on the order of log base 2 of n, where n is the height of the tree. This means that if you have 250,000 records in the table the number of operations required to search are log base 2 ( 250,000 ) or approximately 18 operations.
When you load all of the information into a data reader and then use a LINQ expression it's a linear operation, (O) n, where n is the length of the list. So worst case, it's going to be 250,000 operations. If you use a DataView there will be indexes that can be used to help with searching, which will drastically improve performance.
At the end of the day if there will not be too many requests submitted against the database server leverage the query optimizer to do this. As long as the LIKE operation isn't performed with a wildcard at the front of the string (i.e. LIKE %some_string) (negates the use of an index) and there is an index on the table you will have really fast performance. If there are just too many requests that will be submitted to the database server, either put all of the information into a DataView so an index can be used, or use a dictionary as Tim suggested above, which has a search time of O(1) (on the order of one), assuming the dictionary is implemented using a hash table.

You'd be wanting to preload things and build yourself a data structure called a trie
It's memory-intensive, but it's what the doctor ordered in this case.

See my answer to this question. If you need instant response (i.e. as fast as a user types), loading the data into memory can be a very attractive option. It may use a bit of memory, but it is very fast.
Even though there are many characters (250K records * 1000), how many unique values are there? An in-memory structure based off of keys with pointers to records matching those keys really doesn't have to be that big, even accounting for permutations of those keys.
If the data it truly won't fit into memory or changes frequently, keep it in the database and use SQL Server Full Text Indexing, which will handle searches such as this much better than a LIKE. This assumes a fast connection from the application to the database.
Full Text Indexing offers a powerful set of operators/expressions which can be used to make searches more intelligent. It's available with the free SQL Expression Edition, which will handle up to 10GB of data.

If the records can be sorted, you may want to go with a binary search, which is much, much faster for large data sets. There are several implementations in .NET collections, like List<T> and Array.

C# Programming: Maintaining a List that Associates an ID with Information Quickly

In a game that I've been working on, I created a system by which the game polls a specific 'ItemDatabase' file in order to retrieve information about itself based on a given identification number. The identification number represented the point in the database at which the information regarding a specific item was stored. The representation of every item in the database was comprised of 162 bytes. The code for the system was similar to the following:
// Retrieves the information about an 'Item' object given the ID. The
// 'BinaryReader' object contains a file stream to the 'ItemDatabase' file.
public Item(ushort ID, BinaryReader itemReader)
{
// Since each 'Item' object is represented by 162 bytes of information in the
// database, skip 162 bytes per ID skipped.
itemReader.BaseStream.Seek(162 * ID, SeekOrigin.Begin);
// Retrieve the name of this 'Item' from the database.
this.itemName = itemReader.ReadChars(20).ToString();
}
Normally there wouldn't be anything particularly wrong with this system as it queries the desired data and initializes it to the correct variables. However, this process must occur during game time, and, based on research that I've done about the efficiency of the 'Seek' method, this technique won't be nearly fast enough to be incorporated into a game. So, my question is: What's a good way to maintain a list that associates an identification number with information that can be accessed quickly?

You best shot would be a database. SQLite is very portable and does not need to be installed on the system.
If you have loaded all the data into memory, you can use Dictionary<int, Item>. This makes it very easy to add and remove items to the list.
As it seems like your IDs all go from 0 and upwards, it would be really fast with just an array. Just set the index of the item to be the id.

Assuming the information in the "database" is not being changed continuously, couldn't you just read out the various items once-off during the load of the game or level? You could store the data in a variety of ways, such as a Dictionary. The .Net Dictionary is actually what is commonly referred to as a hash table, mapping keys (in this case, your ID field) to objects (which I am guessing is of type "Item"). Lookup times are extremely good (definitely in the millions per second), I doubt you'd ever have issues.
Alternatively, if your ID is a ushort, you could just store your objects in an array with all possible ushort values. An array of 65535 length is not large in today's terms. Array lookups are as fast as you can get.

You could use a Dictionary or if this is used in a multi-threaded app then ConcurrentDictionary .
Extremely fast but a bit more effort in implementing is a MemoryMappedFile .

Should I only store IDs into Redis and then query SQL using the IDs

If I want to display the recent X amount of recent posts\comments on the page, I store a list of IDs using LPUSH and then use LTRIM to keep the list a certain size.
Do I then use the IDs returned from LRANGE to get the details of the post\comments from:
a) Redis using MGET where I pass:
MGET comment.1.author, comment.1.content, comment.1.timestamp, comment.2.author, etc
Is it OK to store all the comments details in Redis like that?
b) SQL Database using the list if IDs in a query

If you want to store the fields individually like that, a hash would seem more appropriate than 5 keys each, i.e. where "comment.1" is the key, with subkeys "author", "content", etc. However, unless you have a reason to just want the IDs, I would be tempted to store the entire thing as a serialized chunk of json or binary - then you just LRANGE and you have the data - no fuss, and minimal trips.
To reduce bandwidth, I'd use something like protobuf-net for the storage (although I may be biased), and if you need a binary safe client BookSleeve will do quite nicely.

Storing user access level in a database

I am storing a list of "Users" in a table. The business logic of the application will have a reference to an object with all the data in this table for the currently logged-in user. And be able to allow the user to perform operations if they have the correct access.
I'm wondering what is the best way to store "access levels?"
One way I'm thinking of storing the access level is as an integer, and using C# "flags" to combine multiple access levels without requiring a bunch of fields, is this wise?
Create = 1
Read = 2
Update = 4
Delete = 8
FullAcc = 16
The other option I'm thinking of, feels less elegent, but I've seen it done a lot:
Read/Write = 1
R/W + Delete= 2
Full Access = 3
The reason I'm wondering, is that it seems like it would be more simple to add additional items to the second method, but at some point, it would become a pain in the ass to maintain. What are your thoughts?

I've always preferred the first approach using flags. The danger is that you get too many levels of permissions and you have to keep extending your enum and start using huge numbers and therefor maybe have to change the data type in your database to a large int. However, for something like permissions the number of options should be fairly limited. The one suggestion I would make is to have FullAcc defined as the sum of Create, Read, Update and Delete instead of as a separate entity. That way you won't have to check if a user has Update OR FullAcc permissions when they are trying to update something.

I would go with Option #1 because it gives me individual flags for each type of access.
I would also recommend that you store history of changes with timestamps.

I'd go the enum route. Its strongly typed, transfers reasonably well between the db and code (ints and enums cast well), you can use the FlagsAttribute to combine security rights, and enums are pretty flexible when it comes to versioning issues (as long as you don't remove or rename previously defined enum values).

Your 'flags' idea is more flexible, allowing you any combination of rights if that ever becomes necessary. The 'FullAcc' item should not be defined as a specific number in your enum, however - it should be a combination of other flags or'd together (like this, with a few left out):
enum Rights { Create, read, Update, FullAcc = Create | Read | Update }
The only pain I see with this is if you add more items to the enum, you have to modify the FullAcc item, and then identify your FullAcc records in the db and update the flag value.