I need to cache some look up tables in memory from sql database. I have hundreds of them. The tables are pretty simple with the following structure.
Tablename= "l_lookupobjectname"
column1Name: ID
Column2Name: Code
Code is mostly a string but can also be an integer in a few cases.
I use entity framework and would like a generic way to load those tables into my web application memory. I do not want to individually load each table by specifying its name.
I'm thinking along having a list of dictionary<in id, dynamic code>.
My problem is:
How do I generate the data access code that will pull all the data to my List of dictionary without having to write repetitive code for all my hundreds of table.
"select ID, Code from all the tables" instead calling this statement for each table.
I'm not concerned about the code for caching the data. This is quite trivial.
Your issue might be types, unless you declare everything is a string or an object (and cast it as needed).
Other than that going with some nested dictionaries seems like your best bet. You can build SQL queries ("select * from {0}"), and just provide a list of tables. Then read each one into a dictionary.
You could use DataSet, but that is quite cumbersome. Probably SqlDataReader is better bet.
You can get column names from it by:
var reader = cmd.ExecuteReader();
var columns = new List<string>();
for(int i=0;i<reader.FieldCount;i++)
{
columns.Add(reader.GetName(i));
}
and then just read it all as strings or objects.
Related
What is a data structure (like list, array, etc...) that could replace a database like SQL?
I would like it to have as many database-like features as possible, like query select and so on...
If is there none, suggest structure how it should be look like
edit:
datatable is good enough i think, thx for the answers
The simplest such data structure would be a record in F# or a class in C#. This would represent your table. A collection of tables would represent a database. You can query this with query expressions (aka Linq), and serialize it as pointed out above. You can also use a DataTable. If you are just looking for an in memory representation of a database you could have that with SQLite.
If you just want to access a database you can do it with the SQLProvider in F#, or Dapper in both F# and C#.
Here is an example with a list of records and a query expression:
open System
type Row = {
Id: bigint
Name: string
Address: string
}
let table = [
{Id = 100I; Name = "Joe"; Address = "NYC"}
{Id = 101I; Name = "Jane"; Address = "KC"}
{Id = 102I; Name = "Jim"; Address = "LA"}
]
let notInNYC =
query {
for user in table do
where (user.Address <> "NYC")
select user.Name
}
|> Seq.toList
//val notInNYC : string list = ["Jane"; "Jim"]
If you are looking to use an actual SQL database, then
(per MSDN):
private static void CreateCommand(string queryString, string connectionString)
{
using (SqlConnection connection = new SqlConnection(connectionString))
{
SqlCommand command = new SqlCommand(queryString, connection);
command.Connection.Open();
command.ExecuteNonQuery();
}
}
If you are looking to not use an actual SQL database and try to save data, relations, etc. directly in your code (not sure why you'd want to do that), you could create your own custom classes for it. You'd want to include some form of table, as well as a search method that could look through the instances of table, etc. There are so many functionalities that you'd have to implement though, so this would be difficult to do if you are trying to replicate all of the functionality of a real SQL db.
Assuming you already know about Entity Framework as an ORM and a gateway to access DBs, here are some alternatives you'd want to have in mind.
One straight forward and quick solution for small data amounts is serialization.
You can choose from:
Json
XML
Binary
Some others.
Serialization allows you to store and retrieve an object graph with not fuss of setting up DBs and connections. But doesn't give sophisticated search and update capabilities.
Another thing you might want to explore is NoSQL databases.
Check out LiteDB to get you started with the concept.
I am looking for the best practices in storing OleDBDataReader's reads. Essentially I want it to retain that same dictionary like reader["Column"]. I am writing an API that returns a data structure made up of "rows." I feel like there must be a better solution than creating an ArrayList of dictionaries but I cannot seem to find a "best practice" for this.
The code below is taken from my current project
using (var commandToQueryDB = new OleDbCommand(query))
{
commandToQueryDB.Connection = Connection;
Connection.Open();
var reader = commandToQueryDB.ExecuteReader();
while (reader.Read())
{
//Insert reader's read in some sort of data structure
}
}
I would like to be able to iterate through the queries and then access each query as a dictionary (e.g query["DistrictName"] if I had a table with DistrictName as a column)
A reader is a "pipe", not a "bucket" - the reader API is not suitable for disconnected data. For that, it depends on whether you know the schema of the data.
If you do know the schema at compile-time, then populate a typed class model - just a List<Foo> will do nicely. There are tools that can make this even simpler, handling the member population for you, etc; "dapper" leaps to mind (although I am biased).
If you do not know the schema in advance, then DataTable may be suitable. I don't usually recommend it, but it does the job here. Just:
table.Load(reader);
is enough to populate a DataTable including schema (columns) and values (rows / cells).
I am developing an HRM application to import and export xml data from database. The application receives exported xml data for the employee entry. I imported the xml file using linq to xml, where I converted the xml into respective objects. Then I want to attach (update) the employee objects.
I tried to use
//linqoper class for importing xml data and converts into IEnumerable employees object.
var emp = linqoper.importxml(filename.xml);
Using (EmployeedataContext db = new EmployeedatContext){
db.attachAllonSubmit(emp);
db.submitchange();
}
But I got error
“An entity can only be attached as modified without original state if it declares as version member or doesn't have an update check policy”.
I have also an option to retrieve each employee, and assign value to the new employee from xml data using this format.
//import IEnumerable of Employee objects
var employees = = linqoper.importxml(filename.xml)
using(Employeedatacontext db = new Employeedatacontext){
foreach(var empobj in employees)
{
Employee emp = db.Employee.where(m=>m.id==empobj.Id);
emp.FirstName=empobj.FirstName;
emp.BirthDate=empobj.BirthDate;
//….continue
}
db.submitChanges();
}
But the problem with the above is I have to iterate through the whole employee objects, which is very tiresome.
So is there any other way, I could attach (update) the employee entity in the database using LINQ to SQL.
I have seen some similar links on SO, but none of them seems to help.
https://stackoverflow.com/questions/898267/linq-to-sql-attach-refresh-entity-object
When linq-to-sql saves the changes to the database, it has to know properties of the object has been changed. It also checks if a potentially conflicting update to the database have been done during the update (optimistic concurrency).
To handle those cases LINQ-to-SQL needs two copies of the object when attaching. One with the original values (as present in the DB) and one with the new, changed values. There is also a more advanced mechanism involving a version member which is mapped to a rowversion column.
The linq-to-sql way to update a set of data is to first read all data from the database, then update the objects retrieved form the database and finally call SubmitChanges(). That would be my first approach in your situation.
If you experience performance problems, then it's time to go outside of linq-to-sql's toolbox. A solution with better performance is to load the new data into a separate staging table (for best performance, use bulk insert). Then run a SQL command or Stored Procedure that does the actual merging of data. The SQL Merge clause is excellent for this kind of updates.
LINQ to SQL is proper ORM, but if you want to take control of create/update/delete in your hand; than you can try some simple ORMs which just provide ways to do CRUD operations. I can recommend one http://crystalmapper.codeplex.com, it is simple yet powerful.
Why CrystalMapper?
I built this for large financial transaction system with lots of insert and update operations. What I need is speed and control of insert/update serving complex business scenarios ... hitting multiple tables just for one transaction.
When I put this to use in social text processing platform, it serves very well there too.
I'm using table adapters and datasets in .NET (version 2.0).
I have two tables like this:
Table1
------
...
TypeId
Table2
------
Id
Name
where Table1.TypeId is linked to Table2.Id.
I've strongly generated these types by using the wizard so now I can do things like this:
Table1Adapter adapter = new Table1Adapter();
Table1DataSet data = adapter.GetData();
foreach(Table1Row row in data) { ... }
// Can now iterate through and display the data
This all works fine. But now I also want the data from Table2. I have noticed that in row there is generated field Table2Row which seems ideal but it is set to null. How do I populate this dataset properly?
Each DataTable in Typed-Dataset has its own TableAdapter. So you'll have to the same step for each DataTable in the dataset.
There does not exists any api that lets you do this auto-fill of the entire typed-dataset or no such code is generated within typed-dataset that supports this. It is also difficult to do this because TableAdapters do not have a common base-class that can let you do this.
If you really need to do this, you'll have to maintain a collection of DataTable type-names and TableAdapter type-names and iterate over the collection to perform the dataset fill.
So I recommend to fill dataset for each table in 'hard-code' manner.
Actually there is a Microsoft Data Provider called `MSDataShape', which provides this functionality. It was flagged for depreciation last when I checked and I don't know what the current status and future plans are, or what replaces it.
Here's an example of a single "SQL command", that will return, I believe, a DataSet with two nicely related DataTable-s:
SHAPE {select * from customers}
APPEND ({select * from orders} AS rsOrders
RELATE customerid TO customerid)
Potential replacement to look at is well-crafted FOR XML query, but I have not played with that.
EDIT:
I think this SO answer does exactly that using XML.
i have a query that return only one row (always) and i want to convert this row to class object (lets say obi)
i have a feeling that using data table to this kind of query is to much
but i dont realy know which other data object to use
data reader?
is there a way to execute sql command to data row ?
DataReader is the best choice here - DataAdapters and DataSets may be overkill for a single row, although, that said, if performance is not critical then keeping-it-simple isn't a bad thing. You don't need to go from DataReader -> DataRow -> your object, just read the values off of the DataReader and you're done.
A datareader lets you query individual fields. If you want the row as a single object, I believe the DataTable/DataRowView family of objects is in fact the way to go.
You might seriously consider taking a look at Linq-to-Sql or Linq-to-Entities.
The appeal of these frameworks is they provide automatic serialization of your database data into objects, abstract away many of the mundane details of connection management, and have better compile-time support by providing strongly-typed properties which you can use without string keys or column ordinals.
When using Linq, the difference between retrieving a single row vs. retrieving multiple rows often only involves appending .Single() or .First() to your query.
At any rate, if you already use or are willing to learn one of these frameworks, you may see the bulk and difficulty of data access code reduce substantially.
With respect to DataReader vs. DataSet/DataTable, it is correct that it takes more cycles to allocate and populate a data table; however, I highly doubt you will notice the difference unless creating an extremely high volume of database calls.
In case it is helpful, here are documentation examples of data access using data readers and data sets.
DataReader
DataSet