I have a SQL query which returns a large set of fields.
I'd like to ultimately output this to a CSV file / XML file, selecting a specific subset of fields based on some configuration (which itself may reside in SQL, or a config file). The subset may be a list of the field names that apply to this configuration.
Trying to figure out how best to achieve this- I had look at using Automapper, mapping my Result Object (which contains all fields) to a dynamic object containing just the fields I want, but I am unsure if this is performant, or if Automapper can take a list of fields to map from config (say, a List of field names).
I have also considered whether to restrict the results from SQL using Entity Framework, but again am unsure on how to do this.
Related
I'm dynamically importing data into a database where I create new tables on the fly and store the metadata so that I can access those tables later via dynamically constructed SQL. My question is, for C#, is there a library out there that I can use that can abstract away some of the details of the SQL itself? The situation I'm running into is with sequences (although there are others). In Oracle accessing a sequence looks like this
select foo.nextVal from dual;
In Postgres...
select currval('foo_id_seq');
For my project I don't know what the final database will be and I don't like the idea of running through the project fixing a bunch of errors due to bad SQL.
I looked at NHibernate and it looks like tools like that (Linq to SQL) require an existing object model in place. I don't have an object model because all of my data is dynamically provided and I don't know the number of columns, data types, etc.
Any suggested approach to this problem is appreciated.
If the data you're trying to store has a dynamic structure, then it really sounds like a relational database may not be the best choice. It's strengths rely on data being statically structured and well defined. You might be better served with a document oriented store like MongoDB which is designed for dynamic schemas. If you used something like MongoDB, I think your question around abstracting query generation for dynamically changing schemas goes away.
That said, some relational databases like SQL Server have good support for XML data types which allow you to specify an arbitrary structure within your static schema. SQL Server also allows you to query directly into XML data types and even index them, which means you can query on the server side without the need for transferring the XML back to the client, deserializing, etc. To decide if this will perform well enough for your needs you'll have to test with data that will represent your production load.
At some point in my code, I'm creating a dictionary of type Dictionary<string, string> and I'm wondering what's the best way to store this in a database in terms of converting it to a string and then back to a dictionary.
Thanks.
There are a number of options here.
You can go the normalization route and use a separate table with a key/value pair of columns.
Some databases provide you with a data type that is similar to what you need. PostgreSQL has an hstore type where you can save any key-value pairs, and MS SQL has an XML data type that can be used as well with some simple massaging of your data before insertion.
Without this type of database-specific assistance, you can just use a TEXT or BLOB column and serialize your dictionary using a DB-friendly format such as JSON, XML or language-specific serialization formats.
The tradeoffs are the following:
A separate table with key/value columns makes for expensive querying and is a PITA in general, but you get the most query flexibility and is portable across databases.
If you use a database-powered dictionary type, you get support in queries (i.e "select rows where an attribute stored in the dictionary matches a certain condition"). Without that, you are left with selecting everything and filtering in your program, but
You lose database portability unless you code a middle layer that abstracts this away, and you lose ease of data manipulation in your code (because things "work" as if there was a column in your database with this data).
NoSQL databases that are "document oriented" are meant exactly for this type of storage. Depending on what you are doing, you might want to look at some options. MongoDB is a popular choice.
The proper choice depends on the querying patterns for the data and other non-functional issues such as database support, etc. If you expand on the functionality you need to implement, I can expand on my answer.
If you really want to store the full dictionary as a single string, then you could serialize your dictionary to JSON (or XML) and store the result to the database.
You have a few options here. You could serialize the object into XML, or JSON as #M4N mentioned. You could also create a table with at least two columns: one for key and one for value.
It really depends on what your domain models look like and how you need to manage the data. If the dictionary values or keys change (IE rename, correction, etc), and needs to be reflected across many objects that are dependent on the data, then creating a sort of lookup table for that maps directly to the dictionary might be best. Otherwise, serializing the data would be one of the best performing options.
I am writing an application in C# that takes data from approximately 100 unique file sources and maps it into existing objects for analysis and storage to a database. These files are typically CSV or TAB. I'm currently using the Lumenworks library for parsing the actual CSV, so I already have the ability to reference fields by name.
Many of these fields are direct 1:1 mapping, but some items need to either be split, concatenated, or otherwise converted.
Rather than use a factory with a concrete class for each source, I see an opportunity to make all of this configurable using a library of common methods for mapping individual fields, reserving the custom datasource specific coding for some of the edge-case mapping.
I'm thinking that I could put an front end on this that would allow a couple of our admins to creating mappings for new datasources on-the-fly.
What would be a good design to consider for this type of task?
I am struggling with how to persist preferences/settings/configurations that are read but also updated by my C# application. My configurations can be
individual ints, strings, byte[] that represent connection strings, directory names, file names,...
collections of strings or other value types.
class object instances that represent a group of mostly value types.
-> Such settings are accessed in different, sometimes unrelated classes.
I am currently pondering what you think is best practice, given the following bullet points:
A self-written key/value store. Internally the key/values are stored in a hashtable and the hashtable is serialized/deserialized for persistence.
Entity Framework: I have access to the MS entity framework and run MySQL server instances.
Text file / XML file
Any other/better ideas?
Exclusion:
I do not want to store any settings in Settings.settings, App.config, or the like
I do not want to use the registry
I do not want to store settings in any obscure roaming directory structures or other hidden directories. The settings should either be stored in a dedicated database/table or in the application executable folder or sub-folder.
Questions:
I am not sure whether to store all configuration objects in one and the same entity or whether I should setup several such entities to store groups of configurations?
what is the best way to persist such lightweight objects that function as settings/configuration and later load and access those?
You can store your settings in WMI namespaces. We had stored one of our product specific settings and configuration there. Also though you have excluded App.config, Application specific settings would be normally stored therein.
I would suggest using XML for this type of scenario.
It's easy to write and read
Human readable, and easy to open and edit manually
With LINQ to XML you can query for specific data and convert that data into strongly typed objects
Widely used and understood, especially for config files
Flexible structure
These are just a few things that come to mind. Here is some example code of querying the file for all protocols of type tcp.
XElement settings = XElement.Load("settings.xml");
IEnumerable<XElement> tcpSettings = settings.Elements("protocol").Where(o => o.Attribute("name").Value == "tcp")
// make some changes to the settings
settings.Save("settings.xml");
Edit:
Serialize Class containing Dictionary member
I need to execute read-only queries against a database that I don't control. My top choice would be Linq to SQL, however, the column names differ slightly between our Dev, QA, and Production environments.
For example, take a FolderName column. We might have:
Dev: u34_FolderName
QA: u74_FolderName
PROD: u56_FolderName
I want to do queries like this:
var query = from c in DepartmentReviews
where c.FolderName == "Test"
Can I use the Entity Framework to solve this problem?
What a potential solution look like? 3 assemblies, 1 for each of my environments? Can I create common interfaces or base classes for each of these and code against those?
Changing schemas are always a problem. Though I am not a big advocate of storing XMLs in a database, your problem sounds like it can make use of this approach.
This solution requires you to make one (and only one schema change).It may/may not be feasable in your application.
In your table you probably would have a column for unique identifier and a column for xml (SQL Server 2005/2008 naturally support it). You can serialize it as XML (you probably would end up using a generic XML Serializer- Serializer<T> whose type you would infer at run time using reflection). Deserialize it and you can get your object. You can read more about it here.
So your query would be like
var myXML = from c in ObjectContext.Table
where c.FolderName == "Test" select MyXmlColumn;
var myType = InferTypeFromConfig();
var serializer = new XmlSerializer<myType>();
var object = (myType )serializer.Deserialize(TheXMLStreamCreatedFrom(myXml));
my $0.02
You may need to use an XmlMappingSource instead of the default AttributeMappingSource. With an XmlMappingSource you can customise the column mappings in an xml file, independent of the main application. This should allow you to have different mappings from dev, qa and production.
Have you considered making your own data access layer? If your queries are limited to a specific set of cases, it wouldn't be a fully-fledged framework. You would have a common vocabulary sit atop the three database schemas, and you would pick the correct raw column name at runtime. You'd just stitch up your queries the old fashioned way.
Although you would have to write your queries manually, you still retain a lot of the client-side power of LINQ if your native data structures are IEnumerable.
Using Entity Framework you could manipulate the EDMX file at build time to achieve the desired result for each environment, see http://msdn.microsoft.com/en-us/library/cc982042.aspx
You could for example create a project that does the pre-processing for the appropriate environment at build time and include this project file in MSBUILD as the first project that gets built.
Or you could generate multiple CSDL, MSL, SSDL files and switch between them at runtime instead of using the default behavior which is to load them from resources that are embedded during the build process.
In the EF 4 you can do this fairly easily with Code First, which is currently in CTP.