Suppose we have the following config file
{
"TeamRoster:People:0:Name": "Bryan",
"TeamRoster:People:0:City": "Toronto",
"TeamRoster:People:0:Interests": "Code",
"TeamRoster:People:1:Name": "Sebastian",
"TeamRoster:People:1:City": "Vancouver",
"TeamRoster:People:1:ApartmentType": "Condo"
}
And the following POCOs to represent this config:
public class TeamRoster
{
public Person[] People { get; set; }
}
public class Person
{
public string Name { get; set; }
public string City { get; set; }
}
Assuming I'm using a ConfigurationBuilder to read the dotnet core config with something like:
IConfiguration config =
new ConfigurationBuilder()
.AddJsonFile("file.json")
.Build();
var myConfig = config.GetSection("TeamRoster").Get<TeamRoster>();
How can I capture the additional properties that weren't part of the Person object (Interests, ApartmentType)?
My reasoning is beyond the trivial example above. In my model, I might have dozens of additional attributes (eg like polymorphic types) and I'd want to capture them without having to define them in the config object. Ideally, I'd want to do this like [JsonExtendedData] but as far as I understand, even though the data is expressed as Json that's just the source. It's read into a config object and then bound.
Is there a way to customize the .net core binding?
To shed some light on this, IConfigurationRoot within Microsoft.Extensions.Configuration abstracts the configuration data from its underlying configuration sources. Out of the box, configuration data can come from Json, Azure KeyVault, Command Line, Environment Variables, Xml, Memory, Ini files, User secrets and individual files (key per file).
Each of these configuration sources loads their data into a dictionary where hierarchical values are separated using ':'. When you fetch data from a IConfiguration object, it attempts to resolve the value from each data source in reverse order. Thus the last added configuration source has the highest precedence.
The mapping of IConfiguration objects to POCO objects is done using the ConfigurationBinder. While the ConfigurationBinder does support resolution of values using TypeConverters, this scenario is only helpful for mapping individual string values within the config dictionary, not entire objects.
The ConfigurationBinder has native support for binding collections, so in theory the following two approaches can be used for the following config.
Separate Fixed Values from Dynamic Values
Represent the data so that the dynamic data is separate from the static data.
{
"auth": {
"providers": [
{
"name": "clientsecret",
"loginUrl": "https://login.microsoftonline.com/mytenant/oauth2",
"fields": {
"client_id": "xxx",
"client_secret": "yyy"
"resource": "https://management.azure.com",
"grant_type": "client_credentials"
}
},
...
]
}
}
When binding the dynamic fields to a Dictionary<string,string>, the Binder maps fields as Key/Value pairs. The key advantage for this approach is we're able to leverage TypeConverters and type-safety with the fixed fields.
public class AuthProvider
{
// FIXED FIELDS
public string Name { get; set; }
public Uri LoginUrl { get; set; }
// DYNAMIC FIELDS
public Dictionary<string,string> Fields { get; set; }
}
Map all Data as Dynamic
The slightly more hackish route is to describe all the fields in the data as siblings and then bind the entire data as a Dictionary.
{
"auth": {
"providers": [
{
"name": "clientsecret",
"loginurl": "https://somewhere",
"client_id": "xxx",
"client_secret": "yyy",
"scope": "https://url",
"grant_type": "client_credentials"
}
]
}
}
We can simply take advantage of binding directly to a dictionary. Note we use lose the type-converter for the Uri field:
public class AuthProvider : Dictionary<string,string>
{
public string Name => this["name"]
public string LoginUrl => this["loginUrl"]
public Dictionary<string,string> Fields
{
get
{
var propertyNames = GetType().GetProperties().Select(x => x.Name).ToArray();
var keyPairs = this.Where(kvp => !propertyNames.Contains(kvp.Key));
return
new Dictionary<string,string>(
keyPairs,
StringComparer.OrdinalIgnoreCase);
}
}
}
Related
I'm new to .net core and I have a problem with using multiple configuration providers for a single POCO. I'm using .net core, not asp.net core.
I tried to simplify the classes for clarity, please ignore any compilation errors. The model for my configuration is as follows:
public class Configurations
{
public EnvironmentConfig EnvironmentConfig {get; set;}
public ServerConfig ServerConfig {get; set;}
}
public class EnvironmentConfig
{
public string EnvironmentStr {get; set;}
}
public class ServerConfig
{
public string Address {get; set;}
public string Username {get; set;}
public string Password {get; set;}
}
I have two providers - appsettings.json and a database. The database is used with other services, so it's not possible to alter the data easily (it can be considered read-only). The appsettings.json file has the following hierarchy:
{
"EnvironmentConfig": {
"EnvironmentStr": "dev"
},
"ServerConfig": {
"Address": "https://example.com"
// the other properties are not here, they're kept only in the database
}
}
The database doesn't have hierarchy and provides only the missing properties (I'll use JSON again for consistency):
{
"Username": "user"
"Password": "password"
}
When I try to get a IOptionsMonitor<Configurations> object, only the Address property is populated inside ServerConfig, like it's only reading from appsettings.json (username and password are at the root level, so it's not binding them correctly):
var configs = host.Services.GetService<IOptionsMonitor<Configurations>>().CurrentValue;
{
"EnvironmentConfig": {
"EnvironmentStr": "dev"
},
"ServerConfig": {
"Address": "https://example.com"
"Username": null,
"Password": null
}
}
And when I try to get a IOptionsMonitor<ServerConfig> object, it binds only the data from the database:
var serverConfig = host.Services.GetService<IOptionsMonitor<ServerConfig>>().CurrentValue;
{
"Address": null,
"Username": "user",
"Password": "password"
}
It looks like I'm not binding it correctly. I tried multiple ways, but it didn't work:
public static IServiceCollection AddConfiguration(this IServiceCollection services, IConfiguration configuration)
{
services
.Configure<ServerConfig>(configuration) // the db properties are also at root level
.Configure<Configurations>(configuration);
return serviceCollection;
}
public static IServiceCollection AddConfiguration(this IServiceCollection services, IConfiguration configuration)
{
services
.AddOptions<ServerConfig>()
.Bind(configuration.GetSection("ServerConfig");
services
.AddOptions<Configurations>()
.Bind(configuration);
return serviceCollection;
}
Could someone explain how to bind it correctly, regardless of the difference in hierarchy, so ServerConfig has all properties populated when inside Configurations?
I am assuming the database is accessed as a custom configuration provider
Update the keys coming from the database to match the desired hierarchy
Database:
"ServerConfig:Username" -> "user"
"ServerConfig:Password" -> "password"
so the configuration framework knows what you are referring to when it merges all the configuration from the multiple configuration providers.
Extract the settings from the database and prepends the hierarchy to the keys when loading configuration.
Simplified example:
public class ServerConfigProvider : ConfigurationProvider {
public ServerConfigProvider (Action<DbContextOptionsBuilder> optionsAction) {
OptionsAction = optionsAction;
}
Action<DbContextOptionsBuilder> OptionsAction { get; }
public override void Load() {
var builder = new DbContextOptionsBuilder<MyEFConfigurationContext>();
OptionsAction(builder);
using (var dbContext = new MyEFConfigurationContext(builder.Options)) {
Data = dbContext.MySettingsTable.ToDictionary(c => $"ServerConfig:{c.Key}", c => c.Value);
}
}
}
Reference Custom configuration provider
I am building a console app and i have to save a couple of settings values and some of them is an array of values A sample section of Appsettings i am thinking is like below
"FolderSettings":{
"source": "c:\\sourcefolder",
"target": "c:\\targetfolder"
},
"FolderJPG":[
{
"name":"foldername",
"width":1450,
"height":1450
}
],
"FolderPNG":[
{
"name":"foldername",
"width":300,
"height":300
},
{
"name":"foldername2",
"width":450,
"height":450
}
]
Is it recommended to save the array of settings values in appsettings file [ FolderJPG and FolderPNG in this case ] or is there any recommended ways to keep such settings in .net core?
Also, how can i fetch the settings values as an array of values? I know the way to read a simple key-value pair will be like
_configuration.GetValue<string>("FolderSettings:source");
But how can I read the array of settings values for FolderJPG, FolderPNG etc correctly ?
Bind it to the model
public class FolderOption
{
public string Name { get; set; }
public int Width{ get; set; }
public int Height{ get; set; }
}
Then
_configuration.GetSection("FolderJPG").Get<List<FolderOption>>();
The appsettings.json can contain any configuration that you need with restriction - it should be JSON.
JSON objects are written in key/value pairs.
Keys must be strings, and
values must be a valid JSON data type (string, number, object, array,
boolean or null).
So, an array is just data type of value
In RavenDB 4.2, I want to create an Index/Map based on a dynamic object. Better put, on dynamic properties which are not known at compile-time.
Here is an example of the raw JSON I'm ingesting:
{
"id": "A",
"detections":
[
{
"steps": [
{
"object": {
"id": "A1",
"target": {
"domain_name": "foobar.com"
}
},
"object": {
"id": "A2",
"target": {
"ipv4": "127.0.0.1"
}
}
}
]
}
]
}
The above sample is ingested from a 3rd party and stored in a RavenDB collection. Roughly translated, the following model has the challenge:
public class Step
{
public string Id { get; set; }
public DateTime When {get; set;}
public dynamic Object { get; set; } // aware that it's not handy naming
}
The pickle in this is that the object.target.X property name is dynamic. They cannot be strong-typed and can be a lot of things, like: domain_name, ipv4, ipv6, dns, shoe_size, hair_colour etc. This is why the entire steps.object is ingested and stored as either System.Object or dynamic.
My objective is to basically do a SelectMany() on each object.target and extract the property name (key) and value. This would make my RavenDB Index something like this:
public class StepsIndex : AbstractIndexCreationTask<Models.Step, StepsIndex.Result>
{
public class Result
{
public DateTime When { get; set; }
public string TargetKey { get; set; }
public string TargetValue { get; set; }
// ... removed other properties for brevity
}
public StepsIndex()
{
Map = steps =>
from block in blocks
from detection in blocks.Detections
from step in detection.Steps
select new Result
{
// extract property name (key), like 'domain_name'
TargetKey = step.Object.target.GetType().GetProperties()[0].Name,
// extract property value, like 'foobar.com'
TargetValue = step.Object.target.GetType().GetProperty(s.Object.target.GetType().GetProperties()[0].Name).GetValue(s.Object.target, null)
};
}
}
Unfortunately this doesn't work due to step.Object being dynamic and resulting in the following error during compile-time:
Error [CS1963] An expression tree may not contain a dynamic operation
Second option I've tried is to cast it to JSON in the expression, which also fails because Raven's projection is not aware of Newtonsoft.Json during runtime:
// Error CS0103: The name 'JObject' does not exist in the current context
// Error CS0103: The name 'JsonConvert' does not exist in the current context
TargetKey = JObject.Parse(JsonConvert.SerializeObject(ass.Object))["target"][0].Value<string>(),
A third option I thought of was perhaps changing the dynamic Object to System.Object Object, but haven't found a neat way to extract the property key/values without knowning the property.
The question: how can I extract these dynamic property keys and values and Map them to a RavenDB index?
RavenDB allows to index dynamic fields:
See:
https://ravendb.net/docs/article-page/4.2/Csharp/indexes/using-dynamic-fields
https://github.com/ravendb/book/blob/v4.0/Ch10/Ch10.md#dynamic-data
I'm having a problem getting data from my appsettings.json.
The file looks like:
"Integrations": {
"System01": {
"Host": "failover://(tcp://localhost:61616)?transport.timeout=2000",
"User": "admin",
"Password": "admin"
},
"System02": {
"Host": "failover://(tcp://localhost:61616)?transport.timeout=2000",
"User": "admin",
"Password": "admin"
},
}
I have the following DTO:
public class IntegrationsConfigurationDto
{
public string Host { get; set; }
public string User { get; set; }
public string Password { get; set; }
}
When trying to read it like:
var config = _configuration.GetValue<IntegrationsConfigurationDto>("Integrations:System01");
I get null. But if I do:
var config = new IntegrationsConfigurationDto();
_config.Bind("Integrations:System01", config);
I get the values correctly in my config variable.
Why does that happen? How can I use GetValue<T> in this scenario?
Thanks in advance.
GetValue only works for simple values, such as string, int, etc - it doesn't traverse the hierarchy of nested configuration.
Reference: Configuration in ASP.NET Core: GetValue
ConfigurationBinder.GetValue<T> extracts a value from configuration with a specified key and converts it to the specified type. An overload permits you to provide a default value if the key isn't found.
Instead of using Bind, use the following to avoid having to create your own instance of IntegrationsConfigurationDto:
var config = _configuration.GetSection("Integrations:System01")
.Get<IntegrationsConfigurationDto>();
Reference: Configuration in ASP.NET Core: Bind to an object graph
ConfigurationBinder.Get<T> binds and returns the specified type. Get<T> is more convenient than using Bind.
I just can't seem to get the syntax correct for multi field mapping in NEST 2.0--if that's the correct terminology. Every example I've found for mapping seems to be <= the 1.x version of NEST. I'm new to Elasticsearch and NEST, and I've been reading their documentation, but the NEST documentation hasn't been completely updated for 2.x.
Basically, I don't need to index or store the entire type. Some fields I need for indexing only, some fields I'll need to index and retrieve, and some I don't need for indexing, just for retrieval.
MyType
{
// Index this & allow for retrieval.
int Id { get; set; }
// Index this & allow for retrieval.
// **Also**, in my searching & sorting, I need to sort on this **entire** field, not just individual tokens.
string CompanyName { get; set; }
// Don't index this for searching, but do store for display.
DateTime CreatedDate { get; set; }
// Index this for searching BUT NOT for retrieval/displaying.
string CompanyDescription { get; set; }
// Nest this.
List<MyChildType> Locations { get; set; }
}
MyChildType
{
// Index this & allow for retrieval.
string LocationName { get; set; }
// etc. other properties.
}
I've have been able to index the entire object and child as-is using the following as an example:
client.Index(item, i => i.Index(indexName));
However, the actual object is a lot larger than this, and I really don't need most of it. I've found this, which looks like what I think I want to do, but in an older version: multi field mapping elasticsearch
I think "mapping" is what I'm going for, but like I said, I'm new to Elasticsearch and NEST and I'm trying to learn the terminology.
Be gentle! :) It's my first time to ask a question on SO. Thanks!
In addition to Colin's and Selçuk's answers, you can also fully control the mapping through the fluent (and object initializer syntax) mapping API. Here's an example based on your requirements
void Main()
{
var pool = new SingleNodeConnectionPool(new Uri("http://localhost:9200"));
var connectionSettings = new ConnectionSettings(pool);
var client = new ElasticClient(connectionSettings);
client.Map<MyType>(m => m
.Index("index-name")
.AutoMap()
.Properties(p => p
.String(s => s
.Name(n => n.CompanyName)
.Fields(f => f
.String(ss => ss
.Name("raw")
.NotAnalyzed()
)
)
)
.Date(d => d
.Name(n => n.CreatedDate)
.Index(NonStringIndexOption.No)
)
.String(s => s
.Name(n => n.CompanyDescription)
.Store(false)
)
.Nested<MyChildType>(n => n
.Name(nn => nn.Locations.First())
.AutoMap()
.Properties(pp => pp
/* properties of MyChildType */
)
)
)
);
}
public class MyType
{
// Index this & allow for retrieval.
public int Id { get; set; }
// Index this & allow for retrieval.
// **Also**, in my searching & sorting, I need to sort on this **entire** field, not just individual tokens.
public string CompanyName { get; set; }
// Don't index this for searching, but do store for display.
public DateTime CreatedDate { get; set; }
// Index this for searching BUT NOT for retrieval/displaying.
public string CompanyDescription { get; set; }
// Nest this.
public List<MyChildType> Locations { get; set; }
}
public class MyChildType
{
// Index this & allow for retrieval.
public string LocationName { get; set; }
// etc. other properties.
}
This produces the mapping
{
"properties": {
"id": {
"type": "integer"
},
"companyName": {
"type": "string",
"fields": {
"raw": {
"type": "string",
"index": "not_analyzed"
}
}
},
"createdDate": {
"type": "date",
"index": "no"
},
"companyDescription": {
"type": "string",
"store": false
},
"locations": {
"type": "nested",
"properties": {
"locationName": {
"type": "string"
}
}
}
}
}
Calling .AutoMap() causes NEST to infer the mapping based on the property types and any attributes applied to them. Then .Properties() overrides any of the inferred mappings. For example
CompanyName is mapped as a multi_field with the field companyName analyzed using the standard analyzer and companyName.raw not analyzed. You can reference the latter in your queries using .Field(f => f.CompanyName.Suffix("raw"))
Locations is mapped as a nested type (automapping by default would infer this as an object type mapping). You can then define any specific mappings for MyChildType using .Properties() inside of the Nested<MyChildType>() call.
As far as I can see, you don't have any complex types that you are trying map. So you can easily use NEST attributes to map your objects.
Check this out:
[Nest.ElasticsearchType]
public class MyType
{
// Index this & allow for retrieval.
[Nest.Number(Store=true)]
int Id { get; set; }
// Index this & allow for retrieval.
// **Also**, in my searching & sorting, I need to sort on this **entire** field, not just individual tokens.
[Nest.String(Store = true, Index=Nest.FieldIndexOption.Analyzed, TermVector=Nest.TermVectorOption.WithPositionsOffsets)]
string CompanyName { get; set; }
// Don't index this for searching, but do store for display.
[Nest.Date(Store=true, Index=Nest.NonStringIndexOption.No)]
DateTime CreatedDate { get; set; }
// Index this for searching BUT NOT for retrieval/displaying.
[Nest.String(Store=false, Index=Nest.FieldIndexOption.Analyzed)]
string CompanyDescription { get; set; }
[Nest.Nested(Store=true, IncludeInAll=true)]
// Nest this.
List<MyChildType> Locations { get; set; }
}
[Nest.ElasticsearchType]
public class MyChildType
{
// Index this & allow for retrieval.
[Nest.String(Store=true, Index = Nest.FieldIndexOption.Analyzed)]
string LocationName { get; set; }
// etc. other properties.
}
After this declaration, to create this mapping in elasticsearch you need to make a call similar to:
var mappingResponse = elasticClient.Map<MyType>(m => m.AutoMap());
With AutoMap() call NEST will read your attributes from your POCO and create a mapping request accordingly.
Also see "Attribute Based Mapping" section from here.
Cheers!
At the time of writing, Nest does not offer a way to map a property in your class to multiple fields in your document mapping using built in attributes. However, it does provide the facilities needed to do anything with your mappings that you could do if you wrote the JSON yourself.
Here's a solution I've put together for my own needs. It shouldn't be hard to use it as the starting point for whatever you need to do.
First, here's an example of the mapping I want to generate
{
"product":{
"properties":{
"name":{
"type":"string",
"index":"not_analyzed",
"fields":{
"standard":{
"type":"string",
"analyzer":"standard"
}
}
}
}
}
}
The product document would then have the name field, which is indexed but not analyzed, and the name.standard field, which uses the standard analyzer.
The C# class that I generate the mapping from looks like this
[ElasticsearchType]
public class Product
{
[WantsStandardAnalysisField]
public string Name { get; set; }
}
Note the WantsStandardAnalysisField attribute. That's a custom attribute with no special properties added. Literally just:
public class WantsStandardAnalysisField : Attribute {}
If I were to use AutoMap as-is, my custom attribute would be ignored and I would get a mapping that has the name field, but not name.standard. Luckily, AutoMap accepts an instance of IPropertyVisitor. A base class called NoopPropertyVisitor implements the interface and does nothing at all, so you can subclass it and override only the methods you care about. When you use a property visitor with AutoMap, it will generate a document mapping for you but give you a chance to modify it before it gets sent to Elastic Search. All we need to do is look for properties marked with our custom attribute and add a field to them.
Here's an example that does that:
public class ProductPropertyVisitor : NoopPropertyVisitor
{
public override void Visit(IStringProperty type, PropertyInfo propertyInfo, ElasticsearchPropertyAttributeBase attribute)
{
base.Visit(type, propertyInfo, attribute);
var wsaf = propertyInfo.GetCustomAttribute<WantsStandardAnalysisField>();
if (wsaf != null)
{
type.Index = FieldIndexOption.NotAnalyzed;
type.Fields = new Properties
{
{
"standard",
new StringProperty
{
Index = FieldIndexOption.Analyzed,
Analyzer = "standard"
}
}
};
}
}
}
As you can see, we can do pretty much anything we want with the generated property, including turning off analysis for the main property and adding a new field with its own settings. For fun, you could add a couple properties to the custom attribute allowing you to specify the name of the field you want and the analyzer to use. You could even modify the code to see if the attribute has been added multiple times, letting you add as many fields as you want.
If you were to run this through any method that generates a mapping using AutoMap, such as:
new TypeMappingDescriptor<Product>().AutoMap(new ProductPropertyVisitor())
You'll get the desired multi-field mapping. Now you can customize mappings to your heart's content. Enjoy!
I think you have at least 2 possibilities to solve your problem:
On indexing: Create something like a metadata model, which is stored just for retrieving. See the _source field to limit the return to this field.
On searching: Specify the fields you want to query: if you don`t want to query the CreatedDate, just don't include it in your search.
In my case I am using both of these approaches to get very fast results :-)