I have a base configuration file eg.
appsettings.json
{
"Values": {
"Test": ["one", "two"]
}
}
and
appsettings.dev.json
{
"Values": {
"Test": ["three"]
}
}
and after transforming, the array would be
["three", "two"]
How do I make sure the transformed array is shrunk to a smaller number of elements rather than each element changing individually?
To understand the cause of such 'strange' behavior for overridden array settings you need to understand how those settings are stored inside configuration providers.
The reality is that all loaded settings are stored in dictionaries, own for each configuration provider. Keys are built from setting paths where nested sections are delimited with a colon.
Array settings are stored in the same dictionary with an index in setting path (:0, :1, ...).
For configuration you described you will have 2 configuration providers with following sets of settings:
provider1[Values:Test:0] = "one"
provider1[Values:Test:1] = "two"
and
provider2[Values:Test:0] = "three"
Now it's clear why the final value of array setting is ["three", "two"]. Values:Test:0 from the second provider overrides the same setting from the first provider, and Values:Test:1 is left untouched.
Unfortunately, there is now a built-in possibility to overcome this problem. Fortunately, .net core configuration model is flexible enough for adjusting this behavior for your needs.
Idea is the following:
Enumerate configuration providers in reverse order.
For each provider get all its setting keys. You could call IConfigurationProvider.GetChildKeys() method recursively for this purpose. See GetProviderKeys() in below snippet.
With a regular expression check whether current key is an array entry.
If it is and some of previous providers overrides this array, then just suppress current array entry by setting it to null value.
If it's unseen array then current provider is marked as the only provider of values for this array. Arrays from all other providers will be suppressed (step #4).
For convenience you could wrap all this logic into extension method on IConfigurationRoot.
Here is a working sample:
public static class ConfigurationRootExtensions
{
private static readonly Regex ArrayKeyRegex = new Regex("^(.+):\\d+$", RegexOptions.Compiled);
public static IConfigurationRoot FixOverridenArrays(this IConfigurationRoot configurationRoot)
{
HashSet<string> knownArrayKeys = new HashSet<string>();
foreach (IConfigurationProvider provider in configurationRoot.Providers.Reverse())
{
HashSet<string> currProviderArrayKeys = new HashSet<string>();
foreach (var key in GetProviderKeys(provider, null).Reverse())
{
// Is this an array value?
var match = ArrayKeyRegex.Match(key);
if (match.Success)
{
var arrayKey = match.Groups[1].Value;
// Some provider overrides this array.
// Suppressing the value.
if (knownArrayKeys.Contains(arrayKey))
{
provider.Set(key, null);
}
else
{
currProviderArrayKeys.Add(arrayKey);
}
}
}
foreach (var key in currProviderArrayKeys)
{
knownArrayKeys.Add(key);
}
}
return configurationRoot;
}
private static IEnumerable<string> GetProviderKeys(IConfigurationProvider provider,
string parentPath)
{
var prefix = parentPath == null
? string.Empty
: parentPath + ConfigurationPath.KeyDelimiter;
List<string> keys = new List<string>();
var childKeys = provider.GetChildKeys(Enumerable.Empty<string>(), parentPath)
.Distinct()
.Select(k => prefix + k).ToList();
keys.AddRange(childKeys);
foreach (var key in childKeys)
{
keys.AddRange(GetProviderKeys(provider, key));
}
return keys;
}
}
The last thing is to call it when building the configuration:
IConfigurationBuilder configurationBuilder = new ConfigurationBuilder();
configurationBuilder.AddJsonFile("AppSettings.json")
.AddJsonFile("appsettings.dev.json");
var configuration = configurationBuilder.Build();
configuration.FixOverridenArrays();
#Matt in my opinion it's just an unnecessary logic. Flow 'KISS'.
appsettings.json should contain only common settings. If your production mode or development mode must have some same values in one key, just duplicate their. Like appsettings.Development.json
"Values": {
"Test": ["one", "two"]
}
and appsettings.Production.json
"Values": {
"Test": ["one", "two","three"]
}
And if you need same values for both modes you should put it in appsettings.json.
"SomeValues": {
"Test": ["1", "2","3"]
}
In final settings you will have for production
"SomeValues": {
"Test": ["1", "2","3"]
},
"Values": {
"Test": ["one", "two","three"]
}
and for development
"SomeValues": {
"Test": ["1", "2","3"]
},
"Values": {
"Test": ["one", "two"]
}
anyway If previous answer solve your problem it's ok, it's just my opinion. Thanks)
I recommend use appsettings.Development.json and appsettings.Production.json to separate environments. And keep common settings in appsettings.json for both environments.
Just rename your appsettings.dev.json to appsettings.Development.json. Add Stage or Prodaction mode of appsettings.{#mode}.json. And modify ConfigurationBuilder in Startup.cs.
.AddJsonFile("appsettings.json", optional: false, reloadOnChange: true)
.AddJsonFile($"appsettings.{env.EnvironmentName}.json", optional: true)
I think it's more common practice and could save your time from unnecessary logic of merging
Related
In my appsettings.json, I have a couple of urls which I'm using them in my project.
"Url1": "http://localhost:154/",
"Url2": "https://localhost:222/",
"Url3": "https://www.googelc.com/"
I'm retrieving them via _configuration.GetSection(). like this:
private readonly IConfiguration _configuration;
IConfigurationSection? URL1 = _configuration.GetSection("Url1");
Problem:
If by any chance Urls which are stored in appsettings.json won't end with "/", like this:
"Url1": "http://localhost:154",
Project won't work properly.
What I want:
I want to change _configuration.GetSection's body which any time "string key" contains "url", it checks the value ends with "/" and if it does not add "/" to the end of url.
why I cant:
here is IConfiguration's content:
namespace Microsoft.Extensions.Configuration
{
//
// Summary:
// Represents a set of key/value application configuration properties.
public interface IConfiguration
{
//
// Summary:
// Gets or sets a configuration value.
//
// Parameters:
// key:
// The configuration key.
//
// Returns:
// The configuration value.
string this[string key]
{
get;
set;
}
//
// Summary:
// Gets the immediate descendant configuration sub-sections.
//
// Returns:
// The configuration sub-sections.
IEnumerable<IConfigurationSection> GetChildren();
//
// Summary:
// Returns a Microsoft.Extensions.Primitives.IChangeToken that can be used to observe
// when this configuration is reloaded.
//
// Returns:
// A Microsoft.Extensions.Primitives.IChangeToken.
IChangeToken GetReloadToken();
//
// Summary:
// Gets a configuration sub-section with the specified key.
//
// Parameters:
// key:
// The key of the configuration section.
//
// Returns:
// The Microsoft.Extensions.Configuration.IConfigurationSection.
//
// Remarks:
// This method will never return null. If no matching sub-section is found with
// the specified key, an empty Microsoft.Extensions.Configuration.IConfigurationSection
// will be returned.
IConfigurationSection GetSection(string key);
}
}
As expected IConfiguration is locked and doesn't have any implementation class.
Main Question
Other than checking it every time when I'm retrieving Urls in code,
What is substitute solution?
Is there a way for changing _Configuration.GetSection?
Is there a way to change appsetting.json at runtime (to change urls
dont end whit "/")?
Temporary Solution
I implemented a static method in a class and called in my Program.cs.
it's just being called once when app starts.
here is the method:
public static void EditAppSettings()
{
try
{
var filePath = Path.Combine(System.IO.Directory.GetCurrentDirectory(), "appsettings.json");
string json = File.ReadAllText(filePath);
dynamic jsonObj = Newtonsoft.Json.JsonConvert.DeserializeObject(json);
foreach (var item in jsonObj)
{
string value = item.Value;
if (value.Contains("http"))
{
if (!value.EndsWith("/"))
value += "/";
item.Value = value;
}
}
string output = Newtonsoft.Json.JsonConvert.SerializeObject(jsonObj, Newtonsoft.Json.Formatting.Indented);
File.WriteAllText(filePath, output);
}
catch (Exception ex)
{
///do nothing
}
}
What solution is doing>
It opens appsettings.json and deserialize its content and accesses each item and if value has "http" and doesn't end with "/", then modifying it and at the end rewrites the appsettings.json's file!
When running integration tests, I would like to use a partially randomly generated connection string for the DbContext where the database part of the connection string has a format like AppName_{RandomGuid}. This would allow multiple integration test instances to run at the same time against a real database without anyone stepping on anyone else, as the integration test databases would be created, and destroyed as needed on the central dev database server. The entire connection string would look like Server=tcp:0.0.0.0,49172;Database=AppName_{RandomGuid};User Id=;Password=;TrustServerCertificate=True.
What I have tried is something like this with the WebApplicationFactory:
private static readonly WebApplicationFactory<Program> WebApplication = new WebApplicationFactory<Program>()
.WithWebHostBuilder(builder =>
{
var testConfigurationSettings = new Dictionary<string, string>
{
{ $"Database:ConnectionString", $"Server=tcp:0.0.0.0,49172;Database=AppName_{Guid.NewGuid()};User Id=;Password=;TrustServerCertificate=True" }
};
var testConfiguration = new ConfigurationBuilder()
.AddInMemoryCollection(testConfigurationSettings)
.Build();
builder.UseConfiguration(testConfiguration);
});
This would work expect that in my Program.cs I have this line:
var builder = WebApplication.CreateBuilder(args);
// THIS LINE
builder.Configuration.AddJsonFile(path: $"appsettings.Development.{Environment.MachineName}.json", optional: true);
builder.Services.AddApplicationServices(builder.Configuration);
var app = builder.Build();
await app.RunAsync();
This allows developers (based on their machine name) to override their own settings as they want when running the application locally; importantly, each dev will have their own specific config file that, at a minimum, has a connection string which points to their own dev specific database. My problem with this solution is the line which adds the developer specific config is overriding the config registered in the WithWebHostBuilder for the integration test, because, the Program.cs executes after the WithWebHostBuilder code.
The first part is my actual problem, and the second is my current attempted solution which is so close, and clean to doing exactly what I want.
This is the solution I am using for now that works, but might be considered a bit hacky. If no other solutions come soon, then I'll mark this as the accepted answer.
Basically, I added a field to my appsettings called IsTesting, and by default it is false, but is set to true when running the integration tests:
private static readonly WebApplicationFactory<Program> WebApplication = new WebApplicationFactory<Program>()
.WithWebHostBuilder(builder =>
{
var testConfigurationSettings = new Dictionary<string, string>
{
{ "IsTesting", "true" },
{ $"Database:ConnectionString", $"Server=tcp:0.0.0.0,49172;Database=AppName_{Guid.NewGuid()};User Id=;Password=;TrustServerCertificate=True" }
};
var testConfiguration = new ConfigurationBuilder()
.AddInMemoryCollection(testConfigurationSettings)
.Build();
builder.UseConfiguration(testConfiguration);
});
This way I can check for the setting in my Program.cs and skip applying the developer configuration files if needed that were previously overriding the integration test settings:
var config = configurationManager.Get<ApplicationConfiguration>();
if (config.IsTesting)
{
return;
}
configurationManager.AddJsonFile(path: $"appsettings.Development.{Environment.MachineName}.json", optional: true);
I'm pushing a stream of data to Azure EventHub with the following code leveraging Microsoft.Hadoop.Avro.. this code runs every 5 seconds, and simply plops the same two Avro serialised items 👍🏼:
var strSchema = File.ReadAllText("schema.json");
var avroSerializer = AvroSerializer.CreateGeneric(strSchema);
var rootSchema = avroSerializer.WriterSchema as RecordSchema;
var itemList = new List<AvroRecord>();
dynamic record_one = new AvroRecord(rootSchema);
record_one.FirstName = "Some";
record_one.LastName = "Guy";
itemList.Add(record_one);
dynamic record_two = new AvroRecord(rootSchema);
record_two.FirstName = "A.";
record_two.LastName = "Person";
itemList.Add(record_two);
using (var buffer = new MemoryStream())
{
using (var writer = AvroContainer.CreateGenericWriter(strSchema, buffer, Codec.Null))
{
using (var streamWriter = new SequentialWriter<object>(writer, itemList.Count))
{
foreach (var item in itemList)
{
streamWriter.Write(item);
}
}
}
eventHubClient.SendAsync(new EventData(buffer.ToArray()));
}
The schema used here is, again, v. simple:
{
"type": "record",
"name": "User",
"namespace": "SerDes",
"fields": [
{
"name": "FirstName",
"type": "string"
},
{
"name": "LastName",
"type": "string"
}
]
}
I have validated this is all good, with a simple view in Azure Stream Analytics on the portal:
So far so good, but i cannot, for the life of me correctly deserialize this in Databricks leverage the from_avro() command under Scala..
Load (the exact same) schema as a string:
val sampleJsonSchema = dbutils.fs.head("/mnt/schemas/schema.json")
Configure EventHub
val connectionString = ConnectionStringBuilder("<CONNECTION_STRING>")
.setEventHubName("<NAME_OF_EVENT_HUB>")
.build
val eventHubsConf = EventHubsConf(connectionString).setStartingPosition(EventPosition.fromEndOfStream)
val eventhubs = spark.readStream.format("eventhubs").options(eventHubsConf.toMap).load()
Read the data..
// this works, and i can see the serialised data
display(eventhubs.select($"body"))
// this fails, and with an exception: org.apache.spark.SparkException: Malformed records are detected in record parsing. Current parse Mode: FAILFAST. To process malformed records as null result, try setting the option 'mode' as 'PERMISSIVE'.
display(eventhubs.select(from_avro($"body", sampleJsonSchema)))
So essentially, what is going on here.. i am serialising the data with the same schema as deserializing, but something is malformed.. the documentation is incredibly sparse on this front (very very minimal on the Microsoft website).
The issue
After additional investigation, (and mainly with the help of this article) I found what my problem was: from_avro(data: Column, jsonFormatSchema: String) expects spark schema format and not avro schema format. The documentation is not very clear on this.
Solution 1
Databricks provides a handy method from_avro(column: Column, subject: String, schemaRegistryUrl: String)) that fetches needed avro schema from kafka schema registry and automatically converts to correct format.
Unfortunately, it is not available for pure spark, nor is it possible to use it without a kafka schema registry.
Solution 2
Use schema conversion provided by spark:
// define avro deserializer
class AvroDeserializer() extends AbstractKafkaAvroDeserializer {
override def deserialize(payload: Array[Byte]): String = {
val genericRecord = this.deserialize(payload).asInstanceOf[GenericRecord]
genericRecord.toString
}
}
// create deserializer instance
val deserializer = new AvroDeserializer()
// register deserializer
spark.udf.register("deserialize_avro", (bytes: Array[Byte]) =>
deserializer.deserialize(bytes)
)
// get avro schema from registry (but I presume that it should also work with schema read from a local file)
val registryClient = new CachedSchemaRegistryClient(kafkaSchemaRegistryUrl, 128)
val avroSchema = registryClient.getLatestSchemaMetadata(topic + "-value").getSchema
val sparkSchema = SchemaConverters.toSqlType(new Schema.Parser().parse(avroSchema))
// consume data
df.selectExpr("deserialize_avro(value) as data")
.select(from_json(col("data"), sparkSchema.dataType).as("data"))
.select("data.*")
I have an application attached to the configuration file:
{
"ProjectModules": [
{
"Version": "1",
"LoginModule": {
"LoginLogic": "Project1.ModulesV1.LoginModule.Logic.LoginLogic"
}
},
{
"Version": "2",
"LoginModule": {
"LoginLogic": "Project1.ModulesV2.LoginModule.Logic.LoginLogic"
}
}
]
}
How to get the value for the "LoginLogic" key and for a specific version?
Here I started to do but it does not take into account that it is a data table
if (_configuration.GetSection("ProjectModules:" + moduleName).Exists())
{
var configSection = _configuration.GetSection("ProjectModules:" + moduleName);
if (configSection[sectionName] != null)
{
part = configSection[sectionName];
}
}
EDIT:
moduleName -> LoginModule
sectionName -> LoginLogic
I need to get the value for the "LoginLogic" key knowing the version "Version"
It's going to be extremely difficult, if not impossible, to do what you want with JSON formatted this way. You need to understand how the configuration system works. No matter what the config source (JSON, environment variables, console arguments, etc.) everything, and I mean everything ends up dumped into a dictionary. Pretty much the entire responsibility of a config provider is to take the source and convert it into a dictionary, which is then returned and merged into the main configuration dictionary.
As such, what you're actually creating here is:
["ProjectModules[0]:Version"] = 1
["ProjectModules[0]:LoginModule:LoginLogic"] = "Project1.ModulesV1.LoginModule.Logic.LoginLogic"
["ProjectModules[1]:Version"] = 2
["ProjectModules[1]:LoginModule:LoginLogic"] = "Project1.ModulesV2.LoginModule.Logic.LoginLogic"
As you can see, there's no real way here to tell exactly which version belongs to which LoginLogic, except for the index of ProjectModules being the same. However, since that's just a string serving as a key in the dictionary, it's not something you can easily filter or search on.
One option would be to change the format a bit if you can. For example, if you instead had JSON like:
{
"ProjectModules": {
"Version1": {
"LoginModule": {
"LoginLogic": "Project1.ModulesV1.LoginModule.Logic.LoginLogic"
}
},
"Version2": {
"LoginModule": {
"LoginLogic": "Project1.ModulesV1.LoginModule.Logic.LoginLogic"
}
}
}
}
Then, you'd end up with:
["ProjectModules:Version1:LoginModule:LoginLogic"] = "Project1.ModulesV1.LoginModule.Logic.LoginLogic"
["ProjectModules:Version2:LoginModule:LoginLogic"] = "Project1.ModulesV2.LoginModule.Logic.LoginLogic"
And, it's easy enough to distinguish then by version.
I have json file Users.json and I want delete one element. This is the structure:
{
"1": {
"Username": "1",
"Name": "1",
"AccessGroup": "Administrators"
},
"2": {
"Username": "2",
"Name": "2",
"AccessGroup": "Supervisors"
},
"3": {
"Username": "3",
"Name": "3",
"AccessGroup": "Administrators"
}
}
And the code:
public void DeleteUser(Users.User user)
{
String filename = USERS_PATH;
try
{
if (File.Exists(filename))
{
var data = DeserializeFromFile<Dictionary<String, User>>(USERS_PATH);
foreach (var item in data)
{
if (user.Username == data[item.Key].Username)
{
data.Remove(user.Username);
break;
}
}
}
}
catch (Exception)
{
throw new ArgumentException("User has not deleted");
}
}
After that iteration the file is the same like before. Where do I wrong? Thanks in advance.
There are a few things wrong with this code:
After loading the file into an object, the Dictionary<string,User> no longer corresponds to the file: if the dictionary is updated, the file isn't and vice versa, you you need to serialize the data again and save it to the file with:
String encoded = JsonConvert.SerializeObject(data);
System.IO.File.WriteAllText(filename,encode);
Dictionary.Remove asks to provide a key, you can remove with the username, it's not guaranteed (especially not in your example file), that the key is the username. The result is that nothing is removed. So you should use:
data.Remove(item.Key);
instead of:
data.Remove(user.Username);
Next, you shouldn't remove data in an iterator. Although this works in this case because you do a break, in general it's a very bad idea to do this since most enumerators are not designed to enumerate over changing collections. In this case you can store a reference to the key to be removed:
var torem = null;
foreach (var item in data) {
if (user.Username == data[item.Key].Username) {
torem = item.Key;
break;
}
}
if(torem != null) {
data.Remove(torem);
}
You can also save some CPU cycles by using item.Value instead of data[item.Key].Username since what you do is a lookup for a value where you already have the pointer so use:
if(user.Username == item.Value.Username)
instead of:
if (user.Username == data[item.Key].Username)
You have to save the data to he same file.
File.WriteAllText(USERS_PATH, JsonConvert.SerializeObject(data));
// I don't know what JSON parser you're using, but replace it with
// whatever that library provides for serializing data/saving to file :)
or something like this :)
Basically you're working on an in-memory copy of what was read from file. If you want to use a JSON like a DB, you might want to write some kind of DAO/Repository that will just expose basic CRUD methods (open file -> perform changes -> write file).
I could be wrong, but don't you have to delete an item by its key, not by a property of the item?
So:
data.Remove(item.Key);