I'm trying to upgrade a monolithic repo so that it is no longer susceptible to this NewtonsSoft.Json Exploit. I'm new to C# so maybe that's why I'm having a little trouble understanding the fix. They say
This can be done globally with he following statement:
JsonConvert.DefaultSettings = () => new JsonSerializerSettings { MaxDepth = 128 };
I think I could just set this in each classes constructor that relies on Newtonsoft, but that would create a whole lot of duplication (example below). Am I totally off, is there a cleaner way to do things?
using Newtonsoft.Json
private class MyClasss
{
public MyClass()
{
// add this line here
JsonConvert.DefaultSettings = () => new JsonSerializerSettings { MaxDepth = 128 };
// other steps
}
// other methods
}
Notes:
I'm working in a monolithic repo full of a bunch of solutions that each contain multiple projects.
We can't update to Json.NET 13.0.1 because of some external dependencies.
We are using .Net 3.1 and there seems to be about 5 entrypoints to our repo.
JsonConvert.DefaultSettings is a public static Func<JsonSerializerSettings>, so you only really need to set it once, on startup.
You have a few options for doing this which should be easier than setting it in every class constructor:
You note that your monolithic repo has 5 entry points, so you could set JsonConvert.DefaultSettings in each Program.cs.
If you have some class that is used by all consumers of your monolithic repo, you could set JsonConvert.DefaultSettings in the static constructor for that class:
public class SomeUniversallyUsedClass
{
static SomeUniversallyUsedClass()
{
// add this line here
JsonConvert.DefaultSettings = () => new JsonSerializerSettings { MaxDepth = 128 };
}
// Remainder of the class
}
You mention you are using .NET 6 .NET Core 3.1. In c# 9.0/.NET 5 and later, you can use a module initializer to set JsonConvert.DefaultSettings once for every module in your monolithic repo like so:
internal class JsonNetModuleInitializer
{
[System.Runtime.CompilerServices.ModuleInitializer]
public static void Initialize()
{
// add this line here
JsonConvert.DefaultSettings = () => new JsonSerializerSettings { MaxDepth = 128 };
}
}
If you are using a version earlier than .NET 5, you could still introduce JsonNetModuleInitializer and call JsonNetModuleInitializer.Initialize() from your 5 entry points and/or the static constructors for your commonly used classes.
Demo fiddle here.
Core2 has a hook for validating options read from appsettings.json:
services.PostConfigure<MyConfig>(options => {
// do some validation
// maybe throw exception if appsettings.json has invalid data
});
This validation code triggers on first use of MyConfig, and every time after that. So I get multiple runtime errors.
However it is more sensible to run validation during startup - if config validation fails I want the app to fail immediately. The docs imply that is how it works, but that is not what happens.
So am I doing it right? If so and this is by design, then how can I change what I'm doing so it works the way I want?
(Also, what is the difference between PostConfigure and PostConfigureAll? There is no difference in this case, so when should I use either one?)
This has been discussed in this dotnet/runtime issue since 2018.
In .NET 6, a ValidateOnStart extension method has been added to Microsoft.Extensions.Hosting
You can use it this way:
services.AddOptions<MyOptions>()
.ValidateDataAnnotations()
.ValidateOnStart(); // Support eager validation
However, ValidateDataAnnotations still does not validate nested properties and this won't be fixed soon (Microsoft issue).
This NuGet package provides a ConfigureAndValidate<TOptions> extension method which validates options at startup.
It is based on Microsoft.Extensions.Options.DataAnnotations. But unlike Microsoft's package, it can even validate nested properties.
It is compatible with .NET Standard 2.0, .NET Core 3.1, .NET 5, .NET 6 and .NET 7.
Documentation & source code (GitHub)
TL;DR
Create your options class(es)
Decorate your options with data annotations
Call ConfigureAndValidate<T>(Action<T> configureOptions) on your IServiceCollection
ConfigureAndValidate will configure your options (calling the base Configure method), but will also check that the built configuration respects the data annotations, otherwise an OptionsValidationException (with details) is thrown as soon as the application is started. No misconfiguration surprise at runtime!
Use
ServiceCollection extension
services.ConfigureAndValidate<TOptions>(configureOptions)
Is syntactic sugar for
services
.AddOptions<TOptions>()
.Configure(configureOptions) // Microsoft
.ValidateDataAnnotationsRecursively() // based on Microsoft's ValidateDataAnnotations, but supports nested properties
.ValidateOnStart() // or ValidateEagerly() in previous versions
.Services
OptionsBuilder extensions
ValidateDataAnnotationsRecursively
This method register this options instance for validation of its DataAnnotations at the first dependency injection. Nested objects are supported.
ValidateOnStart (or ValidateEagerly in previous versions)
This method validates this options instance at application startup rather than at the first dependency injection.
Custom validation
You can combine with your own option validations:
services
.AddOptions<TOptions>()
.Configure(configureOptions)
//...
.Validate(options => { /* custom */ }, message)
.Validate<TDependency1, TDependency2>((options, dependency1, dependency2) =>
{
// custom validation
},
"Custom error message")
//...
.ValidateDataAnnotationsRecursively()
.ValidateOnStart()
Microsoft options validation documentation
There is no real way to run a configuration validation during startup. As you already noticed, post configure actions run, just like normal configure actions, lazily when the options object is being requested. This completely by design, and allows for many important features, for example reloading configuration during run-time or also options cache invalidation.
What the post configuration action is usually being used for is not a validation in terms of “if there’s something wrong, then throw an exception”, but rather “if there’s something wrong, fall back to sane defaults and make it work”.
For example, there’s a post configuration step in the authentication stack, that makes sure that there’s always a SignInScheme set for remote authentication handlers:
options.SignInScheme = options.SignInScheme ?? _authOptions.DefaultSignInScheme ?? _authOptions.DefaultScheme;
As you can see, this will not fail but rather just provides multiple fallbacks.
In this sense, it’s also important to remember that options and configuration are actually two separate things. It’s just that the configuration is a commonly used source for configuring options. So one might argue that it is not actually the job of the options to validate that the configuration is correct.
As such it might make more sense to actually check the configuration in the Startup, before configuring the options. Something like this:
var myOptionsConfiguration = Configuration.GetSection("MyOptions");
if (string.IsNullOrEmpty(myOptionsConfiguration["Url"]))
throw new Exception("MyOptions:Url is a required configuration");
services.Configure<MyOptions>(myOptionsConfiguration);
Of course this easily becomes very excessive, and will likely force you to bind/parse many properties manually. It will also ignore the configuration chaining that the options pattern supports (i.e. configuring a single options object with multiple sources/actions).
So what you could do here is keep your post configuration action for validation, and simply trigger the validation during startup by actually requesting the options object. For example, you could simply add IOptions<MyOptions> as a dependency to the Startup.Configure method:
public void Configure(IApplicationBuilder app, IOptions<MyOptions> myOptions)
{
// all configuration and post configuration actions automatically run
// …
}
If you have multiple of these options, you could even move this into a separate type:
public class OptionsValidator
{
public OptionsValidator(IOptions<MyOptions> myOptions, IOptions<OtherOptions> otherOptions)
{ }
}
At that time, you could also move the logic from the post configuration action into that OptionsValidator. So you could trigger the validation explicitly as part of the application startup:
public void Configure(IApplicationBuilder app, OptionsValidator optionsValidator)
{
optionsValidator.Validate();
// …
}
As you can see, there’s no single answer for this. You should think about your requirements and see what makes the most sense for your case. And of course, this whole validation only makes sense for certain configurations. In particular, you will have difficulties when working configurations that will change during run-time (you could make this work with a custom options monitor, but it’s probably not worth the hassle). But as most own applications usually just use cached IOptions<T>, you likely don’t need that.
As for PostConfigure and PostConfigureAll, they both register an IPostConfigure<TOptions>. The difference is simply that the former will only match a single named option (by default the unnamed option—if you don’t care about option names), while PostConfigureAll will run for all names.
Named options are for example used for the authentication stack, where each authentication method is identified by its scheme name. So you could for example add multiple OAuth handlers and use PostConfigure("oauth-a", …) to configure one and PostConfigure("oauth-b", …) to configure the other, or use PostConfigureAll(…) to configure them both.
On an ASP.NET Core 2.2 project I got this working doing eager validation by following these steps...
Given an Options class like this one:
public class CredCycleOptions
{
[Range(1753, int.MaxValue, ErrorMessage = "Please enter a valid integer Number.")]
public int VerifiedMinYear { get; set; }
[Range(1753, int.MaxValue, ErrorMessage = "Please enter a valid integer Number.")]
public int SignedMinYear { get; set; }
[Range(1753, int.MaxValue, ErrorMessage = "Please enter a valid integer Number.")]
public int SentMinYear { get; set; }
[Range(1753, int.MaxValue, ErrorMessage = "Please enter a valid integer Number.")]
public int ConfirmedMinYear { get; set; }
}
In Startup.cs add these lines to ConfigureServices method:
services.AddOptions();
// This will validate Eagerly...
services.ConfigureAndValidate<CredCycleOptions>("CredCycle", Configuration);
ConfigureAndValidate is an extension method from here.
public static class OptionsExtensions
{
private static void ValidateByDataAnnotation(object instance, string sectionName)
{
var validationResults = new List<ValidationResult>();
var context = new ValidationContext(instance);
var valid = Validator.TryValidateObject(instance, context, validationResults);
if (valid)
return;
var msg = string.Join("\n", validationResults.Select(r => r.ErrorMessage));
throw new Exception($"Invalid configuration for section '{sectionName}':\n{msg}");
}
public static OptionsBuilder<TOptions> ValidateByDataAnnotation<TOptions>(
this OptionsBuilder<TOptions> builder,
string sectionName)
where TOptions : class
{
return builder.PostConfigure(x => ValidateByDataAnnotation(x, sectionName));
}
public static IServiceCollection ConfigureAndValidate<TOptions>(
this IServiceCollection services,
string sectionName,
IConfiguration configuration)
where TOptions : class
{
var section = configuration.GetSection(sectionName);
services
.AddOptions<TOptions>()
.Bind(section)
.ValidateByDataAnnotation(sectionName)
.ValidateEagerly();
return services;
}
public static OptionsBuilder<TOptions> ValidateEagerly<TOptions>(this OptionsBuilder<TOptions> optionsBuilder) where TOptions : class
{
optionsBuilder.Services.AddTransient<IStartupFilter, StartupOptionsValidation<TOptions>>();
return optionsBuilder;
}
}
I plumbed ValidateEargerly extension method right inside ConfigureAndValidate. It makes use of this other class from here:
public class StartupOptionsValidation<T> : IStartupFilter
{
public Action<IApplicationBuilder> Configure(Action<IApplicationBuilder> next)
{
return builder =>
{
var options = builder.ApplicationServices.GetService(typeof(IOptions<>).MakeGenericType(typeof(T)));
if (options != null)
{
// Retrieve the value to trigger validation
var optionsValue = ((IOptions<object>)options).Value;
}
next(builder);
};
}
}
This allows us to add data annotations to the CredCycleOptions and get nice error feedback right at the moment the app starts making it an ideal solution.
If an option is missing or have a wrong value, we don't want users to catch these errors at runtime. That would be a bad experience.
There are easy way to validating with using IStartupFilter and IValidateOptions.
You can just put below code your ASP.NET Core project.
public static class OptionsBuilderExtensions
{
public static OptionsBuilder<TOptions> ValidateOnStartupTime<TOptions>(this OptionsBuilder<TOptions> builder)
where TOptions : class
{
builder.Services.AddTransient<IStartupFilter, OptionsValidateFilter<TOptions>>();
return builder;
}
public class OptionsValidateFilter<TOptions> : IStartupFilter where TOptions : class
{
private readonly IOptions<TOptions> _options;
public OptionsValidateFilter(IOptions<TOptions> options)
{
_options = options;
}
public Action<IApplicationBuilder> Configure(Action<IApplicationBuilder> next)
{
_ = _options.Value; // Trigger for validating options.
return next;
}
}
}
And just chain the ValidateOnStartup method on OptionsBuilder<TOptions>.
services.AddOptions<SampleOption>()
.Bind(Configuration)
.ValidateDataAnnotations()
.ValidateOnStartupTime();
If you want to create custom validator for options class, checkout this article.
This has been implemented in .NET 6. Now you can just write the following:
services.AddOptions<SampleOption>()
.Bind(Configuration)
.ValidateDataAnnotations()
.ValidateOnStart(); // works in .NET 6
No need for external NuGet Packages or extra code.
See OptionsBuilderExtensions.ValidateOnStart<TOptions>
Below is a generic ConfigureAndValidate method to validate immediately and "fail fast".
To summarize the steps:
Call serviceCollection.Configure for your options
Do serviceCollection.BuildServiceProvider().CreateScope()
Get the options instance with scope.ServiceProvider.GetRequiredService<IOptions<T>> (remember to take the .Value)
Validate it using Validator.TryValidateObject
public static class ConfigExtensions
{
public static void ConfigureAndValidate<T>(this IServiceCollection serviceCollection, Action<T> configureOptions) where T : class, new()
{
// Inspired by https://blog.bredvid.no/validating-configuration-in-asp-net-core-e9825bd15f10
serviceCollection.Configure(configureOptions);
using (var scope = serviceCollection.BuildServiceProvider().CreateScope())
{
var options = scope.ServiceProvider.GetRequiredService<IOptions<T>>();
var optionsValue = options.Value;
var configErrors = ValidationErrors(optionsValue).ToArray();
if (!configErrors.Any())
{
return;
}
var aggregatedErrors = string.Join(",", configErrors);
var count = configErrors.Length;
var configType = typeof(T).FullName;
throw new ApplicationException($"{configType} configuration has {count} error(s): {aggregatedErrors}");
}
}
private static IEnumerable<string> ValidationErrors(object obj)
{
var context = new ValidationContext(obj, serviceProvider: null, items: null);
var results = new List<ValidationResult>();
Validator.TryValidateObject(obj, context, results, true);
foreach (var validationResult in results)
{
yield return validationResult.ErrorMessage;
}
}
}
I have an ASP.NET Core Web API project using .Net Framework 4.7 and I'm upgrading to .Net Core 3.1. One of the reasons that I'm upgrading is to use the new System.Text.Json serializer.
Currently, I have some versions of the API based on the route, like:
/api/v1/controller
/api/v2/controller
And I will create a new one (v3) to use the new serializer. But here is the problem: I want to keep using JSON.Net on the older routes, to avoid any possible formating problem with the integrated clients.
Is there an easy way to configure Asp.Net Core to automatically select the correct JSON serializer based on the route?
You could create your own super InputFormatter/OutputFormatter so that it checks the condition at runtime and then make a decision to use System.Text.Json or use Newtonsoft.Json dynamically.
For example, we can check the current action method ( or controller class):
if it has a custom attribute of [UseSystemTextJsonAttribute], then use System.Text.Json
if it has a custom attribute of [UseNewtonsoftJsonAttribute], then use Newtonsoft.Json.
I create a custom InputFormatter for your reference:
// the custom attribute
internal abstract class UseJsonAttribute : Attribute, IAsyncActionFilter
{
public Task OnActionExecutionAsync(ActionExecutingContext context, ActionExecutionDelegate next) => next();
}
internal class UseSystemTextJsonAttribute : UseJsonAttribute { }
internal class UseNewtonsoftJsonAttribute : UseJsonAttribute { }
// Our Super Input Formatter
internal class MySuperJsonInputFormatter : TextInputFormatter
{
public MySuperJsonInputFormatter()
{
SupportedEncodings.Add(UTF8EncodingWithoutBOM);
SupportedEncodings.Add(UTF16EncodingLittleEndian);
SupportedMediaTypes.Add("application/json");
}
public override async Task<InputFormatterResult> ReadRequestBodyAsync(InputFormatterContext context, Encoding encoding)
{
var mvcOpt= context.HttpContext.RequestServices.GetRequiredService<IOptions<MvcOptions>>().Value;
var formatters = mvcOpt.InputFormatters;
TextInputFormatter formatter =null; // the real formatter : SystemTextJsonInput or Newtonsoft
Endpoint endpoint = context.HttpContext.GetEndpoint();
if(endpoint.Metadata.GetMetadata<UseSystemTextJsonAttribute>()!= null)
{
formatter= formatters.OfType<SystemTextJsonInputFormatter>().FirstOrDefault();
//formatter = formatter ?? SystemTextJsonInputFormatter
}
else if( endpoint.Metadata.GetMetadata<UseNewtonsoftJsonAttribute>() != null){
// don't use `Of<NewtonsoftJsonInputFormatter>` here because there's a NewtonsoftJsonPatchInputFormatter
formatter= (NewtonsoftJsonInputFormatter)(formatters
.Where(f =>typeof(NewtonsoftJsonInputFormatter) == f.GetType())
.FirstOrDefault());
}
else{
throw new Exception("This formatter is only used for System.Text.Json InputFormatter or NewtonsoftJson InputFormatter");
}
var result = await formatter.ReadRequestBodyAsync(context,encoding);
return result;
}
}
internal class MySuperJsonOutputFormatter : TextOutputFormatter
{
... // similar to MySuperJsonInputFormatter, omitted for brevity
}
And then configure the Json settings/options in the startup:
services.AddControllers(opts =>{ })
.AddNewtonsoftJson(opts =>{ /**/ })
.AddJsonOptions(opts =>{ /**/ });
Note AddNewtonsoftJson() will remove the builtin SystemTextJsonInputFormatters. So we need configure the MvcOptions manually :
services.AddOptions<MvcOptions>()
.PostConfigure<IOptions<JsonOptions>, IOptions<MvcNewtonsoftJsonOptions>,ArrayPool<char>, ObjectPoolProvider,ILoggerFactory>((opts, jsonOpts, newtonJsonOpts, charPool, objectPoolProvider, loggerFactory )=>{
// configure System.Text.Json formatters
if(opts.InputFormatters.OfType<SystemTextJsonInputFormatter>().Count() ==0){
var systemInputlogger = loggerFactory.CreateLogger<SystemTextJsonInputFormatter>();
opts.InputFormatters.Add(new SystemTextJsonInputFormatter(jsonOpts.Value, systemInputlogger));
}
if(opts.OutputFormatters.OfType<SystemTextJsonOutputFormatter>().Count() ==0){
opts.OutputFormatters.Add(new SystemTextJsonOutputFormatter(jsonOpts.Value.JsonSerializerOptions));
}
// configure Newtonjson formatters
if(opts.InputFormatters.OfType<NewtonsoftJsonInputFormatter>().Count() ==0){
var inputLogger= loggerFactory.CreateLogger<NewtonsoftJsonInputFormatter>();
opts.InputFormatters.Add(new NewtonsoftJsonInputFormatter(
inputLogger, newtonJsonOpts.Value.SerializerSettings, charPool, objectPoolProvider, opts, newtonJsonOpts.Value
));
}
if(opts.OutputFormatters.OfType<NewtonsoftJsonOutputFormatter>().Count()==0){
opts.OutputFormatters.Add(new NewtonsoftJsonOutputFormatter(newtonJsonOpts.Value.SerializerSettings, charPool, opts));
}
opts.InputFormatters.Insert(0, new MySuperJsonInputFormatter());
opts.OutputFormatters.Insert(0, new MySuperJsonOutputFormatter());
});
Now it should work fine.
I've used this approach (create a "super" formatter which then decides which real formatter to use) to allow different routes to have different JSON formatting (camelCase or not in our scenario). This works well and allows us to use a third-party add-in which requires a specific set of formatting rules which differ from our standard...
As the third-party add-in is fully compiled we couldn't use a custom attribute on their controllers; and didn't one to add one to each of ours (!); so instead we examine the route and selected the formatter based on that.
Real life saver in this case :)
Am trying to get Nancy to retain property names as they are. For example FirstName to remain so instead of firstName in Response.AsJson.
I have seen where it is mentioned to set JsonSettings.RetainCasing = true.
I couldn't find it in Nancy.Json or an example where to set this configuration. Any hint where to find this setting and where to place it?
By default Nancy uses SimpleJson.
To configure case retaining just override the configure method like this :
public class MyBootstrapper : DefaultNancyBootstrapper
{
public override void Configure(INancyEnvironment environment)
{
environment.Json(retainCasing: true);
base.Configure(environment);
}
}
I could not make sense of the other answer, but found this solution. Add the following line to use in your startup code. It will prevent Nancy from converting cases on objects.
Nancy.Json.JsonSettings.RetainCasing = true;
I put it right before my host.Start() call:
NancyHost host = new NancyHost(uri, new DefaultNancyBootstrapper(), hostConfigs);
host.Start();
I'm building a small API (for read operations - GET) using NancyFX and C# with .NET 4.0
I'm using Entity Framework 6.0 in order to access a SQL Server 2008R2 database.
I have the following route exposed with Nancy (this is just for testing purposes):
public ExampleNancyModule()
{
Get["/v1/address_types"] = parameters =>
{
var address_types = context.address_type.First();
return Response.AsJson(address_types);
};
}
I'm able to access the route with Postman, however I'm receiving an empty response body. How can I return the object and/or a list of objects with Nancy?
I guess there's more configuration that needs to be done first. I'm new with Nancy, I just started using it this morning. It seems promising!
Thanks for the support.
I found a solution for this case:
I've changed the default Json Serializer that comes with Nancy with NewtonSoft.Json
The code in ExampleNancyModule remains the same, however I've added a Boostrap file to overwrite the default behaviour of Nancy. My Bootstrap.cs file looks like this:
namespace MyProject
{
public class Bootstrap : DefaultNancyBootstrapper
{
protected override void ConfigureApplicationContainer(TinyIoCContainer container)
{
base.ConfigureApplicationContainer(container);
container.Register<JsonSerializer, CustomJsonSerializer>();
}
}
public class CustomJsonSerializer : JsonSerializer
{
public CustomJsonSerializer()
{
this.ContractResolver = new CamelCasePropertyNamesContractResolver();
this.Formatting = Formatting.Indented;
}
}
}
With this I can get a JSON response that respects the attributes and the JSON configuration of my Entity Framework Models. Hope this helps someone else.