We Keep Coding

Set RootAttribute for System.Text.JsonSerializer?

Is there something like XmlRootAttribute that can be used with System.Text.JsonSerializer?
I need to be able to download data from this vendor using both XML an JSON. See sample data here:
{
"categories": [
{
"id": 125,
"name": "Trade Balance",
"parent_id": 13
}
]
}
Note the data elements are wrapped by an array named categories. When downloading using XML I can set the root element to categories and the correct object is returned (see XMLClient below ). When using JSONClient however I cannot (or do not know how) to set the root element. Best workaround I could find is to use JsonDocument which creates a string allocation. I could also create some wrapper classes for the JSON implementation but that is a lot of work that involves not only creating additional DTOs but also requires overwriting many methods on BaseClient. I also don't want to write converters - it pretty much defeats the purpose of using well-known serialization protocols. (Different responses will have a different root wrapper property name. I know the name in runtime, but not necessarily at compile time.)
public class JSONClient : BaseClient
{
protected override async Task<T> Parse<T>(string uri, string root)
{
uri = uri + (uri.Contains("?") ? "&" : "?") + "file_type=json";
var document = JsonDocument.Parse((await Download(uri)), new JsonDocumentOptions { AllowTrailingCommas = true });
string json = document.RootElement.GetProperty(root).GetRawText(); // string allocation
return JsonSerializer.Deserialize<T>(json);
}
}
public class XMLClient : BaseClient
{
protected override async Task<T> Parse<T>(string uri, string root)
{
return (T)new XmlSerializer(typeof(T), new XmlRootAttribute(root)).Deserialize(await Download(uri)); // want to do this - using JsonSerializer
}
}
public abstract class BaseClient
{
protected virtual async Task<Stream> Download(string uri)
{
uri = uri + (uri.Contains("?") ? "&" : "?") + "api_key=" + "xxxxx";
var response = await new HttpClient() { BaseAddress = new Uri(uri) }.GetAsync(uri);
return await response.Content.ReadAsStreamAsync();
}
protected abstract Task<T> Parse<T>(string uri, string root) where T : class, new();
public async Task<Category> GetCategory(string categoryID)
{
string uri = "https://api.stlouisfed.org/fred/category?category_id=" + categoryID;
return (await Parse<List<Category>>(uri, "categories"))?.FirstOrDefault();
}
}

JSON has no concept of a root element (or element names in general), so there's no equivalent to XmlRootAttribute in System.Text.Json (or Json.NET for that matter). Rather, it has the following two types of container, along with several atomic value types:
Objects, which are unordered sets of name/value pairs. An object begins with {left brace and ends with }right brace.
Arrays, which are ordered collections of values. An array begins with [left bracket and ends with ]right bracket.
As System.Text.Json.JsonSerializer is designed to map c# objects to JSON objects and c# collections to JSON arrays in a 1-1 manner, there's no built-in attribute or declarative option to tell the serializer to automatically descend the JSON hierarchy until a property with a specific name is encountered, then deserialize its value to a required type.
If you need to access some JSON data that is consistently embedded in some wrapper object containing a single property whose name is known at runtime but not compile time, i.e.:
{
"someRuntimeKnownWrapperPropertyName" : // The value you actually want
}
Then the easiest way to do that would be to deserialize to a Dictionary<string, T> where the type T corresponds to the type of the expected value, e.g.:
protected override async Task<T> Parse<T>(string uri, string root)
{
uri = uri + (uri.Contains("?") ? "&" : "?") + "file_type=json";
using var stream = await Download(uri); // Dispose here or not? What about disposing of the containing HttpResponseMessage?
var options = new JsonSerializerOptions
{
AllowTrailingCommas = true,
// PropertyNameCaseInsensitive = false, Uncomment if you need case insensitivity.
};
var dictionary = await JsonSerializer.DeserializeAsync<Dictionary<string, T>>(stream, options);
// Throw an exception if the dictionary does not have exactly one entry, with the required name
var pair = dictionary?.Single();
if (pair == null || !pair.Value.Key.Equals(root, StringComparison.Ordinal)) //StringComparison.OrdinalIgnoreCase if required
throw new JsonException();
// And now return the value
return pair.Value.Value;
}
Notes:
By deserializing to a typed dictionary rather than a JsonDocument you avoid the intermediate allocations required for the JsonDocument itself as well as the string returned by GetRawText(). The only memory overhead is for the dictionary itself.
Note also that JsonDocument is disposable. Failure to dispose it will result in the memory not being returned to the pool, which will increase GC impact across various parts of the framework.
Alternatively, you could create a [custom JsonConverter](Failure to properly dispose this object will result in the memory not being returned to the pool, which will increase GC impact across various parts of the framework.) to read through the incoming JSON until a property of the required name is encountered, then deserialize the value to the expected type.
For some examples, see Is there a simple way to manually serialize/deserialize child objects in a custom converter in System.Text.Json? or https://stackoverflow.com/a/62155881/3744182.
Rather than allocating an HttpClient for every call, you should allocate only one and reuse it. See Do HttpClient and HttpClientHandler have to be disposed between requests?.
You are not disposing of the HttpResponseMessage or response content stream. You may want to refactor your code to do that; see When or if to Dispose HttpResponseMessage when calling ReadAsStreamAsync?. You may also want to check response.IsSuccessStatusCode before deserializing.
In .Net 5, HttpClientJsonExtensions can make deserializing JSON returned by HttpClient simpler.

Getting column names dynamically in SSIS script component acting as Destination

Ok, I am writing an avro file using SSIS script component as a destination. Since AVRO needs a schema as well, I need to define the schema. It works fine when I define the schema manually. But I have 10-12 data flow tasks and I do not want to write the schema explicitly. I am trying to see if I can use the BufferWrapper which is auto generated to see if I can read from there but I can't and it always returns blank.
I have tried the solution posted here and also read up this.
But everything returns blank.
I have also come across this. Could that be the reason and if that explanation in the answer posted there is correct, isn't this possible?
So, in my
public override void PreExecute(), I have something like this:
Schema = #"{
""type"":""record"",
""name"":""Microsoft.Hadoop.Avro.Specifications.Counterparts"",
""fields"":
[
{ ""name"":""CounterpartID"", ""type"":""int"" },
{ ""name"":""CounterpartFirstDepositDate"", ""type"":[""string"",""null""] },
{ ""name"":""CounterpartFirstTradeDate"",""type"":[""string"",""null""] },
{ ""name"":""ClientSegmentReportingID"",""type"":[""int"",""null""] },
{ ""name"":""ClientSegmentReportingName"", ""type"":[""string"",""null""] },
{ ""name"":""ContractID"", ""type"":[""int"",""null""] },
{ ""name"":""ContractFirstDepositDate"", ""type"":[""string"",""null""] },
{ ""name"":""ContractFirstTradeDate"",""type"":[""string"",""null""] },
{ ""name"":""ContractClosingOffice"",""type"":[""string"",""null""] },
{ ""name"":""LeadCreationDate"", ""type"":[""string"",""null""] },
{ ""name"":""ContractCountryOfResidence"", ""type"":[""string"",""null""] }
]
}";
}
Instead of manually defining all this schema, I am checking if I could generate it out of the BufferWrapper but this returns blank:
var fields = typeof(Input0Buffer).GetFields().Select(m => new
{
Name = m.Name,
Type = m.FieldType
}).ToList();
Also, if I just do this, that also returns blank
Type myType = typeof(Input0Buffer);
// Get the fields of the specified class.
FieldInfo[] myField = myType.GetFields();
Earlier I was putting these new methods in Pre-Execute but then thought, maybe the buffer isn't initialized by then , so I moved that to Input0_ProcessInputRow method and making sure that it is triggered only once using a counter variable and making this code run only when counter=0, but even that returns blank.
public override void Input0_ProcessInputRow(Input0Buffer Row)
{
if (counter == 0)
{
Type myType = typeof(Input0Buffer);
// Get the fields of the specified class.
FieldInfo[] myField = myType.GetFields();
}
//Processing logic
}
Isn't it possible because of this?
As it talks about it being protected and also not accessible from outside that autogenerated class.

I have finally found the answer here: https://waheedrous.wordpress.com/2014/02/24/ssis-global-replace-for-all-columns-using-a-script-component/
I can finally get the list of columns and datatypes from within the script component. I will only run it the first time (using a counter variable).

Validate object against a schema before serialization

I want to serialize a C# object as JSON into a stream, but to avoid the serialization if the object is not valid according to a schema. How should I proceed with this task using JSON.NET and Json.NET Schema? From what I see there is no method in the JSON.NET library which allows the validation of a C# object against a JSON schema. It seems somewhat weird that there is no direct method to just validate the C# object without encoding it. Do you have any idea why this method is not available?

It seems this API not currently available. At a guess, this might be because recursively generating the JSON values to validate involves most of the work of serializing the object. Or it could just be because no one at Newtonsoft ever designed, specified, implemented, tested, documented and shipped that feature.
If you want, you could file an enhancement request requesting this API, probably as a part of the SchemaExtensions class.
In the meantime, if you do need to test-validate a POCO without generating a complete serialization of it (because e.g. the result would be very large), you could grab NullJsonWriter from Reference to automatically created objects, wrap it in a JSchemaValidatingWriter and test-serialize your object as shown in Validate JSON with JSchemaValidatingWriter. NullJsonWriter doesn't actually write anything, and so using it eliminates the performance and memory overhead of generating a complete serialization (either as a string or as a JToken).
First, add the following static method:
public static class JsonExtensions
{
public static bool TestValidate<T>(T obj, JSchema schema, SchemaValidationEventHandler handler = null, JsonSerializerSettings settings = null)
{
using (var writer = new NullJsonWriter())
using (var validatingWriter = new JSchemaValidatingWriter(writer) { Schema = schema })
{
int count = 0;
if (handler != null)
validatingWriter.ValidationEventHandler += handler;
validatingWriter.ValidationEventHandler += (o, a) => count++;
JsonSerializer.CreateDefault(settings).Serialize(validatingWriter, obj);
return count == 0;
}
}
}
// Used to enable Json.NET to traverse an object hierarchy without actually writing any data.
class NullJsonWriter : JsonWriter
{
public NullJsonWriter()
: base()
{
}
public override void Flush()
{
// Do nothing.
}
}
Then use it like:
// Example adapted from
// https://www.newtonsoft.com/jsonschema/help/html/JsonValidatingWriterAndSerializer.htm
// by James Newton-King
string schemaJson = #"{
'description': 'A person',
'type': 'object',
'properties': {
'name': {'type':'string'},
'hobbies': {
'type': 'array',
'maxItems': 3,
'items': {'type':'string'}
}
}
}";
var schema = JSchema.Parse(schemaJson);
var person = new
{
Name = "James",
Hobbies = new [] { ".Net", "Blogging", "Reading", "XBox", "LOLCATS" },
};
var settings = new JsonSerializerSettings { ContractResolver = new CamelCasePropertyNamesContractResolver() };
var isValid = JsonExtensions.TestValidate(person, schema, (o, a) => Console.WriteLine(a.Message), settings);
// Prints Array item count 5 exceeds maximum count of 3. Path 'hobbies'.
Console.WriteLine("isValid = {0}", isValid);
// Prints isValid = False
Watch out for cases by the way. Json.NET schema is case sensitive so you will need to use an appropriate contract resolver when test-validating.
Sample fiddle.

You cannot do that from the JSON string, you need an object and a schema to compare with first..
public void Validate()
{
//...
JsonSchema schema = JsonSchema.Parse("{'pattern':'lol'}");
JToken stringToken = JToken.FromObject("pie");
stringToken.Validate(schema);

Serializing cyclic object references using DataContractSerializer not working

I'm building an XNA game and I'm trying to save game/map etc. state completely, and then be able to load and resume from exactly the same state.
My game logic consists of fairly complex elements (for serializing) such as references, delegates etc. I've done hours of research and decided that it's the best to use a DataContractSerializer that preserves the object references. (I also got around for delegates but that's another topic) I have no problem serializing and deserializing the state, re-creating the objects, the fields, lists, and even object references correctly and completely. But I've got a problem with cyclic references. Consider this scenario:
class X{
public X another;
}
//from code:
X first = new X();
X second = new X();
first.another = second;
second.another = first;
Trying to serialize X will result in an exception complaining about cyclic references. If I comment out the last line it works fine. Well, I can imagine WHY it is happening, but I have no idea HOW to solve it. I've read somewhere that I can use the DataContract attribute with IsReference set to true, but it didn't change anything for me -- still got the error. (I want to avoid it anyway since the code I'm working on is portable code and may someday run on Xbox too, and portable library for Xbox doesn't support the assembly that DataContract is in.)
Here is the code to serialize:
class DataContractContentWriterBase<T> where T : GameObject
{
internal void Write(Stream output, T objectToWrite, Type[] extraTypes = null)
{
if (extraTypes == null) { extraTypes = new Type[0]; }
DataContractSerializer serializer = new DataContractSerializer(typeof(T), extraTypes, int.MaxValue, false, true, null);
serializer.WriteObject(output, objectToWrite);
}
}
and I'm calling this code from this class:
[ContentTypeWriter]
public class PlatformObjectTemplateWriter : ContentTypeWriter<TWrite>
(... lots of code ...)
DataContractContentWriterBase<TWrite> writer = new DataContractContentWriterBase<TWrite>();
protected override void Write(ContentWriter output, TWrite value)
{
writer.Write(output.BaseStream, value, GetExtraTypes());
}
and for deserialization:
class DataContractContentReaderBase<T> where T: GameObject
{
internal T Read(Stream input, Type[] extraTypes = null)
{
if (extraTypes == null) { extraTypes = new Type[0]; }
DataContractSerializer serializer = new DataContractSerializer(typeof(T), extraTypes, int.MaxValue, false, true, null);
T obj = serializer.ReadObject(input) as T;
//return obj.Clone() as T; //clone falan.. bi bak iste.
return obj;
}
}
and it's being called by:
public class PlatformObjectTemplateReader : ContentTypeReader<TRead>
(lots of code...)
DataContractContentReaderBase<TRead> reader = new DataContractContentReaderBase<TRead>();
protected override TRead Read(ContentReader input, TRead existingInstance)
{
return reader.Read(input.BaseStream, GetExtraTypes());
}
where:
PlatformObjectTemplate was my type to write.
Any suggestions?
SOLUTION: Just a few minutes ago, I've realized that I wasn't marking the fields with DataMember attribute, and before I added the DataContract attribute, the XNA serializer was somehow acting as the "default" serializer. Now, I've marked all the objects, and things are working perfectly now. I now have cyclic references with no problem in my model.

If you don't want to use [DataContract(IsReference=true)] then DataContractSerializer won't help you, because this attribute is the thing that does the trick with references.
So, you should either look for alternative serializers, or write some serialization code that transforms your graphs into some conventional representation (like a list of nodes + a list of links between them) and back, and then serialize that simple structure.
In case you decide to use DataContract(IsReference=true), here's a sample that serializes your graph:
[DataContract(IsReference = true)]
class X{
[DataMember]
public X another;
}
static void Main()
{
//from code:
var first = new X();
var second = new X();
first.another = second;
second.another = first;
byte[] data;
using (var stream = new MemoryStream())
{
var serializer = new DataContractSerializer(typeof(X));
serializer.WriteObject(stream, first);
data = stream.ToArray();
}
var str = Encoding.UTF8.GetString(data2);
}
The str will contain the following XML:
<X z:Id="i1" xmlns="http://schemas.datacontract.org/2004/07/GraphXmlSerialization"
xmlns:i="http://www.w3.org/2001/XMLSchema-instance"
xmlns:z="http://schemas.microsoft.com/2003/10/Serialization/">
<another z:Id="i2">
<another z:Ref="i1"/>
</another>
</X>

XmlReader best practices

I've had a good read through MSDN and the XmlReader related questions on StackOverflow and I haven't yet come across a decent "best practices" example.
I've tried various combinations and each seems to have downsides, but the best I can come up with is as follows:
The XML:
<properties>
<actions:name>start</actions:name>
<actions:value type="System.DateTime">06/08/2011 01:26:49</actions:value>
</properties>
The code:
// Reads past the initial/root element
reader.ReadStartElement();
// Check we haven't hit the end
while (!reader.EOF && reader.NodeType != XmlNodeType.EndElement) {
if (reader.IsStartElement("name", NamespaceUri)) {
// Read the name element
this.name = reader.ReadElementContentAsString();
} else if (reader.IsStartElement("value", NamespaceUri)) {
// Read the value element
string valueTypeName = reader["type"] ?? typeof(string).Name;
Type valueType = Type.GetType(valueTypeName);
string valueString = reader.ReadElementContentAsString();
// Other stuff here that doesn;t matter to the XML parsing
} else {
// We can't do anything with this node so skip over it
reader.Read();
}
}
This is being passed into my class from a .ReadSubTree() call and each class reads its own information. I would prefer it NOT to rely on it being in a specific order.
Before this, I did try several variations.
1) while(reader.Read())
This was taken from various example, but found that it "missed" some elements when .ReadContent*() of element 1 left it on the start of the element 2, .Read read over it to element 3.
2) Removing the .Read() caused it to just get stuck after the first element I read.
3) Several others I long consigned to "failed".
As far as I can see, the code I've settled on seems to be the most accepting and stable but is there anything obvious I'm missing?
(Note the c# 2.0 tag so LINQ/XNode/XElement aren't options)

One approach is to use a custom XmlReader. XmlReader is abstract and XmlReaders can be chained, giving a powerful mechanism to do some domain specific processing in a reader.
Example: XamlXmlReader
Help on XmlWrappingReader
Here's a sample of how it could be implemented (See inline comments):
/// <summary>
/// Depending on the complexity of the Xml structure, a complex statemachine could be required here.
/// Such a reader nicely separates the structure of the Xml from the business logic dependent on the data in the Xml.
/// </summary>
public class CustomXmlReader: XmlWrappingReader
{
public CustomXmlReader(XmlReader xmlReader)
:base(XmlReader.Create(xmlReader, xmlReader.Settings))
{
}
public override bool Read()
{
var b = base.Read();
if (!b)
return false;
_myEnum = MyEnum.None;
if("name".Equals(this.Name))
{
_myEnum = MyEnum.Name;
//custom logic to read the entire element and set the enum, name and any other properties relevant to your domain
//Use base.Read() until you've read the complete "logical" chunk of Xml. The "logical" chunk could be more than a element.
}
if("value".Equals(this.Value))
{
_myEnum = Xml.MyEnum.Value;
//custom logic to read the entire element and set the enum, value and and any other properties relevant to your domain
//Use base.Read() until you've read the complete "logical" chunk of Xml. The "logical" chunk could be more than a element.
}
return true;
}
//These properties could return some domain specific values
#region domain specific reader properties.
private MyEnum _myEnum;
public MyEnum MyEnum
{
get { return _myEnum; }
}
#endregion
}
public enum MyEnum
{
Name,
Value,
None
}
public class MyBusinessAppClass
{
public void DoSomething(XmlReader passedInReader)
{
var myReader = new CustomXmlReader(passedInReader);
while(myReader.Read())
{
switch(myReader.MyEnum)
{
case MyEnum.Name:
//Do something here;
break;
case MyEnum.Value:
//Do something here;
break;
}
}
}
}
A word of caution : This might be over engineering for some simple Xml processing that you've shown here. Unless, you have more that two elements that need custom processing, this approach is not advised.