I'm trying to persist the following class to DynamoDB using the .NET SDK:
public class MyClass
{
public string Id { get; set; }
public string Name { get; set; }
public object Settings { get; set; }
}
The problem is with the Settings property. It can be any type of object, and I do not know in advance what might be assigned to it. When I try to persist it to DynamoDB, I get the following exception:
System.InvalidOperationException: 'Type System.Object is unsupported, it has no supported members'
Both the Document Model and Object Persistence Model methods result in the same exception.
Is there a way to persist these objects in DynamoDB? Other databases like MongoDB and Azure DocumentDB will do this without any issue, and they can be deserialized to either the proper type with a discriminator, or as a dynamic JSON object.
You can use the general approach documented here: https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/DynamoDBContext.ArbitraryDataMapping.html
Here's my implementation for any arbitrary object:
public class DataConverter : IPropertyConverter
{
public object FromEntry(DynamoDBEntry entry)
{
var primitive = entry as Primitive;
if (primitive == null || !(primitive.Value is String) || string.IsNullOrEmpty((string)primitive.Value))
throw new ArgumentOutOfRangeException();
object ret = JsonConvert.DeserializeObject(primitive.Value as string);
return ret;
}
public DynamoDBEntry ToEntry(object value)
{
var jsonString = JsonConvert.SerializeObject(value);
DynamoDBEntry ret = new Primitive(jsonString);
return ret;
}
}
Then annotate your property like this:
[DynamoDBProperty(typeof(DataConverter))]
public object data { get; set; }
Little improvement to the previous answer: make converter generic so that you can deserialize to the correct type, like this:
public class SerializeConverter<T> : IPropertyConverter
{
public object FromEntry(DynamoDBEntry entry)
{
var primitive = entry as Primitive;
if (primitive is not { Value: string value } || string.IsNullOrEmpty(value))
throw new ArgumentException("Data has no value", nameof(entry));
return JsonConvert.DeserializeObject<T>(value);
}
public DynamoDBEntry ToEntry(object value) =>
new Primitive(JsonConvert.SerializeObject(value));
}
Usage:
[DynamoDBProperty(typeof(SerializeConverter<YourType>))]
public YourType data{ get; set; }
I struggled to find a good solution for interacting with thoroughly unstructured data, then eventually realized that the DynamoDBContext really isn't designed for that.
For anyone else who gets to this point, my advice is to drop to a lower abstraction level and use the AmazonDynamoDBClient directly with Dictionary<string, AttributeValue> objects.
Related
I am trying to create a generic wrapper library(C#/.NET) for AWS DynamoDB which can act as DAL(Data Access Layer). The applications consuming this library will not be tightly coupled with AWS libraries as there is a possibility that it can be changed later.
The structure of methods to be exposed from wrapper class are
InsertItem< T>(object) , UpdateItem< T>(object) , DeleteItem< T>(id/object),
List< T> GetAll() , T GetByParameter< T>(Id).
I see that there are three approaches to consume AWS DynamoDB services using AWSSDK.
Approach (1) : Low level Access - convert model to aws hashmap input structure and invoke getItem()/putItem() .
Approach (2) : High Level Access using Document - convert model to aws document model and passing document object to aws.
Approach (3) : High Level Access using Persistence - Using attribute DynamoDBTable in model to map model to dynamoDb table and using linq operation to get/update table.
In approach(1) & (2), i find it difficult to map the model to dynamoDB table. In approach (3), I see that i need to include the DynamoDB attributes in model class in application which would make it tightly coupled.
Is there any way to create mapping in runtime in this cases or is there any other approach?
I also thought whether i can json serialize/deserialize the model and inset into dynamoDB(In this case there would be only 2 columns - id, json body for any model).
Please correct me if i am wrong or missing something.
What follows is the solution I'm moving forward with, minus some extraneous details. The biggest challenge is reading data of arbitrary types, since you can't easily tell just from the JSON.
In my solution, the consuming application that chooses the types to write also knows how to identify which type to deserialize to when reading. This is necessary because I need to return multiple logs of different types, so it makes more sense to return the JSON to the consumer and let them deal with it. If you want to contain this in the DAL, you could add a Type column in the DB and convert it using Reflection, though I think you'd still have issues returning data for multiple types in one call.
The interface we present for consumption has read/write methods (you can add whatever else you need). Note that writing allows specification of T, but reading requires the caller to deserialize, as mentioned above.
public interface IDataAccess
{
Task WriteAsync<T>(Log<T> log) where T : class, new();
Task<IEnumerable<LogDb>> GetLogsAsync(long id);
}
A LogDb class contains the Persistence attributes:
[DynamoDBTable("TableName")]
public class LogDb
{
[DynamoDBHashKey("Id")]
public long Id{ get; set; }
[DynamoDBProperty(AttributeName = "Data", Converter = typeof(JsonStringConverter))]
public string DataJson { get; set; }
}
A generic Log<T> class used for strongly-typed writing. The ToDb() method is called in the IDataAccess implementation to actually write to the DB. The constructor taking in a LogDb would be used by a consuming application that had identified the appropriate type for deserialization:
public class Log<T>
where T : class, new()
{
public Log() { }
public Log(LogDb logDb)
{
Data = licensingLogDb.OldDataJson.Deserialize<T>();
Id = licensingLogDb.Id;
}
public long Id { get; set; }
public T Data { get; set; }
public LogDb ToDb()
{
string dataJson = Data.Serialize();
return new LogDb
{
DataJson = dataJson,
Id = Id
};
}
}
The JsonStringConverter used in the attributes on LogDb converts the JSON string in the DataJson property to and from a DynamoDB Document:
public class JsonStringConverter : IPropertyConverter
{
public DynamoDBEntry ToEntry(object value)
{
string json = value as string;
return !String.IsNullOrEmpty(json)
? Document.FromJson(json)
: null;
}
public object FromEntry(DynamoDBEntry entry)
{
var document = entry.AsDocument();
return document.ToJson();
}
}
A helper class provides the Serialize and Deserialize extensions, which use JSON.NET's JsonConvert.Serialize/Deserialize, but with null checks:
public static class JsonHelper
{
public static string Serialize(this object value)
{
return value != null
? JsonConvert.SerializeObject(value)
: String.Empty;
}
public static T Deserialize<T>(this string json)
where T : class, new()
{
return !String.IsNullOrWhiteSpace(json)
? JsonConvert.DeserializeObject<T>(json)
: null;
}
}
Generic method convert Dynamo table to c# class as an extension function.
public static List<T> ToMap<T>(this List<Document> item)
{
List<T> model = (List<T>)Activator.CreateInstance(typeof(List<T>));
foreach (Document doc in item)
{
T m = (T)Activator.CreateInstance(typeof(T));
var propTypes = m.GetType();
foreach (var attribute in doc.GetAttributeNames())
{
var property = doc[attribute];
if (property is Primitive)
{
var properties = propTypes.GetProperty(attribute);
if (properties != null)
{
var value = (Primitive)property;
if (value.Type == DynamoDBEntryType.String)
{
properties.SetValue(m, Convert.ToString(value.AsPrimitive().Value));
}
else if (value.Type == DynamoDBEntryType.Numeric)
{
properties.SetValue(m, Convert.ToInt32(value.AsPrimitive().Value));
}
}
}
else if (property is DynamoDBBool)
{
var booleanProperty = propTypes.GetProperty(attribute);
if (booleanProperty != null)
booleanProperty.SetValue(m, property.AsBoolean());
}
}
model.Add(m);
}
return model;
}
I have to store complex object into hash of redis cash.I am using stackexchange.redis to do this.My Class is like below.
public class Company
{
public string CompanyName { get; set; }
public List<User> UserList { get; set; }
}
public class User
{
public string Firstname { get; set; }
public string Lastname { get; set; }
public string Twitter { get; set; }
public string Blog { get; set; }
}
My code snippet to store data in redis is:
db.HashSet("Red:10000",comapny.ToHashEntries());
//Serialize in Redis format:
public static HashEntry[] ToHashEntries(this object obj)
{
PropertyInfo[] properties = obj.GetType().GetProperties();
return properties
.Where(x => x.GetValue(obj) != null) // <-- PREVENT NullReferenceException
.Select(property => new HashEntry(property.Name, property.GetValue(obj)
.ToString())).ToArray();
}
I could store the data in redis but not as i want.I am geting result as shown in below image.
I want UserList value in json format.So,how can i do this.
Probably the easiest path is checking if each property value is a collection (see the comments in my modified version of your method):
public static HashEntry[] ToHashEntries(this object obj)
{
PropertyInfo[] properties = obj.GetType().GetProperties();
return properties
.Where(x => x.GetValue(obj) != null) // <-- PREVENT NullReferenceException
.Select
(
property =>
{
object propertyValue = property.GetValue(obj);
string hashValue;
// This will detect if given property value is
// enumerable, which is a good reason to serialize it
// as JSON!
if(propertyValue is IEnumerable<object>)
{
// So you use JSON.NET to serialize the property
// value as JSON
hashValue = JsonConvert.SerializeObject(propertyValue);
}
else
{
hashValue = propertyValue.ToString();
}
return new HashEntry(property.Name, hashValue);
}
)
.ToArray();
}
It seems that there is something wrong with serializing. The best way of converting between JSON and .NET object is using the JsonSerializer:
JsonConvert.SerializeObject(fooObject);
You can see more details from Serializing and Deserializing JSON.
Also there is another good way,you can try to use IRedisTypedClient which is a part of ServiceStack.Redis.
IRedisTypedClient - A high-level 'strongly-typed' API available
on Service Stack's C# Redis Client to make all Redis Value operations
to apply against any c# type. Where all complex types are
transparently serialized to JSON using ServiceStack JsonSerializer -
The fastest JSON Serializer for .NET.
Hope this helps.
I am trying to serialize an object in XML so that I will be able to deserialize it in silverlight. Pasted below is the code.
StringBuilder builder = new StringBuilder();
var serializer = new XmlSerializer(data.GetType());
var writer = XmlWriter.Create(builder);
serializer.Serialize(writer, data); //error here
string xml = builder.ToString();
The code is throwing this error for data
"System.RuntimeType is inaccessible due to its protection level. Only public types can be processed."
The object data is actually
List<List<LabelValueTypeGroup>>
where LabelValueType is declared as
[Serializable]
[XmlRoot]
public struct LabelValueTypeGroup/* : IEnumerable*/
{
public string Label { get; set; }
public Type Type { get; set; }
public string Value { get; set; }
public ComparisonType? ComparisonType { get; set; }
public IEnumerator GetEnumerator()
{
yield return this.Label;
yield return this.Type;
yield return this.Value;
yield return this.ComparisonType;
}
}
What am I doing wrong? Is List not serialize-able or what? Why I am doing it this way is because, I want to pass an object from WCF to Silverlight that is resulting from a select query, that would only select the columns found in an xml file. So I have to create a sort of dynamic serialize-able object which could be de-serialized in Silverlight. Javascriptserializer and JSON are not accessible in Silverlight project.
It's probably the public Type Type { get; set; } field, holding a reference to a RuntimeType instance which causes the error. RuntimeType is not serializable, that is what the error message tells you.
I want to create a key value table in my database along the lines of
public class KeyValue {
public string Id { get; set; }
public dynamic Value {get; set; }
}
Using a slightly modified SqlProvider I have no problems getting CreateTable<KeyValue>() to generate varchar(1024) Id, varchar(max) Value.
I have no issues saving objects to it. The problem is when I load the objects
var content = dbConn.GetById<KeyValue>("about");
content.Value at this point is a string.
Looking at the database record, the text for value does not appear to store any type information.
Is there really anything I can do better other than manually invoking ServiceStack.Text and call deserialize with the appropriate type information?
I do not need absolute dynamic, my actual use case is for polymorphism with a base class instead of dynamic. So I don't really care what type Value is whether it's the base class, dynamic, object, etc. Regardless other than using the class
public class KeyValue {
public string Id { get; set; }
public MySpecificChildType Value {get; set; }
}
I haven't been able to get anything other than a string back for Value. Can I tell OrmLite to serialize the type information to be able to correctly deserialize my objects or do I just have to do it manually?
Edit: some further information. OrmLite is using the Jsv serializer defined by ServiceStack.Text.TypeSerializer and is in no way pluggable in the BSD version. If I add a Type property to my KeyValue class with the dynamic Value I can do
var value = content.Value as string;
MySpecificChildType strongType =
TypeSerializer.DeserializeFromString(content, content.Type);
I just really want a better way to do this, I really don't like an object of 1 type going into the db coming back out with a different type (string).
I haven't worked much with the JsvSerializer but with the JsonSerializer you can achieve this (in a few different ways) and as of ServiceStack 4.0.11 you can opt to use the JsonSerializer instead, see https://github.com/ServiceStack/ServiceStack/blob/master/release-notes.md#v4011-release-notes.
Example
public abstract class BaseClass {
//Used for second example of custom type lookup
public abstract string Type { get; set; }
}
public class ChildA : BaseClass {
//Used for second example of custom type lookup
public override string Type { get; set; }
public string PropA { get; set; }
}
And then in your init/bootstrap class you can configure the serializer to emit the type information needed for proper deserialization:
public class Bootstrapper {
public void Init() {
ServiceStack.Text.JsConfig.ExcludeTypeInfo = false;
ServiceStack.Text.JsConfig.IncludeTypeInfo = true;
}
}
If you wish to use something other that the default "__type" attribute that ServiceStack uses (if you for example want to have a friendly name identifying the type rather then namespace/assembly) you can also configure your own custom type lookup as such
public class Bootstrapper {
public void Init() {
ServiceStack.Text.JsConfig.ExcludeTypeInfo = false;
ServiceStack.Text.JsConfig.IncludeTypeInfo = true;
ServiceStack.Text.JsConfig.TypeAttr = "type";
ServiceStack.Text.JsConfig.TypeFinder = type =>
{
if ("CustomTypeName".Equals(type, StringComparison.OrdinalIgnoreCase))
{
return typeof(ChildA);
}
return typeof(BaseClass);
}
}
}
I'm trying to deserialize json to an object model where the collections are represented as IList<T> types.
The actual deserializing is here:
JavaScriptSerializer serializer = new JavaScriptSerializer();
return serializer.Deserialize<IList<Contact>>(
(new StreamReader(General.GetEmbeddedFile("Contacts.json")).ReadToEnd()));
Before i post the exception i'm getting you should know what the implicit conversions are. This is the Contact type:
public class Contact
{
public int ID { get; set; }
public string Name { get; set; }
public LazyList<ContactDetail> Details { get; set; }
//public List<ContactDetail> Details { get; set; }
}
And this is the ContactDetail type:
public class ContactDetail
{
public int ID { get; set; }
public int OrderIndex { get; set; }
public string Name { get; set; }
public string Value { get; set; }
}
The important thing to know with the LazyList<T> is that it implements IList<T>:
public class LazyList<T> : IList<T>
{
private IQueryable<T> _query = null;
private IList<T> _inner = null;
private int? _iqueryableCountCache = null;
public LazyList()
{
this._inner = new List<T>();
}
public LazyList(IList<T> inner)
{
this._inner = inner;
}
public LazyList(IQueryable<T> query)
{
if (query == null)
throw new ArgumentNullException();
this._query = query;
}
Now this LazyList<T> class definition was fine until i tried deserializing Json into it. The System.Web.Script.Serialization.JavaScriptSerializer seems to want to serialize lists to List<T> which makes sense coz of it's age but i need them in the type IList<T> so they will cast into my LazyList<T> (at least that's where i think i am going wrong).
I get this exception:
System.ArgumentException: Object of type 'System.Collections.Generic.List`1[ContactDetail]' cannot be converted to type 'LazyList`1[ContactDetail]'..
When i try using List<ContactDetail> in my Contact type (as you can see commented above) it seems to work. But i dont want to use List<T>'s. I even tried having my LazyList<T> inheriting from List<T> which seemed to execute but passing the List<T>'s internal T[] to my implementation was a nightmare and i simply don't want the bloat of List<T> anywhere in my model.
I also tried some other json libraries to no avail (it's possible i may not be using these to their full potential. I more or less replaced the references and attempted to repeat the code quoted at the top of this question. Maybe passing settings params will help??).
I dont know what to try now. Do i go with another deserializer? Do i tweak the deserializing itself? Do i need to change my types to please the deserializer? Do i need to worry more about implicit casting or just implement another interface?
It is not possible to deserialize directly to an interface, as interfaces are simply a contract. The JavaScriptSerializer has to deserialize to some concrete type that implements IList<T>, and the most logical choice is List<T>. You will have to convert the List to a LazyList, which given the code you posted, should be easy enough:
var list = serializer.Deserialize<IList<Contact>>(...);
var lazyList = new LazyList(list);
Unfortunately you will probably need to fix your class, as there is no way for a deserializer to know that it should be of type IList, since List is an implementation of IList.
Since the deserializers at http://json.org have source available you could just modify one to do what you want.
I ended up using the Json.NET lib which has good linq support for custom mapping. This is what my deserializing ended up looking like:
JArray json = JArray.Parse(
(new StreamReader(General.GetEmbeddedFile("Contacts.json")).ReadToEnd()));
IList<Contact> tempContacts = (from c in json
select new Contact
{
ID = (int)c["ID"],
Name = (string)c["Name"],
Details = new LazyList<ContactDetail>(
(
from cd in c["Details"]
select new ContactDetail
{
ID = (int)cd["ID"],
OrderIndex = (int)cd["OrderIndex"],
Name = (string)cd["Name"],
Value = (string)cd["Value"]
}
).AsQueryable()),
Updated = (DateTime)c["Updated"]
}).ToList<Contact>();
return tempContacts;