We have a data model with defined properties, but one of the properties allows for dynamic metadata (a list of maps or dictionaries). Using the document model, this property maps fine to a list of Document, however, when I'm having trouble getting this dynamic property to map to anything using DataModel. Is there a way to map dynamic data to documents inside a model class property?
Attempting to map it as a list of dictionaries (which matches the structure of the metadata) fails with the below error:
public List<Dictionary<string, object>> Events { get; set; }
Unable to convert [Amazon.DynamoDBv2.DocumentModel.Document] of type
Amazon.DynamoDBv2.DocumentModel.Document to
System.Collections.Generic.Dictionary`
Using a type of List<Document> got me the closest, which it now lists out 39 documents, but all the Documents have 0 keys, 0 values.
public List<Document> Events { get; set; }
Ex:
document["Events"].AsListOfDocument().First(); // works, contains the keys and values
datamodel.Events.First(); // does not work, it is an empty document
I know this is a quite old but hopefully this might help somebody else struggling with this.
Hi Devon,
In case you are trying to create the object to send it to your DynamoDB you can try mapping arbitrary data just as stated in AWS documentation for .NET
Here is the code from the documentation:
try
{
DynamoDBContext context = new DynamoDBContext(client);
// 1. Create a book.
DimensionType myBookDimensions = new DimensionType()
{
Length = 8M,
Height = 11M,
Thickness = 0.5M
};
Book myBook = new Book
{
Id = 501,
Title = "AWS SDK for .NET Object Persistence Model Handling Arbitrary Data",
ISBN = "999-9999999999",
BookAuthors = new List<string> { "Author 1", "Author 2" },
Dimensions = myBookDimensions
};
context.Save(myBook);
Once you have your data in the database you can append a new map to the list with something on the lines of:
var request = new UpdateItemRequest
{
TableName = "TableName",
Key = new Dictionary<string, AttributeValue>() { { "PartitionKey", new AttributeValue { S = "value" } } },
ExpressionAttributeNames = new Dictionary<string, string>()
{
{ "#A", "A" }
},
ExpressionAttributeValues = new Dictionary<string, AttributeValue>()
{
{
":val", new AttributeValue
{
L = new List<AttributeValue>()
{
{
new AttributeValue
{
M = new Dictionary<string, AttributeValue>()
{
{ "Address", new AttributeValue{S = "Value" } },
{ "Latitude", new AttributeValue { S = position.Latitude.ToString() } },
{ "Longitude", new AttributeValue { S = position.Longitude.ToString() } }
}
}
}
}
}
}
},
UpdateExpression = "SET #A = list_append(#A, :val)"
};
try
{
var response = await client.UpdateItemAsync(request);
}
catch (Exception ex)
{
Debug.WriteLine(ex.Message);
}
This did the trick for me, I hope it does for someone else.
.
.
.
PS: BTW this is my first answer in Stackoverflow and it feels nice to try to contribute when I have came here multiple times for answers that saved me time jajajaja
Related
I am using free tier of AWZ.
Trying to insert item in DynamoDb Table. Below code doesn't throw any exception but also not creating new item in table.
Please let me know what i am doing wrong here.
Note: Using AWZ free tier.
public async Task Additem()
{
try
{
AmazonDynamoDBClient client = new AmazonDynamoDBClient();
string tableName = "Car";
var request = new PutItemRequest
{
TableName = tableName,
Item = new Dictionary<string, AttributeValue>()
{
{ "Id", new AttributeValue { N = "201" }},
{ "Name", new AttributeValue { S = "Car2" }},
{ "Price", new AttributeValue { N = "200" }}
}
};
var result = await client.PutItemAsync(request);
}
catch ( Exception ex)
{
throw ex;
}
}
I am trying to retrieve the data from Dynamo DB based on a search criteria using Scan request. I am following the steps mentioned in http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/LowLevelDotNetScanning.html" page. My DynamoDB table contains more than 1 million records. I know that by using the do-while loop and ExclusiveStartKey we can fetch the records from dynamo DB. but in my case I can not wait till search process is complete as this will hang my angularJS UI. instead I want to progressively load the data with out waiting for the search process to complete. How we can do that.?
sample request:
var lastEvaluatedKey = new Dictionary<string, AttributeValue>(); ;
AmazonDynamoDBClient amazonDynamoDbClient= new AmazonDynamoDBClient()
var filterExpression = "#aws_s3_bucket = :v_aws_s3_bucket and contains(#aws_s3_key,:v_aws_s3_key)";
var projectExpression = "#aws_s3_key,filename,#region,aws_s3_bucket,#projecttype,folder,#siteid,locationname,createdon,modifiedon";
do
{
var request = new ScanRequest
{
TableName = "Test1",
ExclusiveStartKey=lastEvaluatedKey,
FilterExpression = filterExpression,
ExpressionAttributeNames = new Dictionary<string, string>
{
{ "#region", "region" },
{ "#siteid", "siteid" },
{ "#projecttype", "projecttype" },
{ "#aws_s3_key", "aws_s3_key" },
{ "#aws_s3_bucket", "aws_s3_bucket" }
},
ExpressionAttributeValues = new Dictionary<string, AttributeValue> {
{":v_aws_s3_bucket", new AttributeValue { S = "sampleBucket"}},
{":v_aws_s3_key", new AttributeValue { S = "92226"}}
},
ConsistentRead = true,
ProjectionExpression = projectExpression
};
response = amazonDynamoDbClient.Scan(request);
lastEvaluatedKey = response.LastEvaluatedKey;}while(lastEvaluatedKey!=null && lastEvaluatedKey.count()!=0)
I tried executing the above request with out using do-while loop and saved the ExclusiveStartKey for the next request it throws error the "The provided starting key is invalid: One or more parameter values were invalid: Null attribute value types must have the value of true".
any help on this issue will be helpful...
The error you're getting appears to be because you're setting ExclusiveStartKey on the request without setting any values for its parameters. Notice how you aren't updating request.ExclusiveStartKey after you get your response. Obviously, if you don't do that, the scan won't know where to pick up again when you hit your limit. See below.
AmazonDynamoDBClient amazonDynamoDbClient= new AmazonDynamoDBClient()
var filterExpression = "#aws_s3_bucket = :v_aws_s3_bucket and contains(#aws_s3_key,:v_aws_s3_key)";
var projectExpression = "#aws_s3_key,filename,#region,aws_s3_bucket,#projecttype,folder,#siteid,locationname,createdon,modifiedon";
ScanRequest request = new ScanRequest
{
TableName = "Test1",
FilterExpression = filterExpression,
ExpressionAttributeNames = new Dictionary<string, string>
{
{ "#region", "region" },
{ "#siteid", "siteid" },
{ "#projecttype", "projecttype" },
{ "#aws_s3_key", "aws_s3_key" },
{ "#aws_s3_bucket", "aws_s3_bucket" }
},
ExpressionAttributeValues = new Dictionary<string, AttributeValue> {
{":v_aws_s3_bucket", new AttributeValue { S = "sampleBucket"}},
{":v_aws_s3_key", new AttributeValue { S = "92226"}}
},
ConsistentRead = true,
ProjectionExpression = projectExpression
};
do
{
response = amazonDynamoDbClient.Scan(request);
request.ExclusiveStartKey = response.LastEvaluatedKey;
} while (response.lastEvaluatedKey.Count != 0);
i have searched for something similiar in stackoverflow and couldnt find anything which will give me some hint.
i have following code:
DATA val1 = new DATA();
val1.Name = "KeyValue";
val1.Value = "805373069";
DATA val2 = new DATA();
val2.Name = "Tel";
val2.Value = "0123456789";
DATA val3 = new DATA();
val3.Name = "TargetID";
val3.Value = "43301";
DATA val4 = new DATA();
val4.Name = "ServiceLevel";
val4.Value = "Y";
DATA val5 = new DATA();
val5.Name = "TypeId";
val5.Value = "13505";
DATA val6 = new DATA();
val6.Name = "DateTime";
val6.Value = System.DateTime.Now.ToString("ddMMyyyyHHmmssffftt");
DATA val7 = new DATA();
val7.Name = "DbDateTime";
val7.Value = System.DateTime.Now.ToString("ddMMyyyyHHmmssffftt");
and once all the objects are populated i put them in Single array.
i.e. to be used somewhere else
DATA[] array = {val1,val2,val3,val4,val5,val6,val7};
and Proxy class which i cant change is:
public partial class DATA {
private string nameField;
private string valueField;
public string Name {
get {
return this.nameField;
}
set {
this.nameField = value;
this.RaisePropertyChanged("Name");
}
}
public string Value {
get {
return this.valueField;
}
set {
this.valueField = value;
this.RaisePropertyChanged("Value");
}
}
Now what i have tried and failed to make it easier is used Dictionary and also jagged array and multi dimensional array which didnt worked as i hoped.
can someone give me hint of a better solution then having 7 different objects created, as this data is dynamic i have to do this runtime data population.
suggestions please?
You could just declare the objects in-line as part of the array declaration, if all you're trying to do is avoid having the variables:
DATA[] array = {
new DATA { Name = "something", Value = "something else" },
new DATA { Name = "something", Value = "something else" },
new DATA { Name = "something", Value = "something else" },
new DATA { Name = "something", Value = "something else" }
};
Anywhere that you have a variable, you can instead have the operation which created that variable. The order of operations will result in evaluating to the same thing. Where you'd need a variable is where you want to use the same instance of something multiple times, or the same value without having to re-calculate it.
Put all data in a dictionary if you want to make sure names must not be duplicated:
var data = new Dictionary<string, string>();
// fill dictionary:
data.Add("name1", /*value*/);
data.Add("name2", /*value*/);
data.Add("name3", /*value*/);
data.Add("name4", /*value*/);
Then convert it to array:
return data.Select(d => new Data(){ Name = d.Key, Value = d.Value}).ToArray();
Make sure you have included using System.Linq in top.
UPDATE:
As #LukeH suggested, You can simply use collection initializer like this:
var data = new Data[]
{
new Data(){ Name = "Sylvester", Value = /*value*/ },
new Data(){ Name = "Whiskers", Value = /*value*/ },
new Data(){ Name = "Sasha", Value = /*value*/ }
};
Which doesn't prevent duplicate names for Data type instances.
You can create extension method something like this to overcome the problem of assign properties values rather then duplication,
static class Extensions
{
public static void AddDataObject(this List<DATA> dataList, params string[] values)
{
dataList.Add(new DATA() { Name = values[0], Value = values[1] });
}
}
and passing that values as per given below,
List<DATA> dataList = new List<DATA>();
dataList.AddDataObject("KeyValue", "805373069");
dataList.AddDataObject("Tel", "0123456789");
Here in above example I used List instead of array, you can change according to your requirements
You could initialize an anonymous object and then convert to an array of data like this:
var data = new {
KeyValue="805373069",
Tel="0123456789",
TargetID="43301",
ServiceLevel="Y",
TypeId="13505",
DateTime=System.DateTime.Now.ToString("ddMMyyyyHHmmssffftt"),
DbDateTime=System.DateTime.Now.ToString("ddMMyyyyHHmmssffftt")
};
var array = data.GetType()
.GetProperties()
.Select(x=>new DATA{Name=x.Name,Value=(string)x.GetValue(data)})
.ToArray();
You could also do it like this:
var data = new {
KeyValue="805373069",
Tel="0123456789",
TargetID="43301",
ServiceLevel="Y",
TypeId="13505",
DateTime=System.DateTime.Now.ToString("ddMMyyyyHHmmssffftt"),
DbDateTime=System.DateTime.Now.ToString("ddMMyyyyHHmmssffftt")
};
var array=System.Web.Mvc.HtmlHelper.AnonymousObjectToHtmlAttributes(data)
.Select(x=>new DATA {Name=x.Key,Value=(string)x.Value})
.ToArray();
If you need to take an array of data and convert it back into a class object (not anonymous), you can do the first method, just in reverse as well. Or put extension methods on it to convert from/to your data array.
static class Extensions
{
public static DATA[] ToDataArray(this object data)
{
return data.GetType()
.GetProperties()
.Select(x=>new DATA{Name=x.Name,Value=(string)x.GetValue(data)})
.ToArray();
}
}
var data = new {
KeyValue="805373069",
Tel="0123456789",
TargetID="43301",
ServiceLevel="Y",
TypeId="13505",
DateTime=System.DateTime.Now.ToString("ddMMyyyyHHmmssffftt"),
DbDateTime=System.DateTime.Now.ToString("ddMMyyyyHHmmssffftt")
};
var array=data.ToDataArray();
However, David's answer is better.
I started out with Mongo client doing some nifty queries and aggretations.. but now that I want to use it in .NET/C#, I see that I can't simply run the query as text field..
Furthermore, after resorting to building an Aggregation Pipeline, and running the collection.Aggregate() function, I'm getting a result set, but I have no idea how to traverse it..
Can anyone help guide me here?
Here's my code:
var coll = db.GetCollection("animals");
var match = new BsonDocument {
{ "$match", new BsonDocument {{"category","cats"}} }
};
var group = new BsonDocument{
{
"$group", new BsonDocument{
{"_id", "$species"},
{"AvgWeight", new BsonDocument{{"$avg", "$weight"}}} }
}
};
var sort = new BsonDocument{{"$sort", new BsonDocument{{"AvgWeight", -1}}}};
var pipeline = new[] { match, group, sort };
var args = new AggregateArgs { Pipeline = pipeline };
var res = coll.Aggregate(args);
foreach (var obj in res)
{
// WHAT TO DO HERE??
}
Also, I should say that I'm a little rusty with C# / ASP.NET / MVC so any room for simplification would be much appreciated.
Your result is IEnumerable of BsonDocument, you can Serialize them to C# objects using the BSonSerializer. And this code snippet just writes them to your console, but you can see that you have typed objects
List<Average> returnValue = new List<Average>();
returnValue.AddRange(documents.Select(x=> BsonSerializer.Deserialize<Average>(x)));
foreach (var obj in returnValue)
{
Console.WriteLine("Species {0}, avg weight: {1}",returnValue._Id,returnValue.AvgWeight);
}
And then have a class called Average, where the property name match the names in the BSonDocument, if you want to rename then (because _Id is not so nice in c# terms concerning naming conventions), you can add a $project BsonDocument to your pipeline.
public class Average
{
public string _Id { get; set; }
public Double AvgWeight {get; set; }
}
$project sample (add this in your pipeline just before sort
var project = new BsonDocument
{
{
"$project",
new BsonDocument
{
{"_id", 0},
{"Species","$_id"},
{"AvgWeight", "$AvgWeight"},
}
}
};
I have a class named "IndexModel":
public class IndexModel
{
[ElasticProperty(Index= FieldIndexOption.NotAnalyzed, Store = true)]
public string ModelNumber{ get; set; }
}
following is how i setup the elastic client:
var uri = new Uri("http://localhost:9200");
var config = new ConnectionSettings(uri);
var client = new ElasticClient(config);
client.Map<IndexModel>(m => m.MapFromAttributes());
I can see the mapped result from response:
Request {
"indexmodel": {
"properties": {
"modelNumber": {
"type": "string",
"store": true,
"index": "not_analyzed"
},
}
}
}
and i have one index record for this type, the value of "ModelNumber" property is "test-123", and following is my query:
var result = client.Search<IndexModel>(s => s.Query(new TermQuery() { Field = Property.Path<IndexModel>(it => it.ModelNumber), Value = "test-123"}));
here is the final mapped request i got:
Method: POST,
Url: http://localhost:9200/_search,
Request: {
"query": {
"term": {
"modelNumber": {
"value": "test-123"
}
}
}
}
But i can not get the result, if i change the value of "ModelNumber" property to "test123", re-index it, and search it by keywords "test123", then it's works, so i think the analyzer still analyzed the "ModelNumber" property, can someone help me, thanks.
I had the same problem, the solution is first create the index then put the mapping and at last add your data.
Add Type Attribute to your model field
[ElasticProperty(OmitNorms = true, Index = FieldIndexOption.NotAnalyzed)]
var node = new Uri("http://192.168.0.56:9200/");
var settings = new ConnectionSettings(node, defaultIndex: "ticket");
var client = new ElasticClient(settings);
var createIndexResult = client.CreateIndex("ticket");
var mapResult = client.Map<TicketElastic>(c => c.MapFromAttributes().IgnoreConflicts().Type("TicketElastic").Indices("ticket"));