We Keep Coding

DynamoDb converting Attribute Values to Types

Is there a more efficient way of converting dynamo db data into concrete types? For example, when I query the data everything is in:
List<Dictionary<string, AttributeValue>>
Is it possible to easily convert the type without having to loop through each item and doing this all manually?
For example I am doing:
return items.Select(item => new Connection
{
ConnectionId = Guid.Parse(item["connectionId"].S),
ClientId = item["clientId"].S,
ProviderId = item["providerId"].S,
Scopes = item["scopes"].SS.ToArray(),
CredentialsId = item["credentialsId"].S,
Evidences = ToEvidences(item["consentEvidences"].L)
})
.ToList();
This then returns a list of my type Connection however I am explicitly mapping each field. Is there an easier way or a helper library that can do the mapping?

I think you'll have luck with the higher-level .NET Document model. It presents more natural data types.
https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/DotNetSDKMidLevel.html

The easiest way I have found is to use the Document.FromAttributeMap function to convert it to a Document object and then convert it again to the .NET type using the DynamoDBContext.FromDocument method as shown below.
public async Task<IEnumerable<WeatherForecast>> GetAll(string cityName)
{
var queryRequest = new QueryRequest()
{
TableName = nameof(WeatherForecast),
KeyConditionExpression = "CityName = :cityName",
ExpressionAttributeValues = new Dictionary<string, AttributeValue>()
{
{":cityName", new AttributeValue(cityName)},
}
};
var response = await _dynamoDbClient.QueryAsync(queryRequest);
return response.Items.Select(a =>
{
var doc = Document.FromAttributeMap(a);
return _dynamoDbContext.FromDocument<WeatherForecast>(doc);
});
}

C# Create Deedle DataFrame from JSON response

I'm having a bit of trouble loading a JSON response from this request into a Deedle DataFrame:https://sampleserver6.arcgisonline.com/arcgis/rest/services/Earthquakes_Since1970/FeatureServer/0/query?where=OBJECTID%3C10&returnGeometry=false&f=json
In the JSON, what I'm interested are the features. More specifically, for each feature there are attributes - I want essentially just a collection of those attributes to load into a DataFrame. In this particular case, there is only one attribute "name" so my expectation is that the resulting DataFrame would have a column "name" with the values shown.
I've tried using json2csharp and creating my own class, but the result either doesn't have the column header/values or the values are missing. I'm not really sure what I'm doing wrong or if I'm even approaching this the right way. My understanding from the Deedle documentation is that it should be possible to create a DataFrame from a collection of objects: https://bluemountaincapital.github.io/Deedle/csharpframe.html#Creating-and-loading-data-frames. Certainly, using the Enumerable example listed on the page works as expected.
Here is the pertinent section of my code:
string url = "https://sampleserver6.arcgisonline.com/arcgis/rest/services/Earthquakes_Since1970/FeatureServer/0/query?";
WebClient wc = new WebClient();
wc.QueryString.Add("where", "OBJECTID<10");
wc.QueryString.Add("returnGeometry", "false");
wc.QueryString.Add("f", "json");
var data = wc.UploadValues(url, "POST", wc.QueryString);
var responseString = UnicodeEncoding.UTF8.GetString(data);
JObject o = JObject.Parse(responseString);
dynamic x = JsonConvert.DeserializeObject(responseString);
var testObjList = new List<dynamic>();
foreach (dynamic element in x.features)
{
testObjList.Add(new myClass { name = element.attributes.name});
Console.WriteLine($"{element.attributes.name}");
}
var dfObjects = Frame.FromRecords(testObjList);
dfObjects.Print();
var df = Frame.FromRecords(test);
df.Print(); // No headers or values shown
where myClass is just this:
public class myClass{
public string name { get; set; }
}
Any help/pointers would be much appreciated!

The Frame.FromRecords operation relies on static type information to figure out what properties the class has. In your case, you define the list of objects as List<dynamic> - this is compiled as Object and so Deedle does not see any members.
To fix this, all you need to do is to define the type as a list of myClass objects:
var testObjList = new List<myClass>();
A more compact approach using an anonymous type would work too:
var testObjList =
((IEnumerable<dynamic>)x.features).Select(element =>
new { name = element.attributes.name });
var dfObjects = Frame.FromRecords(testObjList);

Bulk collection inside the right Index path in ElasticSearch using NEST in a .NET Core application

I am trying to bulk a collection of elements inside an index of ElasticSearch using NEST inside a .NET Core application.
Currently what I have is working, and the elements are saved, but Is not saved where I try to do
My client creation:
protected ElasticClient GetClient()
{
var node = new Uri("http://localhost:9200/");
var settings = new ConnectionSettings(node)
.DefaultIndex("TestIndex")
.PrettyJson(true);
return new ElasticClient(settings);
}
Here is how I create the descriptor for bulk all the data
protected BulkDescriptor GenerateBulkDescriptor<T>(IEnumerable<T> elements, string indexName) where T: class, IIndexable
{
var bulkIndexer = new BulkDescriptor();
foreach (var element in elements)
bulkIndexer.Index<T>(i => i
.Document(element)
.Id(element.Id)
.Index(indexName));
return bulkIndexer;
}
Finally, once I have this, here is how I index the data
var descriptor = GenerateBulkDescriptor(indexedElements, "indexed_elements");
var response = GetClient().Bulk(descriptor);
But, If I see how It's stored in the Elastic index using this, that is what I have:
How can I know if is created under TestIndex index? Because as far as I can see, there is just one index created
Thank you a lot in advance

When defining the index operations on the BulkDescriptor, you are explicitly setting the index to use for each operation
foreach (var element in elements)
bulkIndexer.Index<T>(i => i
.Document(element)
.Id(element.Id)
.Index(indexName));
where indexName is "indexed_elements". This is why all documents are indexed into this index and you do not see any in "TestIndex".
The Bulk API allows multiple operations to be defined, which may include indexing documents into different indices. When the index is specified directly on an operation, that will be the index used. If all index operations on a Bulk API call are to take place against the same index, you can omit the index on each operation and instead, specify the index to use on the Bulk API call directly
var defaultIndex = "default_index";
var pool = new SingleNodeConnectionPool(new Uri("http://localhost:9200"));
var settings = new ConnectionSettings(pool)
.DefaultIndex(defaultIndex);
var client = new ElasticClient(settings);
var people = new []
{
new Person { Id = 1, Name = "Paul" },
new Person { Id = 2, Name = "John" },
new Person { Id = 3, Name = "George" },
new Person { Id = 4, Name = "Ringo" },
};
var bulkResponse = client.Bulk(b => b
.Index("people")
.IndexMany(people)
);
which sends the following request
POST http://localhost:9200/people/_bulk
{"index":{"_id":"1","_type":"person"}}
{"id":1,"name":"Paul"}
{"index":{"_id":"2","_type":"person"}}
{"id":2,"name":"John"}
{"index":{"_id":"3","_type":"person"}}
{"id":3,"name":"George"}
{"index":{"_id":"4","_type":"person"}}
{"id":4,"name":"Ringo"}
Note that the URI is /people/bulk and that each JSON object representing an operation does not contain an "_index".
If you omit the .Index() on Bulk API call, it will use the DefaultIndex configured on ConnectionSettings:
var bulkResponse = client.Bulk(b => b
.IndexMany(people)
);
which yields
POST http://localhost:9200/_bulk
{"index":{"_id":"1","_index":"default_index","_type":"person"}}
{"id":1,"name":"Paul"}
{"index":{"_id":"2","_index":"default_index","_type":"person"}}
{"id":2,"name":"John"}
{"index":{"_id":"3","_index":"default_index","_type":"person"}}
{"id":3,"name":"George"}
{"index":{"_id":"4","_index":"default_index","_type":"person"}}
{"id":4,"name":"Ringo"}
You can also specify a default index to use for a given POCO type on ConnectionSettings with DefaultMappingFor<T>(), where T is your POCO type.

After som tests and attemps, I have found a solution.
First of all, it was a problem with the index configured, once I set it in lower case, the index was working fine indexing data inside.
Then, I had the problem of index data in a specific "path" inside the same index, finalyy I found the Type solution from NEST, taking also advantage of the DefaultMappingFor suggested by Russ in the previous answer.
Client definition:
var node = new Uri(_elasticSearchConfiguration.Node);
var settings = new ConnectionSettings(node)
.DefaultMappingFor<IndexedElement>(m => m
.IndexName(_elasticSearchConfiguration.Index)
.TypeName(nameof(IndexedElement).ToLower()))
.PrettyJson(true)
.DisableDirectStreaming();
var client = new ElasticClient(settings);
Then, the BulkDescriptior creation:
var bulkIndexer = new BulkDescriptor();
foreach (var element in elements)
bulkIndexer.Index<IndexedElement>(i => i
.Document(element)
.Type(nameof(IndexedElement).ToLower()))
.Id(element.Id)
);
And finally, data bulk:
client.Bulk(bulkIndexer);
Now, If I perform a call to the index, I can see this
{
"testindex": {
"aliases": {},
"mappings": {
"indexedelement": {
[...]
}
Thank you Russ for your help and for who have had a look to the post.
UPDATE
Finally, it seems that the unique problem was regarding the default index, that it must be lowercase, so, specify the type with the name of the POCO itself is not neccesary, like #RussCam has truly detected in comments above. After changing thedefault index to lowercase, all the different possibilities worked fine.
Thank you all again

How to use Solrnet to search multiple collection? C#

Q1. How to use Solrnet to search multiple collection?
Q2. I created a method to add data to Solr,
And if i want to dynamic assign sechma add data to Solr,
how to modify it?
public void SolrFeeder(SchemaFieldList DataList)
{
var solrFacility = new SolrNetFacility(SolrServer);
var container = new WindsorContainer();
container.AddFacility("solr", solrFacility);
var solr = container.Resolve<ISolrOperations<SchemaField>>();
foreach (var item in DataList.SchemaFieldList)
{
solr.Add(item);
}
solr.Commit();
}

The standard syntax for searching across collections is to provide the name of the collections in the query - i.e. if you're querying collection1, you can still append a parameter named collection which contains a list of the collections you want to search, collection=collection1,collection2,collection3.
You can use the syntax for "Additional Parameters" in SolrNet to add custom arguments to a query:
ISolrOperations<Product> solr = ...
var products = solr.Query(SolrQuery.All, new QueryOptions {
ExtraParams = new Dictionary<string, string> {
{"collection", "collection1,collection2,collection3"}
}
});

LINQ and creating NON anonymous return values

I think I understand returning records of an anonymous type from But in this I want to create NEW CatalogEntries, and set them from the values selected. (context is a Devart LinqConnect database context, which lets me grab a view).
My solution works, but it seems clumsy. I want to do this in one from statement.
var query = from it in context.Viewbostons
select it;
foreach (GPLContext.Viewboston item in query)
{
CatalogEntry card = new CatalogEntry();
card.idx = item.Idx;
card.product = item.Product;
card.size = (long)item.SizeBytes;
card.date = item.Date.ToString();
card.type = item.Type;
card.classification = item.Classification;
card.distributor = item.Distributor;
card.egplDate = item.EgplDate.ToString();
card.classificationVal = (int)item.ClassificationInt;
card.handling = item.Handling;
card.creator = item.Creator;
card.datum = item.Datum;
card.elevation = (int)item.ElevationFt;
card.description = item.Description;
card.dirLocation = item.DoLocation;
card.bbox = item.Bbox;
card.uniqID = item.UniqId;
values.Add(card);
}
CatalogResults response = new CatalogResults();
I just tried this:
var query2 = from item in context.Viewbostons
select new CatalogResults
{ item.Idx,
item.Product,
(long)item.SizeBytes,
item.Date.ToString(),
item.Type,
item.Classification,
item.Distributor,
item.EgplDate.ToString(),
(int)item.ClassificationInt,
item.Handling,
item.Creator,
item.Datum,
(int)item.ElevationFt,
item.Description,
item.DoLocation,
item.Bbox,
item.UniqId
};
But I get the following error:
Error 79 Cannot initialize type 'CatalogService.CatalogResults' with a
collection initializer because it does not implement
'System.Collections.IEnumerable' C:\Users\ysg4206\Documents\Visual
Studio
2010\Projects\CatalogService\CatalogService\CatalogService.svc.cs 91 25 CatalogService
I should tell you what the definition of the CatalogResults is that I want to return:
[DataContract]
public class CatalogResults
{
CatalogEntry[] _results;
[DataMember]
public CatalogEntry[] results
{
get { return _results; }
set { _results = value; }
}
}
My mind is dull today, apologies to all. You are being helpful. The end result is going to be serialized by WCF to a JSON structure, I need the array wrapped in a object with some information about size, etc.

Since .NET 3.0 you can use object initializer like shown below:
var catalogResults = new CatalogResults
{
results = context.Viewbostons
.Select(it => new CatalogEntry
{
idx = it.Idx,
product = it.Product,
...
})
.ToArray()
};
So if this is only one place where you are using CatalogEntry property setters - make all properties read-only so CatalogEntry will be immutable.
MSDN, Object initializer:
Object initializers let you assign values to any accessible fields or properties of an
object at creation time without having to explicitly invoke a constructor.

The trick here is to create a IQueryable, and then take the FirstOrDefault() value as your response (if you want a single response) or ToArray() (if you want an array). The error you are getting (Error 79 Cannot initialize type 'CatalogService.CatalogResults' with a collection initializer because it does not implement 'System.Collections.IEnumerable') is because you're trying to create an IEnumerable within the CatalogEntry object (by referencing the item variable).
var response = (from item in context.Viewbostons
select new CatalogEntry()
{
idx = item.Idx,
product = item.Product,
size = (long)item.SizeBytes,
...
}).ToArray();

You don't have to create anonymous types in a Linq select. You can specify your real type.
var query = context.Viewbostons.Select( it =>
new CatalogEntry
{
idx = it.idx,
... etc
});

This should work:
var query = from it in context.Viewbostons
select new CatalogEntry()
{
// ...
};