I'm looking to be able to reuse some of the transform expressions from indexes so I can perform identical transformations in my service layer when the document is already available.
For example, whether it's by a query or by transforming an existing document at the service layer, I want to produce a ViewModel object with this shape:
public class ClientBrief
{
public int Id { get; set; }
public string FullName { get; set; }
public string Email { get; set; }
// ellided
}
From this document model:
public class Client
{
public int Id { get; private set; }
public CompleteName Name { get; private set; }
public Dictionary<EmailAddressKey, EmailAddress> Emails { get; private set; }
// ellided
}
public class CompleteName
{
public string Title { get; set; }
public string GivenName { get; set; }
public string MiddleName { get; set; }
public string Initials { get; set; }
public string Surname { get; set; }
public string Suffix { get; set; }
public string FullName { get; set; }
}
public enum EmailAddressKey
{
EmailAddress1,
EmailAddress2,
EmailAddress3
}
public class EmailAddress
{
public string Address { get; set; }
public string Name { get; set; }
public string RoutingType { get; set; }
}
I have an expression to transform a full Client document to a ClientBrief view model:
static Expression<Func<IClientSideDatabase, Client, ClientBrief>> ClientBrief = (db, client) =>
new ClientBrief
{
Id = client.Id,
FullName = client.Name.FullName,
Email = client.Emails.Select(x => x.Value.Address).FirstOrDefault()
// ellided
};
This expression is then manipulated using an expression visitor so it can be used as the TransformResults property of an index (Client_Search) which, once it has been generated at application startup, has the following definition in Raven Studio:
Map:
docs.Clients.Select(client => new {
Query = new object[] {
client.Name.FullName,
client.Emails.SelectMany(x => x.Value.Address.Split(new char[] {
'#'
})) // ellided
}
})
(The Query field is analysed.)
Transform:
results.Select(result => new {
result = result,
client = Database.Load(result.Id.ToString())
}).Select(this0 => new {
Id = this0.client.__document_id,
FullName = this0.client.Name.FullName,
Email = DynamicEnumerable.FirstOrDefault(this0.client.Emails.Select(x => x.Value.Address))
})
However, the transformation expression used to create the index can then also be used in the service layer locally when I already have a Client document:
var brief = ClientBrief.Compile().Invoke(null, client);
It allows me to only have to have one piece of code that understands the mapping from Client to ClientBrief, whether that code is running in the database or the client app. It all seems to work ok, except the query results all have an Id of 0.
How can I get the Id property (integer) properly populated in the query?
I've read a number of similar questions here but none of the suggested answers seem to work. (Changing the Ids to strings from integers is not an option.)
I have a hard time following your sample fully, Really the best way to dig in to this would be with a failing self-contained unit test.
Nonetheless, let's see if I can pull out the important bits.
In the transform, you have two areas where you are working with the id:
...
client = Database.Load(result.Id.ToString())
...
Id = this0.client.__document_id,
...
The result.Id in the first line and the Id = in the second line are expected to be integers.
The Database.Load() expects a string document key and that is also what you see in __document_id.
The confusion comes from Raven's documentation, code, and examples all use the terms id and key interchangeably, but this is only true when you use string identifiers. When you use non-string identifiers, such as ints or guids, the id may be 123, but the document key is still clients/123.
So try changing your transform so it translates:
...
client = Database.Load("clients/" + result.Id)
...
Id = int.Parse(this0.client.__document_id.Split("/")[1]),
...
... or whatever the c# equivalent linq form would be.
Related
I am cassandra for custom logging my .netcore project, i am using CassandraCSharpDriver.
Problem:
I have created UDT for params in log, and added list of paramUDT in Log table as frozen.
But i am getting error: Non-frozen UDTs are not allowed inside collections. I don't know why ia m getting this error because i am using Frozen attribute on list i am using in Log Model.
logSession.Execute($"CREATE TYPE IF NOT EXISTS {options.Keyspaces.Log}.{nameof(LogParamsCUDT)} (Key text, ValueString text);");
Here is model:
public class Log
{
public int LoggingLevel { get; set; }
public Guid UserId { get; set; }
public string TimeZone { get; set; }
public string Text { get; set; }
[Frozen]
public IEnumerable<LogParamsCUDT> LogParams { get; set; }
}
Question where i am doing wrong, is my UDT script not correct or need to change in model.
Thanks in advance
I've tried using that model and Table.CreateIfNotExists ran successfully.
Here is the the code:
public class Program
{
public static void Main()
{
var cluster = Cluster.Builder().AddContactPoint("127.0.0.1").Build();
var session = cluster.Connect();
session.CreateKeyspaceIfNotExists("testks");
session.ChangeKeyspace("testks");
session.Execute($"CREATE TYPE IF NOT EXISTS testks.{nameof(LogParamsCUDT)} (Key text, ValueString text);");
session.UserDefinedTypes.Define(UdtMap.For<LogParamsCUDT>($"{nameof(LogParamsCUDT)}", "testks"));
var table = new Table<Log>(session);
table.CreateIfNotExists();
table.Insert(new Log
{
LoggingLevel = 1,
UserId = Guid.NewGuid(),
TimeZone = "123",
Text = "123",
LogParams = new List<LogParamsCUDT>
{
new LogParamsCUDT
{
Key = "123",
ValueString = "321"
}
}
}).Execute();
var result = table.First(l => l.Text == "123").Execute();
Console.WriteLine(JsonConvert.SerializeObject(result));
Console.ReadLine();
table.Where(l => l.Text == "123").Delete().Execute();
}
}
public class Log
{
public int LoggingLevel { get; set; }
public Guid UserId { get; set; }
public string TimeZone { get; set; }
[Cassandra.Mapping.Attributes.PartitionKey]
public string Text { get; set; }
[Frozen]
public IEnumerable<LogParamsCUDT> LogParams { get; set; }
}
public class LogParamsCUDT
{
public string Key { get; set; }
public string ValueString { get; set; }
}
Note that I had to add the PartitionKey attribute or else it wouldn't run.
Here is the CQL statement that it generated:
CREATE TABLE Log (
LoggingLevel int,
UserId uuid,
TimeZone text,
Text text,
LogParams frozen<list<"testks"."logparamscudt">>,
PRIMARY KEY (Text)
)
If I remove the Frozen attribute, then this error occurs: Cassandra.InvalidQueryException: 'Non-frozen collections are not allowed inside collections: list<testks.logparamscudt>'.
If your intention is to have a column like this LogParams frozen<list<"testks"."logparamscudt">> then the Frozen attribute will work. If instead you want only the UDT to be frozen, i.e., LogParams list<frozen<"testks"."logparamscudt">>, then AFAIK the Frozen attribute won't work and you can't rely on the driver to generate the CREATE statement for you.
All my testing was done against cassandra 3.0.18 using the latest C# driver (3.10.1).
We are using elastic search just for document search in our application so we don't have any one expert in it. I was able to use TermQuery, SimpleQueryStringQuery and MatchPhraseQuery successfully. But I found out in documentation that using From & Size for pagination is not good for production and Search After is recommended.
But my implementation return null. It is confusing for me what should be in <Project> parameter as shown in Nest API Object Initializer Syntax in docs here.
My code looks like this:
var request = new SearchRequest<ElasticSearchJsonObject._Source>
{
//Sort = new List<ISort>
//{
// new SortField { Field = Field<ElasticSearchJsonObject>(p=>)}
//},
SearchAfter = new List<object> {
},
Size = 20,
Query = query
};
Reality is I don't understand this. Over here ElasticSearchJsonObject._Source is the class to map returned results.
My documents are simple text documents and I only want documents sorted according to score so document Id is not relevant.
There was already a question like this on SO but I can't find it somehow.
Update
After looking at answer I updated my code and though query obtained does work. It return result in kibana but not in NEST.
This is the new updated code:
var request = new SearchRequest<ElasticSearchJsonObject.Rootobject>
{
Sort = new List<ISort>
{
new SortField { Field = "_id", Order = SortOrder.Descending}
},
SearchAfter = new List<object> {
"0fc3ccb625f5d95b973ce1462b9f7"
},
Size = 1,
Query = query
};
Over here I am using size=1 just for test as well as hard code _id value in SearchAfter.
The query generated by NEST is:
{
"size": 1,
"sort": [
{
"_id": {
"order": "desc"
}
}
],
"search_after": [
"0fc3ccb625f5d95b973ce1462b9f7"
],
"query": {
"match": {
"content": {
"query": "lahore",
"fuzziness": "AUTO",
"prefix_length": 3,
"max_expansions": 10
}
}
}
}
The response from the ES does say successful but no results are returned.
Results do return in Kibana
Query status is successful
But...
Total returned is 0 in NEST
Sort value is null in kibana I used TrackScores = true to solve this issue
Here is the debug information:
Valid NEST response built from a successful low level call on POST: /extract/_source/_search?typed_keys=true
# Audit trail of this API call:
- [1] HealthyResponse: Node: http://localhost:9200/ Took: 00:00:00.1002662
# Request:
{"size":1,"sort":[{"_id":{"order":"desc"}}],"search_after":["0fc3ccb625f5d95b973ce1462b9f7"],"query":{"match":{"content":{"query":"lahore","fuzziness":"AUTO","prefix_length":3,"max_expansions":10}}}}
# Response:
{"took":3,"timed_out":false,"_shards":{"total":5,"successful":5,"skipped":0,"failed":0},"hits":{"total":0,"max_score":null,"hits":[]}}
So please tell me where I am wrong and what can be the problem and how to solve it.
Update 2:
Code in Controller:
Connection String:
var node = new Uri("http://localhost:9200");
var settings = new ConnectionSettings(node);
settings.DisableDirectStreaming();
settings.DefaultIndex("extract");
var client = new ElasticClient(settings);
Query:
var query = (dynamic)null;
query = new MatchQuery
{
Field = "content",
Query = content,
Fuzziness = Fuzziness.Auto,
PrefixLength = 3,
MaxExpansions = 10
};
Query Builder
var request = new SearchRequest<ElasticSearchJsonObject.Rootobject>
{
Sort = new List<ISort>
{
new SortField { Field = "_id", Order = SortOrder.Descending}
},
SearchAfter = new List<object> {
documentid //sent as parameter
},
Size = 1, //for testing 1 other wise 10
TrackScores = true,
Query = query
};
JSON Query
I use this code to get query I posted above. This query is then passed to kibana with GET <my index name>/_Search and there it works
var stream = new System.IO.MemoryStream();
client.SourceSerializer.Serialize(request, stream);
var jsonQuery = System.Text.Encoding.UTF8.GetString(stream.ToArray());
ES Response
string responseJson = "";
ElasticSearchJsonObject.Rootobject response = new ElasticSearchJsonObject.Rootobject();
var res = client.Search<object>(request);
if (res.ApiCall.ResponseBodyInBytes != null)
{
responseJson = System.Text.Encoding.UTF8.GetString(res.ApiCall.ResponseBodyInBytes);
try
{
response = JsonConvert.DeserializeObject<ElasticSearchJsonObject.Rootobject>(responseJson);
}
catch (Exception)
{
var model1 = new LoginSignUpViewModel();
return PartialView("_NoResultPage", model1);
}
}
This is where things go wrong. Above debug information was captured from response
ElasticSearchJsonObject
Some how I think problem might be here somewhere? The class is generated by taking response from NEST in Search request.
using System;
using System.Collections.Generic;
using System.Linq;
using System.Web;
namespace ESAPI
{
public class ElasticSearchJsonObject
{
public class Rootobject
{
public int took { get; set; }
public bool timed_out { get; set; }
public _Shards _shards { get; set; }
public Hits hits { get; set; }
}
public class _Shards
{
public int total { get; set; }
public int successful { get; set; }
public int skipped { get; set; }
public int failed { get; set; }
}
public class Hits
{
public int total { get; set; }
public float max_score { get; set; }
public Hit[] hits { get; set; }
}
public class Hit
{
public string _index { get; set; }
public string _type { get; set; }
public string _id { get; set; }
public float _score { get; set; }
public _Source _source { get; set; }
}
public class _Source
{
public string content { get; set; }
public Meta meta { get; set; }
public File file { get; set; }
public Path path { get; set; }
}
public class Meta
{
public string title { get; set; }
public Raw raw { get; set; }
}
public class Raw
{
public string XParsedBy { get; set; }
public string Originator { get; set; }
public string dctitle { get; set; }
public string ContentEncoding { get; set; }
public string ContentTypeHint { get; set; }
public string resourceName { get; set; }
public string ProgId { get; set; }
public string title { get; set; }
public string ContentType { get; set; }
public string Generator { get; set; }
}
public class File
{
public string extension { get; set; }
public string content_type { get; set; }
public DateTime last_modified { get; set; }
public DateTime indexing_date { get; set; }
public int filesize { get; set; }
public string filename { get; set; }
public string url { get; set; }
}
public class Path
{
public string root { get; set; }
public string _virtual { get; set; }
public string real { get; set; }
}
}
}
I am sure this can be used to get response.
Please note that in case of simple search this code works:
so for this query below my code is working:
var request = new SearchRequest
{
From = 0,
Size = 20,
Query = query
};
Using from/size is not recommended for deep pagination because of the amount of documents that need to be fetched from all shards for a deep page, only to be discarded when finally returning an overall ordered result set. This operation is inherent to the distributed nature of Elasticsearch, and is common to many distributed systems in relation to deep pagination.
With search_after, you can paginate forward through documents in a stateless fashion and it requires
the documents returned from the first search response are sorted (documents are sorted by _score by default)
passing the values for the sort fields of the last document in the hits from one search request as the values for "search_after": [] for the next request.
In the Search After Usage documentation, a search request is made with sort on NumberOfCommits descending, then by Name descending. The values to use for each of these sort fields are passed in SearchAfter(...) and are the values of Project.First.NumberOfCommits and Project.First.Name properties, respectively. This tells Elasticsearch to return documents that have values for the sort fields that correspond to the sort constraints for each field, and relate to the values supplied in the request. For example, sort descending on NumberOfCommits with a supplied value of 775 means that Elasticsearch should only consider documents with a value less than 775 (and to do this for all sort fields and supplied values).
If you ever need to dig further into any NEST documentation, click the "EDIT" link on the page:
which will take you to the github repository of the documentation, with the original asciidoc markdown for the page:
Within that page will be a link back to the original NEST source code from which the asciidoc was generated. In this case, the original file is SearchAfterUsageTests.cs in the 6.x branch
UPDATE 2:
It seems to be a problem with the containing value in Format.
The Debugger shows me "{{MinValue: 6, MaxValue:44}}" for property Format.
When I change this to .Format = new MyFormat(6,44) it works fine.. so maybe its a problem with the web api 2 I'm using ... but anyway it should just push the source-object to the destination-object ;)
UPDATE:
I removed the originial post and added some real code here. the other classes were only dummies to explain my problem.
By the way: The Problem only occurrs when I add the "Format" Property of type object to the class. otherwise it works fine...
I tried to create an simple example to keep things easy ;) but here are parts of my real code
private void InitAutoMapper()
{
if (_mappingInitialized) //static init
return;
//will prevent Mapper to try to map constructor-parameters. With Copy-Constructors otherwise this would cause infiniteloops and stackoverflowexceptions.
Mapper.Configuration.DisableConstructorMapping();
//From Client to Server
...
Mapper.CreateMap<SDK.Model.Form.Field, Field>();
//From Server to Client
...
Mapper.CreateMap<Field, SDK.Model.Form.Field>();
...
//checks if the configuration was correct.
Mapper.AssertConfigurationIsValid();
_mappingInitialized = true;
}
and this is the call
public string CreateEntityField(SDK.Model.Form.Field field)
{
var mappedField = Mapper.Map<Field>(field);
...
}
my Field-class looks like this (source and destination classes look exactly the same. they are just separated in two different namespaces to have the possibility to add different properties in the future.
public class Field : IEntityRelatedEntity, IModificationTrackObject
{
public string Id { get; set; }
public string Name { get; set; }
public string DisplayName { get; set; }
public string Description { get; set; }
public DateTime CreatedOn { get; set; }
public DateTime ModifiedOn { get; set; }
public int Status { get; set; }
public int SubStatus { get; set; }
...many more fields
public FieldType Type { get; set; }
public object Format { get; set; }
public FieldRequirement Required { get; set; }
public Field()
{
Name = null;
Description = string.Empty;
DisplayName = null;
Type = FieldType.Text;
Required = FieldRequirement.Optional;
CreatedOn = new DateTime();
ModifiedOn = new DateTime();
}
public Field(Field copy)
{
Id = copy.Id;
Format = copy.Format;
...
}
}
and this is the exception I get (sorry parts of this exception are in german but it means that the source and destination-Type of Format are not the same)
exception =
{
Mapping types:\r\nJObject -> JObject
Newtonsoft.Json.Linq.JObject -> Newtonsoft.Json.Linq.JObject
Destination path:
Field.Format.Format
Source value:
{
"MinValue": "2",
"MaxValue": "100"
}
}
The Mapper should just copy the source to the destination for property "Format" and shouldn't care about whats in there...
one more thing: when i Ignore the Format-Property everything works fine, too.
Mapper.CreateMap<SDK.Model.Form.Field, Field>().ForMember(m => m.Format, opt => opt.Ignore());
I have a similar structure to the one below
Base class
public class BaseClass
{
public string Name { get; set; }
public string Address { get; set; }
public int Age { get; set; }
public Guid Guid { get; set; }
public string Hometown { get; set; }
}
Derived Class
public class DerivedClass : BaseClass
{
public List<DerivedClassDataItem> Data { get; set; }
}
Data class
public class DerivedClassDataItem
{
public string Datum1 { get; set; }
public string Datum2 { get; set; }
public string Datum3 { get; set; }
public string Datum4 { get; set; }
public int Datum5 { get; set; }
public DateTime Datum6 { get; set; }
}
What is the best practice to return specific set of info from the DerivedClass?
a potential set could be:
Name, Address, Guid and then a Data list that only contains Datum1 and Datum4
I could see anonymousTypes, Tuples or another set of class(es), all to be valid approaches.
My concern about creating new set of classs for the set returned is that the class(s) structure will be similar to the structure of the three mentioned above except it will have fewer selected members, which to me, does not sound ideal. (duplicate code and structure)
Using anonymousTypes was my initial solution to tackle this, something like
List<DerivedClass> list = new List<DerivedClass>();
var mySet = list.Select(d => new
{
Name = d.Name,
Address = d.Address,
.
.
.
.
.
Data = d.Data.Select(item => new
{
Datum1 = item.Datum1,
Datum4 = item.Datum4
})
});
but again, that was a headache for us to track through httpResponse and through out API calls.
Should I go with Tuple?
Any insights as to what is the best practice for doing this?
Edit
I am using this set of data to be a response returned by a API/GET call. I will send the set back using HttpRespose and then the framework will transform that into json
this is an actual method we have now
private void populateReturnFile()
{
var returnFileAnonymous = new
{
Vendor = this.Vendor,
OrganizationName = this.OrganizationName,
User = this.User,
Platform = this.Platform,
DictionaryType = this.DictionaryType,
UseCaseId = this.UseCaseId,
Data = this.Data.Select(d => new
{
MigrationTermId = d.MigrationTermId,
ImoLexicalCode = d.ImoLexicalCode
})
};
this.returnFile = returnFileAnonymous;
}
Then my GET will return the retunFile (this is a very simple method, i have remove irrelevant code)
[HttpGet]
public HttpResponseMessage Get(Guid migrationFileId)
{
ProblemList problemList = ProblemList.GetProblemList(migrationFileId);
return Request.CreateResponse(HttpStatusCode.OK, problemList.ReturnFile, new JsonMediaTypeFormatter());
}
If API calls is where you are using these classes, then I personally like to keep it simple and avoid complex inheritance hierarchy. Remember, simple code is good code.
I would make a separate class for each api request/response call. For very simple api calls (ajax requests for example) I like to use anonymous types, but for controllers that only handle API calls I like to create separate classes, organized in a nice folder structure.
Everyone has their "style" but as long as you strive for simplicity your code will be maintainable.
I've got a list of objects that look like this:
public class AllTablesView
{
public string ClientId { get; set; }
public string ClientName { get; set; }
public string ContractNumber { get; set; }
public string ContractDate { get; set; }
public string OrderId { get; set; }
public string CarColor { get; set; }
public string CarPlateNumber { get; set; }
public string InvoiceType { get; set; }
public string InvoiceValue { get; set; }
}
The actual class is bigger than this, having properties of at least 12 different entities, and they should look like this when converted:
public class Client
{
public string ClientId { get; set; }
public string ClientName { get; set; }
public IEnumerable<Contract> Contracts { get; set; }
}
public class Contract
{
public string ContractNumber { get; set; }
public string ContractDate { get; set; }
public IEnumerable<Order> Orders { get; set; }
}
public class Car
{
public string CarColor { get; set; }
public string CarPlateNumber { get; set; }
public IEnumerable<Invoice> Invoices { get; set; }
}
public class Invoice
{
public string InvoiceType { get; set; }
public string InvoiceValue { get; set; }
}
Like I said, this goes on a little bit, but the structure is the same. I'd like to know how I could extract the nested objects structure from the "AllTablesView" class.
I've attempted to use Linq's Group By but then I had to group by every entity and then loop through, it wasn't practical considering the amount of entities.
One note is that I have some logic when creating some of the objects, as an example: the invoice type is defined based on the client id.
EDIT:
One thing I've forgot and will probably impact the answer. The data that fills these properties come from the database, and is not grouped. Which means that the client id will repeat multiple times, but it is essentially the same client, so I cannot create it again. This is true for every entity I've exposed here, probably not so much for the invoice.
The goal is to create a json file which has this "tree" represented, without duplication.
An alternative approach to represent nested object structure is to use XML. So, for example, you could easily build up an XML string representing clients and their corresponding contracts like this:
XElement xeBody =
new XElement("Clients",
// for each client in the list of all tables
from client in lstAllTables
//...group by client id into client group
group client by client.ClientId into clientGroup
//... for each client int the group create a <Client> element
select new XElement("Client",
new XAttribute("ClientId", clientGroup.First().ClientId),
new XAttribute("ClientName", clientGroup.First().ClientName),
// for each contract in the current client group create a <Contract> element
from contract in clientGroup
select new XElement("Contract",
new XAttribute("ContractNumber", contract.ContractNumber),
new XAttribute("ContractDate", contract.ContractDate)
)
)
);
With this as input:
List<AllTablesView> lstAllTables = new List<AllTablesView>()
{
new AllTablesView() { ClientId = "1", ClientName = "Bob", ContractNumber = "BC1001", ContractDate = "2014-12-07" },
new AllTablesView() { ClientId = "1", ClientName = "Bob", ContractNumber = "BC1002", ContractDate = "2014-12-08" },
new AllTablesView() { ClientId = "1", ClientName = "Bob", ContractNumber = "BC1003", ContractDate = "2014-12-08" },
new AllTablesView() { ClientId = "2", ClientName = "Jim", ContractNumber = "AD1003", ContractDate = "2014-12-08" },
new AllTablesView() { ClientId = "2", ClientName = "Jim", ContractNumber = "AD1004", ContractDate = "2014-12-08" }
};
you get the following XML:
<Clients>
<Client ClientId="1" ClientName="Bob">
<Contract ContractNumber="BC1001" ContractDate="2014-12-07" />
<Contract ContractNumber="BC1002" ContractDate="2014-12-08" />
<Contract ContractNumber="BC1003" ContractDate="2014-12-08" />
</Client>
<Client ClientId="2" ClientName="Jim">
<Contract ContractNumber="AD1003" ContractDate="2014-12-08" />
<Contract ContractNumber="AD1004" ContractDate="2014-12-08" />
</Client>
</Clients>
You can fiddle with the XML generation code above to produce the desired xml structure, including the rest of the classes referred to in the OP and effectively avoiding any data duplication.
You could finally use Newtonsoft Json to serialize XML to Json.
I'm not aware of a way to automatically extract a class from another class. You might try adding extension methods instead that extract the information you need, for example:
public static class ExtensionMethods
{
public static Client GetClient(this AllTablesView view)
{
return new Client()
{
ClientID = view.ClientID,
ClientName = view.ClientName,
};
}
}
Then getting your classes out looks a little simpler:
Client client = myAllTableView.GetClient();
Likely not the answer you were hoping for, but beyond some code auto-generation tool, I don't see LINQ necessarily helping out here.
Edit: For grouping manually, you could try something like the following to check if it already exists:
Dictionary<int, Client> clients = new Dictionary<int, Client>();
foreach (AllTableView tableView in AllTableViewCollection)
{
if (clients.ContainsKey(tableView.ClientID) == false)
{
clients.Add(tableView.GetClient());
}
}
This is, of course, doing it yourself. Someone else may have a better idea for a way to automate this through LINQ.