We Keep Coding

Best way to send Customer and Order data from SQL server to MongoDB along with hyperlinks

Problem Statement:
I need to get Customer and all their corresponding Orders stored in the following format in MongoDB.
Current data resides in 2 different tables - Customers and Orders in MSSQL database.
Expected Collection in MongoDb:
{
"_data": [
{
"customer": {
"customerID": "1240014902244000006",
"firstName": "Jerry",
"surname": "Galoski",
"orders": [
{
"_data": {
"orderId": "1240014912244000016",
"suburb": "Delhi",
"postcode": "110051",
"country": "India"
},
"_links": [
{
"rel": "self",
"href": "/customer/1240014902244000006/orders",
"action": "GET"
},
{
"rel": "customer",
"href": "/customer/1240014902244000006",
"action": "GET"
}
]
},
{
"_data": {
"orderId": "1240014912244000024",
"suburb": "Delhi",
"postcode": "110051",
"country": "India"
},
"_links": [
{
"rel": "customer",
"href": "/customer/1240014902244000006/orders",
"action": "GET"
}
]
}
]
},
"_links": [
{
"rel": "self",
"href": "/customer/1240014902244000006",
"action": "GET"
},
{
"rel": "orders",
"href": "/customer/1240014902244000006/orders",
"action": "GET"
}
]
}
]
}
Steps taken to achieve this:
First step is to collate all orders data, which I'm doing using the below SQL query:
SELECT CustomerId, AllOrderIds =
STUFF(
(
SELECT DISTINCT ', ' + CAST(e2.OrderId AS VARCHAR(MAX))
FROM CustomerOrderRelation e2
WHERE e1.CustomerId = e2.CustomerId
FOR XML PATH('')
), 1,1,''
) FROM CustomerOrderRelation cor;
Then I use the above subquery in the customers sql query to get the complete output as shown.
Then in the C# code, I will process the Customer records one by one, to generate the hyperlinks for each.
I need to generate hyperlinks for Order records also.
Once I have the hyperlinks generated in C#, I will call InsertMany command (MongoDB) to insert data. As a result, I will have 2 collections created in MongoDb - one each for Customer and Order.
Finally, I'll be using MongoDb's MapReduce functionality to merge 2 collections to get the data.
db.customers.mapReduce(mapCustomers, reduce, {"out": {"reduce": "customerId"}});
db.orders.mapReduce(mapOrders, reduce, {"out": {"reduce": "customerId"}});
db.customerAndOrders.find().pretty();
I need to do the merge because, I am not finding a clever way to do following operations:
Get customers and their consolidated orders in one single entity
Generate Hyperlinks for customers and all their orders
Push this data into MongoDb with customer details and their order details
Suggestion required:
Is there a better way to accomplish the above?

How to customize word template on c# for docs generate

I have a word template with .docx format,
my question:
Can I get a list of tag names from the rich text content control
that I have declared in the template like inside table and other else? and how?
(this is the example of template)
Can I call a parameter from the return response database using a
string? and how to retrieve deeper level data like this? (example are below)
{
"id": "293",
"user": "315",
"userNavigation": {
"id": "314",
"name": "insomnia"
},
"department": [
{
"id": "2",
"name": "Tech"
},
{
"id": "1",
"name": "Bio"
},
],
}
I've used two libraries
OpenXml
TemplateEngine.Docx: https://bitbucket.org/unit6ru/templateengine/src/master/
I do not use third party services because they are not paid.

Handle singular and plural search terms in Azure Cognitive Search

We're using Azure Cognitive Search as our search engine for searching for images. The analyzer is Lucene standard and when a user searches for "scottish landscapes" some of our users claim that their image is missing. They will then have to add the keyword "landscapes" in their images so that the search engine can find them.
Changing the analyzer to "en-lucene" or "en-microsoft" only seemed to have way smaller search results, which we didn't like for our users.
Azure Cognitive Search does not seem to distinguish singular and plural words. To resolve the issue, I created a dictionary in the database, used inflection and tried manipulating the search terms:
foreach (var term in terms)
{
if (ps.IsSingular(term))
{
// check with db
var singular = noun.GetSingularWord(term);
if (!string.IsNullOrEmpty(singular))
{
var plural = ps.Pluralize(term);
keywords = keywords + " " + plural;
}
}
else
{
// check with db
var plural = noun.GetPluralWord(term);
if (!string.IsNullOrEmpty(plural))
{
var singular = ps.Singularize(term);
keywords = keywords + " " + singular;
}
}
}
My solution is not 100% ideal but it would be nicer if Azure Cognitive Search can distinguish singular and plural words.
UPDATE:
Custom Analyzers may be the answer to my problem, I just need to find the right token filters.
UPDATE:
Below is my custom analyzer. It removes html constructs, apostrophes, stopwords and converts them to lowercase. The tokenizer is MicrosoftLanguageStemmingTokenizer and it reduces the words to its root words so it's apt for plural to singular scenario (searching for "landscapes" returns "landscapes" and "landscape")
"analyzers": [
{
"name": "p4m_custom_analyzer",
"#odata.type": "#Microsoft.Azure.Search.CustomAnalyzer",
"charFilters": [
"html_strip",
"remove_apostrophe"
],
"tokenizer": "custom_tokenizer",
"tokenFilters": [
"lowercase",
"remove_stopwords"
]
}
],
"charFilters": [
{
"name": "remove_apostrophe",
"#odata.type":"#Microsoft.Azure.Search.MappingCharFilter",
"mappings": ["'=>"]
}
],
"tokenizers": [
{
"name": "custom_tokenizer",
"#odata.type":"#Microsoft.Azure.Search.MicrosoftLanguageStemmingTokenizer",
"isSearchTokenizer": "false"
}
],
"tokenFilters": [
{
"name": "remove_stopwords",
"#odata.type": "#Microsoft.Azure.Search.StopwordsTokenFilter"
}
]
I have yet to figure out the other way around. If the user searches for "apple" it should return "apple" and "apples".

Both en.lucene and en.microsoft should have helped with this, you shouldn't need to manually expand inflections on your side. I'm surprised to hear you see less recall with them. Generally speaking I would expect higher recall with those than the standard analyzer. Do you by any chance have multiple searchable fields with different analyzers? That could interfere. Otherwise, it would be great to see a specific case (a query/document pair along with the index definition) to investigate further.
As a quick test, I used this small index definition:
{
"name": "inflections",
"fields": [
{
"name": "id",
"type": "Edm.String",
"searchable": false,
"filterable": true,
"retrievable": true,
"sortable": false,
"facetable": false,
"key": true
},
{
"name": "en_ms",
"type": "Edm.String",
"searchable": true,
"filterable": false,
"retrievable": true,
"sortable": false,
"facetable": false,
"key": false,
"analyzer": "en.microsoft"
}
]
}
These docs:
{
"id": "1",
"en_ms": "example with scottish landscape as part of the sentence"
},
{
"id": "2",
"en_ms": "this doc has one apple word"
},
{
"id": "3",
"en_ms": "this doc has two apples in it"
}
For this search search=landscapes I see these results:
{
"value": [
{
"#search.score": 0.9631388,
"id": "1",
"en_ms": "example with scottish landscape as part of the sentence"
}
]
}
And for search=apple I see:
{
"value": [
{
"#search.score": 0.51188517,
"id": "3",
"en_ms": "this doc has two apples in it"
},
{
"#search.score": 0.46152657,
"id": "2",
"en_ms": "this doc has one apple word"
}
]
}

Elasticsearch terms query problem for "text" or "keyword" type fields

"hits" : [
{
"id": 1,
"sampleArrayData": ["x"]
},
{
"id": 2,
"sampleArrayData": ["y"]
},
{
"id": 3,
"sampleArrayData": ["z"]
},
{
"id": 4,
"sampleArrayData": ["x", "y", "z"]
},
{
"id": 5,
"sampleArrayData": ["z", "w"]
}
]
It's a sample data of index.
I want to search this index by sampleArrayData field which includes one or more values I will give dynamically.
For example if want to search this index by ["x","y"] parameters. I must get data which includes "x", or "y" or both "x", "y". (First three records from this index).
We could do this by using terms query on the old elastic search versions like below.
{
"query": {
"bool": {
"must": [
{
"terms": {
"sampleArrayData": [ "x", "y"]
}
}
]
}
}
}
But we can not use terms query for "text" or "keyword" type fields on the elasticsearch current versions.
How can I dynamically search this index with unknown number of parameters on the C# NEST library?

MongoDB, C#. Update multiple array elements with different conditions

There is an entity with custom fields that can change their value. The fields are stored as an array.
public class SomeEntity :...
{
...
public List<CustomField> FieldList { get; set; } = new List<CustomField>();
...
}
I use code like this to update one field:
var filterBuilder = Builders<SomeEntity>.Filter;
var updateFilter =
filterBuilder.Eq(e => e.EntityId, entityToUpdate.EntityId)
& filterBuilder.ElemMatch(e => e.FieldList, f => f.FieldId == updateField.FieldId);
var updateDef = Builders<SomeEntity>.Update.Set(e => e.FieldList[-1].Value, updateField.Value);
var updateRes = await GetEntityCollection().UpdateOneAsync(updateFilter, updateDef);
Sometimes it becomes necessary to update the values of several fields in the database at the same time, but I cannot find a way to do this in one action.
Is it possible in MongoDB to first search for a document using a filter, and then update/delete several elements of its array by their Id's (changes for each element may be different) in one action?
Is it possible to do the same for several documents (for example, remove fields with certain identifiers from all objects matching the filter)?
An example of an object before the change:
{
...
"fieldList": [
{
"id": "3bf2c235-82c3-40e4-91dc-46dc4c1ed177",
"type": 0,
"value": 10
},
{
"id": "5909dabd-fe8f-4edb-a642-c052e23082d8",
"type": 1,
"value": "some value"
},
{
"id": "66805403-d508-4b99-82f3-fa2ed828c19e",
"type": 3,
"value": "2019-08-01T12:00:00"
}
]
}
An example of an object after modification (only one document is updated):
{
...
"fieldList": [
{
"id": "3bf2c235-82c3-40e4-91dc-46dc4c1ed177",
"type": 0,
"value": 500
},
{
"id": "5909dabd-fe8f-4edb-a642-c052e23082d8",
"type": 1,
"value": "new value"
},
{
"id": "66805403-d508-4b99-82f3-fa2ed828c19e",
"type": 3,
"value": "2020-09-10T10:00:00"
}
]
}
An example of an object after a massive change (the document matched the filter, by analogy, all documents that would fit the filter should be updated - delete two fields, add a default value for some field):
{
...
"fieldList": [
{
"id": "3bf2c235-82c3-40e4-91dc-46dc4c1ed177",
"type": 0,
"value": 500,
"defaultValue": 150
}
]
}