Cosmos DB Indexed search problems - c#

I have a Cosmos DB with a collection with 134k documents and every document contains 2 string properties (named ChemicalName and EchNo) and 3 string collections (named CasNos, IndexNos and ReachNos).
Document JSON:
{
"id": "9cb924ad-a100-4fce-9dc9-30ec968de585",
"ChemicalName": "MyChemical",
"EchaNumber": 1,
"EcNo": "111-222-3",
"CasNos": [
"4444-55-6"
],
"IndexNos": [
"777-888-99-0"
],
"ReachNos": [
"01-0123456789-38-0000",
"01-0123456789-38-0001",
"01-0123456789-38-0002",
"01-0123456789-38-0003",
"01-0123456789-38-0004",
"01-0123456789-38-0006",
"01-0123456789-38-0007",
"01-0123456789-38-0008",
"01-0123456789-38-XXXX"
],
"IndexScrapeDate": "2018-01-19T12:30:20.0404174Z",
"ChemicalScrapeDate": null,
"_rid": "96VvAOgIVwABAAAAAAAAAA==",
"_self": "dbs/96VvAA==/colls/96VvAOgIVwA=/docs/96VvAOgIVwABAAAAAAAAAA==/",
"_etag": "\"00007e4d-0000-0000-0000-5a69b49b0000\"",
"_attachments": "attachments/",
"_ts": 1516876955
}
Collection have index definition:
{
"indexingMode": "consistent",
"automatic": true,
"includedPaths": [
{
"path": "/*",
"indexes": [
{
"kind": "Range",
"dataType": "Number",
"precision": -1
},
{
"kind": "Hash",
"dataType": "String",
"precision": 3
}
]
},
{
"path": "/EchaNumber/?",
"indexes": [
{
"kind": "Range",
"dataType": "Number",
"precision": -1
},
{
"kind": "Hash",
"dataType": "String",
"precision": 3
}
]
},
{
"path": "/ChemicalName/?",
"indexes": [
{
"kind": "Range",
"dataType": "String",
"precision": -1
},
{
"kind": "Range",
"dataType": "Number",
"precision": -1
}
]
},
{
"path": "/CasNos/[]/?",
"indexes": [
{
"kind": "Range",
"dataType": "String",
"precision": -1
},
{
"kind": "Range",
"dataType": "Number",
"precision": -1
}
]
},
{
"path": "/EcNo/?",
"indexes": [
{
"kind": "Range",
"dataType": "String",
"precision": -1
},
{
"kind": "Range",
"dataType": "Number",
"precision": -1
}
]
},
{
"path": "/IndexNos/[]/?",
"indexes": [
{
"kind": "Range",
"dataType": "String",
"precision": -1
},
{
"kind": "Range",
"dataType": "Number",
"precision": -1
}
]
},
{
"path": "/ReachNos/[]/?",
"indexes": [
{
"kind": "Range",
"dataType": "String",
"precision": -1
},
{
"kind": "Range",
"dataType": "Number",
"precision": -1
}
]
}
],
"excludedPaths": []
}
I implemented the code in Dotnet Core 2.0 and are using the SQL API.
When I search the collection I use the following SQL to find the ids of the docs I need:
SELECT VALUE c.id FROM IndexCollection c JOIN cas IN c.CasNos JOIN indx IN c.IndexNos WHERE CONTAINS(c.ChemicalName,"searchVal") OR STARTSWITH(cas,"searchVal") OR STARTSWITH(c.EcNo,"searchVal") OR STARTSWITH(indx,"searchVal")
I verified it by searching for a known partial match in CasNos string collection property.
I then needed to add search for ReachNos as well and added it to the search:
SELECT VALUE c.id FROM IndexCollection c JOIN cas IN c.CasNos JOIN indx IN c.IndexNos JOIN rch IN c.ReachNos WHERE CONTAINS(c.ChemicalName,"searchVal") OR STARTSWITH(cas,"searchVal") OR STARTSWITH(c.EcNo,"searchVal") OR STARTSWITH(indx,"searchVal") OR STARTSWITH(rch,"searchVal")
...But as soon as I add ReachNos to search it stops finding any result tho I use same searchval as above.
If I modify the SQL to only search the ReachNos property it will find results from that given I enter a known value from that property.
Can anyone pls. tell me why this is not working?
I tried all combinations of the first SQL and no matter what properties I searched for it worked fine until I added ReachNos property.
EDIT: I should add I have tested on local Cosmos DB Emulator and azure.
EDIT2: Added document JSON
/Thanx

Related

JSON Schema required properties in array are not validate [duplicate]

I want to make a schema of json file.It's for an array of products.
The json schema is similar as below:
{
"$schema": "http://json-schema.org/draft-04/schema#",
"title": "Product set",
"type": "array",
"items": {
"title": "Product",
"type": "object",
"properties": {
"id": {
"description": "The unique identifier for a product",
"type": "number"
},
"name": {
"type": "string"
},
"price": {
"type": "number",
"minimum": 0,
"exclusiveMinimum": true
},
"tags": {
"type": "array",
"items": {
"type": "string"
},
"minItems": 1,
"uniqueItems": true
},
"dimensions": {
"type": "object",
"properties": {
"length": {"type": "number"},
"width": {"type": "number"},
"height": {"type": "number"}
},
"required": ["length", "width", "height"]
},
"warehouseLocation": {
"description": "Coordinates of the warehouse with the product",
"$ref": "http://json-schema.org/geo"
}
},
"required": ["id", "name", "price"]
}
}
The array should at least one item in it. How can I define the minimum of the array?
Do I need to add the minimun defination?
To set the minimum # of item in an array, use the "minItems".
See:
https://datatracker.ietf.org/doc/html/draft-fge-json-schema-validation-00#section-5.3.3
and
http://jsonary.com/documentation/json-schema/?section=keywords/Array%20validation
{
"$schema": "http://json-schema.org/draft-04/schema#",
"title": "Product",
"description": "A product from Acme's catalog",
"type": "object",
"properties": {
...
"tags": {
"type": "array",
"items": {
"type": "string"
},
"minItems": 1,
"maxItems": 4,
"uniqueItems": true
}
},
"required": ["id", "name", "price"]
}
It looks like draft v4 permits what you are looking for. From http://json-schema.org/example1.html:
{
"$schema": "http://json-schema.org/draft-04/schema#",
"title": "Product",
"description": "A product from Acme's catalog",
"type": "object",
"properties": {
...
"tags": {
"type": "array",
"items": {
"type": "string"
},
"minItems": 1,
"uniqueItems": true
}
},
"required": ["id", "name", "price"]
}
Notice that the "tags" property is defined as an array, with a minimum number of items (1).
I suppose no, at least looking to working draft the minimum is applied only for numeric values, not arrays.
5.1. Validation keywords for numeric instances (number and integer)
...
5.1.3. minimum and exclusiveMinimum
So you should be good with min/maxItems for arrays.

How to check if any object in an array has a property in Azure Logic App

I am building a Azure Logic App which recieves vouchers array. I need to parse the body and stop execution if any voucher has the property canBeUsed to true. For achieving I want to use until instead of forEach because If I find any voucher with the property to true I want to stop iterating, I dont need to keep iterating.
In c# I would do this return response.Any(b => b.CanBeUsed);
The JSON received:
{
"vouchers": [
{
"amountPending": {
"currency": {
"code": "EUR",
"symbol": "€",
"id": 978
},
"amount": 1517
},
"amountPendingWithoutBonus": {
"currency": {
"code": "EUR",
"symbol": "€",
"id": 978
},
"amount": 1517
},
"canBeUsed": false,
"cannotBeUsedReasons": [],
"firstName": "jason",
"lastName": "denis",
"expirationDate": "2021-11-30T00:00:00",
"issuedDate": "2020-05-31T00:00:00",
"issuedReason": "Cancellation",
"voucherType": "Covid19",
"usages": []
},
{
"amountPending": {
"currency": {
"code": "EUR",
"symbol": "€",
"id": 123
},
"amount": 134
},
"amountPendingWithoutBonus": {
"currency": {
"code": "EUR",
"symbol": "€",
"id": 9718
},
"amount": 551
},
"canBeUsed": true,
"cannotBeUsedReasons": [],
"firstName": "paul",
"lastName": "frank",
"expirationDate": "2021-12-30T00:00:00",
"issuedDate": "2020-09-31T00:00:00",
"issuedReason": "Cancellation",
"voucherType": "Covid19",
"usages": []
}
]
}
What I have done:
I am stuck on accessing to the array inside the Until Customer Has Not Vouchers That Can Be Used.
I want to achieve this:
HasVouchersThatCanBeUsed = vouchers[i].canBeTrue;
You can use the Filter array action to filter the vouchers array by #item()?['canBeUsed'] equal to #true. Then you can check if the result array is empty.
You don't need the loop, unless you want to process the array items individually.

How JSON data can be converted to Avro schema in C#?

I don't want to use Microsoft.Hadoop.Avro.
Is there any way to convert JSON data directly to Avro schema in C# and asp.net core API.
Example: Json Data:
{"name":"John", "age":30, "car":null}
Avro Schema:
"name": "MyClass",
"type": "record",
"namespace": "com.acme.avro",
"fields": [
{
"name": "name",
"type": "string"
},
{
"name": "age",
"type": "int"
},
{
"name": "car",
"type": [
"string",
"null"
]
}
]
}```

The underlying connection was closed - webAPI ,WCF

I have a REST Service calling a WCF service. The method in WCf service returns data as expected. Here is the Json format of the c#object.
[
{
"$id": "1",
"children": [],
"id": 1,
"name": "1",
"owner": {
"userId": 1,
"username": "testuser",
"firstName": null,
"lastName": null
},
"parent": null,
"permissions": [],
"type": 0
}
]
When There is a complex object the WCf service is throwing an error "The underlying connection was closed: The connection was closed unexpectedly"
[
{
"$id": "1",
"Children": [
{
"$id": "2",
"Children": [],
"Id": 603268262,
"Name": "testfolder",
"Owner": {
"UserId": 555,
"Username": "testuser",
"FirstName": null,
"LastName": null
},
"Parent": {
"$ref": "1"
},
"Permissions": null,
"Type": 0
}
],
"Id": 1,
"Name": "555",
"Owner": {
"UserId": 555,
"Username": "testuser",
"FirstName": null,
"LastName": null
},
"Parent": null,
"Permissions": null,
"Type": 0
}
]
I have tried most answers on this SO question. Why do I get the error only when a complex object is returned?
you need to check data contracts ,take a look at this : http://msdn.microsoft.com/en-us/library/aa347850.aspx
Found the answer just after posting the question. I did notice that there is a object holding a reference to a object in the Json
"Parent": {
"$ref": "1"
}
This is preventing the object from serialized. I did a deep copy and that fixed the issue

How do i parse json? [duplicate]

This question already has answers here:
Parsing JSON using Json.net
(5 answers)
Closed 3 years ago.
I am using Newtonsoft Json library to parse json but i don't know how to use it. I parsed the string using JObject. When i output the value of JObject instance in immediate window i get this :-
json
{
"data": [
{
"id": "id",
"from": {
"name": "name",
"id": "someotherid"
},
"name": "pic",
"description": "desc",
"link": "linktosite",
"privacy": "everyone",
"count": 1,
"type": "normal",
"created_time": "2010-10-22T14:54:32+0000",
"updated_time": "2010-10-22T14:55:41+0000"
},
{
"id": "id2",
"from": {
"name": "name",
"id": "someotherid"
},
"name": "Profile Pictures",
"link": "link",
"privacy": "everyone",
"count": 6,
"type": "profile",
"created_time": "2010-10-12T14:27:58+0000",
"updated_time": "2011-01-01T18:38:14+0000"
},
{
"id": "id3",
"from": {
"name": "name",
"id": "829741958"
},
"name": "T",
"link": "link",
"privacy": "everyone",
"count": 5,
"type": "normal",
"created_time": "2010-05-01T03:03:39+0000",
"updated_time": "2010-05-01T03:19:13+0000",
"comments": {
"data": [
{
"id": "id",
"from": null,
"message": "message",
"created_time": "2010-08-28T18:27:10+0000",
"likes": 1
}
]
}
}
],
"paging": {
"previous": "paginglink",
"next": "otherpaginglink"
}
}
Count: 2
Type: Object
What should i do further to have the values from this jobject?
I personally prefer the JavaScriptSerializer for use with JSON in the .NET environment. By default it will return a Dictionary result, but can be used to parse in to a custom object (or you could make use of the dynamic datatype).
Some other posts on SO with JSON & JavaScript Serializer as topic

Categories