Scroll for all records ElasticSearch C# Nest - c#

Query to scroll at the matching records from the query
this is the query of nest in C# to get all the records from nest C# find many questions which can solve it by using different method linq method but i want to do this this way any suggestions help would be appreciated
string[] MERCHANTNO = MerchantId.Split(",");
var mustClause = new List<QueryContainer>();
var filterClause = new List<QueryContainer>();
var filters = new List<QueryContainer>();
filters.Add(new TermsQuery{
Field = new Field("MERCHANTNO"),
Terms = MERCHANTNO,
});
Logger.LogInformation(clsName, funcName, "Filter Clause is:", filters);
var SearchRequest = new SearchRequest<AcquirerDTO>(idxName) {
Size = 10000,
SearchType = Elasticsearch.Net.SearchType.QueryThenFetch,
Scroll = "5m",
Query = new BoolQuery { Must = filters }
};
var searchResponse = await _elasticClient.SearchAsync<AcquirerDTO>( SearchRequest );

The code for Scroll all the Records you have in ElasticSearch is
Filter
filters.Add(new TermsQuery {
Field = new Field("MERCHANTNO"), >>> Value needs to be searched
Terms = MERCHANTNO,
});
Date Range Filter
filterClause.Add(new DateRangeQuery {
Boost = 1.1,
Field = new Field("filedate"),
GreaterThanOrEqualTo = DateMath.Anchored(yesterday),
LessThanOrEqualTo = DateMath.Anchored(Today),
Format = "yyyy-MM-dd",
TimeZone = "+01:00"
});
Search Request for scrolling
var SearchRequest = new SearchRequest<AcquirerDTO>(idxName) {
From = 0,
Scroll = scrollTimeoutMinutes,
Size = scrollPageSize,
Query = new BoolQuery
{
Must = filters,
Filter = filterClause
}
};
var searchResponse = await _elasticClient.SearchAsync<AcquirerDTO>(SearchRequest);
if (searchResponse.ApiCall.ResponseBodyInBytes != null) {
var requestJson = System.Text.Encoding.UTF8.GetString(searchResponse.ApiCall.RequestBodyInBytes);
var JsonFormatQuery = JsonConvert.SerializeObject(JsonConvert.DeserializeObject(requestJson), Formatting.Indented);
}
This is the code for Scrolling all the results in kibana
List<AcquirerDTO> results = new List<AcquirerDTO>();
if (searchResponse.Documents.Any())
results.AddRange(searchResponse.Documents);
string scrollid = searchResponse.ScrollId;
bool isScrollSetHasData = true;
while (isScrollSetHasData)
{
ISearchResponse<AcquirerDTO> loopingResponse = _elasticClient.Scroll<AcquirerDTO>(scrollTimeoutMinutes, scrollid);
if (loopingResponse.IsValid)
{
results.AddRange(loopingResponse.Documents);
scrollid = loopingResponse.ScrollId;
}
isScrollSetHasData = loopingResponse.Documents.Any();
}
var records = results;

Related

How to query the Google Analytics Data API (GA4) using an event_category and an event_label?

I'm using .NET 5 and the nuget package Google.Analytics.Data.V1Beta to query Google Analytics. But I need to query events using the event_category and the event_label and I get this error: "Field customEvent:event_category is not a valid dimension". This is my code:
var click = new FilterExpression
{
Filter = new Filter
{
FieldName = "ga:event_category",
StringFilter = new Filter.Types.StringFilter
{
CaseSensitive = false,
MatchType = Filter.Types.StringFilter.Types.MatchType.Exact,
Value = "click"
}
},
};
var request = new RunReportRequest
{
Property = $"properties/{_settings.GoogleAnalytics.PropertyId}",
Dimensions = {new Dimension {Name = "eventName"}, new Dimension {Name = "customEvent:event_category" } },
DimensionFilter = click,
Metrics = {new Metric {Name = "eventCount"}},
DateRanges =
{
new DateRange
{StartDate = startDate.ToString("yyyy-MM-dd"), EndDate = endDate.ToString("yyyy-MM-dd")}
}
};
var response = await _analyticsClient.GetClient().RunReportAsync(request);
How do I pass event_category and event_label to the query?

MS Dynamics - QueryExpression with ConditionOperator.In result in no result but works with ConditionOperator.Equal

I'm trying to make a query in order to retrieve all record containing one of the text in a string list.
QueryExpression query = new QueryExpression("account")
{
ColumnSet = new ColumnSet("primarycontactid", "new_text"),
NoLock = true,
Criteria =
{
Conditions =
{
new ConditionExpression()
{
AttributeName = "new_text",
Operator = ConditionOperator.In,
Values = { texts.ToArray() }
}
}
}
};
This code execute without issue, but don't return any record.
I also tried the following code, which resulted in the return of multiple record.
QueryExpression query = new QueryExpression("account")
{
ColumnSet = new ColumnSet("primarycontactid", "new_text"),
NoLock = true,
Criteria =
{
Conditions =
{
new ConditionExpression()
{
AttributeName = "new_text",
Operator = ConditionOperator.Equal,
Values = { texts.ToArray()[0] }
}
}
}
};
I also tried, without error, but with no return.
QueryExpression query = new QueryExpression("account")
{
ColumnSet = new ColumnSet("primarycontactid", "new_text"),
NoLock = true,
Criteria =
{
Conditions =
{
new ConditionExpression()
{
AttributeName = "new_text",
Operator = ConditionOperator.Equal,
Values = { texts.ToArray() }
}
}
}
};
How can I do in order to query with a list of values ?
The below syntax should work.
QueryExpression q = new QueryExpression("account");
q.Criteria.AddCondition("new_text", ConditionOperator.In, new object[] { "value1", "value2" });
Alternate version:
q.Criteria.AddCondition("new_text", ConditionOperator.In, "value1", "value2");
Read more
Here is one more approach.
Make your texts as list and then convert it to comma separated string and use this string in your condition
IList texts = new List{"1","2","testing"}; string joined = string.Join(",", texts);
Then you can use it as below
QueryExpression query = new QueryExpression("account") { ColumnSet = new ColumnSet("primarycontactid", "new_text"), NoLock = false, Criteria = { Conditions = { new ConditionExpression() { AttributeName = "new_text", Operator = ConditionOperator.In, Values = { joined } } } } };

c# Nest and Elasticsearch Aggregations

Does anyone know how to do multiple aggregations with nest?
I have found quite a few examples unfortunately none of them work.
Here's what I have:
Vehicles fields = new Vehicles();
//create a terms query
var query = new TermsQuery
{
IsVerbatim = true,
Field = "VehicleOwnerId",
Terms = new string[] { 25 },
};
var aggregations = new Dictionary<string, IAggregationContainer>
{
{ "years", new AggregationContainer
{
Terms = new TermsAggregation(nameof(fields.Year))
{
Field = new Field(nameof(fields.Year))
}
}
}
//,
//{ "makes", new AggregationContainer
// {
// Terms = new TermsAggregation("Make")
// {
// Field = new Field(nameof(fields.Make))
// }
// }
//}
};
//create the search request
var searchRequest = new SearchRequest
{
Query = query,
From = 0,
Size = 100,
Aggregations = aggregations
};
var result = client.SearchAsync<InventoryLiveView>(searchRequest).Result;
var years = result.Aggregations.Terms("years");
Dictionary<string, long> yearCounts = new Dictionary<string, long>();
foreach (var item in years.Buckets)
{
yearCounts.Add(item.Key, item.DocCount ?? 0);
}
If I just execute the code like this it works. Years returns the aggregates as expected. If I try to add another field (like the one commented out above) it fails and I get zero records.
How can I get multiple aggregates in one query? I see examples of it all over, but none of the examples I've tried seem to work and most seem to be outdated (including some in the Nest documentation).
I have also tried this approach which is pretty close to the documentation.
//create the search request
var searchRequest = new SearchRequest
{
Query = query,
From = 0,
Size = 100,
//Aggregations = aggregations
Aggregations = new AggregationDictionary
{
{
"childAgg", new ChildrenAggregation("childAgg", typeof(Vehicles ))
{
Aggregations = new AggregationDictionary
{
{"years", new TermsAggregation(nameof(fields.VehicleYear))},
{"makes", new TermsAggregation(nameof(fields.VehicleMakeName))},
{"models", new TermsAggregation(nameof(fields.VehicleModelName))},
}
}
}
}
};
var result = client.SearchAsync<Vehicles>(searchRequest).Result;
This just produces a null reference exception.
I guess I'll never have too worry about getting to proud as a programmer :)
It's too often that the solution to the problem makes me feel stupid when it reveals itself.
So my issue was that the field I was trying to use in the aggregation was text and couldn't be used. I switched everything to the ID fields and multiple aggregations work as expected.
So this version of the code works like a champ:
Vehicle fields = new Vehicle ();
//create a terms query
var query = new TermsQuery
{
IsVerbatim = true,
Field = "VehicleOwnerId",
Terms = new string[] { "30" },
};
string[] Fields = new[]
{
nameof(fields.Year),
nameof(fields.MakeId),
nameof(fields.ModelId)
};
var aggregations = new Dictionary<string, IAggregationContainer>();
foreach (string sField in Fields)
{
var termsAggregation = new TermsAggregation(sField)
{
Field = sField
};
aggregations.Add(sField, new AggregationContainer { Terms = termsAggregation });
}
//create the search request
var searchRequest = new SearchRequest
{
Query = query,
From = 0,
Size = 10,
Aggregations = aggregations
};
var result = client.SearchAsync<InventoryLiveView>(searchRequest).Result;
var years = result.Aggregations.Terms(nameof(fields.Year));
Dictionary<string, long> yearCounts = new Dictionary<string, long>();
foreach (var item in years.Buckets)
{
yearCounts.Add(item.Key, item.DocCount ?? 0);
}
The exact error from elasticsearch, which I saw using postman was:
Fielddata is disabled on text fields by default. Set fielddata=true on [MakeName] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead.
Here is my example using SearchDescriptors. My only problem is how to serialize returned results into a proper Key Value list. Is Looping through a fields list the best way to return results.
SearchDescriptor<Advert> agghDescriptor = new SearchDescriptor<Advert>();
agghDescriptor.Aggregations(ag => ag.Terms("make", a => a.Field(f => f.Make)) &&
ag.Terms("region", a => a.Field(f => f.Region)) &&
ag.Terms("city", a => a.Field(f => f.City)) &&
ag.Terms("category", a => a.Field(f => f.Category)) &&
ag.Terms("application", a => a.Field(f => f.Application)) &&
ag.Terms("portalId", a => a.Field(f => f.PortalId)) &&
ag.Terms("isActiveAuctionAdvert", a => a.Field(f => f.IsActiveAuctionAdvert)) &&
ag.Terms("isBargainAccount", a => a.Field(f => f.IsBargainAccount)) &&
ag.Terms("condition", a => a.Field(f => f.Condition))
);
agghDescriptor.Size(0);
var json2 = _client.RequestResponseSerializer.SerializeToString(agghDescriptor);
var aggregationResult = _client.Search<Advert>(agghDescriptor);
List<string> fields = new List<string>();
fields.Add("make");
fields.Add("category");
fields.Add("region");
List<Aggregation> aggregations = new List<Aggregation>();
foreach (var field in fields)
{
var aggrs = aggregationResult.Aggregations.Terms(field);
List<AggregateItem> aggregateItems = new List<AggregateItem>();
foreach (var item in aggrs.Buckets)
{
aggregateItems.Add(new AggregateItem()
{
Count = item.DocCount ?? 0,
Key = item.Key
});
}
aggregations.Add(new Aggregation()
{
Name = field,
Aggregates = aggregateItems
});
}

AWS-DynomoDB Two parameter for filter the result using c#

How can we achieve to get the result from dynomodb using two parameter? I am using below code it's does not work.
var _request = new QueryRequest
{
TableName = "Attendence",
KeyConditionExpression = "Roster_EmpID = :Roster_EmpID and Roster_CreatedDateTime between :v_start and :v_end",
ExpressionAttributeValues = new Dictionary<string, AttributeValue> {
{":Roster_EmpID", new AttributeValue { S = result.empId }}
,{":v_start", new AttributeValue { S = result.fromDate.ToString(AWSSDKUtils.ISO8601DateFormat) }}
,{":v_end", new AttributeValue { S = result.toDate.ToString(AWSSDKUtils.ISO8601DateFormat) }}
},
IndexName = "Roster_EmpID-index"
};
var _response = await _client.QueryAsync(_request);
But it's not return the result, Please help me to get the result. I already wasted 2 days, still not able to find the answer.
I think you may looking for FilterExpression
KeyConditionExpression = "Roster_EmpID = :Roster_EmpID ",
FilterExpression = "Roster_CreatedDateTime between :v_start and :v_end",

How to add/update milestones in a feature using rally rest API and C#?

I am not able to add or update milestones field for the Features in the Rally. If anyone having the code available using C# to update the same, please share with me. I am searching and doing from last one week with no luck.
When I am trying to add/Update milestones in the Features. I am getting the error as "Could not read: Could not read referenced object null". My code is as follows:-
public DynamicJsonObject UpdateFeaturesbyName(string fea, string bFun)
{
//getting list of Feature.
Request feat = new Request("PortfolioItem/Feature");
feat.Query = new Query("Name", Query.Operator.Equals, fea);
QueryResult TCSResults = restApi.Query(feat);
foreach (var res in TCSResults.Results)
{
var steps = res["Milestones"];
Request tsteps = new Request(steps);
QueryResult tstepsResults = restApi.Query(tsteps);
foreach (var item in tstepsResults.Results)
{
}
if (res.Name == fea)
{
var targetFeature = TCSResults.Results.FirstOrDefault();
DynamicJsonObject toUpdate = new DynamicJsonObject();
//toUpdate["Milestones"] = "";
// CreateResult createResult = restApi.Create(steps._ref, toUpdate);
// String contentRef = steps._ref;
//String contentRef = createResult._ref;
string[] value = null;
string AccCri = string.Empty;
if (!string.IsNullOrWhiteSpace(bFun))
{
value = bFun.Split(new string[] { "\r\n", "\n" }, StringSplitOptions.None);
foreach (string item in value)
{
//if (string.IsNullOrWhiteSpace(AccCri))
// AccCri = item;
//else
// AccCri = AccCri + "<br/>" + item;
if (!string.IsNullOrWhiteSpace(item))
{
//Query for Milestone.
Request ms = new Request("Milestone");
ms.Fetch = new List<string>() { "Name", "ObjectID" };
ms.Query = new Query("Name", Query.Operator.Equals, item);
QueryResult msResults = restApi.Query(ms);
var targetMLResult = msResults.Results.FirstOrDefault();
long MLOID = targetMLResult["ObjectID"];
DynamicJsonObject tarML = restApi.GetByReference("Milestone", MLOID, "Name", "_ref", "DisplayColor");
DynamicJsonObject targetML = new DynamicJsonObject();
targetML["Name"] = tarML["Name"];
//targetML["_ref"] = tarML["_ref"];
targetML["_ref"] = "/milestone/" + Convert.ToString(MLOID);
targetML["DisplayColor"] = tarML["DisplayColor"];
// Grab collection of existing Milestones.
var existingMilestones = targetFeature["Milestones"];
long targetOID = targetFeature["ObjectID"];
// Milestones collection on object is expected to be a System.Collections.ArrayList.
var targetMLArray = existingMilestones;
var tagList2 = targetMLArray["_tagsNameArray"];
tagList2.Add(targetML);//
//targetMLArray.Add(targetML);
targetMLArray["_tagsNameArray"] = tagList2;
toUpdate["Milestones"] = targetMLArray;
OperationResult updateResult = restApi.Update(res._ref, toUpdate);
bool resp = updateResult.Success;
}
}
}
//toUpdate["c_AcceptanceCriteria"] = AccCri;
//OperationResult updateResult = restApi.Update(res._ref, toUpdate);
}
}
var features = TCSResults.Results.Where(p => p.Name == fea).FirstOrDefault();
var featuresref = features._ref;
return features;
}
Now that v3.1.1 of the toolkit has been released you can use the AddToCollection method to do this.
Otherwise, you can still always just update the full collection. The value should be an arraylist of objects with _ref properties.
Check out this example (which adds tasks to defects, but should be very similar to what you're doing): https://github.com/RallyCommunity/rally-dot-net-rest-apps/blob/master/UpdateTaskCollectionOnDefect/addTaskOnDefect.cs

Categories