We Keep Coding

Scroll for all records ElasticSearch C# Nest

Query to scroll at the matching records from the query
this is the query of nest in C# to get all the records from nest C# find many questions which can solve it by using different method linq method but i want to do this this way any suggestions help would be appreciated
string[] MERCHANTNO = MerchantId.Split(",");
var mustClause = new List<QueryContainer>();
var filterClause = new List<QueryContainer>();
var filters = new List<QueryContainer>();
filters.Add(new TermsQuery{
Field = new Field("MERCHANTNO"),
Terms = MERCHANTNO,
});
Logger.LogInformation(clsName, funcName, "Filter Clause is:", filters);
var SearchRequest = new SearchRequest<AcquirerDTO>(idxName) {
Size = 10000,
SearchType = Elasticsearch.Net.SearchType.QueryThenFetch,
Scroll = "5m",
Query = new BoolQuery { Must = filters }
};
var searchResponse = await _elasticClient.SearchAsync<AcquirerDTO>( SearchRequest );

The code for Scroll all the Records you have in ElasticSearch is
Filter
filters.Add(new TermsQuery {
Field = new Field("MERCHANTNO"), >>> Value needs to be searched
Terms = MERCHANTNO,
});
Date Range Filter
filterClause.Add(new DateRangeQuery {
Boost = 1.1,
Field = new Field("filedate"),
GreaterThanOrEqualTo = DateMath.Anchored(yesterday),
LessThanOrEqualTo = DateMath.Anchored(Today),
Format = "yyyy-MM-dd",
TimeZone = "+01:00"
});
Search Request for scrolling
var SearchRequest = new SearchRequest<AcquirerDTO>(idxName) {
From = 0,
Scroll = scrollTimeoutMinutes,
Size = scrollPageSize,
Query = new BoolQuery
{
Must = filters,
Filter = filterClause
}
};
var searchResponse = await _elasticClient.SearchAsync<AcquirerDTO>(SearchRequest);
if (searchResponse.ApiCall.ResponseBodyInBytes != null) {
var requestJson = System.Text.Encoding.UTF8.GetString(searchResponse.ApiCall.RequestBodyInBytes);
var JsonFormatQuery = JsonConvert.SerializeObject(JsonConvert.DeserializeObject(requestJson), Formatting.Indented);
}
This is the code for Scrolling all the results in kibana
List<AcquirerDTO> results = new List<AcquirerDTO>();
if (searchResponse.Documents.Any())
results.AddRange(searchResponse.Documents);
string scrollid = searchResponse.ScrollId;
bool isScrollSetHasData = true;
while (isScrollSetHasData)
{
ISearchResponse<AcquirerDTO> loopingResponse = _elasticClient.Scroll<AcquirerDTO>(scrollTimeoutMinutes, scrollid);
if (loopingResponse.IsValid)
{
results.AddRange(loopingResponse.Documents);
scrollid = loopingResponse.ScrollId;
}
isScrollSetHasData = loopingResponse.Documents.Any();
}
var records = results;

Nest aggregation not working correctly

I have a use case where I need to do aggregation on multiple columns using C#.
I am using NEST libraries for this and I am facing the following issue
Query C# :
var searchRequest = new SearchRequest
{
SearchType = SearchType.Count,
Filter = filter,
Aggregations = new Dictionary<string, IAggregationContainer>
{
{ "a", new AggregationContainer
{
ExtendedStats = new ExtendedStatsAggregator()
{
Field = "a"
}
}
},
{ "b", new AggregationContainer
{
ExtendedStats = new ExtendedStatsAggregator()
{
Field = "b"
}
}
}
}
};
When I receive response from NEST, however I am getting only result for one aggregation. I am looking at SearchResult.Agg dictionary but it has only one entry for one aggregation field instead of two.
Let me know if I am missing soemthing or is it some issue with NEST libraries

if you are using term aggregation then you need to use aggregation with filter.
var qryRes1 = client.Search<object>(x => x
.Aggregations(ag => ag
.Filter("filter", (flt => flt
.Filter(f =>
{
FilterContainer filter = null;
filter &= f.Query(qr => qr.Term(wl => wl.OnField("a").Value("the value you need to filter for field a")));
return filter;
})
.Aggregations(agr => agr
.Terms("b", tr =>
{
TermsAggregationDescriptor<object> trmAggDescriptor = new TermsAggregationDescriptor<object>();
trmAggDescriptor.Field("b");
return trmAggDescriptor;
}))))
));
var terms = qryRes1.Aggs.Filter("filter").Terms("b");

RavenDB Collection "in" Collection query

I need to preform a query that check if a collection is in given collection, just like the regular in operation but for collections.
class Post
{
public string[] Tags {get;set;}
}
session.Queury<Post>.Where(x=>x.Tags.in(new[]{".net","c#","RavenDB"})).ToList();
so if i have in my DB:
new Post{Tags= new[]{"C#",".net"}};
it will be returned
but if i have:
new Post{Tags= new[]{"C#",".net","SQLServer"}};
it will not be returned.
Update:
what i am trying to do is this:
session.Query<Post>()
.Where(x => x.Tags.All(y => y.In(new[] { "C#", ".net", "RavenDB" })))
.ToList();
but i got System.NotSupportedException.

I manage to find a solution:
static void Main(string[] args)
{
var sessionStore = new EmbeddableDocumentStore
{
RunInMemory = true,
UseEmbeddedHttpServer = true,
Conventions =
{
DefaultQueryingConsistency = ConsistencyOptions.AlwaysWaitForNonStaleResultsAsOfLastWrite
}
};
sessionStore.Initialize();
using (var session = sessionStore.OpenSession())
{
var allTags = new[] {"C#", ".net", "RavenDB", "Linux", "Mac"};
var tagsCollection = new[] {"C#", ".net", "RavenDB"};
var complementTagsCollection = allTags.Except(tagsCollection).ToList();
session.Store(new Post
{
Tags = new List<string>{"C#",".net"}
});
session.SaveChanges();
// Posts where all their tags are in tagsCollection
var result = session.Query<Post>().Where(x => !x.Tags.In(complementTagsCollection)).ToList();
}
}

The way IN works, it matches ANY of them.
If you want to match all you have to do a separate check for each.

Creating multiple results from Linq query

I have a fairly unique situation, I have never needed to this before anyways. I have a Linq query that returns data from a database using EF4.1. I want to create multiple similar (same signature) anonymous (or even named if necessary) results from each query result.
Here's the code i'm using now:
var data = getMyData().Select(x =>
new
{
GoalName = x.GoalType.Name,
Start = x.StartDate,
End = x.EndDate,
x.StartValue,
x.CheckIns
}).ToList();
var r1 = data.Select(x =>
new
{
title = x.GoalName,
start = x.Start.ToString(),
end = x.End.ToString(),
className = "hidden",
type = "goal"
});
var r2 = data.Select(x =>
new
{
title = string.Format("Start: {0:0.##}", x.StartValue),
start = x.Start.ToString(),
end = x.Start.ToString(),
className = "",
type = ""
});
var r3 = data.Select(x =>
new
{
title = "End",
start = x.End.ToString(),
end = x.End.ToString(),
className = "",
type = ""
});
var r4 = data.SelectMany(x => x.CheckIns)
.Select(y =>
new
{
title = y.CheckInValue.Value.ToString(),
start = y.CheckInDateTime.ToString(),
end = y.CheckInDateTime.ToString(),
className = "",
type = ""
});
var result = r1.Union(r2).Union(r3).Union(r4);
Now maybe this is as good a way as any, but I can't help feeling that i'm missing something.
Is there a better solution?

What you have is actually OK I think.
But StevenzNPaul's suggestion not that bad, here's how you can use the let keyword to store the different projections, then select the results individually (for brevity, I did not project all the fields, but you get the point):
var query = from x in data
let result1 = new {title = x.GoalName, start = x.Start}
let result2 = new {title = string.Format("Start: {0:0.##}", x.StartValue), start = x.Start}
let result3 = new {title = "End", start = x.End}
let checkins = x.CheckIns.Select(checkin => new { title = "...", start = checkin.Start })
from result in new[] { result1, result2, result3 }.Concat(checkins)
select result;
Obviously, whether this is better is a matter of preference. Also, this will result in a different ordering, which may or may not be a problem for you.

You can create an iterator using yield which also has the advantage of being evaluated lazily (doesn't require the ToList()). I created a typed class Result to hold the query results
private IEnumerable<Result> PerformQuery()
{
var results= getMyData().Select(x => new {GoalName = x.GoalType.Name,
Start = x.StartDate, End = x.EndDate, x.StartValue, x.CheckIns});
foreach (var result in results)
{
yield return new Result() { Title = result.GoalName, Start = result.Start.ToString(), End = result.End.ToString(), ClassName = "Hidden", Type = "Goal" };
yield return new Result() { Title = String.Format("Start: {0:0.##}",result.StartValue), Start = result.Start.ToString(), End = result.Start.ToString() }
yield return new Result() { Title = "End", Start = result.End.ToString(), End = result.End.ToString() };
foreach (var checkIn in result.CheckIns)
yield return new Result() { Title = checkIn.CheckInValue.Value.ToString(), Start = checkIn.CheckInDateTime.ToString(), End = checkIn.CheckInDateTime.ToString() };
}
}

try using let keyword it will work for you.

How can I write a LINQ to SQL query to update tags?

I have an image site where users can tag photos much like you can tag a question on Stackoverflow.
I have the following tables:
Images [ID, URL, etc]
Tags [ID, TagName]
ImageTag [TagID, ImageID]
I want to write a method with the signature:
public void UpdateImageTags(int imageId, IEnumerable<string> currentTags)
This method will do the following:
Create any new Tags in currentTags that don't already exist in the Tags table.
Get the old ImageTag's for an image.
Delete any ImageTag's that no longer exist in the currentTags
Add any ImageTag's that are new between the currentTags and oldTags.
Here is my attempt at that method:
public void UpdateImageTags(int imageId, IEnumerable<string> currentTags)
{
using (var db = new ImagesDataContext())
{
var oldTags = db.ImageTags.Where(it => it.ImageId == imageId).Select(it => it.Tag.TagName);
var added = currentTags.Except(oldTags);
var removed = oldTags.Except(currentTags);
// Add any new tags that need created
foreach (var tag in added)
{
if (!db.Tags.Any(t => t.TagName == tag))
{
db.Tags.InsertOnSubmit(new Tag { TagName = tag });
}
}
db.SubmitChanges();
// Delete any ImageTags that need deleted.
var deletedImageTags = db.ImageTags.Where(it => removed.Contains(it.Tag.TagName));
db.ImageTags.DeleteAllOnSubmit(deletedImageTags);
// Add any ImageTags that need added.
var addedImageTags = db.Tags.Where(t => added.Contains(t.TagName)).Select(t => new ImageTag { ImageId = imageId, TagId = t.TagId });
db.ImageTags.InsertAllOnSubmit(addedImageTags);
db.SubmitChanges();
}
}
However, this fails on the line:
db.ImageTags.DeleteAllOnSubmit(deletedImageTags);
With the error:
Local sequence cannot be used in LINQ to SQL implementations of query
operators except the Contains operator.
Is there an easier way I can handle the operation of adding new tags, deleting old ImageTags, adding new ImageTags in LINQ to SQL?

Seems like this would be easiest
public void UpdateImageTags(int imageId, IEnumerable<string> currentTags)
{
using (var db = new ImagesDataContext())
{
var image = db.Images.Where(it => it.ImageId == imageId).First()
image.Tags.Clear();
foreach(string s in currentTags)
{
image.Tags.Add(new Tag() { TagName = s});
}
db.SubmitChanges();
}
}
This might have to be modified slightly for LinqtoSQL. EF is what i have been using most recently. Also this is dependent on Lazy loading being enabled. If it is not, you will have to force the include of the image tags.

Here is a helper method to deal with many-to-many relationships:
public static void UpdateReferences<FK, FKV>(
this EntitySet<FK> refs,
Expression<Func<FK, FKV>> fkexpr,
IEnumerable<FKV> values)
where FK : class
where FKV : class
{
Func<FK, FKV> fkvalue = fkexpr.Compile();
var fkmaker = MakeMaker(fkexpr);
var fkdelete = MakeDeleter(fkexpr);
var fks = refs.Select(fkvalue).ToList();
var added = values.Except(fks);
var removed = fks.Except(values);
foreach (var add in added)
{
refs.Add(fkmaker(add));
}
foreach (var r in removed)
{
var res = refs.Single(x => fkvalue(x) == r);
refs.Remove(res);
fkdelete(res);
}
}
static Func<FKV, FK> MakeMaker<FKV, FK>(Expression<Func<FK, FKV>> fkexpr)
{
var me = fkexpr.Body as MemberExpression;
var par = Expression.Parameter(typeof(FKV), "fkv");
var maker = Expression.Lambda(
Expression.MemberInit(Expression.New(typeof(FK)),
Expression.Bind(me.Member, par)), par);
var cmaker = maker.Compile() as Func<FKV, FK>;
return cmaker;
}
static Action<FK> MakeDeleter<FK, FKV>(Expression<Func<FK, FKV>> fkexpr)
{
var me = fkexpr.Body as MemberExpression;
var pi = me.Member as PropertyInfo;
var assoc = Attribute.GetCustomAttribute(pi, typeof(AssociationAttribute))
as AssociationAttribute;
if (assoc == null || !assoc.DeleteOnNull)
{
throw new ArgumentException("DeleteOnNull must be set to true");
}
var par = Expression.Parameter(typeof(FK), "fk");
var maker = Expression.Lambda(
Expression.Call(par, pi.GetSetMethod(),
Expression.Convert(Expression.Constant(null), typeof(FKV))), par);
var cmaker = maker.Compile() as Action<FK>;
return cmaker;
}
Usage:
IEnumerable<Tag> values = ...;
Image e = ...;
e.ImageTags.UpdateReferences(x => x.Tag, tags);