I have documents stored in cosmos db, I have multiple documents for a same "stationkey"(partition key), in this example stationkey "ABC" has more than one documents with "yymm" has "2018-02" & "2018-01" e.t.c,
query that i am trying is get all "avg" & "dd" fields along with "yymm" for the given stationkey and yymm filter combination
I am trying to query using C#, I am trying to get "avg", "dd" & "yymm" fields from "data" array, the query that I have written is giving entire "data" array.
var weatherQuery = this.docClient.CreateDocumentQuery<WeatherStation>(docUri, queryOptions)
.Where(wq => wq.stationName == stationKey && lstYearMonthFilter.Contains(wq.yearMonth))
.Select(s => s.data);
what is the best way to query specific fields in from a document array?
So you got the data in s => s.data. To get only the avg from the array you have to do another projection as following:
.Select (s => s.data.Select ( a => a.avg ))
Modifying my answer as you say you don't find 'Select' on 'data'.
Define a class MyDocument as such:
public class Datum
{
[JsonProperty("dd")]
public string dd;
[JsonProperty("max")]
public int max;
[JsonProperty("min")]
public int min;
[JsonProperty("avg")]
public int avg;
}
public class MyDocument : Document
{
[JsonProperty("id")]
public string id;
[JsonProperty("data")]
public Datum[] data;
}
modify your code accordingly
IDocumentQuery<MyDocument> query = client.CreateDocumentQuery<MyDocument>(UriFactory.CreateDocumentCollectionUri(_database, _collection),
new FeedOptions { MaxItemCount = -1, EnableCrossPartitionQuery = true, MaxDegreeOfParallelism = 199, MaxBufferedItemCount = 100000})
.Where(predicate)
.AsDocumentQuery();
while (query.HasMoreResults)
{
FeedResponse<MyDocument> feedResponse = await query.ExecuteNextAsync<MyDocument>();
Console.WriteLine (feedResponse.Select(x => x.data.Select(y => y.avg)));
}
HTH
You can select only specific fields from the array items using a double-nested anonymous class - see the altered SelectMany below. This will return yymm with every Datum, so may not be as efficient as just selecting the entire array - definitely measure the RU/s in both cases.
var weatherQuery = this.docClient.CreateDocumentQuery<WeatherStation>(docUri, queryOptions)
.Where(wq => wq.stationName == stationKey && lstYearMonthFilter.Contains(wq.yearMonth))
.SelectMany(x => x.data.Select(y => new { x.yymm, data = new[] { new { y.dd, y.avg } } }))
.AsDocumentQuery();
var results = new List<WeatherStation>();
while (weatherQuery.HasMoreResults)
{
results.AddRange(await weatherQuery.ExecuteNextAsync<WeatherStation>());
}
var groupedResults = results
.GroupBy(x => x.yymm)
.Select(x => new { x.First().yymm, data = x.SelectMany(y => y.data).ToArray() })
.Select(x => new WeatherStation() { yymm = x.yymm, data = x.data });
Related
I have the following query:
var countA=await _importContext.table1.CountAsync(ssc => ssc.ImportId == importId)
var countB=await _importContext.table2.CountAsync(ssc => ssc.ImportId == importId)
var countC=await _importContext.table3.CountAsync(ssc => ssc.ImportId == importId)
var countD=await _importContext.table4.CountAsync(ssc => ssc.ImportId == importId)
There are 9 more count from different tables. Is there a way to summarize the query in terms of optimizing & removing redundancy?
I tried wrapping up the queries like:
var result = new
{
countA = context.table1.Count(),
countB = context.table2.Count(),
.....
};
but this takes more time than the first one above.
You can't really optimise it as you seem to need the counts from all of the tables. Your second method of getting the data still calls the database the same amount of times as the first but also creates an object containing all of the counts so is likely to take longer.
The only thing you can really do to make it faster is to get the data in parallel but this might be overkill. I would just go with your first option unless it's really slow.
You can create such query via gouping by constant and Concat operator:
Helper class:
public class TableResult
{
public string Name { get; set; }
public int Count { get; set; }
}
Query:
var query = _importContext.table1.Where(ssc.ImportId == importId).GroupBy(x => 1).Select(g => new TableResult { Name = "table1", Count = g.Count() })
.Concat(_importContext.table2.Where(ssc.ImportId == importId).GroupBy(x => 1).Select(g => new TableResult { Name = "table2", Count = g.Count() }))
.Concat(_importContext.table3.Where(ssc.ImportId == importId).GroupBy(x => 1).Select(g => new TableResult { Name = "table3", Count = g.Count() }))
.Concat(_importContext.table4.Where(ssc.ImportId == importId).GroupBy(x => 1).Select(g => new TableResult { Name = "table4", Count = g.Count() }));
var result = await query.ToListAsync();
I have this document, a post :
{Content:"blabla",Tags:["test","toto"], CreatedOn:"2019-05-01 01:02:01"}
I want to have a page that displays themost used tags since the last 30 days.
So far I tried to create an index like this
public class Toss_TagPerDay : AbstractIndexCreationTask<TossEntity, TagByDayIndex>
{
public Toss_TagPerDay()
{
Map = tosses => from toss in tosses
from tag in toss.Tags
select new TagByDayIndex()
{
Tag = tag,
CreatedOn = toss.CreatedOn.Date,
Count = 1
};
Reduce = results => from result in results
group result by new { result.Tag, result.CreatedOn }
into g
select new TagByDayIndex()
{
Tag = g.Key.Tag,
CreatedOn = g.Key.CreatedOn,
Count = g.Sum(i => i.Count)
};
}
}
And I query it like that
await _session
.Query<TagByDayIndex, Toss_TagPerDay>()
.Where(i => i.CreatedOn >= firstDay)
.GroupBy(i => i.Tag)
.OrderByDescending(g => g.Sum(i => i.Count))
.Take(50)
.Select(t => new BestTagsResult()
{
CountLastMonth = t.Count(),
Tag = t.Key
})
.ToListAsync()
But this gives me the error
Message: System.NotSupportedException : Could not understand expression: from index 'Toss/TagPerDay'.Where(i => (Convert(i.CreatedOn, DateTimeOffset) >= value(Toss.Server.Models.Tosses.BestTagsQueryHandler+<>c__DisplayClass3_0).firstDay)).GroupBy(i => i.Tag).OrderByDescending(g => g.Sum(i => i.Count)).Take(50).Select(t => new BestTagsResult() {CountLastMonth = t.Count(), Tag = t.Key})
---- System.NotSupportedException : GroupBy method is only supported in dynamic map-reduce queries
Any idea how can I make this work ? I could query for all the index data from the past 30 days and do the groupby / order / take in memory but this could make my app load a lot of data.
The results from the map-reduce index you created will give you the number of tags per day. You want to have the most popular ones from the last 30 days so you need to do the following query:
var tagCountPerDay = session
.Query<TagByDayIndex, Toss_TagPerDay>()
.Where(i => i.CreatedOn >= DateTime.Now.AddDays(-30))
.ToList();
Then you can the the client side grouping by Tag:
var mostUsedTags = tagCountPerDay.GroupBy(x => x.Tag)
.Select(t => new BestTagsResult()
{
CountLastMonth = t.Count(),
Tag = t.Key
})
.OrderByDescending(g => g.CountLastMonth)
.ToList();
#Kuepper
Based on your index definition. You can handle that by the following index:
public class TrendingSongs : AbstractIndexCreationTask<TrackPlayedEvent, TrendingSongs.Result>
{
public TrendingSongs()
{
Map = events => from e in events
where e.TypeOfTrack == TrackSubtype.song && e.Percentage >= 80 && !e.Tags.Contains(Podcast.Tags.FraKaare)
select new Result
{
TrackId = e.TrackId,
Count = 1,
Timestamp = new DateTime(e.TimestampStart.Year, e.TimestampStart.Month, e.TimestampStart.Day)
};
Reduce = results => from r in results
group r by new {r.TrackId, r.Timestamp}
into g
select new Result
{
TrackId = g.Key.TrackId,
Count = g.Sum(x => x.Count),
Timestamp = g.Key.Timestamp
};
}
}
and the query using facets:
from index TrendingSongs where Timestamp between $then and $now select facet(TrackId, sum(Count))
The reason for the error is that you can't use 'GroupBy' in a query made on an index.
'GroupBy' can be used when performing a 'dynamic query',
i.e. a query that is made on a collection, without specifying an index.
See:
https://ravendb.net/docs/article-page/4.1/Csharp/client-api/session/querying/how-to-perform-group-by-query
I solved a similar problem, by using AdditionalSources that uses dynamic values.
Then I update the index every morning to increase the Earliest Timestamp. await IndexCreation.CreateIndexesAsync(new AbstractIndexCreationTask[] {new TrendingSongs()}, _store);
I still have to try it in production, but my tests so far look like it's a lot faster than the alternatives. It does feel pretty hacky though and I'm surprised RavenDB does not offer a better solution.
public class TrendingSongs : AbstractIndexCreationTask<TrackPlayedEvent, TrendingSongs.Result>
{
public DateTime Earliest = DateTime.UtcNow.AddDays(-16);
public TrendingSongs()
{
Map = events => from e in events
where e.TypeOfTrack == TrackSubtype.song && e.Percentage >= 80 && !e.Tags.Contains(Podcast.Tags.FraKaare)
&& e.TimestampStart > new DateTime(TrendingHelpers.Year, TrendingHelpers.Month, TrendingHelpers.Day)
select new Result
{
TrackId = e.TrackId,
Count = 1
};
Reduce = results => from r in results
group r by new {r.TrackId}
into g
select new Result
{
TrackId = g.Key.TrackId,
Count = g.Sum(x => x.Count)
};
AdditionalSources = new Dictionary<string, string>
{
{
"TrendingHelpers",
#"namespace Helpers
{
public static class TrendingHelpers
{
public static int Day = "+Earliest.Day+#";
public static int Month = "+Earliest.Month+#";
public static int Year = "+Earliest.Year+#";
}
}"
}
};
}
}
i'm new to linq and i've tried this function which should check for duplicates. what i want to do is to check through my list of buildingobjects to check if any buildingobject contains an objectID identical to another buildingObject in the list. finally i want to use the GUID of the buildingObject which had a duplicate, and print it to my log for the user to see.
public class FMBuildingObject
{
public Int64 ObjectId { get; set; }
public string GUID { get; set; }
}
the building object is bigger, but this is the values i'm using.
next i'm trying to use the buildingobjects, find duplicates and then print the GUID out. however i can't figure out how to access that GUID.
var query =
buildingObjects
.GroupBy(x => new { x })
.Select(group => new { Name = group.Key, Count = group.Count() })
.OrderByDescending(x => x.Count);
foreach (var q in query)
{
var updateLog = new LogServiceModel()
{
LogType = LogTypes.Warning, Parameters = { {?GUID?}}, LogTitle = "You have used two different classifications on a same Buildingobject in {0}. "
};
logService.Create(updateLog);
}
This will return a List<string> containing the GUID's of the objects which has duplicate ObjectId's:
var result = buildingObjects
.GroupBy(b => b.ObjectId)
.Where(g => g.Count() > 1)
.SelectMany(g => g.Select(b => b.GUID)).ToList();
You can get duplicate objects by:
var query = buildingObjects.GroupBy(x => new { x.ObjectId, x.GUID })
.Where(g => g.Count() > 1)
.Select(group => new { Name = group.Key, group.Key.GUID, group.Key.ObjectId }); //I don't know what is Name and why it's equal to Key
Then inside your foreach loop:
foreach (var q in query)
{
var updateLog = new LogServiceModel()
{
LogType = LogTypes.Warning, Parameters = q.GUID, LogTitle = $"You have used two different classifications on a same Buildingobject, Id: {q.ObjectId}. "
};
logService.Create(updateLog);
}
Or you can simply do:
var query = buildingObjects.GroupBy(x => new { x.ObjectId, x.GUID })
.Where(g => g.Count() > 1)
.Select(group => new LogServiceModel()
{
LogType = LogTypes.Warning,
Parameters = group.Key.GUID,
LogTitle = $"You have used two different classifications on a same Buildingobject, Id: {group.Key.ObjectId}."
});
Let's say I have the following two lists..
var unscoredList = new List<string> { "John", "Robert", "Rebecca" };
var scoredList = new List<WordScore>
{
new WordScore("John", 10),
new WordScore("Robert", 40),
new WordScore("Rebecca", 30)
};
Is there a way I can sort the values in unscoredList by the values in scoredList where the word with the highest score appears first in unscoredList?
Below is the WordScore class if required..
public class WordScore {
public string Word;
public int Score;
public WordScore(string word, int score) {
Word = word;
Score = score;
}
}
If you don't need an in-place sort, you can do this:
var scoreLookup = scoredList.ToDictionary(l => l.Word, l => l.Score);
var result = unscoredList.OrderByDescending(l => scoreLookup[l]);
Alternatively, you can use:
unscoredList.Sort((l,r) => scoreLookup[r].CompareTo(scoreLookup[l]));
And of course, there should be some sanity checks done (duplicate values in scoredList, values in unscoredList which are not in scoredList, etc).
var test = unscoredList.OrderByDescending(x => scoredList
.Where(y => y.Word == x.ToString())
.Select(z => z.Word)
.FirstOrDefault()).ToList();
This returns another list, but not a big deal.
I am trying to send an array of ranges to an Aggregation Descriptor, but the Lambda expression expects a comma delimited expresions
.Aggregations(agg =>
{
AggregationDescriptor ag = agg.Terms("objectTypes", ot => ot.Field("doc.objectType"));
if (!parameters.ContainsKey("userID"))
ag = ag.Terms("users", ot => ot.Field("doc.entryUserID"));//.Field("doc.sourceUserID")))
ag.Terms("commentTypes", ot => ot.Field("doc.commentType"));
if (!parameters.ContainsKey("dateRange"))
{
Dictionary<string, SearchDateRange> dateMap = GetDateRangeMap();
ag.DateRange("dates", dr => dr.Field("doc.date").Format("yyyy-MM-dd")
.Ranges(r1 => r1.Key("Today").From(dateMap["Today"].startDate.Value.ToString("yyyy-MM-dd")).To("now"),
r2 => r2.Key("SinceWednesday").From(dateMap["Today"].startDate.Value.ToString("yyyy-MM-dd")).To("now"),
r3 => r3.Key("ThisYear").From(dateMap["ThisYear"].startDate.Value.ToString("yyyy-MM-dd")).To("now"),
r3 => r3.Key("Last2Years").From(dateMap["Last2Years"].startDate.Value.ToString("yyyy-MM-dd")).To("now"),
r4 => r4.Key("Last3Years").From(dateMap["Last3Years"].startDate.Value.ToString("yyyy-MM-dd")).To("now")
));
}
The above code works.
Below I would like to use the Range[] array and pass it to the aggregate discriptor but I can't, but I can create an AggregationContainer with the range array. How do I marry these two pieces together?
if (!parameters.ContainsKey("revenueRange") && docTypes.Contains(CouchbaseDocumentType.Company))
{
Dictionary<string, SearchNumberRange> numMap = GetMoneyRangeMap();
Range<double>[] ranges = numMap.Select(m =>
{
var r = new Range<double>().Key(m.Key);
if (m.Value.low.HasValue) r.From(m.Value.low.Value);
if (m.Value.high.HasValue) r.To(m.Value.high.Value);
return r;
}).ToArray();
AggregationContainer agr = new AggregationContainer
{
Range = new RangeAggregator { Field = "doc.lastFinancial.revenueUSD", Ranges = ranges }
};
}
return ag;
}
)
I created simple example to show you how you can achieve this.
var funcs = new List<Func<Range<double>, Range<double>>>();
funcs.Add(range => new Range<double>().From(1).To(2));
funcs.Add(range => new Range<double>().From(3).To(4));
var searchResponse = client.Search<Document>(
s => s.Aggregations(agg => agg.Range("range", descriptor => descriptor.Field(f => f.Number).Ranges(funcs.ToArray()))));
Document class:
public class Document
{
public int Id { get; set; }
public double Number { get; set; }
}
Hope you won't have problem with putting it into your context.