How can I get names of all the keys in a MongoDB collection using c#.
I am using mongocsharpdriver.
I am able to get all the records using
var collection1 = database1.GetCollection("Games").FindAll();
Now I need the key names to display/use it. I need key names of collection that I have fetched.
e.g. If I have collection which
{ "_id" : ObjectId("c3"), "GameID" : 20, "GameName" : "COD5", "Cost" : 100}
{ "_id" : ObjectId("c4"), "GameID" : 21, "GameName" : "NFS", "Publisher" : "EA"}
{ "_id" : ObjectId("c5"), "GameID" : 22, "GameName" : "CS", "Cost" : 200}
So I should get list of keys like GameID, GameName, Cost, Publisher.
I also went through MongoDB Get names of all keys in collection but was not able to implement it, didnot understood it & got problem with mapreduce.
Inspired from the link in your question:
string map = #"function() {
for (var key in this) { emit(key, null); }
}";
string reduce = #"function(key, stuff) { return null; }";
string finalize = #"function(key, value){
return key;
}";
MapReduceArgs args = new MapReduceArgs();
args.FinalizeFunction = new BsonJavaScript(finalize);
args.MapFunction = new BsonJavaScript(map);
args.ReduceFunction = new BsonJavaScript(reduce);
var results = collection1.MapReduce(args);
foreach (BsonValue result in results.GetResults().Select(item => item["_id"]))
{
Console.WriteLine(result.AsString);
}
A very inefficient, but simple way to do this is
HashSet<string> keys = new HashSet<string>();
foreach (var rover in collection1.FindAll())
{
rover.Names.ToList().ForEach(p => keys.Add(p));
}
Keep in mind that finding the set of keys, no matter how it's implemented, will always have to iterate the entire collection, so it will be terribly slow on larger collections.
It makes sense to use Map/Reduce for this problem on larger collections, because that avoids all the data transfer and deserialization overhead that is incurred by the solution I posted above, but you should generally try to avoid doing something like this at all. At least, don't do it where it's needed synchronously.
If you somehow need to know the set of all fields quickly, you're better off keeping track of the fields during writes and store the list of fields in a separate collection somewhere.
Related
I am trying to search for a text in elasticsearch using nest 7.10.1. I want to search in two different indexes, I get a response in the form of documents, but I cannot access its properties because the result has the combination of two indexes. Below is the code I tried. Both the indexes has same properties. What do I use in the foreach loop to access the key and values of the result documents.
public void searchIndices(string query) {
var response = client.Search<object>(
s => s.Index("knowledgearticles_index, index2")
.Query(q => q.Match(m => m.Field("locationName")
.Query(query))));
Console.WriteLine(response.Documents);
foreach(object r in response.Documents) {
}
}
I am using elasticsearch 7.10.2
Each raw hit coming back in the search response has the _index meta field associated with it:
"hits" : {
"total" : {
"value" : 91,
"relation" : "eq"
},
"hits" : [
{
"_index" : "knowledgearticles_index", <---
"_type" : "_doc",
"_id" : "r_oLl3cBZOT6A8Qby8Qd",
"_score" : 1.0,
"_source" : {
...
}
}
Now, in NEST,
.Documents is a convenient shorthand for retrieving the _source for each hit
-- meaning that you'll have lost access to the meta properties.
So the trick is to use a loop like this instead:
foreach (var hit in response.HitsMetadata.Hits) {
Console.WriteLine(hit);
}
If the two indices that you're searching over contain different JSON structures, then the T in Search<T> will need to be a type that the differing JSON structures can be deserialize to. object will work as per the example in your question, but then there is no typed access for any of the properties.
A simple approach to overcoming this is to hook up the JsonNetSerializer and then use JObject for T
var pool = new SingleNodeConnectionPool(new Uri("http://localhost:9200"));
var connectionSettings =
new ConnectionSettings(pool, sourceSerializer: JsonNetSerializer.Default);
var client = new ElasticClient(connectionSettings);
var response = client.Search<JObject>(s => s
.Index("knowledgearticles_index, index2")
.Query(q => q
.Match(m => m
.Field("locationName")
.Query(query)
)
)
);
We now have a way of accessing properties on JObject, but will still need to handle the type of each property.
I'm trying to get all of the "last" values from this JSON here:
{"btc":{
"usd": {
"bitfinex": {
"last": "1191.60",
"volume": "1.99324e+7"
},
"bitstamp": {
"last": "1193.06",
"volume": "8.73693e+6"
},
"btce": {
"last": "1174.27",
"volume": "6.03521e+6"
}
}
}
But for some reason I can only access "btc" and "usd". I can't get anything out of it including the "last" values. Here is the code i'm using:
private string GetPrice()
{
WebClient wc = new WebClient();
var data = wc.DownloadString("http://preev.com/pulse/units:btc+usd/sources:bitfinex+bitstamp+btce");
JObject o = JObject.Parse(data);
string response = o["btc"].ToString();
return response;
}
If I change it to:
o["last"].ToString();
It just doesn't return anything. Can someone please provide me with a solution? I also tried making a key/value dict out of it and looping over each pair. Did not work.
The JObject structure is similar to a class with properties, so the first-level indexer ["btc"] returns another object that you have to query for its own properties ["usd"]
You can also opt for using JObject.SelectToken, generally not a bad idea. Other answers have shown how to chain the indexers but that's hard to read and maintain. Instead you can do:
jObj.SelectToken("btc[0].usd[0].bitstamp[0].last").ToString();
Further you can use the power of this syntax for other queries:
// a list o all the 'last' values
jObj.SelectTokens("btc.usd.*.last").Select(t=>t.ToString()).ToList();
Another advantage, if you're building a more complex system, is that you could put the queries in a config file or attributes etc to make them more manageable or deploy logic changes without rebuilding.
Yet another approach would be to build your own class structure and deserialize your json into it, so you have strongly typed values (double instead of string for the values for example)
public class btc {
public usd usd {get;set;}
}
public class usd....
var btcLoaded = JsonConvert.DeserializeObject<btc>(jsonString);
var lastBitstamp = btc.usd.bitstamp.last;
Use: o["btc"]["usd"]["bitfinex"]["last"].ToString() to get the 'last' value of 'bitfinex'.
After parsing the JSON whenever you index the o variable you are indexing from the root of the JSON. In order to access nested properties like 'last' you will need to index into the next level of the JSON as such:
var bitfinex = o["btc"]["usd"]["bitfinex"]["last"].ToString();
var bitstamp = o["btc"]["usd"]["bitstamp"]["last"].ToString();
var btce = o["btc"]["usd"]["btce"]["last"].ToString();
To reduce the repetition you could iterate over the properties under the btc.usd field.
if u want to all last values use this..
decimal[] lastValues = obj.SelectTokens("$..last").ToArray()
.Select(a => a.Parent.ToObject<decimal>()).ToArray();
if u want to dictionary, use this..
var dictionary = obj["btc"]["usd"].Select(a =>
new
{
Key = ((JProperty)a).Name,
Value = a.First["last"].ToObject<decimal>()
})
.ToDictionary(a => a.Key, a => a.Value);
I'm working with an API with an example output of the following
{
"id": 12345678
"photo-url-1280": "http://68.media.tumblr.com/5076e813bdc463409f59172b2c152487/tumblr_ocy8peMYvo1uur5s9o1_1280.png",
"photo-url-500": "http://67.media.tumblr.com/5076e813bdc463409f59172b2c152487/tumblr_ocy8peMYvo1uur5s9o1_500.png",
"photo-url-400": "http://68.media.tumblr.com/5076e813bdc463409f59172b2c152487/tumblr_ocy8peMYvo1uur5s9o1_400.png",
"photo-url-250": "http://67.media.tumblr.com/5076e813bdc463409f59172b2c152487/tumblr_ocy8peMYvo1uur5s9o1_250.png",
"photo-url-100": "http://67.media.tumblr.com/5076e813bdc463409f59172b2c152487/tumblr_ocy8peMYvo1uur5s9o1_100.png",
"photo-url-75": "http://67.media.tumblr.com/5076e813bdc463409f59172b2c152487/tumblr_ocy8peMYvo1uur5s9o1_75sq.png",
}
Those last items are very related, so I'd like to move them into their own object.
As of now, getting a specified image would be super messy. Getting the 400px version might look like this -
myPhotoObject.Photo400
However, if I was able to move these URLs into an object of their own, I could more cleanly call it like the following:
myPhotoObject.Photo.400
or even introduce more friendly methods
myPhoto.Photo.GetClosestTo(451);
There is a pattern in the URLs here - after an _ character, the identifying size is shown e.g. ..._1280.png and ..._500.png
Would the best way to get these be to write a getter property that just appends a list of known sizes to a URL? Would this way be any less/more efficient than using purely the JSON.net converters?
I suggest to parse your JSON into a dictionary first, as described here, and then manually copy it to a more structured object:
Dictionary<string, string> dict = JsonConvert.DeserializeObject<Dictionary<string, string>>(json);
var myImageObject = new { Id = "", Images = new Dictionary<int, string>()};
foreach(var key in dict.Keys) {
if(key == "id") {
myImageObject.Id = dict[key]
}
else {
var imageSize = ParseForImageSize(key);
myImageObject.Images.Add(imageSize, dict[key])
}
}
I have a User class that accumulates lots of DataTime entries in some List<DateTime> Entries field.
Occasionally, I need to get last 12 Entries (or less, if not reached to 12). It can get to very large numbers.
I can add new Entry object to dedicated collection, but then I have to add ObjectId User field to refer the related user.
It seems like a big overhead, for each entry that holds only a DateTime, to add another field of ObjectId. It may double the collection size.
As I occasionally need to quickly get only last 12 entries of 100,000 for instance, I cannot place these entries in a per-user collection like:
class PerUserEntries {
public ObjectId TheUser;
public List<DateTime> Entries;
}
Because it's not possible to fetch only N entries from an embedded array in a mongo query, AFAIK (if I'm wrong, it would be very gladdening!).
So am I doomed to double my collection size or is there a way around it?
Update, according to #profesor79's answer:
If your answer works, that will be perfect! but unfortunately it fails...
Since I needed to filter on the user entity as well, here is what I did:
With this data:
class EndUserRecordEx {
public ObjectId Id { get; set; }
public string UserName;
public List<EncounterData> Encounters
}
I am trying this:
var query = EuBatch.Find(u => u.UserName == endUser.UserName)
.Project<BsonDocument>(
Builders<EndUserRecordEx>.Projection.Slice(
u => u.Encounters, 0, 12));
var queryString = query.ToString();
var requests = await query.ToListAsync(); // MongoCommandException
This is the query I get in queryString:
find({ "UserName" : "qXyF2uxkcESCTk0zD93Sc+U5fdvUMPow" }, { "Encounters" : { "$slice" : [0, 15] } })
Here is the error (the MongoCommandException.Result):
{
{
"_t" : "OKMongoResponse",
"ok" : 0,
"code" : 9,
"errmsg" : "Syntax error, incorrect syntax near '17'.",
"$err" : "Syntax error, incorrect syntax near '17'."
}
}
Update: problem identified...
Recently, Microsoft announced their DocumentDB protocol support for MongoDB. Apparently, it doesn't support yet all projection operators. I tried it with mLab.com, and it works.
You can use PerUserEntries as this is a valuable document structure.
To get part of that array we need to add projection to query, so we can get only x elements and this is done server side.
Please see snippet below:
static void Main(string[] args)
{
// To directly connect to a single MongoDB server
// or use a connection string
var client = new MongoClient("mongodb://localhost:27017");
var database = client.GetDatabase("test");
var collection = database.GetCollection<PerUserEntries>("tar");
var newData = new PerUserEntries();
newData.Entries = new List<DateTime>();
for (var i = 0; i < 1000; i++)
{
newData.Entries.Add(DateTime.Now.AddSeconds(i));
}
collection.InsertOne(newData);
var list =
collection.Find(new BsonDocument())
.Project<BsonDocument>
(Builders<PerUserEntries>.Projection.Slice(x => x.Entries, 0, 3))
.ToList();
Console.ReadLine();
}
public class PerUserEntries
{
public List<DateTime> Entries;
public ObjectId TheUser;
public ObjectId Id { get; set; }
}
Hello I have a mongoDB Collection called Nodes whose structure is as follows:
{
"_id" : new BinData(3, "Ljot2wCBskWG9VobsLA0zQ=="),
"logicalname" : "Test Node",
"host" : "eh5tmb054pc",
"port" : 104,
"appendtimestamp" : false,
"studies" : ["1.3.12.2.1107"],
"tests" : [],
"mainentries" : [{
"_id" : "1.3.12.2.1107",
"Index" : 0,
"Type" : "Study"
}]
}
I created a new key called "mainentries" which is now storing the "studies" and "tests". So in order to support my new versions without hassle, I now want to write a method in my Setup Helper, which would enable me to read this collection - Check if studies,tests exists , If yes add the key "mainentries" and remove the studies/tests key.
My question is: What kind of query must I use to reach each collection of Nodes to check for the fields and update. I am using the MongoDB-CSharp community driver.
Would appreciate any help and pointers.
You can simply check whether the field(s) still exist(s):
var collection = db.GetCollection<Node>("nodes");
var nodes = collection.Find(Query.And( // might want Query.Or instead?
Query<Node>.Exists(p => p.Tests),
Query<Node>.Exists(p => p.Studies)).SetSnapshot();
foreach(var node in nodes) {
// maybe you want to move the Tests and Studies to MainEntries here?
node.MainEntries = new List<MainEntries>();
node.Test = null;
node.Studies = null;
collection.Update(node);
}
If you don't want to migrate the data, but just remove the fields and create new ones, you can also do in a simple batch update using $exists, $set and $remove