Searching across the internet how to retrieve a subset of fields in MongoDB, using C# official driver (but using LinQ as the base architecture) I found how to do this in MongoDB shell.
// selecting only "field" of a collection
db.collection.find( { field : 'value' }, { field: 1 } );
Then, I found at C# LinQ Tutorial the Select method, which is equivalent to this:
collection.AsQueryable<T>().Select(x => new { x.field });
However, the tutorial says the method "is used to project a new result type from the matching documents".
How to ensure this method will retrieve only the subset of fields and not the entire result and then select only the subset into a new object?
Will the driver build the query command before retrieve the results?
The driver does not currently retrieve a subset of the fields. If you need that functionality, you'll need to do it manually. The ticket for this functionality is here: https://jira.mongodb.org/browse/CSHARP-456. Feel free to leave feedback or vote it up if you need this.
This is cheating... but:
//This actual implementation is untested and may contain small errors.
//The helper method has been tested and *should* work.
public static IMongoQuery GetMongoQuery<T>(this IQueryable<T> query)
{
return ((MongoQueryable<T>)query).GetMongoQuery();
}
var temp =
from x in DB.Foo.AsQueryable<Test>()
where x.SomeField > 5;
select (x.OtherField);
return temp.GetMongoQuery().ToJson();
Related
I have some code which makes calls to a SharePoint Managed Metadata Service that starts off like this :
var clientContext = new ClientContext("http://mysharepointsite/")
{ AuthenticationMode = ClientAuthenticationMode.Default};
var taxonomySession = TaxonomySession.GetTaxonomySession(clientContext);
var termStore = taxonomySession.GetDefaultSiteCollectionTermStore();
which I have no problem with. However, after this we have :
clientContext.Load(termStore,
store => store.Name,
store => store.Groups.Include(
group => group.Name,
group => group.TermSets.Include(
termSet => termSet.Name,
termSet => termSet.Terms.Include(
term => term.Name)
)
)
);
Can anyone please help me understand what is going on here?
At first I thought this was some kind of LINQ query, but then I would expect the class to have the line using System.Linq;, which it does not.
I just noticed that in Visual Studio there is some IntelliSense which says the call is structured like this : void ClientruntimeContext.Load<T>(T clientObject, params System.Linq.Expressions.Expression<Func<T, object>>[] retrievals) - which makes it seem like it is using Linq in some way
I understand that the code is in some way "loading" the termstore data in the managed metadata service from the given sharepoint site, but I don't quite understand what exactly that syntax is doing.
I got the code sample from here, and it does exactly what I want it to do but I would feel a lot more comfortable if I actually understood that syntax!
The documentation was also not particularly helpful, as it just defines Load()s parameters as <T>, which could be anything!
Any advice or recommended reading is much appreciated, thank you!
ClientRuntimeContext.Load<T> Method
The second parameter of this method specifies what properties of target client object (first parameter) should be retrieved using lambda expressions.
Examples
In the following query, all the properties except collection properties such as TermStore.Groups of TermStore client object will be retrieved
ctx.Load(termStore);
In the next query only the explicitly specified list of properties (TermStore.Name, TermStore.Groups) will be retrieved for TermStore client object:
ctx.Load(termStore, store => store.Name, store => store.Groups);
The next question arise, how to specify what properties of collection client object to retrieve, Include<TSource>(IQueryable<TSource>, \[\]) method method comes to the rescue here.
Include<TSource>(IQueryable<TSource>, \[\]) method
This method is used to limit what properties are returned from a collection of objects ( for performance purposes)
Example
The following expression
ctx.Load(termStore, store => store.Groups.Include( g => g.Name));
tells to construct the query to return TermStore client object which includes TermStore.Groups property but not the default properties of Group client object, only Group.Name property.
I am using EF 6 and .net 4.5.
I have a "Documents" table with the PK "DocumentId" (Int) and another column, "Title". I have another table called "DocumentFilters" with the columns "DocumentId" (Int) and "DocGUID" (Guid). There is a FK on "Doc Id" between the two tables. It is a one to many Id's relationship between the Documents and DocumentFilters tables. The Model looks like this:
I'm trying to write a Dynamic LINQ query using the DynamicLibrary.cs library from the NuGet package.
With regular Linq, the query looks something like this in c#:
DocDBContext db = new DocDBContext();
var documents = db.Documents.Include(d => d.DocumentFilters);
Guid gTest = Guid.Parse("xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx");
documents = documents.Where(d => d.DocumentFilters.Select(f => f.DocGUID).Contains(gTest));
This works perfectly and retrieves the documents delimited by the GUID filter that I'm interested in. However, I need to place this within a filter string dynamically, where it may contain other filters. Is it possible to do that? I've tried code like:
documents = documents.Where("DocumentFilters.Select(DocGUID).Contains(#0)", gTest);
Error: No applicable aggregate method 'Select' exists
or
documents = documents.Where("DocumentFilters.DocGUID=#0", gTest);
Error: GUID property does not exist on DocumentFilters.
(Which makes sense because DocumentFilters is a collection).
I also thought it might be worth to give this a shot:
documents = documents.Where("DocumentFilters.AsQueryable().Select(DocGUID).Contains(#0)", gTest);
But again, it doesn't look like the library supports AsQueryable and throws
Error: No applicable aggregate method 'AsQueryable' exists.
Is it possible to write such a restriction on a property of an enumerable object within the Documents table?
So one solution was to modify Dynamic.cs by adding "Select" to the IEnumerableSignatures interface in the ExpressionParser class. And additionally add it to the ParseAggregate() method:
if (signature.Name == "Min" || signature.Name == "Max")
{
typeArgs = new Type[] { elementType, args[0].Type };
}
else if (signature.Name == "Select")
{
typeArgs = new Type[] { elementType, Expression.Lambda(args[0], innerIt).Body.Type };
}
However, after this I still needed to overload the Contains() method to work with GUIDs. I didn't bother to do that. Instead I realized I can simply use the already supported .Any() instead of .Select()!
documents = documents.Where("DocumentFilters.Any(DocGUID.Equals(#0))", gTest); //!!!
I also had to change the column name from "GUID" to "DocGUID" because when writing GUID.Equals() in the restriction string, the parser mistook it for the GUID type and not the column name, and threw the error that Equals() is not a valid extension for the type GUID.
Sorry for wasting everyone's time!
Try to use Dynamic Linq, it can be used against strings.
I am designing a fluent API for writing SQL. Keep in mind one of my goals is to have API not suggest functions that can't be called in that part of the chain. For instance if you just got done defining a field in the select clause you can't call Where until you called From first. A simple query looks like this:
string sql = SelectBuilder.Create()
.Select()
.Fld("field1")
.From("table1")
.Where()
.Whr("field1 > field2")
.Whr("CURRENT_TIMESTAMP > field3")
.Build()
.SQL;
My problem comes with recursion in SQL code. Say you wanted to have a field contain another SQL statement like below:
string sql = SelectBuilder.Create()
.Select()
.Fld("field1")
.SQLFld()
.Select
.Count("field6")
.From("other table")
.EndSQLFld()
.FLd("field2")
.From("table1")
.Where()
.Whr("field1 > field2")
.Whr("CURRENT_TIMESTAMP > field3")
.Build()
.SQL;
I am using method chaining to build my fluent API. It many ways it is a state machine strewn out across many classes which represent each state. To add this functionality I would need to copy essentially every state I already have and wrap them around the two SQLFld and EndSQLFld states. I would need yet another copy if you were one more level down and were embedding a SQL statement in to a field of the already embedded SQL statement. This goes on to infinity, so with an infinitely deep embedded SQL query I would need an infinite number of classes to represent the infinite states.
I thought about writing a SelectBuilder query that was taken to the point of the Build method and then embedding that SelectBuilder in to another SelectBuilder and that fixes my infinity problem, but it is not very elegant and that is the point of this API.
I could also throw out the idea that the API only offers functions when they are appropriate but I would really hate to do that. I feel like that helps you best discover how to use the API. In many fluent APIs it doesn't matter which order you call what, but I want the API to appear as close to the actual SQL statement as possible and enforce its syntax.
Anyone have any idea how to solve this issue?
Glad to see you are trying fluent interfaces, I think they are a very elegant and expressive.
The builder pattern is not the only implementation for fluent interfaces. Consider this design, and let us know what you think =)
This is an example and I leave to you the details of your final implementation.
Interface design example:
public class QueryDefinition
{
// The members doesn't need to be strings, can be whatever you use to handle the construction of the query.
private string select;
private string from;
private string where;
public QueryDefinition AddField(string select)
{
this.select = select;
return this;
}
public QueryDefinition From(string from)
{
this.from = from;
return this;
}
public QueryDefinition Where(string where)
{
this.where = where;
return this;
}
public QueryDefinition AddFieldWithSubQuery(Action<QueryDefinition> definitionAction)
{
var subQueryDefinition = new QueryDefinition();
definitionAction(subQueryDefinition);
// Add here any action needed to consider the sub query, which should be defined in the object subQueryDefinition.
return this;
}
Example usage:
static void Main(string[] args)
{
// 1 query deep
var def = new QueryDefinition();
def
.AddField("Field1")
.AddField("Filed2")
.AddFieldWithSubQuery(subquery =>
{
subquery
.AddField("InnerField1")
.AddField("InnerFiled2")
.From("InnerTable")
.Where("<InnerCondition>");
})
.From("Table")
.Where("<Condition>");
// 2 queries deep
var def2 = new QueryDefinition();
def2
.AddField("Field1")
.AddField("Filed2")
.AddFieldWithSubQuery(subquery =>
{
subquery
.AddField("InnerField1")
.AddField("InnerField2")
.AddFieldWithSubQuery(subsubquery =>
{
subsubquery
.AddField("InnerInnerField1")
.AddField("InnerInnerField2")
.From("InnerInnerTable")
.Where("<InnerInnerCondition>");
})
.From("InnerInnerTable")
.Where("<InnerCondition>");
})
.From("Table")
.Where("<Condition>");
}
You can't "have only applicable methods available" without either sub-APIs for the substructures or clear bracketing/ending of all inner structural levels (SELECT columns, expressions in WHERE clause, subqueries).
Even then, running it all through a single API will require it to be stateful & "modal" with "bracketing" methods, to track whereabouts in the decl you are. Error reporting & getting these right will be tedious.
Ending bracketing by "fluent" methods, to me, seems non-fluent & ugly. This would result in a ugly appearence of EndSelect, EndWhere, EndSubquery etc. I'd prefer to build substructures (eg SUBQUERY for select) into a local variable & add that.
I don't like the EndSQLFld() idiom, which terminates the Subquery implicitly by terminating the Field. I'd prefer & guess it would be better design to terminate the subquery itself which is the complex part of the nested structure -- not the field.
To be honest, trying to enforce ordering of a "declarative" API for a "declarative" language (SQL) seems to be a waste of time.
Probably what I'd consider closer to an ideal usage:
SelectBuilder select = SelectBuilder.Create("CUSTOMER")
.Column("ID")
.Column("NAME")
/*.From("CUSTOMER")*/ // look, I'm just going to promote this onto the constructor.
.Where("field1 > field2")
.Where("CURRENT_TIMESTAMP > field3");
SelectBuilder countSubquery = SelectBuilder.Create("ORDER")
.Formula("count(*)");
.Where("ORDER.FK_CUSTOMER = CUSTOMER.ID");
.Where("STATUS = 'A'");
select.Formula( countSubquery, "ORDER_COUNT");
string sql = SelectBuilder.SQL;
Apologies to the Hibernate Criteria API :)
Searching across the internet how to retrieve a subset of fields in MongoDB, using C# official driver (but using LinQ as the base architecture) I found how to do this in MongoDB shell.
// selecting only "field" of a collection
db.collection.find( { field : 'value' }, { field: 1 } );
Then, I found at C# LinQ Tutorial the Select method, which is equivalent to this:
collection.AsQueryable<T>().Select(x => new { x.field });
However, the tutorial says the method "is used to project a new result type from the matching documents".
How to ensure this method will retrieve only the subset of fields and not the entire result and then select only the subset into a new object?
Will the driver build the query command before retrieve the results?
The driver does not currently retrieve a subset of the fields. If you need that functionality, you'll need to do it manually. The ticket for this functionality is here: https://jira.mongodb.org/browse/CSHARP-456. Feel free to leave feedback or vote it up if you need this.
This is cheating... but:
//This actual implementation is untested and may contain small errors.
//The helper method has been tested and *should* work.
public static IMongoQuery GetMongoQuery<T>(this IQueryable<T> query)
{
return ((MongoQueryable<T>)query).GetMongoQuery();
}
var temp =
from x in DB.Foo.AsQueryable<Test>()
where x.SomeField > 5;
select (x.OtherField);
return temp.GetMongoQuery().ToJson();
So here's my dilemma. I'm trying to utilize Dynamic LINQ to parse a search filter for retrieving a set of records from an Azure table. Currently, I'm able to get all records by using a GenericEntity object defined as below:
public class GenericEntity
{
public string PartitionKey { get; set; }
public string RowKey { get; set; }
Dictionary<string, object> properties = new Dictionary<string, object>();
/* "Property" property and indexer property omitted here */
}
I'm able to get this completely populated by utilizing the ReadingEntity event of the TableServiceContext object (called OnReadingGenericEvent). The following code is what actually pulls all the records and hopefully filter (once I get it working).
public IEnumerable<T> GetTableRecords(string tableName, int numRecords, string filter)
{
ServiceContext.IgnoreMissingProperties = true;
ServiceContext.ReadingEntity -= LogType.GenericEntity.OnReadingGenericEntity;
ServiceContext.ReadingEntity += LogType.GenericEntity.OnReadingGenericEntity;
var result = ServiceContext.CreateQuery<GenericEntity>(tableName).Select(c => c);
if (!string.IsNullOrEmpty(filter))
{
result = result.Where(filter);
}
var query = result.Take(numRecords).AsTableServiceQuery<GenericEntity>();
IEnumerable<GenericEntity> res = query.Execute().ToList();
return res;
}
I have TableServiceEntity derived types for all the tables that I have defined, so I can get all properties/types using Reflection. The problem with using the GenericEntity class in the Dynamic LINQ Query for filtering is that the GenericEntity object does NOT have any of the properties that I'm trying to filter by, as they're really just dictionary entries (dynamic query errors out). I can parse out the filter for all the property names of that particular type and wrap
"Property[" + propName + "]"
around each property (found by using a type resolver function and reflection). However, that seems a little... overkill. I'm trying to find a more elegant solution, but since I actually have to provide a type in ServiceContext.CreateQuery<>, it makes it somewhat difficult.
So I guess my ultimate question is this: How can I use dynamic classes or generic types with this construct to be able to utilize dynamic queries for filtering? That way I can just take in the filter from a textbox (such as "item_ID > 1023000") and just have the TableServiceEntity types dynamically generated.
There ARE other ways around this that I can utilize, but I figured since I started using Dynamic LINQ, might as well try Dynamic Classes as well.
Edit: So I've got the dynamic class being generated by the initial select using some reflection, but I'm hitting a roadblock in mapping the types of GenericEntity.Properties into the various associated table record classes (TableServiceEntity derived classes) and their property types. The primary issue is still that I have to initially use a specific datatype to even create the query, so I'm using the GenericEntity type which only contains KV pairs. This is ultimately preventing me from filtering, as I'm not able to do comparison operators (>, <, =, etc.) with object types.
Here's the code I have now to do the mapping into the dynamic class:
var properties = newType./* omitted */.GetProperties(
System.Reflection.BindingFlags.Instance |
System.Reflection.BindingFlags.Public);
string newSelect = "new(" + properties.Aggregate("", (seed, reflected) => seed += string.Format(", Properties[\"{0}\"] as {0}", reflected.Name)).Substring(2) + ")";
var result = ServiceContext.CreateQuery<GenericEntity>(tableName).Select(newSelect);
Maybe I should just modify the properties.Aggregate method to prefix the "Properties[...]" section with the reflected.PropertyType? So the new select string will be made like:
string newSelect = "new(" + properties.Aggregate("", (seed, reflected) => seed += string.Format(", ({1})Properties[\"{0}\"] as {0}", reflected.Name, reflected.PropertyType)).Substring(2) + ")";
Edit 2: So now I've hit quite the roadblock. I can generate the anonymous types for all tables to pull all values I need, but LINQ craps out on my no matter what I do for the filter. I've stated the reason above (no comparison operators on objects), but the issue I've been battling with now is trying to specify a type parameter to the Dynamic LINQ extension method to accept the schema of the new object type. Not much luck there, either... I'll keep you all posted.
I've created a simple System.Refection.Emit based solution to create the class you need at runtime.
http://blog.kloud.com.au/2012/09/30/a-better-dynamic-tableserviceentity/
I have run into exactly the same problem (with almost the same code :-)). I have a suspicion that the ADO.NET classes underneath somehow do not cooperate with dynamic types but haven't found exactly where yet.
So I've found a way to do this, but it's not very pretty...
Since I can't really do what I want within the framework itself, I utilized a concept used within the AzureTableQuery project. I pretty much just have a large C# code string that gets compiled on the fly with the exact object I need. If you look at the code of the AzureTableQuery project, you'll see that a separate library is compiled on the fly for whatever table we have, that goes through and builds all the properties and stuff we need as we query the table. Not the most elegant or lightweight solution, but it works, nevertheless.
Seriously wish there was a better way to do this, but unfortunately it's not as easy as I had hoped. Hopefully someone will be able to learn from this experience and possibly find a better solution, but I have what I need already so I'm done working on it (for now).