We Keep Coding

Aggregate function issue with latest version of MongoC# Driver

I have .NET Application that uses MongoDB. Current driver I am using is 1.9.2. I am trying to upgrade it to 2.7.0.
I am having some difficulty in getting an Aggregate query to work in the new version:
The working code in version 1.9.2 of the driver is:
public IEnumerable<Car> GetCarsModifiedInPeriod(DateTimeOffset dateFrom, DateTimeOffset dateTo)
{
var matchRequestFromDate = new BsonDocument
{
{
"$match",
new BsonDocument
{
{
// Filter out those too recently modified
"LastUpdatedOn.0", new BsonDocument {{"$gte", dateFrom.Ticks}}
}
}
}
};
var matchRequestToDate = new BsonDocument
{
{
"$match",
new BsonDocument
{
{
// Filter out those too recently modified
"LastUpdatedOn.0", new BsonDocument {{"$lte", dateTo.Ticks}}
}
}
}
};
var cars = collection.Aggregate(new AggregateArgs
{
Pipeline = new[] { matchRequestFromDate, matchRequestToDate},
AllowDiskUse = true,
// Setting the OutputMode to Cursor allows us to return Mongo Doc Size > 16 MB - in the case when a large date
// range is used or a large number of cars were modified in a short period of time
OutputMode = AggregateOutputMode.Cursor
}).Select(r => r.Values.Select(c => c.AsObjectId.ToString()).First());
var returnData = collection.AsQueryable().Where(c => cars.Contains(c.Id)).Select(c => c);
return returnData;
}
With a breakpoint set on returnData for the two periods specified I am getting a count of 25 cars which is what I expect.
This is how I have attempted to re-write for 2.7.0 version of driver:
public IEnumerable<Car> GetCarsModifiedInPeriod(DateTimeOffset dateFrom, DateTimeOffset dateTo)
{
var matchRequestFromDate = new BsonDocument
{
{
"$match",
new BsonDocument
{
{
// Filter out those too recently modified
"LastUpdatedOn.0", new BsonDocument {{"$gte", dateFrom.Ticks}}
}
}
}
};
var matchRequestToDate = new BsonDocument
{
{
"$match",
new BsonDocument
{
{
// Filter out those too recently modified
"LastUpdatedOn.0", new BsonDocument {{"$lte", dateTo.Ticks}}
}
}
}
};
var pipeline = new[] {matchRequestFromDate, matchRequestToDate};
//var mongoPipeline = new AggregateArgs { Pipeline = pipeline, AllowDiskUse = true, OutputMode = AggregateOutputMode.Cursor };
var aggregate = collection.Aggregate(); //.Match(mongoPipeline);
aggregate.Options.AllowDiskUse = true;
aggregate.Options.UseCursor = true;
foreach (var pipe in pipeline)
{
aggregate.AppendStage<BsonDocument>(pipe);
}
var returnData = aggregate.ToList();
return returnData;
}
If I set a breakpoint in returnData in this method I am getting a count of around 10K cars so it doesnt look like I am correctly applying the same matches

Is there a reason you are doing everything in BsonDocuments? There are methods that would make your life a lot easier, for example something like this.
collection.Aggregate(new AggregateOptions() { AllowDiskUse = true, UseCursor = true })
.Match(Builders<BsonDocument>.Filter.Gte("LastUpdatedOn.0", dateFrom.Ticks) & Builders<BsonDocument>.Filter.Lte("LastUpdatedOn.0", dateFrom.Ticks))
.ToListAsync()
You could tidy the filtering up more as well by using the right class for the collection and the builders.
Looking at the query, I'm not sure you even need to be using an aggregate unless you are doing more than a match. It could simply be a find.

Error in Json Serialization "There is already an open DataReader associated with this Command"

Json serialization command is giving this error. I used Newtonsoft.Json to avoid the cyclic reference error on serialization.
private IQueryable<Study> GetStudiesData()
{
var currUser = UserManager.FindById(User.Identity.GetUserId());
var currUserRole = currUser.Roles.First();
IQueryable<Study> studies;
if (User.IsInRole("SuperAdmin")) {
studies = db.Studies; //all studies from all centers
}
else {
var assignedStudies = db.Studies.Where(s => s.AssignedUserID == currUser.Id);
studies = db.Studies.Where(s => s.User.CenterId == currUser.CenterId && s.RoleID == currUserRole.RoleId)
.Concat(assignedStudies);
}
return studies;
}
//ajax call to this function gives error
public ActionResult GetStudies(int pageSize = 10, int pageNum = 1)
{
var studies = GetStudiesData();
var studiesCount = studies.Count();
var studiesPaged = studies.OrderBy(s=>s.PatientID).Skip(pageSize*pageNum).Take(pageSize);
var result = new { TotalRows = studiesCount, Rows = studiesPaged };
//this line gives error
//There is already an open DataReader associated with this Command which must be closed first.
var data = Newtonsoft.Json.JsonConvert.SerializeObject(result, Newtonsoft.Json.Formatting.None,
new Newtonsoft.Json.JsonSerializerSettings()
{
ReferenceLoopHandling = Newtonsoft.Json.ReferenceLoopHandling.Ignore
});
return Json(data, JsonRequestBehavior.AllowGet);
}
I tried result = studies, result = studiesPaged, result = studiesCount in var data = Newtonsoft.Json.JsonConvert.SerializeObject(result, ...).
Only studiesPaged is giving the error.
I can solve it using MultipleActiveResultSets=True; but where am I doing the second query?
Update
public ActionResult GetStudies(int pageSize = 10, int pageNum = 1)
{
var studies = GetStudiesData().ToList(); //solved, created another issue
//other code
}
Now the Json returned looks like a Json is again serialized to string.
"{\"TotalRows\":5,\"Rows\":[{\"AssignedUser\":null, ..... }]}"
When parsed by javascript it is all splitted to an array of characters.
Update2
Changed
//data is a json string after serialization
return Json(data, JsonRequestBehavior.AllowGet); //Serializing the Json
to
return Content(data, "application/json");

Assigning it to a list, and then querying it should prevent this:
public ActionResult GetStudies(int pageSize = 10, int pageNum = 1)
{
var studies = GetStudiesData().ToList();
var studiesCount = studies.Count();
var studiesPaged = studies.OrderBy(s=>s.PatientID).Skip(pageSize*pageNum).Take(pageSize);
var result = new { TotalRows = studiesCount, Rows = studiesPaged };
//this line gives error
//There is already an open DataReader associated with this Command which must be closed first.
var data = Newtonsoft.Json.JsonConvert.SerializeObject(result, Newtonsoft.Json.Formatting.None,
new Newtonsoft.Json.JsonSerializerSettings()
{
ReferenceLoopHandling = Newtonsoft.Json.ReferenceLoopHandling.Ignore
});
return Json(data, JsonRequestBehavior.AllowGet);
}

var assignedStudies = db.Studies.Where(s => s.AssignedUserID == currUser.Id);
studies = db.Studies.Where(s => s.User.CenterId == currUser.CenterId &&
s.RoleID == currUserRole.RoleId).Concat(assignedStudies);
You have two queries to DbSet<Studies> here

Nest aggregation not working correctly

I have a use case where I need to do aggregation on multiple columns using C#.
I am using NEST libraries for this and I am facing the following issue
Query C# :
var searchRequest = new SearchRequest
{
SearchType = SearchType.Count,
Filter = filter,
Aggregations = new Dictionary<string, IAggregationContainer>
{
{ "a", new AggregationContainer
{
ExtendedStats = new ExtendedStatsAggregator()
{
Field = "a"
}
}
},
{ "b", new AggregationContainer
{
ExtendedStats = new ExtendedStatsAggregator()
{
Field = "b"
}
}
}
}
};
When I receive response from NEST, however I am getting only result for one aggregation. I am looking at SearchResult.Agg dictionary but it has only one entry for one aggregation field instead of two.
Let me know if I am missing soemthing or is it some issue with NEST libraries

if you are using term aggregation then you need to use aggregation with filter.
var qryRes1 = client.Search<object>(x => x
.Aggregations(ag => ag
.Filter("filter", (flt => flt
.Filter(f =>
{
FilterContainer filter = null;
filter &= f.Query(qr => qr.Term(wl => wl.OnField("a").Value("the value you need to filter for field a")));
return filter;
})
.Aggregations(agr => agr
.Terms("b", tr =>
{
TermsAggregationDescriptor<object> trmAggDescriptor = new TermsAggregationDescriptor<object>();
trmAggDescriptor.Field("b");
return trmAggDescriptor;
}))))
));
var terms = qryRes1.Aggs.Filter("filter").Terms("b");

elasticsearch NEST client, filed with attribute "not_analyzed" still be analyzed when search keywords contains hyphen

I have a class named "IndexModel":
public class IndexModel
{
[ElasticProperty(Index= FieldIndexOption.NotAnalyzed, Store = true)]
public string ModelNumber{ get; set; }
}
following is how i setup the elastic client:
var uri = new Uri("http://localhost:9200");
var config = new ConnectionSettings(uri);
var client = new ElasticClient(config);
client.Map<IndexModel>(m => m.MapFromAttributes());
I can see the mapped result from response:
Request {
"indexmodel": {
"properties": {
"modelNumber": {
"type": "string",
"store": true,
"index": "not_analyzed"
},
}
}
}
and i have one index record for this type, the value of "ModelNumber" property is "test-123", and following is my query:
var result = client.Search<IndexModel>(s => s.Query(new TermQuery() { Field = Property.Path<IndexModel>(it => it.ModelNumber), Value = "test-123"}));
here is the final mapped request i got:
Method: POST,
Url: http://localhost:9200/_search,
Request: {
"query": {
"term": {
"modelNumber": {
"value": "test-123"
}
}
}
}
But i can not get the result, if i change the value of "ModelNumber" property to "test123", re-index it, and search it by keywords "test123", then it's works, so i think the analyzer still analyzed the "ModelNumber" property, can someone help me, thanks.

I had the same problem, the solution is first create the index then put the mapping and at last add your data.
Add Type Attribute to your model field
[ElasticProperty(OmitNorms = true, Index = FieldIndexOption.NotAnalyzed)]
var node = new Uri("http://192.168.0.56:9200/");
var settings = new ConnectionSettings(node, defaultIndex: "ticket");
var client = new ElasticClient(settings);
var createIndexResult = client.CreateIndex("ticket");
var mapResult = client.Map<TicketElastic>(c => c.MapFromAttributes().IgnoreConflicts().Type("TicketElastic").Indices("ticket"));

How can I specify Product/Service for an Invoice Line Item for QBO IPP .NET SDK V3?

I'm trying to specify a Product/Service list item for invoice line items of invoices that I am importing to QuickBooks Online (QBO) company file and I am getting an error.
Error I receive is:
Intuit.Ipp.Exception.IdsException: InternalServerError ---> Intuit.Ipp.Exception.EndpointNotFoundException: Ids service endpoint was not found.
The exception doesn't give any further breakdown as to if what I'm doing is valid or not.
My unit test method:
[TestMethod()]
public void CreateTest()
{
Entities.Invoice invoice = new Entities.Invoice();
invoice.ReferenceId = Guid.NewGuid().ToString("N").Substring(0, 10);
invoice.CreatedDate = DateTime.Now;
invoice.CustomerId = 1;
invoice.LineItems.Add(new InvoiceLine() { ItemName = "Initial Funding", Description = "Initial Funding", Amount = 5500 });
invoice.LineItems.Add(new InvoiceLine() { ItemName = "Lien Fee", Description = "Lien Fee", Amount = 100 });
IPPRestProfile restProfile = new IPPRestProfile(realmId, accessToken, accessTokenSecret, Intuit.Ipp.Core.IntuitServicesType.QBO, consumerKey, consumerSecret);
IPP.Invoices target = new IPP.Invoices(restProfile);
Intuit.Ipp.Data.Invoice actual = target.Create(invoice);
if (actual != null)
{
Console.WriteLine("QB Invoice ID: {0}", actual.Id);
Console.WriteLine("QB Sync Token: {0}", actual.SyncToken);
Console.WriteLine("================================================");
ObjectDumper.Write(actual, 4);
}
}
The method that the unit test calls:
public Intuit.Ipp.Data.Invoice Create(Entities.Invoice invoice)
{
// Check pre-conditions
if (invoice == null) { throw new ArgumentException("Invoice object is required.", "invoice"); }
var qboInvoice = new Intuit.Ipp.Data.Invoice();
BuildInvoiceEntity(qboInvoice, invoice);
return _Service.Add(qboInvoice) as Intuit.Ipp.Data.Invoice;
}
And finally the build invoice method:
private void BuildInvoiceEntity(Intuit.Ipp.Data.Invoice qboInvoice, Entities.Invoice invoice)
{
if (qboInvoice != null && invoice != null)
{
IQuickBooksHeader header = invoice as IQuickBooksHeader;
if (String.IsNullOrEmpty(header.Id))
{
qboInvoice.DocNumber = invoice.ReferenceId;
qboInvoice.TxnDate = invoice.CreatedDate;
qboInvoice.TxnDateSpecified = true;
// Customer
qboInvoice.CustomerRef = new ReferenceType()
{
type = objectNameEnumType.Customer.ToString(),
Value = invoice.CustomerId.ToString()
};
// AR Account
qboInvoice.ARAccountRef = new ReferenceType()
{
type = objectNameEnumType.Account.ToString(),
name = "Accounts Receivable"
};
}
if (invoice.LineItems.Count > 0)
{
Intuit.Ipp.Data.Line[] invoiceLineCollection = new Intuit.Ipp.Data.Line[invoice.LineItems.Count];
for (int i = 0; i < invoice.LineItems.Count; i++)
{
var line = invoice.LineItems[i];
var qboInvoiceLine = new Intuit.Ipp.Data.Line()
{
Amount = line.Amount,
AmountSpecified = true,
Description = line.Description,
DetailType = LineDetailTypeEnum.SalesItemLineDetail,
DetailTypeSpecified = true,
AnyIntuitObject = new SalesItemLineDetail()
{
ItemRef = new ReferenceType()
{
name = line.ItemName,
},
ItemElementName = ItemChoiceType.UnitPrice,
AnyIntuitObject = line.Amount
}
};
invoiceLineCollection[i] = qboInvoiceLine;
}
qboInvoice.Line = invoiceLineCollection;
}
}
}
If I remove this piece of code from my build method:
ItemRef = new ReferenceType()
{
name = line.ItemName,
},
the invoice is successfully added with the default "Services" list item for the Product/Service of the invoice line items.
The online documentation for the IPP .NET SDK V3 is vague on what to specify for ReferenceType. What is wrong about just specifying the name of the list item? If I'm wrong about how I'm trying to specify a Product/Service list item for invoice line items, what is the correct way?

After days of researching, I never did find an answer as to why I can't just use the name like I wanted to, even though it works that way when specifying an AccountRef. But I digress, here is my solution:
// Hold a collection of QBO items
private ReadOnlyCollection<Item> _Items;
// I set the collection in the constructor only once
public Invoices(Entities.IPPRestProfile restProfile)
{
if (restProfile == null)
throw new ArgumentException("IPPRestProfile object is required.", "restProfile");
OAuthRequestValidator oAuthValidator = new OAuthRequestValidator(restProfile.OAuthAccessToken, restProfile.OAuthAccessTokenSecret,
restProfile.ConsumerKey, restProfile.ConsumerSecret);
ServiceContext context = new ServiceContext(restProfile.RealmId, restProfile.DataSource, oAuthValidator);
_Service = new DataService(context);
_Items = (new QueryService<Item>(context)).ExecuteIdsQuery("SELECT * FROM Item", QueryOperationType.query);
}
Whenever I build my invoice I query the collection for the Id of the item by name:
private void BuildInvoiceEntity(Intuit.Ipp.Data.Invoice qboInvoice, Entities.Invoice invoice)
{
...
// Get the Id value of the item by name
string itemTypeId = _Items.Where(o => o.Name == line.ItemName).FirstOrDefault().Id;
// Specify the Id value in the item reference of the SalesItemLineDetail
var qboInvoiceLine = new Intuit.Ipp.Data.Line()
{
Amount = (decimal)amount,
AmountSpecified = true,
Description = line.Description,
DetailType = LineDetailTypeEnum.SalesItemLineDetail,
DetailTypeSpecified = true,
AnyIntuitObject = new SalesItemLineDetail()
{
ItemRef = new ReferenceType() { Value = itemTypeId },
AnyIntuitObject = (decimal)line.Rate,
ItemElementName = ItemChoiceType.UnitPrice
}
};
...
}
Hopefully this helps point someone in the right direction to a possible better solution.