We Keep Coding

An unhandled exception of type 'System.StackOverflowException' occurred in EntityFramework dll

I get this exception when I try to process 270k records. It fails at 12k. Can someone explain to me what I am missing?
The database is SQL and I am using EF 6. I am using predicate builder to build my where clause.
The idea being
select * from table where ((a = 'v1' and b = 'v2') or (a = 'v11' and b = 'v21') or (a = 'v12' and b = 'v22') ..)
I don't see anywhere that I still hold reference to my object that represents EF class. I am creating a POCO for the result I want to send back to view.
Any ideas?
Also I am using CommandTimeout of 10000 and the point where it fails, when I run the query with same paramters in sql management studio, it returns 400 rows.
When I ran profiler, I noticed a few seconds before I got the error, memory usage shot up to 1GB+
Thanks
public List<SearchResult> SearchDocuments(List<SearchCriteria> searchCriterias)
{
List<SearchResult> results = new List<SearchResult>();
var fieldSettings = GetData() ;// make a call to database to get this data
using (var context = CreateContext())
{
var theQuery = PredicateBuilder.False<ViewInSqlDatabase>();
int skipCount = 0;
const int recordsToProcessInOneBatch = 100;
while (searchCriterias.Skip(skipCount).Any())
{
var searchCriteriasBatched = searchCriterias.Skip(skipCount).Take(recordsToProcessInOneBatch);
foreach (var searchCriteria in searchCriteriasBatched)
{
var queryBuilder = PredicateBuilder.True<ViewInSqlDatabase>();
// theQuery
if (searchCriteria.State.HasValue)
queryBuilder = queryBuilder.And(a => a.State == searchCriteria.State.Value);
if (!string.IsNullOrWhiteSpace(searchCriteria.StateFullName))
queryBuilder = queryBuilder.And(a => a.StateName.Equals(searchCriteria.StateFullName, StringComparison.CurrentCultureIgnoreCase));
if (searchCriteria.County.HasValue)
queryBuilder = queryBuilder.And(a => a.County == searchCriteria.County.Value);
if (!string.IsNullOrWhiteSpace(searchCriteria.CountyFullName))
queryBuilder = queryBuilder.And(a => a.CountyName.Equals(searchCriteria.CountyFullName, StringComparison.CurrentCultureIgnoreCase));
if (!string.IsNullOrWhiteSpace(searchCriteria.Township))
queryBuilder = queryBuilder.And(a => a.Township == searchCriteria.Township);
// and so on...for another 10 parameters
theQuery = theQuery.Or(queryBuilder.Expand());
}
// this is where I get error after 12k to 15k criterias have been processed
var searchQuery = context.ViewInSqlDatabase.AsExpandable().Where(theQuery).Distinct().ToList();
foreach (var query in searchQuery)
{
var newResultItem = SearchResult.Create(query, fieldSettings); // POCO object with no relation to database
if (!results.Contains(newResultItem))
results.Add(newResultItem);
}
skipCount += recordsToProcessInOneBatch;
}
}
return results.Distinct().OrderBy(a => a.State).ThenBy(a => a.County).ThenBy(a => a.Township).ToList();
}

Fourat is correct that you can modify your query to context.SearchResults.Where(x => ((x.a == 'v1' &&x.b == 'v2') || (x.a = 'v11' &&x.b = 'v21') || (x.a = 'v12' && x.b = 'v22')).Distinct().OrderBy(a => a.State).ThenBy(a => a.County).ThenBy(a => a.Township).ToList(); What this do with make the database do the heavy lifting for you and you
I would also suggest that you use lazy evaluation instead of forcing it into a list if you can.

What is best and rapid way for calculate this query?

I'm beginner in c# and linq ,write this query in c#:
var query1 = (from p in behzad.Customer_Care_Database_Analysis_Centers
select p).ToArray();
for (Int64 i = 0; i < query1.Count(); i++)
{
var query2 = (from tt in behzad.Customer_Care_Database_Analysis_DETAILs
where tt.fileid == FILE_ID && tt.code_markaz ==query1[i].code_markaz //"1215" //query1[i].code_markaz.ToString().Trim() //&& tt.code_markaz.ToString().Trim() == query1[i].code_markaz.ToString().Trim()
select new
{
tt.id
}).ToArray();
if (query2.Count() > 0)
{
series1.Points.Add(new SeriesPoint(query1[i].name_markaz, new double[] { query2.Count() }));
counter += 15;
}
}//end for
but up code is very slow,i have about 1000000 Customer_Care_Database_Analysis_Centers and about 20 million record into the Customer_Care_Database_Analysis_DETAILs table,which is best query for up code?thanks.

Your current code first gets a lot of records into memory, then executes a new query for each record - where you only use the count of items, even though you again get everything.
I think (untested) that the following will perform better:
var query = from center in behzad.Customer_Care_Database_Analysis_Centers
join details in behzad.Customer_Care_Database_Analysis_DETAILs
on center.code_markaz equals details.code_markaz
where details.fileid == FILE_ID
where details.Any()
select new { Name = center.name_markaz, Count = details.Count()};
foreach(var point in query)
{
series1.Points.Add(new SeriesPoint(point.Name, new double[] { point.Count };
counter += 15;
}
Instead of a lot of queries, execute just one query that will get just the data needed
Instead of getting everything into memory first (with ToArray()), loop through it as it arrives - this saves a lot of memory

LINQ to Entities does not recognize the method and this method cannot be translated into a store expression

Here's the Linq to Entities Statement which I've written.
public static List<Model.User> GetNearestUsers(int userid,int count,ref Model.HangoutDBEntities context)
{
Model.Location myLocation=GetCurrentLocation(userid,ref context);
return context.Locations.Where(o => EntityFunctions.DiffMinutes(DateTime.Now, o.DateTimeStamp) <= 30).OrderBy(o => Core.Location.Distance.CalculateDistance(myLocation.Latitude, myLocation.Longitude, o.Latitude, o.Longitude)).Select(o => o.User).ToList();
}
And here's the CalculateDistance Method
public static double CalculateDistance(decimal lat1,decimal lon1,decimal lat2,decimal lon2)
{
try
{
var R = 6371; // Radius of the earth in km
var dLat = DegreeToRadian((double)(lat2 - lat1)); // Javascript functions in radians
var dLon = DegreeToRadian((double)(lon2 - lon1));
var a = Math.Sin(dLat / 2) * Math.Sin(dLat / 2) +
Math.Cos(DegreeToRadian((double)lat1)) * Math.Cos(DegreeToRadian((double)lat2)) *
Math.Sin(dLon / 2) * Math.Sin(dLon / 2);
var c = 2 * Math.Atan2(Math.Sqrt(a), Math.Sqrt(1 - a));
var d = R * c; // Distance in km
return (double)d;
}
catch
{
return 0.0;
}
}
public static double DegreeToRadian(double angle)
{
return Math.PI * angle / 180.0;
}
Please let me know if there are any workarounds for this. I get an Exception
LINQ to Entities does not recognize the method 'Double
CalculateDistance(System.Decimal, System.Decimal, System.Decimal,
System.Decimal)' method, and this method cannot be translated into a
store expression.
I know custom functions don't work with Linq 2 Entities but I have no idea on the workarounds for this. Can you please help me out. :)

load the data into a list and then perform the Caclulation on the List
var origList = context.Locations.Where(o => EntityFunctions.DiffMinutes(DateTime.Now, o.DateTimeStamp) <= 30).ToList()
var orderedList = origList.OrderBy(o => Core.Location.Distance.CalculateDistance(myLocation.Latitude, myLocation.Longitude, o.Latitude, o.Longitude)).Select(o => o.User).ToList();
Or;
context.Locations
.Where(o => EntityFunctions.DiffMinutes(DateTime.Now, o.DateTimeStamp) <= 30).ToList()
.OrderBy(o => Core.Location.Distance.CalculateDistance(myLocation.Latitude, myLocation.Longitude, o.Latitude, o.Longitude))
.Select(o => o.User).ToList();
The thing to note is the ToList after the where clause.
Upon calling the ToList(), the data is loaded into memory and the ordering becomes a task for Linq To Objects

Take a look at https://stackoverflow.com/a/5971677/292787
you can define a custom method ..., and tell Entity Framework how to translate that method to SQL

I recently had to solve this problem. The trick is to use SqlFunctions from System.Data.Entity.SqlServer instead of System.Math for most of the statements (except Math.Pow).
The alternate LINQ style (from/where/let/orderby etc.) is useful because the let keyword allows you decompose the problem and keep things readable.
var closestLocation = await (
from q in qryable
where q.LAT.HasValue && q.LONG.HasValue
// Convert derees to radians (pi / 180 = 0.01745...)
let dbLatitudeRadians = q.LAT.Value * 0.0174532925199433
let dbLongitudeRadians = q.LONG.Value * 0.0174532925199433
let requestLatitudeRadians = requestLatitude * 0.0174532925199433
let requestLongitudeRadians = requestLongitude * 0.0174532925199433
let deltaLatitudeRadians = requestLatitudeRadians - dbLatitudeRadians
let deltaLongitudeRadians = requestLongitudeRadians - dbLongitudeRadians
let sinHalfLatitudeDistance = SqlFunctions.Sin(deltaLatitudeRadians / 2) ?? 0.0
let sinHalfLongitudeDistance = SqlFunctions.Sin(deltaLongitudeRadians / 2) ?? 0.0
let cosThisLatitude = SqlFunctions.Cos(dbLatitudeRadians) ?? 0.0
let cosThatLatitude = SqlFunctions.Cos(requestLatitudeRadians) ?? 0.0
let expr = Math.Pow(sinHalfLatitudeDistance, 2) + cosThisLatitude * cosThatLatitude * Math.Pow(sinHalfLongitudeDistance, 2)
let sqrtExpr = Math.Pow(expr, 0.5)
let sqrt1MinusExpr = Math.Pow(1 - expr, 0.5)
let distanceInMeters = 6376500 * 2 * SqlFunctions.Atan2(sqrtExpr, sqrt1MinusExpr)
orderby distanceInMeters
select q
).FirstOrDefaultAsync(cancellationToken).ConfigureAwait(false);

Problem calling string manipulation method within linq-to-sql query

I'm having a frustrating issue trying to use LINQ to call a string manipulation method. I've done lots of searching now and have tried various method to get the line noted as 'FAILS' below to work. It currently throws an exception.
Some things I've tried:
a) Initially the creation of the concatenated key was in the same query, didn't change anything
b) Converting the non-string fields to strings (another whole can of works with .ToString not working in linq. String.Concat and String.Format were tried, work ok in some cases but not when you try to refer to that value later on)
c) Using the concat etc instead of the '+' to join the things together.
As you can see it seems fairly tolerant of appending strings to non-strings, but not when that method is invoked.
There are lots of rows so I'd prefer not to convert the data to a list/array etc, but if that's the only option then any suggestions appreciated.
Many thanks! - Mark
var vouchers = from v in db.Vouchers
select new
{
v.Amount,
v.Due_Date,
v.Invoice_Date,
v.PO_CC,
v.Vendor_No_,
v.Invoice_No_,
invoiceNumeric = MFUtil.StripNonNumeric(v.Invoice_No_)
};
var keyedvouchers = from vv in vouchers
select new
{
thekey = vv.Vendor_No_ + "Test", // works with normal string
thekey2 = vv.Amount + "Test", // works with decimal
thekey3 = vv.Invoice_Date + "Test", // works with date
thekey4 = vv.invoiceNumeric, // works
thekey5 = vv.invoiceNumeric + "Test" // FAILS
};
-- The method to strip chars ---
public static string StripNonNumeric(string str)
{
StringBuilder sb = new StringBuilder();
foreach (char c in str)
{
// only append if its withing the acceptable boundaries
// strip special chars: if ((c >= '0' && c <= '9') || (c >= 'A' && c <= 'Z') | || (c >= 'a' && c <= 'z') | c == '.' || c == '_')
// strip any nonnumeric chars
if (c >= '0' && c <= '9')
{
sb.Append(c);
}
}
return sb.ToString();
}
-- The Exception Message--
System.InvalidOperationException was unhandled by user code
Message=Could not translate expression 'Table(Voucher).Select(v => new <>f__AnonymousType07(Amount = v.Amount, Due_Date = v.Due_Date, Invoice_Date = v.Invoice_Date, PO_CC = v.PO_CC, Vendor_No_ = v.Vendor_No_, Invoice_No_ = v.Invoice_No_, invoiceNumeric = StripNonNumeric(v.Invoice_No_))).Select(vv => new <>f__AnonymousType15(thekey = (vv.Vendor_No_ + "Test"), thekey2 = (Convert(vv.Amount) + "Test"), thekey3 = (Convert(vv.Invoice_Date) + "Test"), thekey4 = vv.invoiceNumeric, thekey5 = (vv.invoiceNumeric + "Test")))' into SQL and could not treat it as a local expression.

It's because it tries to build an SQL query of the expression and the MFUtil.StripNonNumeric cannot be translated into SQL.
Try returning it first and then convert the reult into a list and then use a second query to convert it.
var vouchers_temp = from v in db.Vouchers
select new
{
v.Amount,
v.Due_Date,
v.Invoice_Date,
v.PO_CC,
v.Vendor_No_,
v.Invoice_No_
};
var vouchers = vouchers_temp.ToList().Select( new {
Amount,
Due_Date,
Invoice_Date,
PO_CC,
Vendor_No_,
Invoice_No_,
invoiceNumeric = MFUtil.StripNonNumeric(Invoice_No_)
});

It FAILS to work, because it is not suppose to work.
Create a SQL-side function and call that in the query.

Simple rating algorithm to sorting results according to user query

I'm developing a very basic web search engine that has several parts. After retrieving results according to a user query, I want to calculate rate of each result and then sort results by calculated rate. Here is my query:
var tmpQuery = (from urls in _context.Urls
join documents in _context.Documents
on urls.UrlId equals documents.DocumentId
let words = (from words in _context.Words
join hits in _context.Hits
on words.WordId equals hits.WordId
where hits.DocumentId == documents.DocumentId
select words.Text)
select new { urls, documents, words });
var results = (from r in tmpQuery.AsEnumerable()
where r.urls.ResolvedPath.Contains(breakedQuery, KeywordParts.Url, part) ||
r.documents.Title.Contains(breakedQuery, KeywordParts.Title, part) ||
r.documents.Keywords.Contains(breakedQuery, KeywordParts.Keywords, part) ||
r.documents.Description.Contains(breakedQuery, Description, part) ||
r.words.Contains(breakedQuery, KeywordParts.Content, part)
select new SearchResult()
{
UrlId = r.urls.UrlId,
Url = r.urls.ResolvedPath,
IndexedOn = r.documents.IndexedOn,
Title = r.documents.Title,
Description = r.documents.Description,
Host = new Uri(r.urls.ResolvedPath).Host,
Length = r.documents.Length,
Rate = 0CalculateRating(breakedQuery, r.urls.ResolvedPath, r.documents.Title, r.documents.Keywords, r.documents.Description, r.words)
}).AsEnumerable()
.OrderByDescending(result => result.Rate)
.Distinct(new SearchResultEqualityComparer());
and rate is calculated by this method:
private int CalculateRating(IEnumerable<string> breakedQuery, string resolvedPath, string title, string keywords, string description, IEnumerable<string> words)
{
var baseRate = 0;
foreach (var query in breakedQuery)
{
/*first I'm breaking up user raw query (Microsoft -Apple) to list of broken
queries (Microsoft, -Apple) if broken query start with - that means
results shouldn't have*/
var none = (query.StartsWith("-"));
string term = query.Replace("-", "");
var pathCount = Calculate(resolvedPath, term);
var titleCount = Calculate(title, term);
var keywordsCount = Calculate(keywords, term);
var descriptionCount = Calculate(description, term);
var wordsCount = Calculate(words, term);
var result = (pathCount * 100) + (titleCount * 50) + (keywordsCount * 25) + (descriptionCount * 10) + (wordsCount);
if (none)
baseRate -= result;
else
baseRate += result;
}
return baseRate;
}
private int Calculate(string source, string query)
{
if (!string.IsNullOrWhiteSpace(source))
return Calculate(source.Split(' ').AsEnumerable<string>(), query);
return 0;
}
private int Calculate(IEnumerable<string> sources, string query)
{
var count = 0;
if (sources != null && sources.Count() > 0)
{
//to comparing two strings
//first case sensitive
var elements = sources.Where(source => source == query);
count += elements.Count();
//second case insensitive (half point of sensitive)
count += sources.Except(elements).Where(source => source.ToLowerInvariant() == query.ToLowerInvariant()).Count() / 2;
}
return count;
}
Please guide me to improve performance (speed of my search engine is very very low)

I expect this is down to your from urls in _context.Urls - with no Where on this you're getting a lot of data to then throw away when building up your results. How many items are in tmpQuery / results?