I have one problem. If i'm using LinqToSql, my program load my database in memory.
little example:
//pageNumber = 1; pageSize = 100;
var result =
(
from a in db.Stats.AsEnumerable()
where (DictionaryFilter(a, sourceDictionary) && DateFilter(a, beginTime, endTime) && ErrorFilter(a, WarnLevel))
select a
);
var size = result.Count(); // size = 1007
var resultList = result.Skip((pageNumber-1)*pageSize).Take(pageSize).ToList();
return resultList;
DictionaryFilter, DateFilter and ErrorFilter are functions that filter my datebase.
after this my program use ~250Mb of Ram.
if i dont use:
var size = result.Count();
My program use ~120MB Ram.
Before use this code, my program use ~35MB Ram.
How can I use count and take functions not loading all my datebase in memory?
static bool DateFilter(Stat table, DateTime begin, DateTime end)
{
if ((table.RecordTime >= begin.ToFileTime()) && (table.RecordTime <= end.ToFileTime()))
{
return true;
}
return false;
}
static bool ErrorFilter(Stat table, bool[] WarnLevel)
{
if (WarnLevel[table.WarnLevel]) return true;
else return false;
}
static bool DictionaryFilter(Stat table, Dictionary<GetSourcesNameResult, bool> sourceDictionary)
{
foreach (var word in sourceDictionary)
{
if (table.SourceName == word.Key.SourceName)
{
return word.Value;
}
}
//
return false;
}
Simple: don't use .AsEnumerable(). That means "switch to LINQ-to-Objects". Before that, db.Stats was IQueryable<T>, which is a composable API, and would do what you expect.
That, however, means that you can't use C# methods like DictionaryFilter and DateFilter, and must instead compose things in terms of the Expression API. If you can illustrate what they do I can probably advise further.
With your edit, the filtering can be tweaked, for example:
static IQueryable<Stat> ErrorFilter(IQueryable<Stat> source, bool[] WarnLevel) {
// extract the enabled indices (match to values)
int[] levels = WarnLevel.Select((val, index) => new { val, index })
.Where(pair => pair.val)
.Select(pair => pair.index).ToArray();
switch(levels.Length)
{
case 0:
return source.Where(x => false);
case 1:
int level = levels[0];
return source.Where(x => x.WarnLevel == level);
case 2:
int level0 = levels[0], level1 = levels[1];
return source.Where(
x => x.WarnLevel == level0 || x.WarnLevel == level1);
default:
return source.Where(x => levels.Contains(x.WarnLevel));
}
}
the date filter is simpler:
static IQueryable<Stat> DateFilter(IQueryable<Stat> source,
DateTime begin, DateTime end)
{
var from = begin.ToFileTime(), to = end.ToFileTime();
return source.Where(table => table.RecordTime >= from
&& table.RecordTime <= to);
}
and the dictionary is a bit like the levels:
static IQueryable<Stat> DictionaryFilter(IQueryable<Stat> source,
Dictionary<GetSourcesNameResult, bool> sourceDictionary)
{
var words = (from word in sourceDictionary
where word.Value
select word.Key.SourceName).ToArray();
switch (words.Length)
{
case 0:
return source.Where(x => false);
case 1:
string word = words[0];
return source.Where(x => x.SourceName == word);
case 2:
string word0 = words[0], word1 = words[1];
return source.Where(
x => x.SourceName == word0 || x.SourceName == word1);
default:
return source.Where(x => words.Contains(x.SourceName));
}
}
and:
IQueryable<Stat> result = db.Stats;
result = ErrorFilter(result, WarnLevel);
result = DateFiter(result, beginTime, endTime);
result = DictionaryFilter(result, sourceDictionary);
// etc - note we're *composing* a filter here
var size = result.Count(); // size = 1007
var resultList = result.Skip((pageNumber-1)*pageSize).Take(pageSize).ToList();
return resultList;
The point is we're now using IQueryable<T> and Expression exclusively.
The following SO question might explain things: Understanding .AsEnumerable in Linq To Sql
.AsEnumerable() loads the entire table.
Related
I have a code that's working right now, but it doesn't check if the characters are in order, it only checks if they're there. How can I modify my code so the the characters 'gaoaf' are checked in that order in the string?
Console.WriteLine("5.feladat");
StreamWriter sw = new StreamWriter("keres.txt");
sw.WriteLine("gaoaf");
string s = "";
for (int i = 0; i < n; i++)
{
s = zadatok[i].nev+zadatok[i].cim;
if (s.Contains("g") && s.Contains("a") && s.Contains("o") && s.Contains("a") && s.Contains("f") )
{
sw.WriteLine(i);
sw.WriteLine(zadatok[i].nev + zadatok[i].cim);
}
}
sw.Close();
You can convert the letters into a pattern and use Regex:
var letters = "gaoaf";
var pattern = String.Join(".*",letters.AsEnumerable());
var hasletters = Regex.IsMatch(s, pattern, RegexOptions.IgnoreCase);
For those that needlessly avoid .*, you can also solve this with LINQ:
var ans = letters.Aggregate(0, (p, c) => p >= 0 ? s.IndexOf(c.ToString(), p, StringComparison.InvariantCultureIgnoreCase) : p) != -1;
If it is possible to have repeated adjacent letters, you need to complicate the LINQ solution slightly:
var ans = letters.Aggregate(0, (p, c) => {
if (p >= 0) {
var newp = s.IndexOf(c.ToString(), p, StringComparison.InvariantCultureIgnoreCase);
return newp >= 0 ? newp+1 : newp;
}
else
return p;
}) != -1;
Given the (ugly) machinations required to basically terminate Aggregate early, and given the (ugly and inefficient) syntax required to use an inline anonymous expression call to get rid of the temporary newp, I created some extensions to help, an Aggregate that can terminate early:
public static TAccum AggregateWhile<TAccum, T>(this IEnumerable<T> src, TAccum seed, Func<TAccum, T, TAccum> accumFn, Predicate<TAccum> whileFn) {
using (var e = src.GetEnumerator()) {
if (!e.MoveNext())
throw new Exception("At least one element required by AggregateWhile");
var ans = accumFn(seed, e.Current);
while (whileFn(ans) && e.MoveNext())
ans = accumFn(ans, e.Current);
return ans;
}
}
Now you can solve the problem fairly easily:
var ans2 = letters.AggregateWhile(-1,
(p, c) => s.IndexOf(c.ToString(), p+1, StringComparison.InvariantCultureIgnoreCase),
p => p >= 0
) != -1;
Why not something like this?
static bool CheckInOrder(string source, string charsToCheck)
{
int index = -1;
foreach (var c in charsToCheck)
{
index = source.IndexOf(c, index + 1);
if (index == -1)
return false;
}
return true;
}
Then you can use the function like this:
bool result = CheckInOrder("this is my source string", "gaoaf");
This should work because IndexOf returns -1 if a string isn't found, and it only starts scanning AFTER the previous match.
To preface this I am pulling records from a database. The CaseNumber column will have a unique identifier. However, multiple cases related to ONE Event will have very similar case numbers in which the last two digits will be the next following number. Example:
TR42X2330789
TR42X2330790
TR42X2330791
TR51C0613938
TR51C0613939
TR51C0613940
TR51C0613941
TR51C0613942
TR52X4224749
As you can see we would have to group these records into three groups. Currently my function is really messy and I it does not account for the scenario in which a group of case numbers is followed by another group of case numbers. I was wondering if anybody had any suggestions as to how to tackle this. I was thinking about putting all the case numbers in an array.
int i = 1;
string firstCaseNumber = string.Empty;
string previousCaseNumber = string.Empty;
if (i == 1)
{
firstCaseNumber = texasHarrisPublicRecordInfo.CaseNumber;
i++;
}
else if (i == 2)
{
string previousCaseNumberCode = firstCaseNumber.Remove(firstCaseNumber.Length - 3);
int previousCaseNumberTwoCharacters = Int32.Parse(firstCaseNumber.Substring(Math.Max(0, firstCaseNumber.Length - 2)));
string currentCaseNumberCode = texasHarrisPublicRecordInfo.CaseNumber.Remove(texasHarrisPublicRecordInfo.CaseNumber.Length - 3);
int currentCaselastTwoCharacters = Int32.Parse(texasHarrisPublicRecordInfo.CaseNumber.Substring(Math.Max(0, texasHarrisPublicRecordInfo.CaseNumber.Length - 2)));
int numberPlusOne = previousCaseNumberTwoCharacters + 1;
if (previousCaseNumberCode == currentCaseNumberCode && numberPlusOne == currentCaselastTwoCharacters)
{
//Group offense here
i++;
needNewCriminalRecord = false;
}
else
{
//NewGRoup here
}
previousCaseNumber = texasHarrisPublicRecordInfo.CaseNumber;
i++;
}
else
{
string beforeCaseNumberCode = previousCaseNumber.Remove(previousCaseNumber.Length - 3);
int beforeCaselastTwoCharacters = Int32.Parse(previousCaseNumber.Substring(Math.Max(0, previousCaseNumber.Length - 2)));
string currentCaseNumberCode = texasHarrisPublicRecordInfo.CaseNumber.Remove(texasHarrisPublicRecordInfo.CaseNumber.Length - 3);
int currentCaselastTwoCharacters = Int32.Parse(texasHarrisPublicRecordInfo.CaseNumber.Substring(Math.Max(0, texasHarrisPublicRecordInfo.CaseNumber.Length - 2)));
int numberPlusOne = beforeCaselastTwoCharacters + 1;
if (beforeCaseNumberCode == currentCaseNumberCode && numberPlusOne == currentCaselastTwoCharacters)
{
i++;
needNewCriminalRecord = false;
}
else
{
needNewCriminalRecord = true;
}
}
If you do not really care about performance you can use LINQ .GroupBy() and .ToDictionary() methods and create dictionary with lists. Something among the lines of :
string[] values =
{
"TR42X2330789",
"TR42X2330790",
"TR42X2330791",
"TR51C0613938",
"TR51C0613939",
"TR51C0613940",
"TR51C0613941",
"TR51C0613942",
"TR52X4224749"
};
Dictionary<string, List<string>> grouppedValues = values.GroupBy(v =>
new string(v.Take(9).ToArray()), // key - first 9 chars
v => v) // value
.ToDictionary(g => g.Key, g => g.ToList());
foreach (var item in grouppedValues)
{
Console.WriteLine(item.Key + " " + item.Value.Count);
}
Output :
TR42X2330 3
TR51C0613 5
TR52X4224 1
I would create a general puropose extension method:
static IEnumerable<IEnumerable<T>> GroupByConsecutiveKey<T, TKey>(this IEnumerable<T> list, Func<T, TKey> keySelector, Func<TKey, TKey, bool> areConsecutive)
{
using (var enumerator = list.GetEnumerator())
{
TKey previousKey = default(TKey);
var currentGroup = new List<T>();
while (enumerator.MoveNext())
{
if (!areConsecutive(previousKey, keySelector(enumerator.Current)))
{
if (currentGroup.Count > 0)
{
yield return currentGroup;
currentGroup = new List<T>();
}
}
currentGroup.Add(enumerator.Current);
previousKey = keySelector(enumerator.Current);
}
if (currentGroup.Count != 0)
{
yield return currentGroup;
}
}
}
And now you would use it like:
var grouped = data.GroupByConsecutiveKey(item => item, (k1, k2) => areConsecutive(k1, k2));
A quick hack for areConsecutive could be:
public static bool Consecutive(string s1, string s2)
{
if (s1 == null || s2 == null)
return false;
if (s1.Substring(0, s1.Length - 2) != s2.Substring(0, s2.Length - 2))
return false;
var end1 = s1.Substring(s1.Length - 2, 2);
var end2 = s2.Substring(s2.Length - 2, 2);
if (end1[1]!='0' && end2[1]!='0')
return Math.Abs((int)end1[1] - (int)end2[1]) == 1;
return Math.Abs(int.Parse(end1) - int.Parse(end2)) == 1;
}
Note that I am considering that Key can take any shape. If the alphanumeric code has the same pattern always then you can probably make this method a whole lot prettier or just use regular expressions.
I have a list which contains the name of suppliers. Say
SuppId Supplier Name
----------------------------------
1 Aardema & Whitelaw
2 Aafedt Forde Gray
3 Whitelaw & Sears-Ewald
using following LINQ query
supplierListQuery = supplierListQuery.Where(x => x.SupplierName.Contains(SearchKey));
I can return records correctly in the following conditions,
1) If i am using search string as "Whitelaw & Sears-Ewald" it will return 3rd record.
2) If i am using "Whitelaw" or "Sears-Ewald" it will return 3rd record.
But how can i return 3rd record if i am giving search string as "Whitelaw Sears-Ewald". It always returns 0 records.
Can i use ALL to get this result, but i dont know how to use it for this particular need.
What I usually do in this situation is split the words into a collection, then perform the following:
var searchopts = SearchKey.Split(' ').ToList();
supplierListQuery = supplierListQuery
.Where(x => searchopts.Any(y=> x.SupplierName.Contains(y)));
This works for me:
IEnumerable<string> keyWords = SearchKey.Split('');
supplierListQuery = supplierListQuery
.AsParallel()
.Where
(
x => keyWords.All
(
keyword => x.SupplierName.ContainsIgnoreCase(keyword)
)
);
Thank you all for your quick responses. But the one which worked or a easy fix to handle this was timothyclifford's note on this. Like he said i alterd my answer to this
string[] filters = SearchKey.ToLower().Split(new[] { ' ' });
objSuppliersList = (from x in objSuppliersList
where filters.All(f => x.SupplierName.ToLower().Contains(f))
select x).ToList();
Now it returns the result for all my serach conditions.
Because "Whitelaw" appears in both you will get both records. Otherwise there is no dynamic way to determine you only want the last one. If you know you only have these 3 then append .Last() to get the final record.
supplierListQuery = supplierListQuery.Where(x => x.SupplierName.Contains(SearchKey.Split(' ')[0]));
You need to use some sort of string comparer to create your own simple Search Engine and then you can find strings that are most likely to be included in your result :
public static class SearchEngine
{
public static double CompareStrings(string val1, string val2)
{
if ((val1.Length == 0) || (val2.Length == 0)) return 0;
if (val1 == val2) return 100;
double maxLength = Math.Max(val1.Length, val2.Length);
double minLength = Math.Min(val1.Length, val2.Length);
int charIndex = 0;
for (int i = 0; i < minLength; i++) { if (val1.Contains(val2[i])) charIndex++; }
return Math.Round(charIndex / maxLength * 100);
}
public static List<string> Search(this string[] values, string searchKey, double threshold)
{
List<string> result = new List<string>();
for (int i = 0; i < values.Length; i++) if (CompareStrings(values[i], searchKey) > threshold) result.Add(values[i]);
return result;
}
}
Example of usage :
string[] array = { "Aardema & Whitelaw", "Aafedt Forde Gray", "Whitelaw & Sears-Ewald" };
var result = array.Search("WhitelawSears-Ewald", 80);
// Results that matches this string with 80% or more
foreach (var item in result)
{
Console.WriteLine(item);
}
Output: Whitelaw & Sears-Ewald
If you want an easy (not very handy) solution,
var result = supplierListQuery
.Select(x => normalize(x.SupplierName))
.Where(x => x.Contains(normalize(SearchKey)));
string normalize(string inputStr)
{
string retVal = inputStr.Replace("&", "");
while (retVal.IndexOf(" ") >= 0)
{
retVal = retVal.Replace(" ", " ");
}
return retVal;
}
Provided a list, I want to select all items between the 2 given. (including the begin and end params)
My current solution is as follows:
private IEnumerable<string> GetAllBetween(IEnumerable<string> list, string begin, string end)
{
bool isBetween = false;
foreach (string item in list)
{
if (item == begin)
{
isBetween = true;
}
if (item == end)
{
yield return item;
yield break;
}
if (isBetween)
{
yield return item;
}
}
}
But surely there must be a pretty linq query that accomplishes the same thing?
You can nearly use SkipWhile and TakeWhile, but you want the last item as well - you want the functionality of TakeUntil from MoreLINQ. You can then use:
var query = source.SkipWhile(x => x != begin)
.TakeUntil(x => x == end);
static IEnumerable<T> GetAllBetween<T>( this List<T> list, T a, T b )
{
var aOffset = list.IndexOf( a );
var bOffset = list.IndexOf( b );
// what to do if one or all items not found?
if( -1 == aOffset || -1 == bOffset )
{
// for this example I will return an empty array
return new T[] { };
}
// what to do if a comes after b?
if( aOffset > bOffset )
{
// for this example i'll simply swap them
int temp = aOffset;
aOffset = bOffset;
bOffset = temp;
}
return list.GetRange( aOffset, bOffset - aOffset );
}
I think a simple Skip, Take should do it. I normally take it for paging ASP.NET resultsites.
var startIndex = list.IndexOf(begin);
var endIndex = list.IndexOf(end);
var result = list.Skip(startIndex + 1).Take(endIndex - 1 - startIndex);
Many thanks to leppie:
Currently I got
Expression<Func<vwMailMerge,bool>> whereClause= null;
List<vwMailMerge> mailMergeItems = null;
int personType = mailMergeSettings.PersonType.ToInteger();
if (personType > 0)
{
whereClause = this.MailMergeWhereClause(whereClause, f => f.MemberTypeId == personType);
}
if (mailMergeSettings.PersonIds != null)
{
var personIds = mailMergeSettings.PersonIds.ToGuidArray();
if (personIds != null && personIds.Length > 0)
{
var personList = personIds.ToList();
whereClause = this.MailMergeWhereClause(whereClause, f => personList.Contains(f.UserId));
}
}
mailMergeItems = this.ObjectContext.vwMailMerges.Where(whereClause).ToList();
private Expression<Func<vwMailMerge, bool>> MailMergeWhereClause(params Expression<Func<vwMailMerge, bool>>[] wheres)
{
if (wheres.Length == 0)
{
return x => true;
}
Expression result;
if (wheres[0] == null)
{
result = wheres[1].Body;
return Expression.Lambda<Func<vwMailMerge, bool>>(result, wheres[1].Parameters);
}
else
{
result = wheres[0].Body;
for (int i = 1; i < wheres.Length; i++)
{
result = Expression.And(result, wheres[i].Body);
}
return Expression.Lambda<Func<vwMailMerge, bool>>(result, wheres[0].Parameters);
}
}
}
When it gets to "mailMergeItems =" it drops and gives error: "The parameter 'f' was not bound in the specified LINQ to Entities query expression."
I've noticed that when checking only for people, or only for membertypeId, it works properly.. but combined the 2nd gives a error on it's "f=>" I think.
You cant use Func, you need to use Expression<Func>.
The + can be done via Expression.And.
Update (not tested):
Expression<Func<vwMailMerge, bool>> whereClause = null;
...
Expression<Func<vwMailMerge, bool>> MailMergeWhereClause(
params Expression<Func<vwMailMerge, bool>>[] wheres)
{
if (wheres.Length == 0) return x => true;
Expression result = wheres[0].Body;
for (int i = 1; i < wheres.Length; i++)
{
//probaby needs a parameter fixup, exercise for reader
result = Expression.And(result, wheres[i].Body);
}
return Expression.Lambda<Func<vwMailMerge,bool>>(result, wheres[0].Parameters);
}
Update 2:
The above approach fails as I expected. It might be easy to solve on .NET 4 using the ExpressionVistor class. For .NET 3.5 (or if aforementioned is too hard) the following should work.
The approach is the append the where clauses in the IQueryable directly so you end up with:
somequery.Where(x => x.foo).Where(x => x.bar).Where(x => x.baz)
IOW, you can just add them as required, but it will require some changes to the logic/flow of the code you pasted.
You could reformat your question better with the code tool.
However it looks like you could approach the problem in this way to avoid all those func expressions floating around:
this.ObjectContext.vwMailMerges.Where(mm=>IsValidMailMerge(mm,personType)).ToList()
private bool IsValidMailMerge(YourType mailmerge, YourType2 personType)
{
if(...) // type specific criteria here
return true;
else
return false;
}