I have a Frequently Asked Question (FAQ) database with columns: id, Question, Answer, Category, and Keywords.
I want to take input from a user and search my database for matches. I ultimately want to take the search string and retrieve all records where any of the search string is found in either the Question or Keyword columns.
Since I'm relatively new at Linq to Sql, I'm trying to get just a single column search working first, and then try to get to a double column search. My code is still failing. I've seen other posts similar the topic, but the recommended solutions do not work exactly for my situation. Attempts to tweak those solutions have failed.
Problematic portion of my code:
private FAQViewModel getSearchResultFAQs(string search)
{
FAQViewModel vm = new FAQViewModel();
vm.isSearch = true;
string[] searchTerms = search.Split(new[] { ' ' }, StringSplitOptions.RemoveEmptyEntries);
//var ques = db.FAQs.Where(q => q.Keywords.Intersect(searchTerms).Any());
var ques = db.FAQs.Where(q => searchTerms.Any(t => q.Keywords.Contains(t)));
List<FAQ> list = new List<FAQ>();
foreach(var qs in ques)
{
FAQ f = new FAQ();
f.Question = qs.Question;
f.Answer = qs.Answer;
f.Category = qs.Category;
f.Keywords = qs.Keywords;
f.id = qs.id;
list.Add(f);
}
CategoryModel cm = new CategoryModel();
cm.faqs = list;
vm.faqs.Add(cm);
return vm;
}
This code fails. q.Keywords is underlined in red stating
string does not contain a definition for 'Intersect'
Any assistance would be greatly appreciated.
EDIT:
I commented out the bad line of code and used Gilad's first recommendation. I now get the following error:
"Local sequence cannot be used in LINQ to SQL implementations of query operators except the Contains operator."
It doesn't like the foreach codeblock:
foreach(var qs in ques)
I'm really out of my depth here but its so close to working.
Related
I have a document that looks essentially like this:
{
"Name": "John Smith",
"Value": "SomethingIneed",
"Tags: ["Tag1" ,"Tag2", "Tag3"]
}
My goal is to write a query where I find all documents in my database whose Tag property contains all of the tags in a filter.
For example, in the case above, my query might be ["Tag1", "Tag3"]. I want all documents whose tags collection contains Tag1 AND Tag3.
I have done the following:
tried an All Contains type linq query
var tags = new List<string>() {"Test", "TestAccount"};
var req =
Client.CreateDocumentQuery<Contact>(UriFactory.CreateDocumentCollectionUri("db", "collection"))
.Where(x => x.Tags.All(y => tags.Contains(y)))
.ToList();
Created a user defined function (I couldn't get this to work at all)
var tagString = "'Test', 'TestAccount'";
var req =
Client.CreateDocumentQuery<Contact>(UriFactory.CreateDocumentCollectionUri("db", "collection"),
$"Select c.Name, c.Email, c.id from c WHERE udf.containsAll([${tagString}] , c.Tags)").ToList();
with containsAll defined as:
function arrayContainsAnotherArray(needle, haystack){
for(var i = 0; i < needle.length; i++){
if(haystack.indexOf(needle[i]) === -1)
return false;
}
return true;
}
Use System.Linq.Dynamic to create a predicate from a string
var query = new StringBuilder("ItemType = \"MyType\"");
if (search.CollectionValues.Any())
{
foreach (var searchCollectionValue in search.CollectionValues)
{
query.Append($" and Collection.Contains(\"{searchCollectionValue}\")");
}
}
3 actually worked for me, but the query was very expensive (more than 2000 RUs on a collection of 10K documents) and I am getting throttled like crazy. My result set for the first iteration of my application must be able to support 10K results in the result set. How can I best query for a large number of results with an array of filters?
Thanks.
The UDF could be made to work but it would be a full table scan and so not recommended unless combined with other highly-selective criteria.
I believe the most performant (index-using) approach would be to split it into a series of AND statements. You could do this programmatically building up your query string (being careful to fully escape and user-provided data for security reasons). So, the resulting query would look like:
SELECT *
FROM c
WHERE
ARRAY_CONTAINS(c.Tags, "Tag1") AND
ARRAY_CONTAINS(c.Tags, "Tag3")
i got a piece of code to add filter with Lucene.net but good explanation was not there to understand the code. so here i paste the code for explanation.
List<SearchResults> Searchresults = new List<SearchResults>();
string indexFileLocation = #"C:\o";
Lucene.Net.Store.Directory dir = Lucene.Net.Store.FSDirectory.GetDirectory(indexFileLocation);
string[] searchfields = new string[] { "fname", "lname", "dob", "id"};
IndexSearcher indexSearcher = new IndexSearcher(dir);
Filter fil= new QueryWrapperFilter(new TermQuery( new Term(field, "5/12/1998")));
var hits = indexSearcher.Search(QueryMaker(searchString, searchfields), fil);
for (int i = 0; i < hits.Length(); i++)
{
SearchResults result = new SearchResults();
result.fname = hits.Doc(i).GetField("fname").StringValue();
result.lname = hits.Doc(i).GetField("lname").StringValue();
result.dob = hits.Doc(i).GetField("dob").StringValue();
result.id = hits.Doc(i).GetField("id").StringValue();
Searchresults.Add(result);
}
i need explanation for the below two line
Filter fil= new QueryWrapperFilter(new TermQuery( new Term(field, "5/12/1998")));
var hits = indexSearcher.Search(QueryMaker(searchString, searchfields), fil);
i just like to know first lucene search & pull all data and after implement filter or from the beginning lucene pull data based on filter? please guide. thanks.
i just like to know first lucene search & pull all data and after implement filter or from the beginning lucene pull data based on filter? please guide. thanks.
Lucene.Net will perform your search AND your filtered query and after it, it will "merge" the result. The reason to do it I believe is to cache the filtered query, because it will be more likely to have a hit on the next time than the search query.
We are using Full Text Index Searching on a Company Name field.
We are using EF for the data layer, and I have been asked to not use stored procs.
Here is the method in my Data Access layer:
public Task<List<Company>> SearchByName(string searchText)
{
return DataContext.Company.SqlQuery(
"select CompanyId AS Id, * from Company.Company AS c where contains(c.Name, #SearchText)",
new SqlParameter("#SearchText", ParseSearchTextForMultiwordSearch(searchText)))
.ToListAsync();
}
We wanted to split the words out in the search and then concatonate them together for an AND search. This means that a query like "My Company" would actually be searched against the index for the words "My" and "Company".
This code does the merging of terms for the select query above.
public string ParseSearchTextForMultiwordSearch(string searchText)
{
var words = GetValidSearchTerms(searchText);
var quotedWords = words.Select(x => string.Format("\"{0}*\"", x));
return string.Join(" AND ", quotedWords);
}
Everything works great until you start adding in "key words". So far, we have figured out and, or, or not included in a search return 0 results. There is no error, there just are no results.
Here is our method that "blacklists" certain words so they are left out of the search query.
private static List<string> GetValidSearchTerms(string searchText)
{
//AND and OR are keywords used by SQL Server Full Text Indexing.
var blacklist = new string[] {
"and",
"or",
"not"
};
//Filter them out here
var words = searchText.Split(' ');
var validWords = words.Where(x => !blacklist.Contains(x));
return validWords.ToList();
}
The problem is that we just discovered another "keyword" that seems to be causing the issue. "Do" causes no results to come back. I can just dd it to the blacklist, but as this thing grows it is starting to feel like the wrong way to handle this.
Is there a better way to handle this?
EDIT:
A couple other scenarios
If I do not massage the search string at all, searching on the word "not" causes an error "Null or empty full-text predicate."
Same scenario, just applying the string as is, if I make a company "Company Do Not Delete", any versions of the string that have Do or Not in them return 0 results.
It's been a while since I posted this question and after a few iterations I came up with some search logic that works for our needs.
First off, we have a business rule that requires searching to include an ampersand. Full Text indexing seems to drop off the &, making the results that return incorrect. So, I had to special case any & searches to use a like statement instead.
I left my code to do do as above where it parses out a blacklist of words and tries a CONTAINS search. If that fails for any reason, I perform a FREETEXT search instead.
public async Task<List<Company>> SearchByName(string searchText)
{
var results = new List<Company>();
if (string.IsNullOrWhiteSpace(searchText))
return results;
if (searchText.IndexOf("&") >= 0)
{
var likeQuery = string.Format("%{0}%", searchText);
results = await DataContext.Company.SqlQuery("SELECT CompanyId AS Id, IsEligible AS IsReadOnly, *" +
" FROM Company.Company AS con" +
" WHERE con.Name LIKE #SearchText",
new SqlParameter("#SearchText", likeQuery))
.ToListAsync();
}
else
{
var terms = ParseSearchTextForMultiwordSearch(searchText);
if (string.IsNullOrWhiteSpace(terms))
return results;
// SqlQuery does not take any column mappings into account (https://entityframework.codeplex.com/workitem/233)
// So we have to manually map the columns in the select statement
var sqlQueryFormat = "SELECT CompanyId AS Id, IsEligible AS IsReadOnly, *" +
" FROM Company.Company AS con" +
" WHERE {0}(con.Name, #SearchText)";
var sqlQuery = string.Format(sqlQueryFormat, "CONTAINS");
var errored = false;
try
{
results = await DataContext.Company.SqlQuery(sqlQuery,
new SqlParameter("#SearchText", terms))
.ToListAsync();
}
catch
{
//catch the error but do nothing with it
errored = true;
}
//when the contains search fails due to some unknown error, use Freetext as a backup
if (errored)
{
sqlQuery = string.Format(sqlQueryFormat, "FREETEXT");
results = await DataContext.Company.SqlQuery(sqlQuery,
new SqlParameter("#SearchText", terms))
.ToListAsync();
}
}
return results;
}
I have a string list(A) of individualProfileId's (GUID) that can be in any order(used for displaying personal profiles in a specific order based on user input) which is stored as a string due to it being part of the cms functionality.
I also have an asp c# Repeater that uses a LinqDataSource to query against the individual table. This repeater needs to use the ordered list(A) to display the results in the order specified.
Which is what i am having problems with. Does anyone have any ideas?
list(A)
'CD44D9F9-DE88-4BBD-B7A2-41F7A9904DAC',
'7FF2D867-DE88-4549-B5C1-D3C321F8DB9B',
'3FC3DE3F-7ADE-44F1-B17D-23E037130907'
Datasource example
IndividualProfileId Name JobTitle EmailAddress IsEmployee
3FC3DE3F-7ADE-44F1-B17D-23E037130907 Joe Blo Director dsd#ad.com 1
CD44D9F9-DE88-4BBD-B7A2-41F7A9904DAC Maxy Dosh The Boss 1
98AB3AFD-4D4E-4BAF-91CE-A778EB29D959 some one a job 322#wewd.ocm 1
7FF2D867-DE88-4549-B5C1-D3C321F8DB9B Max Walsh CEO 1
There is a very simple (single-line) way of doing this, given that you get the employee results from the database first (so resultSetFromDatabase is just example data, you should have some LINQ query here that gets your results).
var a = new[] { "GUID1", "GUID2", "GUID3"};
var resultSetFromDatabase = new[]
{
new { IndividualProfileId = "GUID3", Name = "Joe Blo" },
new { IndividualProfileId = "GUID1", Name = "Maxy Dosh" },
new { IndividualProfileId = "GUID4", Name = "some one" },
new { IndividualProfileId = "GUID2", Name = "Max Walsh" }
};
var sortedResults = a.Join(res, s => s, e => e.IndividualProfileId, (s, e) => e);
It's impossible to have the datasource get the results directly in the right order, unless you're willing to write some dedicated SQL stored procedure. The problem is that you'd have to tell the database the contents of a. Using LINQ this can only be done via Contains. And that doesn't guarantee any order in the result set.
Turn the list(A), which you stated is a string, into an actual list. For example, you could use listAsString.Split(",") and then remove the 's from each element. I’ll assume the finished list is called list.
Query the database to retrieve the rows that you need, for example:
var data = db.Table.Where(row => list.Contains(row.IndividualProfileId));
From the data returned, create a dictionary keyed by the IndividualProfileId, for example:
var dic = data.ToDictionary(e => e.IndividualProfileId);
Iterate through the list and retrieve the dictionary entry for each item:
var results = list.Select(item => dic[item]).ToList();
Now results will have the records in the same order that the IDs were in list.
I'm using LINQ to SQL to pull records from a database, sort them by a string field, then perform some other work on them. Unfortunately the Name field that I'm sorting by comes out of the database like this
Name
ADAPT1
ADAPT10
ADAPT11
...
ADAPT2
ADAPT3
I'd like to sort the Name field in numerical order. Right now I'm using the Regex object to replace "ADAPT1" with "ADAPT01", etc. I then sort the records again using another LINQ query. The code I have for this looks like
var adaptationsUnsorted = from aun in dbContext.Adaptations
where aun.EventID == iep.EventID
select new Adaptation
{
StudentID = aun.StudentID,
EventID = aun.EventID,
Name = Regex.Replace(aun.Name,
#"ADAPT([0-9])$", #"ADAPT0$1"),
Value = aun.Value
};
var adaptationsSorted = from ast in adaptationsUnsorted
orderby ast.Name
select ast;
foreach(Adaptation adaptation in adaptationsSorted)
{
// do real work
}
The problem I have is that the foreach loop throws the exception
System.NotSupportedException was unhandled
Message="Method 'System.String Replace(System.String, System.String,
System.String)' has no supported translation to SQL."
Source="System.Data.Linq"
I'm also wondering if there's a cleaner way to do this with just one LINQ query. Any suggestions would be appreciated.
Force the hydration of the elements by enumerating the query (call ToList). From that point on, your operations will be against in-memory objects and those operations will not be translated into SQL.
List<Adaptation> result =
dbContext.Adaptation
.Where(aun => aun.EventID = iep.EventID)
.ToList();
result.ForEach(aun =>
aun.Name = Regex.Replace(aun.Name,
#"ADAPT([0-9])$", #"ADAPT0$1")
);
result = result.OrderBy(aun => aun.Name).ToList();
Implement a IComparer<string> with your logic:
var adaptationsUnsorted = from aun in dbContext.Adaptations
where aun.EventID == iep.EventID
select new Adaptation
{
StudentID = aun.StudentID,
EventID = aun.EventID,
Name = aun.Name,
Value = aun.Value
};
var adaptationsSorted = adaptationsUnsorted.ToList<Adaptation>().OrderBy(a => a.Name, new AdaptationComparer ());
foreach (Adaptation adaptation in adaptationsSorted)
{
// do real work
}
public class AdaptationComparer : IComparer<string>
{
public int Compare(string x, string y)
{
string x1 = Regex.Replace(x, #"ADAPT([0-9])$", #"ADAPT0$1");
string y1 = Regex.Replace(y, #"ADAPT([0-9])$", #"ADAPT0$1");
return Comparer<string>.Default.Compare(x1, y1);
}
}
I didn't test this code but it should do the job.
I wonder if you can add a calculated+persisted+indexed field to the database, that does this for you. It would be fairly trivial to write a UDF that gets the value as an integer (just using string values), but then you can sort on this column at the database. This would allow you to use Skip and Take effectively, rather than constantly fetching all the data to the .NET code (which simply doesn't scale).