We Keep Coding

natural language query processing

I have a NLP (natural language processing application) running that gives me a tree of the parsed sentence, the questions is then how should I proceed with that.
What is the time
\-SBAR - Suborginate clause
|-WHNP - Wh-noun phrase
| \-WP - Wh-pronoun
| \-What
\-S - Simple declarative clause
\-VP - Verb phrase
|-VBZ - Verb, 3rd person singular present
| \-is
\-NP - Noun phrase
|-DT - Determiner
| \-the
\-NN - Noun, singular or mass
\-time
the application has a build in javascript interpreter, and was trying to make the phrase in to a simple function such as
function getReply() {
return Resource.Time();
}
in basic terms, what = request = create function, is would be the returned object, and the time would reference the time, now it would be easy just to make a simple parser for that but then we also have what is the time now, or do you know what time it is. I need it to be able to be further developed based on the english language as the project will grow.
the source is C# .Net 4.5
thanks in advance.

As far as I can see, using dependency parse trees will be more helpful. Often, the number of ways a question is asked is limited (I mean statistically significant variations are limited ... there will probably be corner cases that people ordinarily do not use), and are expressed through words like who, what, when, where, why and how.
Dependency parsing will enable you to extract the nominal subject and the direct as well as indirect objects in a query. Typically, these will express the basic intent of the query. Consider the example of tow equivalent queries:
What is the time?
Do you know what the time is?
Their dependency parse structures are as follows:
root(ROOT-0, What-1)
cop(What-1, is-2)
det(time-4, the-3)
nsubj(What-1, time-4)
and
aux(know-3, Do-1)
nsubj(know-3, you-2)
root(ROOT-0, know-3)
dobj(is-7, what-4)
det(time-6, the-5)
nsubj(is-7, time-6)
ccomp(know-3, is-7)
Both are what-queries, and both contain "time" as a nominal subject. The latter also contains "you" as a nominal subject, but I think expressions like "do you know", "can you please tell me", etc. can be removed based on heuristics.
You will find the Stanford Parser helpful for this approach. They also have this online demo, if you want to see some more examples at work.

How to configure tolkenizers with indexing and searching with Lucene and Nhibernate

This is a question for using Lucene via the NHibernate.Search namespace, which works in conjunction with Lucene.
I'm indexing a Title in the Index: Grey's Anatomy
Title : "Grey's Anatomy"
By using Luke, I see that that title is getting Tokenized into:
Title: anatomy
Title: grey
Now, I get a result if I search for:
"grey" or "grey's"
However, if I search for "greys" then I get nothing.
I would like "greys" to return a result. And I guess this could be an issue with any word with an apostrophe.
So, here are some questions:
Am I right in thinking I could fix this issue either by changing something on the time of index (so, changing the tolkenizer..??) or changing it a query time (query parser?)
If there is a solution, could someone provide a small code sample?
thanks

If you make a classic Term search using Lucene, then greys it's most likely not to show on the results, except that you make a nice tokenizing work when saving, so from where I see it, you have 2 choices or a 3rd beign a combination of them:
Use a Stemmer for indexed data and query. Stemmers are fast, and you can always find an implementation of Porter's stemmer somewhere in Google. Problem is when you look for different languages.
Use Fuzzy queries. Using a Fuzzy Query you can set the edit distance that you want to get "away" from the word being search. The thing is that because 2 words are "close" using an edition distance (i.e, Lehvenstein) doesn't mean that they're the same, but the problem of Grey and Grey's and Greys should be solved with setting an edit distance of 2.
I think you will be able to find a decent implementation of the Porter Stemmer, which is nice right here.
Hope I can help!

Configurable rule based system in C#

I have an algorithm that returns a list of classifications(strings) dependant on the two arguments given to the algorithm: a type variable, and an extra category string that allows certain special classifications to be added to the result list.
The current implementation, is unreadable and unscalable due to the expression of the rules as ifs, and switch statements. Also the rules are hard coded.
A simplified version of the code:
private static List<string> DetermineTypes(Type x, object category) {
List<string> Types = new List<string>();
if (category is DateTime) {
types.Add("1");
types.Add("2");
types.Add("3");
} else if (category is string) {
switch ((string)category) {
case "A":
Types.Add("4");
break;
case "B":
case "C":
case "D":
Types.Add("5");
break;
case "":
Types = DetermineTypesFromX(Types, x);
break;
default:
Types.Add("6");
break;
}
}
return graphTypes;
}
private static List<string> DetermineTypesFromX(List<string> Types, Type x) {
if (x.Equals(typeof(int))) {
Types.Add("7");
} else if (x.Equals(typeof(double))) {
Types.Add("8");
} else if (x.Equals(typeof(System.DateTime))) {
Types.Add("9");
Types.Add("10");
}
return Types;
}
I was thinking that it would be good to maybe specify these with xml, so that a code change wasn't needed for new types/rules, but that is most probably too heavyweight for the situation. Basically I am trying to solve the problem that a new 'Type' may be added at anytime: common case would be that it is one of the 'rules' above, and an unlikely edge case that a new 'rule' branch may have to be added.
I am still to determine whether the work needed it to make it fully dynamic using xml defined rules( or any other way) is worth it compared to the likelihood of the edge cases ever happening and the business environment(schedules etc).
But my main point of the question is how could you elegantly simplify the nested conditional code above? maybe incorporating more flexibility into the design for increased scalability?
I was wondering if using a combination of F# pattern matching might be an appropriate solution? (NB: Never used F# before, have been curious as of late, so thats why I am asking)

A pattern known as dispatch tables has been recently discussed in the following two blog posts and will probably be of interest to you:
Aaron Feng
K. Scott Allen

I wouldn't shy away from a config-based option; it usually has the advantage of not requiring a rebuild. If you don't want that, another option might be type-metadata via an attribute. This would make it trivial to add data for new types (that you write), and you can (indirectly) add attributes to exiting types (int etc) via TypeDescriptor.AddAttributes - as long as you use TypeDescriptor.GetAttributes to get them back out again ;-p
Whether this is a good idea or not... well, reflection (and the twin, TypeDescriptor) can be slow, so if you want to use this in a tight loop I'd look first at something involving a dictionary.

Your problem may be coded in terms of decision tree or decision table
Also, there is posts into Chris Smith's blog about decision trees:
Awesome F# - Decision Trees – Part I and
Awesome F# - Decision Trees – Part II

I would suggest you look at a business rules/inference engine. NxBRE has a good community around it and is quite mature. This may be beyond your immediate requirements but if you expect these rules to increase in complexity over time a BRE will provide a nice framework to keep things under control.

Since you mention F#, here is some F# code with very similar behavior to the C# code:
open System
let DetermineTypesFromX(x:Type) =
if x.Equals(typeof<int>) then
["7"]
elif x.Equals(typeof<double>) then
["8"]
elif x.Equals(typeof<DateTime>) then
["9"; "10"]
else
[]
let DetermineTypes(x:Type, category:obj) =
match category with
| :? DateTime -> ["1"; "2"; "3"]
| :? string as s ->
match s with
| "A" -> ["4"]
| "B" | "C" | "D" -> ["5"]
| "" -> DetermineTypesFromX(x)
| _ -> ["6"]
| _ -> []
That said, I would recommend considering a table-driven approach as an alternative to hard-coded if/switch logic, regardless of whether you move the logic out of the code and into a config file.

I came across similar situation and I've asked a few questions previously in regards to the similar problem that may help you.
The system I did was a configuration driven, rule based dynamic system. All configurations and rules were saved in database. Decision tables were constructed dynamically based on the values and rules retrived from database. Values were then converted and compared in C#. Here's the question I asked about dynamic decision table in C#. And the question regarding dyanmically convert and compare values retrived from databse.
So I end up having something simliar to this in terms of the config table (just an example):
Conditions IsDecision LHS Operator RHS
TTFF False PostCode > 100
TFTF False PostCode < 10000
FTTT True
Note: LHS is the property name of the object.
The above table in plain English:
Condition 1 PostCode > 100 Yes Yes No No
Condition 2 PostCode < 10000 Yes No Yes No
Outcome 1 Yes
Outcome 2 Yes
Outcome 3 Yes
Outcome 4 Yes
Then you have other tables/configs to determine the action for each outcome.
The core parts of the implementation are how to dynamically construct decision table and how to dynamic convert and compare string values, all of which I have provided links to the specific implementations in the above paragraph. I believe you can apply similar concepts in your situations and I hope I've explained the concept in general.
Other Resources:
Martin Fowler's decision tree article.
Luke's post on decision tree.

SubSonic RESTHandler Question

I'm playing with the SubSonic RESTHandler for the first time and it's awesome... There is one quirk tho, that I'm curious about.
RESTHandler.cs (line 319):
//if this column is a string, by default do a fuzzy search
if(comp == Comparison.Like || column.IsString)
{
comp = Comparison.Like;
paramValue = String.Concat("%", paramValue, "%");
}
This little blurp of code forces all searches on string columns to wildcard searches by default. This seems counter intutive, since you've provided a nice set of comparisons we can add to a parameter (_is, _notequal, etc...). Is there a reason this was done? The EvalComparison uses "Comparison.Equals" as it's default, so unless a like is explicitly needed the " || column.IsString" looks like it should be removed since it breaks the ability to use different types of comparisons.
This was driving me crazy, since you can't do a "WHERE Field = X" without modifiying code...
Just curious if this is more of a feature than a bug...
Thanks!
Zach

It's because this is a LIKE operation which for a DB usually allows string operations. The feeling at the time was that if you wanted equals you could just use that.
It's been a while since I've touched this code - if you'd be kind enough to open a bug I'll take a look at it.

It does indeed look like a feature. It's based on the idea that, if I am searching for a string in a column without the wildcards, I must match the string exactly or I get no hits. I suspect that this was done to make programming search textboxes easier.

Determining if enum value is in list (C#)

I am building a fun little app to determine if I should bike to work.
I would like to test to see if it is either Raining or Thunderstorm(ing).
public enum WeatherType : byte
{ Sunny = 0, Cloudy = 1, Thunderstorm = 2, Raining = 4, Snowing = 8, MostlyCloudy = 16 }
I was thinking I could do something like:
WeatherType _badWeatherTypes = WeatherType.Thunderstorm | WeatherType.Raining;
if(currentWeather.Type == _badWeatherTypes)
{
return false;//don't bike
}
but this doesn't work because _badWeatherTypes is a combination of both types. I would like to keep them separated out because this is supposed to be a learning experience and having it separate may be useful in other situations (IE, Invoice not paid reason's etc...).
I would also rather not do: (this would remove the ability to be configured for multiple people)
if(WeatherType.Thunderstorm)
{
return false; //don't bike
}
etc...

Your current code will say whether it's exactly "raining and thundery". To find out whether it's "raining and thundery and possibly something else" you need:
if ((currentWeather.Type & _badWeatherTypes) == _badWeatherTypes)
To find out whether it's "raining or thundery, and possibly something else" you need:
if ((currentWeather.Type & _badWeatherTypes) != 0)
EDIT (for completeness):
It would be good to use the FlagsAttribute, i.e. decorate the type with [Flags]. This is not necessary for the sake of this bitwise logic, but affects how ToString() behaves. The C# compiler ignores this attribute (at least at the moment; the C# 3.0 spec doesn't mention it) but it's generally a good idea for enums which are effectively flags, and it documents the intended use of the type. At the same time, the convention is that when you use flags, you pluralise the enum name - so you'd change it to WeatherTypes (because any actual value is effectively 0 or more weather types).
It would also be worth thinking about what "Sunny" really means. It's currently got a value of 0, which means it's the absence of everything else; you couldn't have it sunny and raining at the same time (which is physically possible, of course). Please don't write code to prohibit rainbows! ;) On the other hand, if in your real use case you genuinely want a value which means "the absence of all other values" then you're fine.

I'm not sure that it should be a flag - I think that you should have an range input for:
Temperature
How much it's raining
Wind strength
any other input you fancy (e.g. thunderstorm)
you can then use an algorithm to determine if the conditions are sufficiently good.
I think you should also have an input for how likely the weather is to remain the same for cycling home. The criteria may be different - you can shower and change more easliy when you get home.
If you really want to make it interesting, collect the input data from a weather service api, and evaulate the decision each day - Yes, I should have cycled, or no, it was a mistake. Then perhaps you can have the app learn to make better decisions.
Next step is to "socialize" your decision, and see whether other people hear you are making the same decisions.

use the FlagsAttribute. That will allow you to use the enum as a bit mask.

You need to use the [Flags] attribute (check here) on your enum; then you can use bitwise and to check for individual matches.

You should be using the Flags attribute on your enum. Beyond that, you also need to test to see if a particular flag is set by:
(currentWeather.Type & WeatherType.Thunderstorm == WeatherType.Thunderstorm)
This will test if currentWeather.Type has the WeatherType.Thunderstorm flag set.

I wouldn't limit yourself to the bit world. Enums and bitwise operators are, as you found out, not the same thing. If you want to solve this using bitwise operators, I'd stick to just them, i.e. don't bother with enums. However, I'd something like the following:
WeatherType[] badWeatherTypes = new WeatherType[]
{
WeatherType.Thunderstorm,
WeatherType.Raining
};
if (Array.IndexOf(badWeatherTypes, currentWeather.Type) >= 0)
{
return false;
}