We Keep Coding

I have a list with over a million objects in it, trying to find the fastest way to search through it

I have a list that stores well over a million objects within it. I need to look through the list and update the objects found through the below query as efficiently as possible.
I was thinking of using a Dictionary or HashSet, but I'm relatively new to C# and could not figure out how to implement these other two approaches. My current code is simply a LINQ statement searching through an IList.
public IList<LandObject> landObjects = new List<LandObject>();
var lObjsToUpdate = from obj in landObjects
where
obj.position.x >= land.x - size.x &&
obj.position.x <= land.x + size.x &&
obj.position.y >= land.y - size.y &&
obj.position.y <= land.y + size.y
select obj;
foreach(var item in lObjsToUpdate)
{
//do what I need to do with records I've found
}
Could anyone be so kind as to suggest how I might approach this efficiently?

The real answer should involve performance tests and comparisons, and depends on your runtime environment (memory vs. cpu, etc).
The first thing I would try, since land and size are constant, you can maintain a simple HashSet<LandObject> of objects that fit the criteria (in addition to a list or set of all objects or just all other objects). Every time a new object is added, check if it fits the criteria and if so - add it to that set of objects. I don't know how good HashSet is at avoiding collisions when working with object references, but you can try to measure it.
BTW, this related question discussing memory limits of HashSet<int> might interest you. With .NET < 4.5 your HashSet should be ok up to several million items. From what I understand, with some configuration .NET 4.5 removes the limitation of 2gb max object size and you'll be able to go crazy, assuming you have a 64-bit machine.

One thing that will probably help with that many iterations is do the calculations once and use different variables inside your query. Also it should help some to increase the range by 1 on each end and eliminate checking for equals:
public IList<LandObject> landObjects = new List<LandObject>();
float xmax = land.x + size.x + 1;
float xmin = land.x - size.x - 1;
float ymax = land.y + size.y + 1;
float ymin = land.y - size.y - 1;
var lObjsToUpdate = from obj in landObjects
where
obj.position.x > xmin &&
obj.position.x < xmax &&
obj.position.y > ymin &&
obj.position.y < ymax
select obj;

Ideally you need a way to partition elements so that you don't have to test every single one to find the ones that fit and the ones that should be thrown out. How you partition will depend on how dense the items are - it might be as simple as partitioning on the integer portion of the X coordinate for instance, or by some suitable scaled value of that coordinate.
Given a method (let's call it Partition for now) that takes an X coordinate and returns a partition value for it, you can filter on the X coordinate fairly quickly as a first-pass to reduce the total number of nodes you need to check. You might need to play with the partition function a little to get the right distribution though.
For example, say that you have floating-point coordinates in the range -100 < X <= 100, with your 1,000,000+ objects distributed fairly uniformly across that range. That would divide the list into 200 partitions of (on average) 5000 entries if partitioned on integer values of X. That means that for every integer step in the X dimension of your search range you only have ~5,000 entries to test.
Here's some code:
public interface IPosition2F
{
float X { get; }
float Y { get; }
}
public class CoordMap<T> where T : IPosition2F
{
SortedDictionary<int, List<T>> map = new SortedDictionary<int,List<T>>();
readonly Func<float, int> xPartition = (x) => (int)Math.Floor(x);
public void Add(T entry)
{
int xpart = xPartition(entry.X);
List<T> col;
if (!map.TryGetValue(xpart, out col))
{
col = new List<T>();
map[xpart] = col;
}
col.Add(entry);
}
public T[] ExtractRange(float left, float top, float right, float bottom)
{
var rngLeft = xPartition(left) - 1;
var rngRight = xPartition(right) + 1;
var cols =
from keyval in map
where keyval.Key >= rngLeft && keyval.Key <= rngRight
select keyval.Value;
var cells =
from cell in cols.SelectMany(c => c)
where cell.X >= left && cell.X <= right &&
cell.Y >= top && cell.Y <= bottom
select cell;
return cells.ToArray();
}
public CoordMap()
{ }
// Create instance with custom partition function
public CoordMap(Func<float, int> partfunc)
{
xPartition = partfunc;
}
}
That will partition on the X coordinate, reducing your final search space. If you wanted to take it a step further you could also partition on the Y coordinate... I'll leave that as an exercise for the reader :)
If your parition function is very finely grained and could result in a large number of partitions, it might be useful to add a ColumnRange function similar to:
public IEnumerable<List<T>> ColumnRange(int left, int right)
{
using (var mapenum = map.GetEnumerator())
{
bool finished = mapenum.MoveNext();
while (!finished && mapenum.Current.Key < left)
finished = mapenum.MoveNext();
while (!finished && mapenum.Current.Key <= right)
{
yield return mapenum.Current.Value;
finished = mapenum.MoveNext();
}
}
}
The ExtractRange method can then use that like so:
public T[] ExtractRange(float left, float top, float right, float bottom)
{
var rngLeft = xPartition(left) - 1;
var rngRight = xPartition(right) + 1;
var cells =
from cell in ColumnRange(rngLeft, rngRight).SelectMany(c => c)
where cell.X >= left && cell.X <= right &&
cell.Y >= top && cell.Y <= bottom
select cell;
return cells.ToArray();
}
I used SortedDictionary for convenience, and because it makes it possible to do an ExtractRange method that is reasonably quick. There are other container types that are possibly better suited to the task.

c# outline numbering

I am trying to develop an algorithm in C# that can take an array list of URL's and output them in an outline numbered list.
As you can imagine I need some help. Does anyone have any suggestions on the logic to use to generate this list?
Example Output:
1 - http://www.example.com/aboutus
1.2 - http://www.example.com/aboutus/page1
1.3 - http://www.example.com/aboutus/page2
1.3.1 - http://www.example.com/aboutus/page2/page3
1.3.1.1 - http://www.example.com/aboutus/page2/page3/page4
1.3.2 - http://www.example.com/aboutus/page5/page6
1.3.2.1 - http://www.example.com/aboutus/page5/page7/page9
1.3.2.2 - http://www.example.com/aboutus/page5/page8/page10
1.4 - http://www.example.com/aboutus/page10
1.4.1 - http://www.example.com/aboutus/page10/page11
1.4.2 - http://www.example.com/aboutus/page10/page12
1.1.5 - http://www.example.com/aboutus/page13
1.1.6 - http://www.example.com/aboutus/page14
1.1.6.1 - http://www.example.com/aboutus/page14/page15
1.1.6.2 - http://www.example.com/aboutus/page14/page16
1.1.6.3 - http://www.example.com/aboutus/page14/page17
... and so on

Take a look at the System.URI class. It should have some methods and properites that should be usefull, like the segments property that splilts the uri into its segmented parts (split by slash bascially). You could create a list of the segment arrays, sort the list, then simply iterate the list adjusting the numbers depending on wether the current list index segments match the previous list index segments.

You'll probably have to strip protocol and query string parameters, so +1 to the advise to use System.URI class to handle that.
As for the printing it in tree-shape - a direct approach is to use a Dictionary<string, string> to keep association of child (key) to a parent (value).
Another way is to take advantage of List<T>.Sort, e.g. like this:
public static void Print(List<string> list)
{
var path = new Stack<string>();
var count = new Stack<int>();
path.Push("");
count.Push(0);
list.Sort(new Comparison<string>(UrlComparison));
foreach (var x in list)
{
while (!x.StartsWith(path.Peek())) { path.Pop(); count.Pop(); }
count.Push(count.Pop() + 1);
foreach(var n in count.Reverse()) Console.Write("{0}.", n);
Console.WriteLine(" {0}", x);
path.Push(x);
count.Push(0);
}
}
Unfortunately, p.campbell is right, a custom comparison is actually required here, which makes this implementation still pretty performant, but more bulky (?:-abuse warning):
public static int UrlComparison(string x, string y)
{
if (x == null && y == null) return 0;
if (x == null) return -1;
if (y == null) return 1;
for(int n = 0; n < Math.Min(x.Length, y.Length); n++)
{
char cx = x[n], cy = y[n];
if(cx == cy) continue;
return
(cx == '/' || cx == '.' || cx == '?') ? -1 :
(cy == '/' || cy == '.' || cy == '?') ? 1 :
(cx > cy) ? 1 : -1;
}
return (x.Length == y.Length) ? 0 : (x.Length > y.Length) ? 1 : -1;
}
PS: Just to put a disclaimer, I feel that Stacks logic is consize, but a bit more complex to understand. In a long-term project, I'd stick with a child-parent dictionary.

May be this Generic Tree Collection will be helpful.
There are few Tree Collections on the Code Project: one, two.

I think you need to implement some sort of tree collection to handle the order. Because if you added a new link called http://www.example.com, that would become 1 instead of http://www.example.com/aboutus.
Then you can print the in-order traversal of the tree and it will be exceedingly simple.

Function to look through list and determine trend

So I have a list of items. Each item on the list has a property called notional. Now, the list is already sorted. What I need to do is, develop a function that sets the type of list to one of the following:
Bullet - notional is the same for every item
Amortizing - notional decreases over the course of the schedule (might stay the same from element to element but it should never go up, and should end lower)
Accreting - notional increases over the course of the schedule (might stay the same from element to element but it should never go down, and should end higher)
Rollercoaster - notional goes up and down (could end the same, higher, or lower, but shouldn't be the same for each element and shouldn't be classfied as the other types)
What would this method look like and what would be the most efficient way to go through the list and figure this out?
Thanks!

This would be a straightforward way to do it:
bool hasGoneUp = false;
bool hasGoneDown = false;
T previous = null; // T is the type of objects in the list; assuming ref type
foreach(var item in list)
{
if (previous == null) {
previous = item;
continue;
}
hasGoneUp = hasGoneUp || item.notional > previous.notional;
hasGoneDown = hasGoneDown || item.notional < previous.notional;
if(hasGoneUp && hasGoneDown) {
return Trend.Rollercoaster;
}
previous = item;
}
if (!hasGoneUp && !hasGoneDown) {
return Trend.Bullet;
}
// Exactly one of hasGoneUp and hasGoneDown is true by this point
return hasGoneUp ? Trend.Accreting : Trend.Amortizing;

Let trendOut = Bullet
Loop from First Item to Last item
2.1. If previous notional < next notional
2.1.a. If trendOut = Amortizing return RollerCoaster
2.1.b. Else set trendOut = Accreting
2.2. if Previous Notional > next notional
2.2.a. If trendOut = Accreting return RollerCoaster
2.2.b. Else set trendOut = Amortizing
return trendOut.

You could probably do something as simple as this
var changeList = new List<Integer>
for(i = 0; i < yourList.Count() - 1; i++)
{
changeList.Add(yourList.Item(i + 1) - yourList.Item(i));
}
//Determine if the nature of the list
var positiveChangeCount = changeList.Where(x => x < 0);
var negativeChangeCount = changeList.Where(x => X > 0);
if (positiveChangeCount = yourList.Count)
{
Accreting;
}
elseif (negativeChangeCount = yourList.Count)
{
Amortizing;
}
elseif (negativeChangeCount + PositiveChangeCount = 0)
{
Bullet;
}
else
{
Rollercoaster;
}

I usually start of by optimizing for simplicity first and then performance. Hence, I would start by making a second list of N-1 elements, whose {elements} are differences between the {notionals} of the first list.
Hence, for the second list, I would expect the following for the list of your needs
Bullet - ALL elements are 0
Amortising - ALL elements stay 0 or negative
Accreting - ALL elements stay 0 or positive
Rollercoaster - Elements oscillate between negative & positive
You can probably optimize it an do it in one pass. Basically, this is a discrete differentiation over your data.

bool OnlyGreaterOrEqual=true;
bool OnlyLessOrEqual=true;
foreach(int i=1;i<itemList.Count;i++){
if(itemList[i].notional>itemList[i-1].notional){
OnlyLessOrEqual=false;
}else if(itemList[i].notional<itemList[i-1].notional){
OnlyGreaterOrEqual=false;
}
}
if(OnlyGreaterOrEqual && OnlyLessOrEqual){
return "Bullet";
}else if(OnlyGreaterOrEqual){
return "Accreting":
}else if(OnlyLessOrEqual){
return "Amortizing";
}else{
return "RollerCoast";
}

This is basically a Linq implementation of Danish's answer. It'll require (worst case) 3 passes through the list, but because they are so small it won't really matter from a performance point of view. (I wrote it to work on a list of ints so you'll have to modify it easily to work with your types).
var tmp = values
.Skip(1)
.Zip( values, (first, second) => first - second )
.ToList();
var up = tmp.Any( t => t > 0 );
var down = tmp.Any( t => t < 0 );
if( up && down )
// Rollercoaster
else if( up )
// Accreting
else if( down )
// Amortizing
else
// Bullet
You could also (ab)use the Aggregate operator and Tuple to do it as one query. However, this will fail if the collection is empty and is a bit weird to use in production code.
var result = values.Skip(1).Aggregate(
Tuple.Create<int, bool, bool>( values.First(), false, false ),
( last, current ) => {
return Tuple.Create(
current,
last.Item2 || (current - last.Item1) > 0,
last.Item3 || (current - last.Item1) < 0 );
});
result will be a tuple that contains:
the last element of the collection (which is of no use)
Item2 will contain a boolean indicating whether any element was bigger than the previous element
Item3 will contain a boolean indicating whether any element was smaller than the previous element
The same switch statement as above can be used to decide which pattern your data follows.

Can I use infinite range and operate over it?

Enumerable.Range(0, int.MaxValue)
.Select(n => Math.Pow(n, 2))
.Where(squared => squared % 2 != 0)
.TakeWhile(squared => squared < 10000).Sum()
Will this code iterate over all of the integer values from 0 to max-range or just through the integer values to satisfy the take-while, where, and select operators?
Can somebody clarify?
EDIT: My first try to make sure it works as expected was dumb one. I revoke it :)

int.MaxValue + 5 overflows to be a negative number. Try it yourself:
unchecked
{
int count = int.MaxValue + 5;
Console.WriteLine(count); // Prints -2147483644
}
The second argument for Enumerable.Range has to be non-negative - hence the exception.
You can certainly use infinite sequences in LINQ though. Here's an example of such a sequence:
public IEnumerable<int> InfiniteCounter()
{
int counter = 0;
while (true)
{
unchecked
{
yield return counter;
counter++;
}
}
}
That will overflow as well, of course, but it'll keep going...
Note that some LINQ operators (e.g. Reverse) need to read all the data before they can yield their first result. Others (like Select) can just keep streaming results as they read them from the input. See my Edulinq blog posts for details of the behaviour of each operator (in LINQ to Objects).

The way to solve these sort of questions in general, is to think about what's going on in steps.
Linq turns the linq code into something that'll be executed by query provider. This could be something like producing SQL code, or all manner of things. In the case of linq-to-objects, it produces some equivalent .NET code. Thinking about what that .NET code will be lets us reason about what will happen.*
With your code you have:
Enumerable.Range(0, int.MaxValue)
.Select(n => Math.Pow(n, 2))
.Where(squared => squared % 2 != 0)
.TakeWhile(squared => squared < 10000).Sum()
Enumerable.Range is slightly more complicated than:
for(int i = start; i != start + count; ++i)
yield return i;
...but that's close enough for argument's sake.
Select is close enough to:
foreach(T item in source)
yield return func(item);
Where is close enough to:
foreach(T item in source)
if(func(item))
yield return item;
TakeWhile is close enough to:
foreach(T item in source)
if(func(item))
yield return item;
else
yield break;
Sum is close enough to:
T tmp = 0;//must be numeric type
foreach(T x in source)
tmp += x;
return tmp;
This simplifies a few optimisations and so on, but is close enough to reason with. Taking each of these in turn, your code is equivalent to:
double ret = 0; // part of equivalent of sum
for(int i = 0; i != int.MaxValue; ++i) // equivalent of Range
{
double j = Math.Pow(i, 2); // equivalent of Select(n => Math.Pow(n, 2))
if(j % 2 != 0) //equivalent of Where(squared => squared %2 != 0)
{
if(j < 10000) //equivalent of TakeWhile(squared => squared < 10000)
{
ret += j; //equaivalent of Sum()
}
else //TakeWhile stopping further iteration
{
break;
}
}
}
return ret; //end of equivalent of Sum()
Now, in some ways the code above is simpler, and in some ways it's more complicated. The whole point of using LINQ is that in many ways its simpler. Still, to answer your question "Will this code iterate over all of the integer values from 0 to max-range or just through the integer values to satisfy the take-while, where, and select operators?" we can look at the above and see that those that don't satisfy the where are iterated through to find that they don't satisfy the where, but no more work is done with them, and once the TakeWhile is satisfied, all further work is stopped (the break in my non-LINQ re-write).
Of course it's only the TakeWhile() in this case that means the call will return in a reasonable length of time, but we also need to think briefly about the others to make sure they yield as they go. Consider the following variant of your code:
Enumerable.Range(0, int.MaxValue)
.Select(n => Math.Pow(n, 2))
.Where(squared => squared % 2 != 0)
.ToList()
.TakeWhile(squared => squared < 10000).Sum()
Theoretically, this will give exactly the same answer, but it will take far longer and far more memory to do so (probably enough to cause an out of memory exception). The equivalent non-linq code here though is:
List<double> tmpList = new List<double>(); // part of ToList equivalent
for(int i = 0; i != int.MaxValue; ++i) // equivalent of Range
{
double j = Math.Pow(i, 2); // equivalent of Select(n => Math.Pow(n, 2))
if(j % 2 != 0) //equivalent of Where(squared => squared %2 != 0)
{
tmpList.Add(j);//part of equivalent to ToList()
}
}
double ret = 0; // part of equivalent of sum
foreach(double k in tmpList)
{
if(k < 10000) //equivalent of TakeWhile(squared => squared < 10000)
{
ret += k; //equaivalent of Sum()
}
else //TakeWhile stopping further iteration
{
break;
}
}
return ret; //end of equivalent of Sum()
Here we can see how adding ToList() to the Linq query vastly affects the query so that every item produced by the Range() call must be dealt with. Methods like ToList() and ToArray() break up the chaining so that non-linq equivalents no longer fit "inside" each other and none can therefore stop the operation of those that come before. (Sum() is another example, but since it's after your TakeWhile() in your example, that isn't an issue).
Another thing that would make it go through every iteration of the range is if you had While(x => false) because it would never actually perform the test in TakeWhile.
*Though there may be further optimisations, esp in the case of SQL code and also while conceptually e.g. Count() is equivalent to:
int c = 0;
foreach(item in src)
++c;
return c;
That this will be turned into a call to the Count property of an ICollection or the Length property of an array means the O(n) above is replaced by an O(1) (for most ICollection implementations) call, which is a massive gain for large sequences.

Your first code will only iterate as long the TakeWhile condition is met. It will not iterate until int.MaxValue.
int.MaxValue + 5 will result in a negative integer. Enumerable.Range throws an ArgumentOutOfRangeException if its second argument is negative. So that's why you get the exception (before any iteration takes place).

C# - Finding the Boundaries of an Image (not the size)

I'm developing an application to split an image grid equally and center the images (based on their similarity). So far, I could manage to fix a grid of images with small sizes, but whenever I try a larger "sprite" size (100x100, for instance), I get Stack Overflow error.
Yes I'm using recursion, but whenever a pixel is checked, I set a boolean to deactivate it, copy it to a list and go on checking the others (in all directions), until the list is filled up with an image from the grid. I'm not sure if this is the best way since for each call, I call the same method 7 times (supposing there are 7 adjacent pixels which weren't checked yet)... until there are no pixels left to check, and I can move on to the next image in the grid.
I tried tracing where the error started happening, it was after checking more or less 1600 pixels and adding them to the List. MyPixel is a class which contains 4 variables: x(int), y(int), color (Color), and checked (bool)
public void processSprite(int i, int j)
{
//OOO
//OXO
//OOO
pixeltemp.Add(new MyPixel(imap.pixels[i, j].x, imap.pixels[i, j].y, imap.pixels[i, j].color));
imap.pixels[i, j].read = true;
//OOO
//OOX
//OOO
try
{
if (!imap.pixels[i + 1, j].read)
{
if (imap.pixels[i + 1, j].color.A == 0) //Found a Border
{
imap.pixels[i + 1, j].read = true;
}
else
{
processSprite(i + 1, j);
}
}
}
//... (code goes on)
}
pixeltemp is the temporary list of pixels which holds the image (List<MyPixel>)
imap contains the entire image (List<MyPixel>)
I guess it's not a memory problem since my app just takes about 16mb tops.
My question is, why do I have this "Stack overflow" error if it's not an infinite recursion? Is there an easier way to do this? I do think my code looks ugly, I just have no idea how to make it better.
Thanks in advance !

Stack overflows aren't caused by infinite recursion but by more recursion (or, rather, call stack) than the process can handle. In your case, each recursive call to processSprite is going to do the same number of recursive calls to processSprite. So in the worst-case scenario of 1600 pixels without finding a boundary, your call tree would look like this:
processSprite(0, j)
processSprite(1, j)
processSprite(2, j)
...
processSprite(1599, j) <-- That's 1600 call frames,
enough for an overflow.
You'll want to reorganize your algorithm into a linear loop that does a depth-first search, perhaps in a spiral pattern if you're wanting to to find a pixel that's closest to the starting point. I'm sure there are already other spiffy algorithms others have already developed to solve this problem.
Edit:
I think I understand better now the problem that you're trying to solve. It sounds like you have an image that may contain multiple image tiles surrounded by 0-alpha pixels and you want to find the bounding rectangles for each of these tiles. That looked like an interesting problem to solve, so I implemented it:
IEnumerable<Rectangle> FindImageTiles(Bitmap compositeImage)
{
var result = new List<Rectangle>();
// Scan for a non-empty region that hasn't already been "captured"
for (var x = 0; x < compositeImage.Width; x++)
{
for (var y = 0; y < compositeImage.Height; y++)
{
// Only process the pixel if we don't have a rectangle that
// already contains this and if it's not empty
if (!result.Any(r => r.Contains(x, y))
&& compositeImage.GetPixel(x, y).A != 0)
{
// Now that we've found a point, create a rectangle
// surrounding that point, then expand outward until
// we have a bounding rectangle that doesn't intersect
// with the tile
var rect = new Rectangle(x - 1, y - 1, 2, 2);
bool foundBounds = false;
while (!foundBounds)
{
var xRange = Enumerable.Range(rect.Left, rect.Right)
.Where(px => px >= 0 && px < compositeImage.Width);
var yRange = Enumerable.Range(rect.Top, rect.Bottom)
.Where(py => py >= 0 && py < compositeImage.Height);
// Adjust the top
if (rect.Top >= 0
&& xRange
.Select(bx => compositeImage.GetPixel(bx, rect.Top))
.Any(p => p.A != 0))
{
rect.Y--;
rect.Height++;
}
else if (rect.Bottom < compositeImage.Height
&& xRange
.Select(bx => compositeImage.GetPixel(bx, rect.Bottom))
.Any(p => p.A != 0))
{
rect.Height++;
}
else if (rect.Left >= 0
&& yRange
.Select(by => compositeImage.GetPixel(rect.Left, by))
.Any(p => p.A != 0))
{
rect.X--;
rect.Width++;
}
else if (rect.Right < compositeImage.Width
&& yRange
.Select(by => compositeImage.GetPixel(rect.Right, by))
.Any(p => p.A != 0))
{
rect.Width++;
}
else
{
foundBounds = true;
}
}
result.Add(rect);
}
}
}
return result;
}