We Keep Coding

Iterating over Linq-to-Entities IEnumerable causes OutOfMemoryException

The part of the code I'm working on receives an
IEnumerable<T> items
where each item contains a class with properties reflecting a MSSQL database table.
The database table has a total count of 953664 rows.
The dataset in code is filtered down to a set of 284360 rows.
The following code throws an OutOfMemoryException when the process reaches about 1,5 GB memory allocation.
private static void Save<T>(IEnumerable<T> items, IList<IDataWriter> dataWriters, IEnumerable<PropertyColumn> columns) where T : MyTableClass
{
foreach (var item in items)
{
}
}
The variable items is of type
IQueryable<MyTableClass>
I can't find anyone with the same setup, and other's solutions that I've found doesn't apply here.
I've also tried paging, using Skip and Take with a page size of 500, but that just takes a long time and ends up with the same result. It seems like objects aren't being released after each iteration. How is that?
How can I rewrite this code to cope with a larger collection set?

Well, as Servy has already said you didn't provide your code so I'll try to make some predictions... (Sorry for my english)
If you have an exception in "foreach (var item in items)" when you are using paging then, I guess, something wrong with paging. I wrote a couple of examples to explain my idea.
if first example I suggest you (just for test) put your filter inside the Save function.
private static void Save<T>(IQueryable<T> items, IList<IDataWriter> dataWriters, IEnumerable<PropertyColumn> columns) where T : MyTableClass
{
int pageSize = 500; //Only 500 records will be loaded.
int currentStep = 0;
while (true)
{
//Here we create a new request into the database using our filter.
var tempList = items.Where(yourFilter).Skip(currentStep * pageSize).Take(pageSize);
foreach (var item in tempList)
{
//If you have an exception here maybe something wrong in your dataWriters or columns.
}
currentStep++;
if (tempList.Count() == 0) //No records have been loaded so we can leave.
break;
}
}
The second example show how to use paging without any changes in the Save function
int pageSize = 500;
int currentStep = 0;
while (true)
{
//Here we create a new request into the database using our filter.
var tempList = items.Where(yourFilter).Skip(currentStep * pageSize).Take(pageSize);
Save(tempList, dataWriters, columns); //Calling saving function.
currentStep++;
if (tempList.Count() == 0)
break;
}
Try both of them and you'll either resolve your problem or find another place where an exception is raised.
By the way, another potential place is your dataWriters. I guess there you store all data that your have been received from the database. Maybe you shouldn't save all data? Just calculate memory size that all objects are required.
P.S. And don't use while(true) in your code. It just an example:)

C# List Sort vs Inserting

I have a list of objects which I sort multiple times throughout code and when the user interacts with the program. I was wondering if it would be better to insert new items into the list rather than add to the end of the list and resort the entire list.
The code below is for importing browser bookmarks - Here I add a bunch of bookmarks to the List (this._MyLinks) which are Link objects and then sort the final List - Which I think is probably best in this given scenario....
public void ImportBookmarks(string importFile)
{
using (var file = File.OpenRead(importFile))
{
var reader = new NetscapeBookmarksReader();
var bookmarks = reader.Read(file);
foreach (var b in bookmarks.AllLinks)
{
bool duplicate = this._MyLinks.Any(link => link._URL == b.Url);
if(duplicate)
{
continue;
}
Link bookmark = new Link();
bookmark._URL = b.Url;
bookmark._SiteName = b.Title;
bookmark.BrowserPath = "";
bookmark.BrowserName = "";
if (bookmark.AddToConfig(true))
{
this._MyLinks.Add(bookmark);
}
}
}
this._MyLinks = this._MyLinks.OrderBy(o => o._SiteName).ToList();
}
Now a user also has the option to add their own links (one at a time). Whenever the user adds a link the ENTIRE list is sorted again using
this._MyLinks = this._MyLinks.OrderBy(o => o._SiteName).ToList();
Is it better from a preformance standpoint (or just generally) to just insert the item directly into it's specified location? If so would you have suggestions on how I can go about doing that?
Thanks!

Since you want a sorted set of data you should be using a more appropriate data structure, specifically a sorted data structure, rather than using an unsorted data structure that you re-sort every time, or that forces you to inefficiently add items to the middle of a list.
SortedSet is specifically designed to maintain a sorted set of data efficiently.

Add element to list before specific element

I have a list of items, lets say 100 items. I need to add another element before the existing element that matches my condition. What is the fastest way and the most performance optimized to do this?
ie.:
foreach (var i in myList)
{
if (myList[i].value == "myValue")
{
myList[i-1] add ("someOtherValue")
}
}
Maybe i should use other container?

First you could find the index of your item using FindIndex method:
var index = myList.FindIndex(x => x.value == "myvalue");
Then Insert at the right point:
myList.Insert(index,newItem);
Note that inserting at a given index pushes everything else forward (think about finding your item at index 0).

Consider using a LinkedList<T>. It has the advantage that inserting or removing items does not require shifting any items. The disadvantage is that items cannot be accessed randomly. You have to traverse the list starting at the first or last item in order to access the items.

myList.Insert(myList.IndexOf("myValue") - 1, "someOtherValue");
You should probably check to make sure myvalue exists first, and it is not in index 0.

int index = myList.IndexOf("myValue");
if (index >= 0)
myList.Insert(index, "myNewValue");
By the way, you should not modify your own collection or list while iterating with for-each (as in your code above).

I presume the list is an array - in which case have you tried doing this with Linq?
string[] mylist = new string[100];
// init the list
List<string> list = keys.ToList();
list.Insert(1,"somethingelse");
mylist = list.ToArray(); // convert back to array if required
if it is a List to begin with, you can skip the conversions and use Insert directly.

How to add just 10 items

I have undoredomanager.
And I need to view in listview only 10 entries.
already seething brain how to do it.
This code is added to the viewlist all records, but I only need the last 10.
lvUndoStack.Items.Clear();
var list = new List<object>();
foreach (var command in UndoRedoManager.UndoCommands)
{
list.Insert(0, command.ToString());
}
lvUndoStack.Items.AddRange(list.ToArray());
lvUndoStack.SelectedIndex = lvUndoStack.Items.Count - 1;
indexSeletedItemUndoStack = lvUndoStack.SelectedIndex;
list = new List<object>();
foreach (var command in UndoRedoManager.RedoCommands)
{
list.Insert(0, command.ToString());
}
lvUndoStack.Items.AddRange(list.ToArray());
importantly - not used linq
Update:
example:
undo1
undo2
undo3
undo4
undo5
undo6
undo7
redo1
redo2
redo3
redo4
redo5
I need obly 10. if start undo4 then you need to show everything in the last or a maximum of 10

If I understand your question, you want (at most) the last 10 undo and the last 10 redo commands. After you get the List of undo commands, use this to get up to the last 10:
if (list.Count <= 10)
{
lvUndoStack.Items.AddRange(list.ToArray());
}
else
{
for (int i = 0; i < 10; i++)
{
lvUndoStack.Items.Add(list[0]);
}
}
And do the same for the redo commands. That's not the best solution, IMO - really more of a kludge, but it should get you going in the right direction.
An even better solution would be to modify/enchance the UndoRedoManager class so that the ListView could call a method to get a list of the last n undo/redo commands. Something like:
public List<Object> GetUndoCommands(int numberOfCommands);
Then the ListView could simply call that method:
lvUndoStack.Items.AddRange(UndoRedoManager.GetUndoCommands(10).ToArray());
And something similar for the redo commands. It removes a bunch of code from your UI layer, and gives you the flexibility to easily switch the max number of items at a later date and just generally (again IMO) seems to be a better way to go.

Bind ComboBox to Two Lists in C#

I have two lists. The first one is hard-coded and the contents never change. The second can be edited by the user to add, change and remove items:
public List<Item> DefaultItems = new List<Item>();
public BindingList<Item> UserItems = new BindingList<Item>();
...
MyTable.DataSource = UserItems;
I would like to bind the contents of both lists, one after the other, to a ComboBox and have it update automatically when the user edits the UserItems list.
The first part I can easily solve with something like:
public List<Items> AllItems
{
get
{
List<Item> Items = new List<Item>();
foreach (Item I in DefaultItems) Items.Add(I);
foreach (Item I in UserItems) Items.Add(I);
return Items;
}
}
...
MyComboBox.DataSource = AllItems;
The problem is that when UserItems changes there is no notification that AllItems has changed so the contents of the combobox remain the same.
I then added an event that is generated when UserItems changes. Now my problem is how to force the ComboBox to refresh. Doing the following:
MyComboBox.DataSource = null;
MyComboBox.DataSource = AllItems;
results in the selecteditem becoming null and the selectedindex becoming -1, which I then have to handle in my code (temporarily remember the current item, restore it afterwards, etc.). It's all becoming very messy and I'm sure there is a clever way of solving this. Is there?
thanks, Andy
UPDATE: I didn't want to add yet more code and complexity just for this in the form of a third party assembly, so I just continued with my messy approach. Thanks.

You need to use a collection that will notify the UI when the collection has been changed.
You can either use the .NET provided BindingList class or, if you want to try something different, you could download the BindingListView class that will wrap your existing collections and provide the UI with the notification it needs.