We Keep Coding

Access one column from a list to use in another query

I am trying to grab a single column from my linq query and put it into another list may not be the right way of doing it so please excuse me if my code is not
good. Wanting to learn more about linq.
List<StockM> _stockm = new List<StockM>();
List<priceLists> _priceList = new List<priceLists>();
stockm = _db.getStockBySupplierCode(_supplierCode);
foreach (var stockitem in _stockm)
{
_priceList = _db.getPriceTypesByProductCode(stockitem.product, "LPS");
stockitem.lpePrice = Convert.ToDecimal(_priceList.Select(s => s.lpsPrice));
}
I think the probelems lies in how I am attempting to select the column out here
_priceList.Select(s => s.lpsPrice)

try using FirstorDefault() as your _priceList is a list and you are assigning it to a value
_priceList.Select(s => s.lpsPrice).FirstOrDefault();
Also, do you need a where to find a particular value that matches? Just curious

There is several things wrong in your code.
First of all, I encourage you to read the .NET convention. There is no leading _ in local variable, the properties must start with an upper case, for examples.
Then, you do not need to instantiate a default instance if you are going to assign another instance to your variable next. You can assign it directly:
List<StockM> _stockm = _db.getStockBySupplierCode(_supplierCode);
Finally, the Select extension method you are using is already doing a foreach:
var result = _priceList.Select(s => s.lpsPrice);
Is basically (not exactly but it is a start to explain it like this) the same as:
var result = new List<decimal>();
foreach(var priceList in _priceList)
{
result.Add(priceList.lpsPrice);
}
That being said, you can improve your code like this:
List<StockM> stockm = _db.getStockBySupplierCode(_supplierCode);
foreach (var stockitem in stockm)
{
List<priceLists> _priceList = _db.getPriceTypesByProductCode(stockitem.product, "LPS");
stockitem.lpePrice = ??? // you cannot use the Select here as it returns a collection of item. What do you really want to achieve?
}

Dynamic Linq Group By

I am currently building reports and have a need to Group columns dynamically, depending on user's choice. Now, assuming that the situation is fixed on all columns, the query would be as follows:
var groupedInvoiceItems = invoiceItems.GroupBy(x => new { x.SalesInvoice.name, x.SalesInvoice.currencyISO, x.CatalogProduct });
Doing so would return results as desired, IGrouping. I would then run a loop to process the necessary data as below:
foreach (var groupedInvoiceItem in groupedInvoiceItems)
{
// Perform work here
}
Now, the headache comes in when I try to make the Grouping dynamic by using Dynamic Linq. The query is as follows:
var groupedInvoiceItems = invoiceItems.GroupBy("new (SalesInvoice.name, SalesInvoice.currencyISO, CatalogProduct)", "it");
The problem with this is that it does not return IGrouping anymore. Hence, my foreach loop no longer works. Is there any solution to the matter? I tried casting IGrouping to the Dynamic query but to no avail. Help is needed urgently.

The result of the GroupBy is an IEnumerable<IGrouping<DynamicClass,InvoiceItem>>, so you can proceed by something like:
foreach (IGrouping<DynamicClass,InvoiceItem> invoiceItemGroup in groupedInvoiceItems)
{
}

You should do the grouping then select the specified attributes to iterate throw them in the foreach loop. Try this out:
var groupedInvoiceItems = invoiceItems.GroupBy("SalesInvoice.name","it").GroupBy("SalesInvoice.currencyISO","it").GroupBy("CatalogProduct","it").Select("new (it.SalesInvoice.name,it.SalesInvoice.currencyISO,it.CatalogProduct)");
Hope this works.

How do I make a streaming LINQ expression that delivers the filtered out items as well as the filtered items?

I am transforming an Excel spreadsheet into a list of "Elements" (this is a domain term). During this transformation, I need to skip the header rows and throw out malformed rows that cannot be transformed.
Now comes the fun part. I need to capture those malformed records so that I can report on them. I constructed a crazy LINQ statement (below). These are extension methods hiding the messy LINQ operations on the types from the OpenXml library.
var elements = sheet
.Rows() <-- BEGIN sheet data transform
.SkipColumnHeaders()
.ToRowLookup()
.ToCellLookup()
.SkipEmptyRows() <-- END sheet data transform
.ToElements(strings) <-- BEGIN domain transform
.RemoveBadRecords(out discard)
.OrderByCompositeKey();
The interesting part starts at ToElements, where I transform the row lookup to my domain object list (details: it's called an ElementRow, which is later transformed into an Element). Bad records are created with just a key (the Excel row index) and are uniquely identifiable vs. a real element.
public static IEnumerable<ElementRow> ToElements(this IEnumerable<KeyValuePair<UInt32Value, Cell[]>> map)
{
return map.Select(pair =>
{
try
{
return ElementRow.FromCells(pair.Key, pair.Value);
}
catch (Exception)
{
return ElementRow.BadRecord(pair.Key);
}
});
}
Then, I want to remove those bad records (it's easier to collect all of them before filtering). That method is RemoveBadRecords, which started like this...
public static IEnumerable<ElementRow> RemoveBadRecords(this IEnumerable<ElementRow> elements)
{
return elements.Where(el => el.FormatId != 0);
}
However, I need to report the discarded elements! And I don't want to muddy my transform extension method with reporting. So, I went to the out parameter (taking into account the difficulties of using an out param in an anonymous block)
public static IEnumerable<ElementRow> RemoveBadRecords(this IEnumerable<ElementRow> elements, out List<ElementRow> discard)
{
var temp = new List<ElementRow>();
var filtered = elements.Where(el =>
{
if (el.FormatId == 0) temp.Add(el);
return el.FormatId != 0;
});
discard = temp;
return filtered;
}
And, lo! I thought I was hardcore and would have this working in one shot...
var discard = new List<ElementRow>();
var elements = data
/* snipped long LINQ statement */
.RemoveBadRecords(out discard)
/* snipped long LINQ statement */
discard.ForEach(el => failures.Add(el));
foreach(var el in elements)
{
/* do more work, maybe add more failures */
}
return new Result(elements, failures);
But, nothing was in my discard list at the time I looped through it! I stepped through the code and realized that I successfully created a fully-streaming LINQ statement.
The temp list was created
The Where filter was assigned (but not yet run)
And the discard list was assigned
Then the streaming thing was returned
When discard was iterated, it contained no elements, because the elements weren't iterated over yet.
Is there a way to fix this problem using the thing I constructed? Do I have to force an iteration of the data before or during the bad record filter? Is there another construction that I've missed?
Some Commentary
Jon mentioned that the assignment /was/ happening. I simply wasn't waiting for it. If I check the contents of discard after the iteration of elements, it is, in fact, full! So, I don't actually have an assignment problem. Unless I take Jon's advice on what's good/bad to have in a LINQ statement.

When the statement was actually iterated, the Where clause ran and temp filled up, but discard was never assigned again!
It doesn't need to be assigned again - the existing list which will have been assigned to discard in the calling code will be populated.
However, I'd strongly recommend against this approach. Using an out parameter here is really against the spirit of LINQ. (If you iterate over your results twice, you'll end up with a list which contains all the bad elements twice. Ick!)
I'd suggest materializing the query before removing the bad records - and then you can run separate queries:
var allElements = sheet
.Rows()
.SkipColumnHeaders()
.ToRowLookup()
.ToCellLookup()
.SkipEmptyRows()
.ToElements(strings)
.ToList();
var goodElements = allElements.Where(el => el.FormatId != 0)
.OrderByCompositeKey();
var badElements = allElements.Where(el => el.FormatId == 0);
By materializing the query in a List<>, you only process each row once in terms of ToRowLookup, ToCellLookup etc. It does mean you need to have enough memory to keep all the elements at a time, of course. There are alternative approaches (such as taking an action on each bad element while filtering it) but they're still likely to end up being fairly fragile.
EDIT: Another option as mentioned by Servy is to use ToLookup, which will materialize and group in one go:
var lookup = sheet
.Rows()
.SkipColumnHeaders()
.ToRowLookup()
.ToCellLookup()
.SkipEmptyRows()
.ToElements(strings)
.OrderByCompositeKey()
.ToLookup(el => el.FormatId == 0);
Then you can use:
foreach (var goodElement in lookup[false])
{
...
}
and
foreach (var badElement in lookup[true])
{
...
}
Note that this performs the ordering on all elements, good and bad. An alternative is to remove the ordering from the original query and use:
foreach (var goodElement in lookup[false].OrderByCompositeKey())
{
...
}
I'm not personally wild about grouping by true/false - it feels like a bit of an abuse of what's normally meant to be a key-based lookup - but it would certainly work.

in foreach loop, should I expect an error since query collection has changed?

For example
var query = myDic.Where(x => !blacklist.Contains(x.Key));
foreach (var item in query)
{
if (condition)
blacklist.Add(item.key+1); //key is int type
ret.add(item);
}
return ret;
would this code be valid? and how do I improve it?
Updated
i am expecting my blacklist.add(item.key+1) would result in smaller ret then otherwise. The ToList() approach won't achieve my intention in this sense.
is there any other better ideas, correct and unambiguous.

That is perfectly safe to do and there shouldn't be any problems as you're not directly modifying the collection that you are iterating over. Though you are making other changes that affects where clause, it's not going to blow up on you.
The query (as written) is lazily evaluated so blacklist is updated as you iterate through the collection and all following iterations will see any newly added items in the list as it is iterated.
The above code is effectively the same as this:
foreach (var item in myDic)
{
if (!blacklist.Contains(item.Key))
{
if (condition)
blacklist.Add(item.key + 1);
}
}
So what you should get out of this is that as long as you are not directly modifying the collection that you are iterating over (the item after in in the foreach loop), what you are doing is safe.
If you're still not convinced, consider this and what would be written out to the console:
var blacklist = new HashSet<int>(Enumerable.Range(3, 100));
var query = Enumerable.Range(2, 98).Where(i => !blacklist.Contains(i));
foreach (var item in query)
{
Console.WriteLine(item);
if ((item % 2) == 0)
{
var value = 2 * item;
blacklist.Remove(value);
}
}

Yes. Changing a collections internal objects is strictly prohibited when iterating over a collection.
UPDATE
I initially made this a comment, but here is a further bit of information:
I should note that my knowledge comes from experience and articles I've read a long time ago. There is a chance that you can execute the code above because (I believe) the query contains references to the selected object within blacklist. blacklist might be able to change, but not query. If you were strictly iterating over blacklist, you would not be able to add to the blacklist collection.

Your code as presented would not throw an exception. The collection being iterated (myDic) is not the collection being modified (blacklist or ret).
What will happen is that each iteration of the loop will evaluate the current item against the query predicate, which would inspect the blacklist collection to see if it contains the current item's key. This is lazily evaluated, so a change to blacklist in one iteration will potentially impact subsequent iterations, but it will not be an error. (blacklist is fully evaluated upon each iteration, its enumerator is not being held.)

fastest way to copy from a collection of one type to another c#

I get a collection of business entities (table data) equal to around 49000 . I am trying to copy certain values from this collection to anothr collection using Add method and manually adding values to its properties
For example
//Returns arounf 49000 VendorProduct collection this refers to the data in the table VendorProduct
List<VendorProduct> productList = GetMultiple(req);
foreach (VendorProduct item in productList)
{
VendorSearchProductWrapper wrapperItem = new VendorSearchProductWrapper();
IEnumerable<ClientProductPreference> prodPrefClient = item.ClientProductPreferences.Where(e => (e.VendorProductId.Equals(item.Id)
&& (e.AccountId.Equals(SuiteUser.Current.AccountId))));
if (prodPrefClient.Count() == 1)
{
foreach (ClientProductPreference it in prodPrefClient)
{
wrapperItem.ClientProdID = it.Id;
wrapperItem.ClientProductDescription = it.Description;
wrapperItem.MarkupPct = it.MarkUpPct;
wrapperItem.SalesPrice = it.SalesPrice;
}
}
wrapperItem.Code = item.Code;
wrapperItem.ListPrice = item.ListPrice;
wrapperItem.Id = item.Id;
wrapperItem.DiscountPct = ValueParser.SafeDecimal(item.Discount, null);
wrapperItem.Discountgroup = item.DiscountGroup.DiscountGroup;
wrapperItem.Description = item.Description;
wrapperItem.Unit = item.Unit;
wrapperItem.ClientName = item.Account.ClientName;
products.Add(wrapperItem);
}
To copy all of these 49000 records it takes lot of time .Infact for 5 mins only 100-200 records get added to the List.
I need to copy these values faster say in about 1 Minute.
Thanks In Advance
Francis P.

IEnumerable<ClientProductPreference> prodPrefClient = item.ClientProductPreferences.Where(e => (e.VendorProductId.Equals(item.Id)
&& (e.AccountId.Equals(SuiteUser.Current.AccountId))));
if (prodPrefClient.Count() == 1)
{
foreach (ClientProductPreference it in prodPrefClient)
{
There is so much wrong in this code.
Try retrieving this value as SingleOrDefault and then checking for NULL. The way you are doing it, is iterating TWICE over same data. First to get count, second to iterate in foreach (that is useless too, because you know collection have only 1 item and creating whole one iterator for one item is crazy)
Is it possible to use some kind of Dictionary?
Check for lazy loading. When you know you will need the data (and I can see you need them), you should eager-load them to reduce ammount of database calls.
Best way to do this is to make dedicated SQL (or LINQ, because its quite simple) query, that would execute completly on DB. This will be fastest solution possible.

This should be happening faster, even with code you have. Are you sure there aren't any lengthy operations in there? Eg. lazy loading, other data calls, ...
Even small ones can have a big impact on a 49000 iteration

I agree with #Bertvan. The iteration should not take that much time (even if the records are 49K).
I would suggest considering the following lines ( which could be creating problem):
if (prodPrefClient.Count() == 1)
Not sure what are you trying to achive here, But this call is iterating a lazy collection. Please consider if you need this check.
wrapperItem.DiscountPct = ValueParser.SafeDecimal(item.Discount, null);
Int/double comparers do eat some processing time. You should also check the total time taken by the operation by removing this line of code.
Hope this would help.