We Keep Coding

Loop across a dataframe in .NET Spark

I have a dataframe(created by reading a csv) in Spark, how do I loop across the rows in this dataframe in C#. There are 10 rows and 3 columns in the dataframe and I would like to get the value for each of the column as I navigate through the rows one by one. Below is what I am trying:
foreach (var obj in df)
{
Console.WriteLine("test");
}
foreach statement cannot operate on variables of type 'DataFrame' because 'DataFrame' does not contain a public instance definition for 'GetEnumerator'

The DataFrame is a reference to the actual data on the spark cluster. If you want to see the actual data (as opposed to running some transformation and writing to the output which is the typical use case) you need to collect the data over to your application.
https://learn.microsoft.com/en-us/dotnet/api/microsoft.spark.sql.dataframe.collect?view=spark-dotnet
foreach (var obj in df.Collect())
{
Console.WriteLine("test");
}
This will give you an enumerable of Row which has Values which is an object array of the actual values.
If you just wanted to see the contents for debugging then you can do:
df.Show();
Show takes two arguments, the first is the number of rows and the second is how many chars width to show in case your data is truncated and you need to see all the columns:
df.Show(100, 10000);

C# How to add string variables with values to the any System.Collections class and retrieve them

So I have a slightly interesting problem right here. What I'm trying to achieve is to add between 1000 to 2000 string variables to the list and later compare them with another strings. So far I can achieve this by using below code:
var list = new List<string>();
var item_1 = "USA";
var item_2 = "Canada";
var item_3 = "Cuba";
...................
var item_N = "Country_N";
list.Add(item_1);
list.Add(item_2);
list.Add(item_3);
.................
list.Add(item_N);
And comparing them with another strings looks like below (using FluentAssertions class):
list[0].Should().Match(stringToCompare_1);
list[1].Should().Match(stringToCompare_2);
list[2].Should().Match(stringToCompare_3);
The biggest problem that I see here is it's sort of hard to memorize what string was added to a particular index of the list (unstoppable scrolling between the code is boring). My question: is there any more elegant way to handle this situation? Something that might look like below (List class method ValueOf is fictional):
var list = new List<string>();
var item_1_USA = "USA";
var item_2_Canada = "Canada";
var item_3_Cuba = "Cuba";
list.ValueOf(item_1_USA).Should().Match(stringToCompare_1);
list.ValueOf(item_1_Canada).Should().Match(stringToCompare_2);
list.ValueOf(item_1_Cuba).Should().Match(stringToCompare_3);
As many suggested (including #Postlagerkarte) to edit the question to clarify what I'm trying to achieve. I'm testing the web app, and while going through every step of this application (imagine booking engine), I need to capture and store different info (like user's First Name, Last Name, Email Address, Phone Number, etc). The amount of the captured info can sometimes exceed 2000 items. Currently I'm using data structure List. Later, at the final step of my application, I need to compare my stored values with what ever present on this final page. For instance: user's first name, that was captured on Step 2 of the booking engine must match with value on final step. List Contains method won't be suitable here as it can validate incorrect information. Accessing the values using list[0]....list[N] is very inconvenient, as I can forget what exactly was stored at that index. I'm solving it now by scrolling through the code to return back. Any wise navigation is appreciated.

You can use a Dictionary<string, string> for this. The key of each value would be a field or info identifier: userFirstName, userAge, etc. As you process data you add it to your dictionary with whatever value you are reading.
When validating, you look up the stored value with the field identifier and compare it to whatever data shows upon your last page.
You can also consider using an enumeration as your key instead of string although if you avoid magic strings literals and use constants you should be ok.

Why can't you use the Contains() method saying
list.Contains(stringToCompare_1);

How to read records from last to top in stack?

Below is my variable of type stack :
var records = Stack<Records>();
Suppose records contains data like below for eg:
3
2
1
5
8
Now I want to loop on this above variable but in reverse order like below:
foreach (var item in records)
{
// item should have 8 then 5 then 1......
}
Note : I don't want to create overhead of converting this in to IEnumerable or List and then reverse loop.

You can use Reverse():
foreach (var item in records.Reverse()) {
...
Additional info for OP:
Stack is implemented as an array.
Stacks and queues are useful when you need temporary storage for
information; that is, when you might want to discard an element after
retrieving its value. Use Queue if you need to access the
information in the same order that it is stored in the collection. Use
System.Collections.Generic.Stack if you need to access the
information in reverse order.

append fields dynamically at runtime based on user input in c#

In c# how would I best handle this scenario, a form contains a large number of checkboxes 100+ which map to columns/fields in the data store, and the user can select any of them. Depending on those they choose when they submit the form I will generate a csv output of the data but only showing the fields(columns) they have requested.
list myData = getAllTheData
var fieldstheychose = "name, mileage, address"
foreach (var item in mydata)
{
var somedata = new StringBuilder();
// make decision to include field or not
somedata.Append(item.name.formatvalue() + ",");
// make decision to include field or not
somedata.Append(item.mileage.GetCSVEncodedValue() + ",");
// make decision to include field or not
somedata.Append(item.address.ToDisplayString() + ",");
// make decision on 100+ fields that might be in the fieldstheychose list depending on their choices
}
return somedata;
I need to at runtime decide which fields will be appended above but I wanted to avoid 100+ if blocks to decide weather to append the data.
I did look at dynamic objects and expando objects but I'm not sure it helps in this scenario.
Any views appreciated.

List<ListItem> selected = new List<ListItem>();
foreach (ListItem item in YourCheckBoxListID)
if (item.Selected)
selected.Add(item);
This will store the ListItems that have been checked assuming you are using a CheckBoxList. From there you can iterate through your selected list and convert the entries into proper csv format.

Give the name of the checkboxes same to what you have in the list mydata. Loop through the list and find your check-boxes using the findControl() function to get the checkbox checked Boolean value. Add it to the list.

How can I modify a repeater to not show data for every row bound to it?

I have a stored proc that returns 4 rows in a table. Each row corresponds to a day a person is staying at a hotel, so when I bind a repeater to it, 4 rows show up. Two of the rows are the same room and the other two are the same room. I want to display credit card information, but not for each individual room, only for every set of rooms that is the same, so it would look something like this:
12/7/2011 Room 1
12/8/2011 Room 1
Credit Card Info
12/7/2011 Room 2
12/8/2011 Room 2
Credit Card Info
I don't want to combine Room information and credit information into a one row each. How can I get this to format the way I want it to?
Currently I am displaying Room Information in an ItemTemplate and inside the ItemTemplate is a tr and td's.

You basically have two options: you can pre-process the data so that what is bound to the repeater represents what you want to present to the user - DJ KRAZE has shown one way to do that.
The other option would be to add an ItemDataBound event handler and manipulate the data and/or the current row at that point.
One drawback of the 2nd approach is that you need access to the prior row in order to make your decision - that may complicate things, especially if you have any paging of the results.

//Create either a HashTable()
//or a Dictionary<int, string[]> I would personally use a Dictionary<>
//create an instance of what ever collection you want to use or array
//List<string>
//Dictionary<int, string[]>() using the room number as the key
// you would add the values to either the List<> or a String[] array or a Dictionary
keey in mind that if you want to create a dynamic string[] array and you do not know how many elements you are going to add just use a List<> or Dictionary<int,string>
and you can always convert that object using .ToArray
so for example
List<string> lstRoomList = new List<string>();
string[] arrayRoom = {};
if you have a DataReader for example and you are doing something like
while (yourdataReader.Read())
{
// here you can decide how you are going to work with the data you
lstRoomList.Add(datareader["roomnubmer"] + " " + datareader[CreditCarInfo];
}
at the end you could assign the values of that list to a dynamic array
arrayRoom = kstRoomList.ToArray(); I would really need to see what you are using and how you are returning the StoredProc results..