Optimizing HTTP request and multiple Split on CSV file - c#

I'm trying to read a CSV file from a website, then split the initial string by \n, then split again by ,.
When I try to print out the content of one of the arrays, it was very slow, it takes almost one second between each Console.WriteLine() that prints each element.
I'm not entirely sure why it takes such great deal of time to print.
Any pointers will help
public List<string[]> list = new List<string[]>();
public List<string[]> Content
{
get
{
using (var url = new WebClient())
{
_content = url.DownloadString("https://docs.google.com/spreadsheets/d/1DDhAd98p5RwXqvV53P2YvaujIQEg28HjeXasrCge9Qo/pub?output=csv");
}
var urlArr = _content.Split('\n');
foreach (var i in urlArr)
{
var contentArr = i.Split(',');
List.Add(contentArr);
}
return list;
}
}
Main
var data = new ReadCSV();
for(var i = 0; i < data.Content[2].Length; i++)
Console.WriteLine(data.Content[2][i]);

You should cache the results in a variable, either in the Content property or before the loop because currently your code downloads and split the string every time in the loop which is why it is taking 1 second
So, your code should look like this:
var data = new ReadCSV();
var content = data.Content[2];
for(var i = 0; i < content.Length; i++)
Console.WriteLine(content[2][i]);

Related

how to find average with strings c#

I need to set a variable as the average of 3 other variables, which are numbers but they are set as strings. How do I do this? I'm using c#, visual studio, windows forms.
The variable i'm trying to set is called skiTime, the variables i'm using to get the average are called skiTime1, skiTime2 and skiTime3.
basically i need the c# version of: skiTime = (skiTime1 + skiTime2 + skiTime3) / 3
The code where I start (declare? I don't know the word to use) the variables
List<string> skiTime1 = new List<string>();
List<string> skiTime2 = new List<string>();
List<string> skiTime3 = new List<string>();
string skiTime
The code where i set the value for the variables:
using (StreamReader sr = new StreamReader("pupilSkiTimes.txt"))
{
string line = "";
while ((line = sr.ReadLine()) != null)
{
string[] components = line.Split("~".ToCharArray(), StringSplitOptions.RemoveEmptyEntries);
skiTime1.Add(components[2]);
skiTime2.Add(components[3]);
skiTime3.Add(components[4]);
}
sr.Close();
}
I need to display skiTime1, skiTime2 and skiTime3 in a data grid view, so i think they need to be strings, if i'm not mistaken. skiTime will only be used in another calculation so maybe it can be turned into an int. I don't really know what i'm doing and only got this far because of tutorials, help.
I can post the whole code if this question is too confusing or doesn't have enough information.
public string CalculateAverage(List<string> skiTime1, List<string> skiTime2, List<string> skiTime3)
{
List<string> allValues = new List<string>();
allValues.AddRange(skiTime1);
allValues.AddRange(skiTime2);
allValues.AddRange(skiTime3);
float totalcount = 0;
float average = 0;
foreach (var value in allValues)
{
totalcount = totalcount + float.Parse(value);
}
average = totalcount / allValues.Count();
return average.ToString();
}
Function for returning the average value
Now call the function where u need like:
string skiTime = CalculateAverage(skiTime1, skiTime2, skiTime3);
You need to parse the strings to decimals then calculate the average:
List<decimal> avgTime = new List<decimal>();
for (var i = 0; i < skiTime1.Length; i++) {
var avg = (decimal.Parse(skiTime1[i]) + decimal.Parse(skiTime2[i]) + decimal.Parse(skiTime3[i])) / 3;
avgTime.Add(avg);
}

How can I split a string to store contents in two different arrays in c#?

The string I want to split is an array of strings.
the array contains strings like:
G1,Active
G2,Inactive
G3,Inactive
.
.
G24,Active
Now I want to store the G's in an array, and Active or Inactive in a different array. So far I have tried this which has successfully store all the G's part but I have lost the other part. I used Split fucntion but did not work so I have tried this.
int i = 0;
for(i = 0; i <= grids.Length; i++)
{
string temp = grids[i];
temp = temp.Replace(",", " ");
if (temp.Contains(' '))
{
int index = temp.IndexOf(' ');
grids[i] = temp.Substring(0, index);
}
//System.Console.WriteLine(temp);
}
Please help me how to achieve this goal. I am new to C#.
If I understand the problem correctly - we have an array of strings Eg:
arrayOfStrings[24] =
{
"G1,Active",
"G2,Inactive",
"G3,Active",
...
"G24,Active"
}
Now we want to split each item and store the g part in one array and the status into another.
Working with arrays the solution is to - traverse the arrayOfStrings.
Per each item in the arrayOfStrings we split it by ',' separator.
The Split operation will return another array of two elements the g part and the status - which will be stored respectively into distinct arrays (gArray and statusArray) for later retrieval. Those arrays will have a 1-to-1 relation.
Here is my implementation:
static string[] LoadArray()
{
return new string[]
{
"G1,Active",
"G2,Inactive",
"G3,Active",
"G4,Active",
"G5,Active",
"G6,Inactive",
"G7,Active",
"G8,Active",
"G9,Active",
"G10,Active",
"G11,Inactive",
"G12,Active",
"G13,Active",
"G14,Inactive",
"G15,Active",
"G16,Inactive",
"G17,Active",
"G18,Active",
"G19,Inactive",
"G20,Active",
"G21,Inactive",
"G22,Active",
"G23,Inactive",
"G24,Active"
};
}
static void Main(string[] args)
{
string[] myarrayOfStrings = LoadArray();
string[] gArray = new string[24];
string[] statusArray = new string[24];
int index = 0;
foreach (var item in myarrayOfStrings)
{
var arraySplit = item.Split(',');
gArray[index] = arraySplit[0];
statusArray[index] = arraySplit[1];
index++;
}
for (int i = 0; i < gArray.Length; i++)
{
Console.WriteLine("{0} has status : {1}", gArray[i] , statusArray[i]);
}
Console.ReadLine();
}
seems like you have a list of Gxx,Active my recomendation is first of all you split the string based on the space, which will give you the array previoulsy mentioned doing the next:
string text = "G1,Active G2,Inactive G3,Inactive G24,Active";
string[] splitedGItems = text.Split(" ");
So, now you have an array, and I strongly recommend you to use an object/Tuple/Dictionary depends of what suits you more in the entire scenario. for now i will use Dictionary as it seems to be key-value
Dictionary<string, string> GxListActiveInactive = new Dictionary<string, string>();
foreach(var singleGItems in splitedGItems)
{
string[] definition = singleGItems.Split(",");
GxListActiveInactive.Add(definition[0], definition[1]);
}
What im achiving in this code is create a collection which is key-value, now you have to search the G24 manually doing the next
string G24Value = GxListActiveInactive.FirstOrDefault(a => a.Key == "G24").Value;
just do it :
var splitedArray = YourStringArray.ToDictionary(x=>x.Split(',')[0],x=>x.Split(',')[1]);
var gArray = splitedArray.Keys;
var activeInactiveArray = splitedArray.Values;
I hope it will be useful
You can divide the string using Split; the first part should be the G's, while the second part will be "Active" or "Inactive".
int i;
string[] temp, activity = new string[grids.Length];
for(i = 0; i <= grids.Length; i++)
{
temp = grids[i].Split(',');
grids[i] = temp[0];
activity[i] = temp[1];
}

Visual Basic C# with Selenium variable in css selector

I am only a week or so into C#, but thanks to forums like this I am able to piece together a bunch of code and get a general understanding. However, I have a problem with using a variable I can't solve.
List<IWebElement> elementList = new List<IWebElement>();
elementList.AddRange(browser.FindElements(By.CssSelector("div[class^='CalendarDays-calendarDays-']>div:nth-child(3)>div>div[class^='CalendarEvent-event-']")));
int t = elementList.Count;
for (int i = 1; i <= t; i++)
{
var item + i = browser.FindElement(By.CssSelector("div[class^='CalendarDays-calendarDays-']>div:nth-child(3)>div>div:nth-child(i)>div>div[class^='CalendarEvent-title']")).GetAttribute("textContent");
}
Problems are "var item + i" and "div:nth-child(i)". The first is automatically creating sequential variables and the second trying to get the nth-child using the variable "i". I can't figure out how to format "i" in either instance.
Any help would be appreciated.
Working Code after edits:
List<IWebElement> elementList = new List<IWebElement>();
elementList.AddRange(browser.FindElements(By.CssSelector("div[class^='CalendarDays-calendarDays-']>div:nth-child(3)>div>div[class^='CalendarEvent-event-']")));
int t = elementList.Count;
List<String> listItems = new List<String>();
for (int i = 1; i <= t; i++)
{
String item = browser.FindElement(By.CssSelector("div[class^='CalendarDays-calendarDays-']>div:nth-child(3)>div>div:nth-child(i)>div>div[class^='CalendarEvent-title']")).GetAttribute("textContent");
listItems.Add(item);
}
Thanks,
Don
You should use a list to store the different items, you have to know the type of GetAttribute. I assume it is String :
List<String> listItems = new List<String>();
for (int i = 1; i <= t; i++)
{
String item = browser.FindElement(By.CssSelector("div[class^='CalendarDays-calendarDays-']>div:nth-child(3)>div>div:nth-child(i)>div>div[class^='CalendarEvent-title']")).GetAttribute("textContent");
listItems.Add(item);
}
then you can access the list items by index listItems[i]
For the second part of the question you have to concatenate the string with the i of the for loop like this:
browser.FindElement(By.CssSelector("div[class^='CalendarDays-calendarDays-']>div:nth-child(3)>div>div:nth-child("+i+")>div>div[class^='CalendarEvent-title']")).GetAttribute("textContent");
You can't do var item + i. Just use the same variable each time, it's inside the loop scope anyway.
To insert the i variable to the locator you can use string interpolation
var item = browser.FindElement(By.CssSelector($"div[class^='CalendarDays-calendarDays-']>div:nth-child(3)>div>div:nth-child({i})>div>div[class^='CalendarEvent-title']")).GetAttribute("textContent");

C# Populate An Array with Values in a Loop

I have a C# console application where an external text file is read. Each line of the file has values separated by spaces, such as:
1 -88 30.1
2 -89 30.1
So line one should be split into '1', '-88', and '30.1'.
What I need to do is to populate an array (or any other better object) so that it duplicate each line; the array should have 3 elements per row. I must be having a brain-lock to not figure it out today. Here's my code:
string line;
int[] intArray;
intArray = new int[3];
int i = 0;
//Read Input file
using (StreamReader file = new StreamReader("Score_4.dat"))
{
while ((line = file.ReadLine()) != null && line.Length > 10)
{
line.Trim();
string[] parts;
parts = line.Split(' ');
intArray[0][i] = parts[0];//error: cannot apply indexing
i++;
}
}
Down the road in my code, I intend to make some API calls to a server by constructing a Json object while looping through the array (or alternate object).
Any idea?
Thanks
If you only need the data to be transferred to JSON then you don't need to process the values of the data, just reformat it to JSON arrays.
As you don't know the number of lines in the input file, it is easier to use a List<>, whose capacity expands automatically, to hold the data rather than an array, whose size you would need to know in advance.
I took your sample data and repeated it a few times into a text file and used this program:
static void Main(string[] args)
{
string src = #"C:\temp\Score_4.dat";
List<string> dataFromFile = new List<string>();
using (var sr = new StreamReader(src))
{
while (!sr.EndOfStream)
{
string thisLine = sr.ReadLine();
string[] parts = thisLine.Split(" ".ToCharArray(), StringSplitOptions.RemoveEmptyEntries);
if (parts.Length == 3)
{
string jsonArray = "[" + string.Join(",", parts) + "]";
dataFromFile.Add(jsonArray);
}
else
{
/* the line did not have three entries */
/* Maybe keep a count of the lines processed to give an error message to the user */
}
}
}
/* Do something with the data... */
int totalEntries = dataFromFile.Count();
int maxBatchSize = 50;
int nBatches = (int)Math.Ceiling((double)totalEntries / maxBatchSize);
for(int i=0;i<nBatches;i+=1)
{
string thisBatchJsonArray = "{\"myData\":[" + string.Join(",", dataFromFile.Skip(i * maxBatchSize).Take(maxBatchSize)) + "]}";
Console.WriteLine(thisBatchJsonArray);
}
Console.ReadLine();
}
to get this output:
{"myData":[[1,-88,30.1],[2,-89,30.1],[1,-88,30.1],[2,-89,30.1],[1,-88,30.1],[2,-89,30.1],[1,-88,30.1],[2,-89,30.1],[1,-88,30.1],[2,-89,30.1],[1,-88,30.1],[2,-89,30.1],[1,-88,30.1],[2,-89,30.1],[1,-88,30.1],[2,-89,30.1],[1,-88,30.1],[2,-89,30.1],[1,-88,30.1],[2,-89,30.1],[1,-88,30.1],[2,-89,30.1],[1,-88,30.1],[2,-89,30.1],[1,-88,30.1],[2,-89,30.1],[1,-88,30.1],[2,-89,30.1],[1,-88,30.1],[2,-89,30.1],[1,-88,30.1],[2,-89,30.1],[1,-88,30.1],[2,-89,30.1],[1,-88,30.1],[2,-89,30.1],[1,-88,30.1],[2,-89,30.1],[1,-88,30.1],[2,-89,30.1],[1,-88,30.1],[2,-89,30.1],[1,-88,30.1],[2,-89,30.1],[1,-88,30.1],[2,-89,30.1],[1,-88,30.1],[2,-89,30.1],[1,-88,30.1],[2,-89,30.1]]}
{"myData":[[1,-88,30.1],[2,-89,30.1],[1,-88,30.1],[2,-89,30.1],[1,-88,30.1],[2,-89,30.1],[1,-88,30.1],[2,-89,30.1],[1,-88,30.1],[2,-89,30.1],[1,-88,30.1],[2,-89,30.1],[1,-88,30.1],[2,-89,30.1],[1,-88,30.1],[2,-89,30.1],[1,-88,30.1],[2,-89,30.1],[1,-88,30.1],[2,-89,30.1]]}
It should be easy to adjust the format as required.
I would create a custom Item class and then populate a list, for easy access and sorting, with self contained items. something like:
public Class MyItem
{
public int first { get; set; }
public int second { get; set; }
public float third { get; set; }
public MyItem(int one, int two, float three)
{
this.first = one;
this.second = two;
this.third = three;
}
}
then you could do:
List<MyItem> mylist = new List<MyItem>();
and then in your loop:
using (StreamReader file = new StreamReader("Score_4.dat"))
{
while ((line = file.ReadLine()) != null && line.Length > 10)
{
line.Trim();
string[] parts;
parts = line.Split(' ');
MyItem item = new Item(Int32.Parse(parts[0]),Int32.Parse(parts[1]),Float.Parse(parts[2]));
mylist.Add(item);
i++;
}
}
As there are numbers like 30.1 so int is not suitable for this, and also it must not be a double[] but double[][]:
string[] lines = File.ReadAllLines("file.txt");
double[][] array = lines.Select(x => s.Split(' ').Select(a => double.Parse(a)).ToArray()).ToArray();
Issue is that int array is single dimensional.
My suggestion is that you can put a class with 3 properties and populate a list of class there. It's better to have class with same property names that you require to build JSON. So that you can easily serialize this class to JSON using some nugets like Newtonsoft and make api calls easily.
Your int array is a single dimensional array yet you're trying to index it like a multidemensional array. It should be something like this:
intArray[i] = parts[0]
(However you'll need to handle converting to int for parts that are fractional)
Alternatively, if you want to use a multidimensional array, you have to declare one.
int[][] intArray = new int[*whatever your expected number of records are*][3]
Arrays have a static size. Since you're reading from a file and may not know how many records there are until your file finishes reading, I recommend using something like a List of Tuples or a Dictionary depending on your needs.
A dictionary will allow you to have quick lookup of your records without iterating over them by using a key value pair, so if you wanted your records to match up with their line numbers, you could do something like this:
Dictionary<int, int[]> test = new Dictionary<int, int[]>();
int lineCount = 1;
while ((line = file.ReadLine()) != null && line.Length > 10)
{
int[] intArray = new int[3];
line.Trim();
string[] parts = line.Split(' ');
for (int i = 0; i < 3; i++)
{
intArray[i] = int.Parse(parts[i]);
}
test[lineCount] = intArray;
lineCount++;
}
This will let you access your values by line count like so:
test[3] = *third line of file*

C# Multithreading Loop Datatable

I have a datatable with 1000 records. Each row has a column with a link.I will loop the datatable and fetch record from the website using the link in the datatable. The code is working fine , but this is taking too much time to retrieve the records. So I need to pass it in multiple threads and fetch records and add all the records to a single datatable. I an using C# , Visual studio 2015.
How can we do using threading C#, Any help appreciated.
Existing code is as below.
for (int i = 0; i < dt.Rows.Count; i++)
{
String years = String.Empty;
dt.Rows[i]["Details"] = GetWebText(dt.Rows[i]["link"].ToString());
}
private String GetWebText(String url)
{
var html = new HtmlAgilityPack.HtmlDocument();
string text= html.LoadHtml(new WebClient().DownloadString(url));
return text;
}
You are going to run in to issues here with the thread-safety of write operations with data tables. So you need to ensure that the operations that you perform are separated nice.
The good thing is that you are actually doing three distinct steps and you can easily break them apart and parallelize the slow part while keeping it thread-safe.
Here's what your code is doing:
var url = dt.Rows[i]["link"].ToString();
var webText = GetWebText(url);
dt.Rows[i]["Details"] = webText;
Let's process the data in these three steps, but only parallize the GetWebText part.
This is how:
var data =
dt
.AsEnumerable()
.Select(r => new { Row = r, Url = r["link"].ToString() })
.AsParallel()
// This `Select` is the only part run in parallel
.Select(x => new { x.Row, WebText = GetWebText(x.Url) })
.ToArray();
foreach (var datum in data)
{
datum.Row["Details"] = datum.WebText;
}
Blocking Collections can solve the problem:
Blocking<string> links= new BlockingCollection<string>();\\ using System.Collections.Concurrent;
Blocking<string> results= new BlockingCollection<string>();
public static void main()
{
//get your datatable
for (int i = 0; i < dt.Rows.Count; i++)
{
ThreadStart t = new ThreadStart(threads);
Thread th = new Thread(t);
th.Start();
}
for (int i = 0; i < dt.Rows.Count; i++)
{
links.add(dt.Rows[i]["link"].ToString());
}
for (int i = 0; i < dt.Rows.Count; i++)
{
dt.Rows[i]["Details"] = results.Take();
}
}
public void threads()
{
while(true)
{
string url= Links.take();//block if links is empty
var html = new HtmlAgilityPack.HtmlDocument();
string text= html.LoadHtml(new WebClient().DownloadString(url));
results.add(text);//add result to the other queue
}
}

Categories