I'm trying to scrape books.toscrape.com everything seems perfect, but it doesn't output anything to console.
I'm sure about the XPaths are correct, and syntax is right.
I don't see any errors, or warnings.
Don't have any clue what i can try for this problem.
using System;
using System.Windows;
using OpenQA.Selenium;
using OpenQA.Selenium.Firefox;
using System.Data;
using System.Collections.Generic;
namespace book_scraping
{
/// <summary>
/// Interaction logic for MainWindow.xaml
/// </summary>
public partial class MainWindow : Window
{
DataTable table = new DataTable();
public MainWindow()
{
InitializeComponent();
}
string user_url;
class Book
{
public string Titlex { get; set; }
public string Price { get; set; }
public string Rate { get; set; }
}
public void Scrape()
{
var books = new List<Book>();
IWebDriver driver = new FirefoxDriver();
user_url = Textbox1.Text;
int.TryParse(Textbox2.Text, out var x);
for (int i = 1; i < x; i++)
{
driver.Url = "http://" + user_url + "/catalogue/" + "page-" + i + ".html";
var element = driver.FindElements(By.XPath("//article[#class='product_pod']"));
foreach (var elements in element) {
var book = new Book
{
Titlex = driver.FindElement(By.XPath("//h3/a")).Text,
Price = driver.FindElement(By.XPath("//p[#class='price_color']")).Text,
Rate = driver.FindElement(By.XPath("//article/p")).GetAttribute("class")?.Replace("star-rating ", ""),
};
foreach (var a in books)
{
Console.WriteLine($"{a.Titlex} {a.Price} {a.Rate}");
}
}
}
}
private void Button_Click(object sender, RoutedEventArgs e)
{
Scrape();
}
}
}
I expect the output of title,price,rate as text like
hello world 50 three, or something similar
Your Books list is empty, try this
books.Add(new Book
{
Titlex = driver.FindElement(By.XPath("//h3/a")).Text,
Price = driver.FindElement(By.XPath("//p[#class='price_color']")).Text,
Rate = driver.FindElement(By.XPath("//article/p")).GetAttribute("class")?.Replace("star-rating ", ""),
});
instead of:
var book = new Book
{
Titlex = driver.FindElement(By.XPath("//h3/a")).Text,
Price = driver.FindElement(By.XPath("//p[#class='price_color']")).Text,
Rate = driver.FindElement(By.XPath("//article/p")).GetAttribute("class")?.Replace("star-rating ", ""),
};
Related
Hello so I am trying to code a simple web view, with asp.net and NEST library, that will take my ElasticSearch database, and show it in textview on button click.
This is the code that I input when my button is clicked,
would you please look at it and tell me am I on a good path or something is not good.
using Elasticsearch.Net;
using Nest;
using System;
using System.Collections.Generic;
using System.Configuration;
using System.Linq;
using System.Web;
using System.Web.UI;
using System.Web.UI.WebControls;
namespace ElasticsearchWeb
{
public class shekspir
{
public string type { get; set; }
public int line_id { get; set; }
public string play_name { get; set; }
public int speech_number { get; set; }
public float line_number { get; set; }
public string speaker { get; set; }
public string text_entry { get; set; }
}
public partial class Default : System.Web.UI.Page
{
public static Uri GetElasticHost()
{
var host = "http://localhost:9200";
return new Uri(host);
}
public static ElasticClient GetElasticClient(ConnectionSettings settings = null)
{
if (settings == null)
{
var node = GetElasticHost();
var pool = new SingleNodeConnectionPool(node);
settings = new ConnectionSettings(pool);
}
settings.DisableDirectStreaming(true);
var client = new ElasticClient(settings);
return client;
}
public static List<shekspir> GetAllShekspir(int ID)
{
var workОfShakespeare = GetElasticClient();
ISearchResponse<shekspir> result = null;
result = workОfShakespeare.Search<shekspir>(x => x
.Index("shekspir")
.Query(q => q
.MatchAll())
.Size(100)
);
List<shekspir> list = new List<shekspir>();
foreach (var r in result.Hits)
{
shekspir a = r.Source;
list.Add(a);
}
return list;
}
protected void Page_Load(object sender, EventArgs e)
{
}
protected void Button1_Click(object sender, EventArgs e)
{
List<shekspir> list = GetAllShekspir (1);
foreach (shekspir u in list)
{
litInfo.Text += u.play_name + ": " + u.text_entry + "<br>";
}
}
}
}
List<shekspir> list = new List<shekspir>();
foreach (var r in result.Hits)
{
shekspir a = r.Source;
list.Add(a);
}
If you just want the returned documents above can be replaced by
var list= result.Documents
So i can't seem to find anything about this online.
So i'm wanting to know how to print data from a text file to the console application through a method.
Here's an example,
This is the class Student, and the method I am wanting to use as a type of layout is DisplayInfo():
class Student
{
public string ID { get; set; }
public double Score { get; set; }
public Student(string SID, double SC)
{
ID = SID;
Score = SC;
}
public void DisplayInfo()
{
Console.WriteLine(ID,": " + Score);
}
}
Here is what i've got so far inside my PrintData void:
public void PrintData()
{
using (StreamReader readtext = new StreamReader("data.txt"))
{
string readMeText = readtext.ReadLine();
}
}
Basically, again, I am wanting to print it in the format shown in DisplayInfo(), inside the PrintData() void.
Let's assume that you have text (demo.txt) file in this format :
s1 140
s2 250
s3 400
Where s1,s2,s3 are studeintID and 140,250,400 are their scores.
Now you need read lines from file while it is not the End of Stream split read line by whitespace and write to console. The following function is doing right that:
public void PrintData()
{
using (var readtext = new StreamReader(#"c:\Users\IIG\Desktop\demo.txt"))
{
while (!readtext.EndOfStream)
{
string currentLine = readtext.ReadLine();
var args = currentLine.Split(' ');
if (args.Length > 1)
{
Console.WriteLine(args[0] + ":" + args[1]);
}
else
{
Console.WriteLine("Invalid line");
}
}
}
}
UPDATE
Here is example how you can use object serialization/deserialization for this:
using System;
using System.Collections.Generic;
using System.IO;
using System.Xml.Serialization;
namespace ConsoleApplication4
{
public class Program
{
static void Main(string[] args)
{
List<Student> students = new List<Student>();
students.Add(new Student("s1", 140));
students.Add(new Student("s2", 200));
students.Add(new Student("s3", 250));
XmlSerializer xmlSerializer = new XmlSerializer(typeof(List<Student>));
if(!File.Exists(#"c:\Users\IIG\Desktop\demo.txt"))
{
using (File.Create(#"c:\Users\IIG\Desktop\demo.txt"))
{
}
}
using (var writer = new StreamWriter(#"c:\Users\IIG\Desktop\demo.txt"))
{
xmlSerializer.Serialize(writer,students);
}
PrintData();
}
[Serializable]
public class Student
{
public string ID { get; set; }
public double Score { get; set; }
public Student()
{
}
public Student(string SID, double SC)
{
ID = SID;
Score = SC;
}
public void DisplayInfo()
{
Console.WriteLine(ID+ ": " + Score);
}
}
public static void PrintData()
{
using (var readtext = new StreamReader(#"c:\Users\IIG\Desktop\demo.txt"))
{
XmlSerializer xmlSerializer = new XmlSerializer(typeof(List<Student>));
List<Student> readStudents = (List<Student>)xmlSerializer.Deserialize(readtext);
foreach (var student in readStudents)
{
student.DisplayInfo();
}
}
}
}
}
Hello I am trying to create a stock log system in c# winforms and i am a bit stuck for ideas on how to read the items back into a list and storing the data into the properties.
I will be reading in from a csv file where each line is 1 item and each property is separated by a comma.
the main Class
using System;
using System.Collections.Generic;
using System.ComponentModel;
using System.Data;
using System.Drawing;
using System.Linq;
using System.Text;
using System.Windows.Forms;
//using System.IO;
//using Microsoft.VisualBasic;
namespace stock_list
{
public partial class Form1 : Form
{
private List<item> itemlist = new List<item>((1));
public Form1()
{
InitializeComponent();
}
private void Form1_Load(object sender, EventArgs e)
{
}
private void btnSave_Click(object sender, EventArgs e)
{
saveitem(Convert.ToInt64(txtStallNumber.Text), Convert.ToInt64(txtStockNumber.Text), txtDescription.Text, Convert.ToDecimal(txtPaidprice.Text), Convert.ToDecimal(txtSoldPrice.Text));
}
private void btnItems_Click(object sender, EventArgs e)
{
readfromfile();
}
private void readfromfile()
{
var reader = new System.IO.StreamReader(#"file.csv", Encoding.UTF8, false);
while (!reader.EndOfStream)
{
//what todo here??
}
}
private void saveitem(long stallnumberpar, long stocknumberpar, string itemdiscriptionpar, decimal boughtpricepar, decimal soldpricepar, decimal profitorlosspar = 0)
{
itemlist.Add(new item { stallnumber = stallnumberpar, stocknumber = stocknumberpar, itemdescription = itemdiscriptionpar, boughtprice = boughtpricepar, soldprice = soldpricepar, profitorloss = soldpricepar - boughtpricepar});
txtDescription.Clear();
txtPaidprice.Clear();
txtSoldPrice.Clear();
txtStallNumber.Text = "";
txtStockNumber.Clear();
txtStallNumber.Focus();
MessageBox.Show("Item Saved");
}
private void btnQuery_Click(object sender, EventArgs e)
{
RunQueryDescription(Microsoft.VisualBasic.Interaction.InputBox("Enter Search Criteria", "Enter Search Criteria", "Default",0,0));
}
private void RunQueryDescription(string description)
{
//List<item> products = new List<item>((1));
var writer = new System.IO.StreamWriter(#"file.csv", true, Encoding.UTF8);
item[] productsarr = new item[itemlist.Count];
int index = 0;
foreach (item product in itemlist)
{
if (product.itemdescription.Contains(description))
{
productsarr[index] = product;
index++;
}
else
{
index++;
continue;
}
}
for (int i = 0; i < productsarr.Length; i++)
{
MessageBox.Show(productsarr[i].stallnumber.ToString() +
productsarr[i].stocknumber.ToString() +
productsarr[i].itemdescription.ToString() +
productsarr[i].boughtprice.ToString() +
productsarr[i].soldprice.ToString() +
productsarr[i].profitorloss.ToString());
writer.Write(productsarr[i].stallnumber.ToString() + "," +
productsarr[i].stocknumber.ToString() + "," +
productsarr[i].itemdescription.ToString() + "," +
productsarr[i].boughtprice.ToString() + "," +
productsarr[i].soldprice.ToString() + "," +
productsarr[i].profitorloss.ToString());
writer.Close();
writer.Dispose();
}
}
}
}
The items class
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Collections;
using System.IO;
namespace stock_list
{
class item
{
public long stallnumber { get; set; }
public long stocknumber { get; set; }
public string itemdescription { get; set; }
public decimal boughtprice { get; set; }
public decimal soldprice { get; set; }
public decimal profitorloss { get; set; }
}
}
EDIT:
Example File
1,1,Vase,1.00,2.00,1.00
Any help will be valued
Thanks in Advance!
You can use the File.ReadAllLines to read the file
private List<item> itemlist = new List<item>();
private void readfromfile()
{
var lines = System.IO.File.ReadAllLines("path");
foreach (string item in lines)
{
var values = item.Split(',');
itemlist.Add(new item()
{
stallnumber = long.Parse(values[0]),
stocknumber = long.Parse(values[1]),
itemdescription = values[2],
//and so on
});
}
}
Try splitting each line on the delimiter (most likely a comma), and then parse out the line and add a new instance of item to your list. Something like this should work inside your loop (untested):
var line = reader.ReadLine();
var values = line.Split(',');
itemlist.Add(new item { stallnumber = Convert.ToInt32(values[0]), ... });
I am using SQLite for a data entry windows 8 app I am working on. I can create the db, insert data, retrieve a column count, and read data, but cannot get the column names.
The underlying framework is from this post.
I read about the PRAGMA table_info(table_name); command but I cannot seem to send and read back this query properly. I have been googling for 3 days!
MainPage.xaml.cs:
using SQLite;
using SqlLiteTest.Model;
using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.Linq;
using Windows.Storage;
using Windows.UI.Popups;
using Windows.UI.Xaml;
using Windows.UI.Xaml.Controls;
namespace SqlLiteTest
{
public sealed partial class MainPage : Page
{
public MainPage()
{
this.InitializeComponent();
txtPath.Text = ApplicationData.Current.LocalFolder.Path;
}
private async void createDB(object sender, RoutedEventArgs e)
{
// access local folder
var qvLocalFolder = Windows.Storage.ApplicationData.Current.LocalFolder;
try
{
//Create a blank carrier file
StorageFile qvLocalFileCarrier = await qvLocalFolder.CreateFileAsync("qvdbLocal.db", CreationCollisionOption.FailIfExists);
//Write the blank carrier file
await FileIO.WriteTextAsync(qvLocalFileCarrier, "");
}
catch { }
// connect
var path = Windows.Storage.ApplicationData.Current.LocalFolder.Path + #"\qvdbLocal.db";
var db = new SQLiteAsyncConnection(path);
// create table
await db.CreateTableAsync<qvdb>();
// insert data
var insertRecords = new List<qvdb>()
{
new qvdb
{
qvdbRecord = 1,
qvdbNotes = "Notes1",
qvdb001 = "Variable 1.1",
qvdb002 = "Variable 2.1"
},
new qvdb
{
qvdbRecord = 1,
qvdbNotes = "Notes1",
qvdb001 = "Variable 1.1",
qvdb002 = "Variable 2.1"
},
new qvdb
{
qvdbRecord = 1,
qvdbNotes = "Notes1",
qvdb001 = "Variable 1.1",
qvdb002 = "Variable 2.1"
},
};
await db.InsertAllAsync(insertRecords);
// read count
var allUsers = await db.QueryAsync<qvdb>("SELECT * FROM qvdb");
var count = allUsers.Any() ? allUsers.Count : 0;
Debug.WriteLine(count);
}
private async void updateDB(object sender, RoutedEventArgs e)
{
var path = Windows.Storage.ApplicationData.Current.LocalFolder.Path + #"\qvdbLocal.db";
var db = new SQLiteAsyncConnection(path);
var tempCell = db.QueryAsync<qvdb>("UPDATE qvdb SET qvdbNotes ='!##$%$%^^&*()+)(*&^%$##!{:L<>?' WHERE qvdbRecord = 10");
await db.UpdateAsync(tempCell);
}
private async void readDB(object sender, RoutedEventArgs e)
{
var path = Windows.Storage.ApplicationData.Current.LocalFolder.Path + #"\qvdbLocal.db";
var db = new SQLiteAsyncConnection(path);
var query = db.Table<qvdb>();
var result = await query.ToListAsync();
foreach (var item in result)
{
MessageDialog dialog = new MessageDialog(string.Format("{0} {1} {2}", item.qvdbRecord, item.qvdbNotes, item.qvdb001));
await dialog.ShowAsync();
}
}
private void readColNames(object sender, RoutedEventArgs e)
{
}
}
}
qvdb.cs
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using SQLite;
namespace SqlLiteTest.Model
{
public class qvdb
{
[PrimaryKey, AutoIncrement]
public int qvdbRecord { get; set; }
[MaxLength(3000)]
public string qvdbNotes { get; set; }
[MaxLength(1000)]
public string qvdb001 { get; set; }
[MaxLength(1000)]
public string qvdb002 { get; set; }
}
}
Thanks CL for the info. I added the class but still do not know how to access them. Some more code...
// this works
// read record count
var allRecords = await db.QueryAsync<qvdb>("SELECT * FROM qvdb");
var countRecords = allRecords.Any() ? allRecords.Count : 0;
this.textboxLog.Text = this.textboxLog.Text + Environment.NewLine + "There are " + countRecords + " records.";
// ??
// read column names
var allColumns = await db.QueryAsync<qvdb>("PRAGMA table_info(qvdb)");
foreach (var item in allColumns) {
//read name
this.textboxLog.Text = this.textboxLog.Text + Environment.NewLine + "columbn names";
}
The records returned by PRAGMA table_info look like this:
public class table_info_record
{
public int cid { get; set; }
public string name { get; set; }
public string type { get; set; }
public int notnull { get; set; }
public string dflt_value { get; set; }
public int pk { get; set; }
}
Use it like this:
db.QueryAsync<table_info_record>("PRAGMA table_info(...)");
o end the loop on CL's advice, this code successfully reads the column names:
// read column names
var query = await db.QueryAsync<table_info_record>("PRAGMA table_info(MY_TABLE_NAME_HERE)");
foreach (var item in query)
{
Debug.WriteLine(string.Format("{0}", item.name) + " is a column.");
}
I'm working on a bigger project atm, but I made this simple example to show you what happens..
using System.Collections.Generic;
using System.Windows;
namespace txt
{
public partial class MainWindow
{
public MainWindow()
{
InitializeComponent();
}
private void Window_Loaded(object sender, RoutedEventArgs e)
{
var obsLst = new List<Info> { new Info { name = "asd" }, new Info { name = "asd" }, new Info { name = "asd" }, new Info { name = "asd" } };
var temp = new List<Info>();
for (var i = 1; i <= 3; i++)
{
temp.Add(obsLst[0]); //I add 3 of the same item from obsLst to temp
}
lst.DataContext = temp; //lst = ListBox
}
}
public class Info
{
public string name { get; set; }
}
}
The ListBox ItemsSource is set to {Binding}..
When I start the application I get 3 txt.Info objects displayed and if I click any of them, 2 or even all of them get selected aswell. From my understanding the problem relies in the fact that the listbox selector cannot differentiate between the items and therefor doesn't know which one I selected.
Here's a picture of what it looks like..
I only clicked on the second txt.Info item.
I found a solution where someone said that I have to specify the DisplayMemberPath, but I can't really do that in the other project because I have a datatemplate for the object.
Any ideas on how I could fix this would be great..
Thx in advance.
EDIT 1:
this works but it's not nice..
using System.Collections.Generic;
using System.Windows;
namespace txt
{
public partial class MainWindow
{
public MainWindow()
{
InitializeComponent();
}
private void Window_Loaded(object sender, RoutedEventArgs e)
{
var obsLst = new List<Info> { new Info { name = "asd" }, new Info { name = "asd" }, new Info { name = "asd" }, new Info { name = "asd" } };
var temp = new List<Container>();
for (var i = 1; i <= 3; i++)
{
var t = new Container();
t.obj = obsLst[0];
temp.Add(t);
}
lst.DataContext = temp;
}
}
public class Info
{
public string name { get; set; }
}
public class Container
{
public Info obj { get; set; }
}
}
In this case you need to set DisplayMemberPath="obj"
Assigning an ID to the object, doesn't work..
using System;
using System.Collections.Generic;
using System.Windows;
namespace txt
{
public partial class MainWindow
{
public MainWindow()
{
InitializeComponent();
}
private void Window_Loaded(object sender, RoutedEventArgs e)
{
var rand = new Random();
var obsLst = new List<Info> { new Info { name = "asd" }, new Info { name = "asd" }, new Info { name = "asd" }, new Info { name = "asd" } };
var temp = new List<Info>();
for (var i = 1; i <= 3; i++)
{
obsLst[0].id = rand.Next(10000);
temp.Add(obsLst[0]);
}
lst.DataContext = temp;
}
}
public class Info
{
public string name { get; set; }
public int id { get; set; }
}
}
I had this problem too a while ago, i fixed it by adding the id # to the item so they're always different.