C#+Selenium web scraping - c#

I want to scrape some data from https://zakup.sk.kz.
First, I initialize my browser:
IWebDriver browser = new ChromeDriver();
browser.Navigate().GoToUrl("https://zakup.sk.kz/#/ext?tabs=lot&adst=PUBLISHED&lst=PUBLISHED&page=1");
After, I click to the frame:
IWebElement click = browser.FindElement(By.ClassName("m-found-item__num"));
click.Click();
In this frame exist data that I want to scrape (I found abs path):
IWebElement tru = browser.FindElement(By.XPath("/html[1]/body[1]/ngb-modal-window[1]/div[1]/div[1]/sk-main-dialog[1]/div[2]/div[6]/div[1]/div[1]/div[7]"));
Console.WriteLine(tru.Text);
After this, I need to switch to the next frame with the same structure, and scrape data:
IWebElement next = browser.FindElement(By.XPath("//div[contains(#class, 'm-modal__arrow')]"));
next.Click();
IWebElement tru2 = browser.FindElement(By.XPath("/html[1]/body[1]/ngb-modal-window[1]/div[1]/div[1]/sk-main-dialog[1]/div[2]/div[6]/div[1]/div[1]/div[7]"));
Console.WriteLine(tru2.Text);
But Selenium not found tru2, I don't know why, because every frame has the same structure? Tell me please, what should I do?

when you click the next arrow the current element will be deleted and added after Ajax request finished, here you need WebDriverWait
IWebElement next = browser.FindElement(By.XPath("//div[contains(#class, 'm-modal__arrow')]"));
next.Click();
// wait max 15 seconds
IWait wait = new WebDriverWait(browser, TimeSpan.FromSeconds(15))
IWebElement tru2 = wait.Until(browser => browser.FindElement(By.XPath("(//div[#class="m-infoblock__layout"])[7]")));
Console.WriteLine(tru2.Text);
note that I use Xpath
(//div[#class="m-infoblock__layout"])[7]

Related

C# - Selenium WebDriver can't get Inspect Element code

(Previous thread)
(Thread for the same problem)
I want to extract information from a website, but in order to do that I need to login. Basically I can't authenticate with a HTTP/Web/Server request so I have to do it via browser, using WebDriver Selenium (Chrome). Now since the information can't be found through "view source-code", driver.PageSource doesn't work for example. Now when I do what has been suggested in the mentioned thread, I still get the wrong source code (there has also been the same problem reported in the comments). I need to get the code from the Inspector.
Any ideas? Thanks in advance!
class Program
{
static void Main(string[] args)
{
string url = "link";
IWebDriver driver = new ChromeDriver();
driver.Navigate().GoToUrl(url);
IWebElement email = driver.FindElement(By.Id("login-email-input-field"));
IWebElement password = driver.FindElement(By.Id("login-password-input"));
email.SendKeys("email");
password.SendKeys("password");
IWebElement login = driver.FindElement(By.Id("login-btn"));
login.Click();
driver.Navigate().GoToUrl(url);
Thread.Sleep(10000);
//Trying to get the inspect element code here
IWebElement element = driver.FindElement(By.Id("id"));
var html = element.GetAttribute("innerHTML");
Console.WriteLine(html);
}
}
}

NoSuchElementException identifying username field within Instagram login page using Selenium and C#

I am getting this error and can't solve it.
This is all the code. I heard adding waiting time might solve the issue but not sure how to add it.
IWebDriver driver = new ChromeDriver();
driver.Navigate().GoToUrl("https://www.instagram.com");
IWebElement userName = driver.FindElement(By.Name("username"));
IWebElement password = driver.FindElement(By.Name("password"));
You could call the Until method from an instance of a WebDriverWait class, which you could tell to wait until it finds, say, element username. For example:
var wait = new WebDriverWait(driver, TimeSpan.FromMilliseconds(10000));
wait.Until(x => x.FindElement(By.Name("username")));
Hope that helps, and if you're crawling Instagram is for the right purpose :)
The identify the username field on Instagram login page you have to induce WebDriverWait for the desired ElementToBeClickable() and you can use either of the following Locator Strategies:
Using ExpectedConditions:
IWebDriver driver = new ChromeDriver();
driver.Navigate().GoToUrl("https://www.instagram.com");
IWebElement userName = new WebDriverWait(driver, TimeSpan.FromSeconds(20)).Until(ExpectedConditions.ElementToBeClickable(By.Name("username")));
Using lambda expression:
IWebDriver driver = new ChromeDriver();
driver.Navigate().GoToUrl("https://www.instagram.com");
IWebElement userName = new WebDriverWait(driver, TimeSpan.FromSeconds(20)).Until(x=> x.FindElement(By.Name("username"));
Your issue is not necessarily your code 100%. It really depends on how you are hitting the site. The way I access IG is to login from another site. When I lick login to IG I get the IG popup. Then I need to:
-switch to the popup
-Login
-Handle all of the other questions
-then switch to main
You may need to tweak this based on what you are doing but this should work.
public void InstagramLogin()
{
Driver.FindElement(By.XPath("//button[#class='uppy-DashboardTab-btn']//div[contains(text(), 'Instagram')]")).Click();
Driver.FindElement(By.XPath("//button[contains(text(), 'Connect to Instagram')]")).Click();
Driver.SwitchToPopup();
Driver.WaitForElementDisplayed_byXPath("//input[#name='username']"));
Driver.FindElement(By.XPath("//input[#name='username']")).SendKeys("MyEmail#gmail.com");
Driver.FindElement(By.XPath("//input[#name='password']")).SendKeys("myPassword");
Driver.FindElement(By.XPath("//button[#type='submit']")).Click();
Driver.WaitForElementDisplayed(By.XPath("//button[text()='Not Now']"));
Driver.FindElement(By.XPath("//button[text()='Not Now']").Click();
bool ele = Driver.IsElementPresent(By.XPath("//button[contains(text(), 'Continue')]"));
if (ele)
{
Driver.FindElement(By.XPath("//button[contains(text(), 'Continue')]")).Click();
Driver.WaitForElementNoLongerDisplayed_byXPath("//button[contains(text(), 'Continue')]");
}
Driver.GoToMainHandle();
}

Why i am getting this error when trying to get the search results panel that contains the link for each result?

I'm getting a lot of errors trying to do this simple app with Selenium and C#
I just want to make an application (in windows application or console application) that opens a browser, enter the google page, search for "APPLES" and bring the first 5 results, using selenium.
This is the code i'm using :
IWebDriver driver = new ChromeDriver();
driver.Navigate().GoToUrl("http://google.com");
driver.Manage().Timeouts().ImplicitWait = TimeSpan.FromSeconds(10);
//IWebElement element = driver.FindElement(By.Id("gbqfq"));
driver.FindElement(By.Name("q")).SendKeys("apples");
//element.SendKeys("apples");
// Get the search results panel that contains the link for each result.
IWebElement resultsPanel = driver.FindElement(By.Id("search"));
// Get all the links only contained within the search result panel.
ReadOnlyCollection<IWebElement> searchResults = resultsPanel.FindElements(By.XPath(".//a"));
// Print the text for every link in the search results.
int resultCNT = 1;
foreach (IWebElement result in searchResults)
{
if (resultCNT <= 5)
{
Console.WriteLine(result.Text);
}
else
{
break;
}
resultCNT++;
}
I'm getting a error that it can not find element search:
OpenQA.Selenium.NoSuchElementException: 'no such element: Unable to locate element: {"method":"css selector","selector":"#search"}
This should do what you need:
IWebDriver driver = new ChromeDriver();
driver.Navigate().GoToUrl("http://google.com");
driver.Manage().Timeouts().ImplicitWait = TimeSpan.FromSeconds(10);
driver.FindElement(By.Name("q")).SendKeys("apples");
// Click on the Search button
driver.FindElement(By.Name("btnK")).Click();
// Use a Css Selector to go down to the actual element, in this case <a>
var results = driver.FindElements(By.CssSelector("#rso > div > div > div.r > a"));
foreach (var item in results)
{
//Extract the page title and the url from the result
var title = item.FindElement(By.TagName("h3")).Text;
var url = item.GetProperty("href");
Console.WriteLine($"{title} | {url}");
}
In short, you had this error because you were not searching for the proper element in the page.

How to select a song from the search result in youtube?

I'm trying to click a song out of a list on Youtube.
I'll try make things easy without sharing all my classes, but still show you the elements I'm using.
IwebDriver _webdriver = new ChromeDriver();
_webdriver.Navigate().GoToUrl("https://www.youtube.com/");
var element = wait.Until(x => x.FindElement(By.Id("search")));
element.SendKeys("Perfect");
var element = wait.Until(x => x.FindElement(By.CssSelector("#search-icon-legacy>yt-icon")));
element.Click();
var content = wait.Until(x => x.FindElement(By.Id("contents")));
var songHREF = content.FindElements(By.CssSelector("#video-title"));
songHREF[2].Click();
So, main thing that happens is that 90% of the runs, the songHREF will click on an object(song's link) that actually located on the main page and not the "results" page.
The other 10% it just fails. It doesn't find the songHREF element(element not visible).
Try to wait until element will be clickable:
var wait = new WebDriverWait(driver, TimeSpan.FromMinutes(1));
wait.Until(ExpectedConditions.ElementIsClickable(songHREF[2]));
songHREF[2].Click()
This will wait a least 1 minute until element will be clickable and only then clicks on it.
Full code would be like this:
IwebDriver _webdriver = new ChromeDriver();
_webdriver.Navigate().GoToUrl("https://www.youtube.com/");
var element = wait.Until(x => x.FindElement(By.Id("search")));
element.SendKeys("Perfect");
var element = wait.Until(x => x.FindElement(By.CssSelector("#search-icon-
legacy>yt-icon")));
element.Click();
// refresh the page
Driver.Instance.Navigate().Refresh();
var content = wait.Until(x => x.FindElement(By.Id("contents")));
var songHREF = content.FindElements(By.CssSelector("#video-title"));
var wait2 = new WebDriverWait(driver, TimeSpan.FromMinutes(1));
wait2.Until(ExpectedConditions.ElementIsClickable(songHREF[2]));
songHREF[2].Click();
I have catched the issue. Sometimes the locators are working wrong locating invisible elements, but simply refresh the page will help to fix it. Note: refresh the page before you are locating elements like in the code snippet above.
Normal result when search the elements:
Sometimes comes this result with the same locator:
And the first 25 elements in this case are invisible. But after refreshing the page the result is like in the first case(as expected).
As per your question once you send a character sequence e.g. Perfect to the search field and initiate search, you can create a List of all the songs and then according to your choice e.g. Song which contains the text Lyrics in the heading you can invoke Click() on it and you can use the following solution:
IwebDriver _webdriver = new ChromeDriver();
_webdriver.Navigate().GoToUrl("https://www.youtube.com/");
new WebDriverWait(_webdriver, TimeSpan.FromSeconds(10)).Until(ExpectedConditions.ElementToBeClickable(By.CssSelector("input#search"))).SendKeys("Perfect");
_webdriver.FindElement(By.CssSelector("button.style-scope.ytd-searchbox>yt-icon")).Click();
IList<IWebElement> contents = new WebDriverWait(_webdriver, TimeSpan.FromSeconds(10)).Until(ExpectedConditions.VisibilityOfAllElementsLocatedBy(By.CssSelector("h3.title-and-badge.style-scope.ytd-video-renderer a")));
foreach (IWebElement content in contents)
if(content.GetAttribute("innerHTML").Contains("Lyrics"))
{
content.Click();
break;
}

How can I use click event on multiple page in the same test cases on Selenium and C#

How can I use click event on multiple page in the same test cases on Selenium and C#
For example:
Go to Google Homepage.
Search the resulting Test
Click on the Search button,
The user gets Redirected to the Searched Result page, but when I am trying to select any link it showing error.
public void opengoogle()
{
ChromeOptions option = new ChromeOptions();
option.AddArgument("--headless");
ChromeDriver wd = new ChromeDriver(option);
try
{
wd.Navigate().GoToUrl("https://www.google.co.in/");
Thread.Sleep(2000);
wd.FindElement(By.CssSelector("#lst-ib")).Click();
Thread.Sleep(2000);
wd.FindElement(By.CssSelector("#lst-ib")).Click();
wd.FindElement(By.CssSelector("#lst-ib")).Clear();
wd.FindElement(By.CssSelector("#lst-ib")).SendKeys("Test");
wd.FindElement(By.CssSelector("#lst-ib")).Submit();
}
finally { }
public void TestMethod1()
{
ChromeOptions option = new ChromeOptions();
option.AddArgument("--headless");
ChromeDriver webDriver = new ChromeDriver(option);
try
{
webDriver.Navigate().GoToUrl("https://www.google.lk/");
WebDriverWait wait = new WebDriverWait(webDriver, new TimeSpan(0, 0, 0, 30));
IWebElement searchTextField= wait.Until(ExpectedConditions.ElementToBeClickable(By.Id("lst-ib")));
searchTextField.SendKeys("Test");
IWebElement searchButton = wait.Until(ExpectedConditions.ElementToBeClickable(By.CssSelector(".lsb")));
searchButton.Click();
wait.Until(ExpectedConditions.TitleContains("Test"));
IWebElement searchResultLink= wait.Until(ExpectedConditions.ElementToBeClickable(By.CssSelector("a[href='http://www.speedtest.net/']")));
searchResultLink.Click();
}
finally { }
}
You can try above code.
It explains following process.
First move to Google search page.
Insert "Test" in to search text field.
Then click on Google search button.
After Navigate to search results page, click on "www.speedtest.net" link.
You can try out this code:
ChromeOptions option = new ChromeOptions();
option.AddArgument("--headless");
ChromeDriver wd = new ChromeDriver(option);
wd.Navigate().GoToUrl("https://www.google.co.in/");
WebDriverWait wait = new WebDriverWait(driver, TimeSpan.FromMinutes(1));
wait.Until(ExpectedConditions.ElementToBeClickable(By.Name("q")));
wd.FindElement(By.Name("q")).SendKeys("Test"+Keys.RETURN);
wait.Until(ExpectedConditions.VisibilityOfElementLocated(By.CssSelector("a[href='http://www.speedtest.net/']")));
driver.FindElement(By.CssSelector("a[href='http://www.speedtest.net/']")).Click();
P.S :
1. I'm not good in C# though I've tried to convert a java code to C#. Hope this will be helpful for you.
2. This will always open the speedtest.net website. If you want something else, you can have it by replacing the cssSelector which I have written.

Categories