So, below is my basic code to try out Browsermob Proxy, and the output generated.
The issue is with the output, which appear to be (1) incomplete in number and (2) not as detailed as I would manually check the network statistics in Dev tools (Firefox or Chrome).
Is it possible to have more detailed info in the HAR file? I would like to know, for example, if a specific Javascript has been loaded (or is there a better solution to this?).
// Supply the path to the Browsermob Proxy batch file
Server server = new Server(#"C:\Users\Frederik\Desktop\SeleniumTestje\browsermob-proxy-2.1.0-beta-6\bin\browsermob-proxy.bat");
server.Start();
Client client = server.CreateProxy();
client.NewHar("google");
var seleniumProxy = new Proxy { HttpProxy = client.SeleniumProxy };
var profile = new FirefoxProfile();
profile.SetProxyPreferences(seleniumProxy);
// Navigate to the page to retrieve performance stats for
IWebDriver driver = new FirefoxDriver(profile);
driver.Navigate().GoToUrl("http://www.google.co.uk");
// Get the performance stats
HarResult harData = client.GetHar();
foreach(Entry e in harData.Log.Entries)
{
Console.WriteLine(e.Request.Url, e.Request.Headers);
}
Output in console:
http://ocsp.digicert.com/
http://ocsp.digicert.com/
http://ocsp.digicert.com/
http://www.google.co.uk/
http://clients1.google.com/ocsp
http://clients1.google.com/ocsp
http://clients1.google.com/ocsp
http://clients1.google.com/ocsp
I had the same issue with Fiddlercore but I have found a solution for that:
Fiddlercore - Requested resource URL's are generic (OSCP-related) instead of actual resource
I ran into this issue as well. It ended up being Chrome's networking tools enables its own cache by default, so BrowserMobProxy is not aware of all of the requests.
To fix the automated .har so that it is identical to a manual one, select the Disable Cache checkbox in the network tools. More info here: Disabling Chrome cache for website development
Related
We've created a Selenium test project that starts the (ASP.NET) web application and runs a couple of tests using the ChromeDriver. Locally this all runs fine (in headless and non-headless mode).
But on the build server (using an Azure DevOps agent) this fails without ever starting the tests. It looks like it fails when starting the ChromeDriver: the driver starts, but then it's immediately followed by 403 errors. It never gets to the part where it actually loads a webpage.
Any ideas where to look?
Answering my own question to document possible solutions.
After some rigorous investigation (which included using the source code to get to the bottom of things) we found out that the proxy server somehow got in the way. It turned out that the ChromeDriver tries to communicate over a local port (e.g. http://localhost:12345), which was redirected through the proxy server. This failed with a 403 error.
This gave us a lead on possible solutions. First we tried to use the .proxybypass file to exclude localhost addresses. This didn't work -- it turns out that this proxy bypass only works for https requests. And the ChromeDriver control commands are sent over http :-(
We then made sure that no proxy was used in our test code. We did this with the following lines:
var options = new ChromeOptions();
options.AddArgument("--no-sandbox");
options.AddArgument("headless");
options.AddArgument("ignore-certificate-errors");
options.Proxy = new Proxy()
{
Kind = ProxyKind.Direct
};
var driver = new ChromeDriver(options);
In addition to these settings (note that some arguments were added to solve other issues and might not apply to your own situation), we also disabled the proxy for other requests:
WebRequest.DefaultWebProxy = null;
HttpClient.DefaultProxy = new WebProxy()
{
BypassProxyOnLocal = true,
};
This allowed our tests to finally run on the build server without the 403 errors.
One last remark (which might be obvious) is to always run your tests in non-headless mode if you encounter any issues. This allowed us to see the "invalid certificate error" which would otherwise be hidden.
im writing a automation tool in visual studio with c#.
im using selenium with chromedriver. i need proxy user pass auth for my session. But i try so many times and so many sources but didnt work.
How to basicly user pass ip port proxy for my chromedriver session. Thanks for help.
There are a couple of options.
Configuring a proxy using ChromeOptions and org.openqa.selenium.Proxy
If you want the solution to be easily portable to run on other systems, you can try configuring a proxy using ChromeOptions and org.openqa.selenium.Proxy. There will be some additional methods in here to configure your proxy, but we need to know what type of proxy and type of authentication is supported.
// Add the WebDriver proxy capability.
Proxy proxy = new Proxy();
proxy.setHttpProxy("myhttpproxy:3337");
options.setCapability("proxy", proxy);
// Add a ChromeDriver-specific capability.
ChromeDriver driver = new ChromeDriver(options);
Create a new chrome profile with proxy configuration
Alternatively, you could create a profile for chrome with the proxy configured and tell chrome driver to work with that. This would be difficult, if not impossible, to port to other systems without manual setup, unless you containerized it, but that's out of scope here.
As of chrome 71, head to chrome settings at chrome://settings/ and choose Manage other people. Create a new person to represent your proxy profile. Configure the proxy using the browser the way you normally would for manually connecting.
Locate the path to your profile, something like: C:\Users\<username>\AppData\Local\Google\Chrome\User Data\Profile 1 if you're on windows. You can easily find by navigating to chrome://version/ and looking for Profile Path
Utilize ChromeOptions to pass the option for setting the user data directory for chrome on startup
ChromeOptions options = new ChromeOptions();
options.addArguments("user-data-dir=/path/to/your/custom/profile");
Some snippets pulled from documentation for chromedriver
I want to load a new Selenium ChromeDriver that is using Chrome as if I open Chrome from my dock (Essentially it'll have all my extensions, history, etc.)
When I use the following code:
ChromeOptions options = new ChromeOptions();
options.AddArgument("user-data-dir=C:\\Users\\User\\AppData\\Local\\Google\\Chrome\\User Data\\");
options.AddArgument("disable-infobars");
options.AddArgument("--start-maximized");
ChromeDriver chromeDriver = new ChromeDriver(options);
It loads the Chrome browser with me signed into my Gmail and with all my extensions, just like I want, but the rest of my code:
chromeDriver.Navigate().GoToUrl("https://www.youtube.com/");
doesn't execute. But when I use the following
ChromeOptions options = new ChromeOptions();
options.AddArgument("user-data-dir=C:\\Users\\Andrea\\AppData\\Local\\Google\\Chrome\\User Data\\Default");
options.AddArgument("disable-infobars");
options.AddArgument("--start-maximized");
ChromeDriver chromeDriver = new ChromeDriver(options);
The rest of my code executes perfectly (Notice the 'Default' added to the end of the first Argument). Any tips or suggestions on how I can get the first block of code (The one without 'Default' on the end) to execute the rest of my program would be great. Thanks!
I know this is an old question, but what worked for me is to do remove the "C:\" and replace all of the backslashes with forward slashes. So, with that from the original question, this should work to load the default profile:
options.AddArgument("user-data-dir=/Users/User/AppData/Local/Google/Chrome/User Data");
The Default Chrome Profile which you use for your regular tasks may contain either/all of the following items:
History
Bookmarks
Cookies
Extensions
Themes
Customized Fonts
All these configurations untill and unless are part of your Test Specification it would be a overkill to load them into the session initiated by Selenium WebDriver. Hence it will be a better approach if you create a dedicated New Chrome Profile for your tests and configure it with all the required configuration.
Here you will find a detailed discussion on How to create and open a Chrome Profile
Once you have created the dedicated New Chrome Profile for your tests you can easily invoke the Chrome Profile as follows:
ChromeOptions options = new ChromeOptions();
options.AddArgument("user-data-dir=C:\\Users\\User\\AppData\\Local\\Google\\Chrome\\User Data\\Profile 2");
options.AddArgument("disable-infobars");
options.AddArgument("--start-maximized");
ChromeDriver chromeDriver = new ChromeDriver(options);
chromeDriver.Navigate().GoToUrl("https://www.youtube.com/");
Here you will find a detailed discussion on How to open URL through default Chrome profile using Python Selenium Webdriver
I have the same issue. I don't know how to fix it, I guess the root cause is white space in profile path.
I know a workaround for this. Just copy the C:\\Users\\Andrea\\AppData\\Local\\Google\\Chrome\\User Data to c:\myUserData (no space in the path).
Then add the argument.
options.AddArgument("user-data-dir=C:\\myUserData");
This is an Old question, but if you are facing this issue, all you have to do is close all tabs, Just shut down the chrome window..
Selenium can't use the data since it is already in use.
Hope you fond this helpful.
I am facing getting The HTTP request to the remote WebDriver server for URL error while executing the selenium script in Chrome.
I am using Selenium with C# and Latest version of Chrome Driver, Chrome(66.0.3359.181) and Selenium(3.12.1)
Give this a try:- add the "no-sandbox" flag to the Chrome options:
var options = new ChromeOptions();
options.AddArgument("no-sandbox");
There are two causes to this exception I've seen:
1.Browser/web driver version mismatch - solved by updating webdriver nuget package to latest usually.
2.Server-side takes too long to load page - solved by either getting a faster server or as per https://code.google.com/p/selenium/issues/detail?id=5071 it looks like you can add a timeout argument when newing up a RemoteWebDriver, in Seleno that happens in Browser, but you don't have to use Browser you can new up the driver yourself to try out the fix. Feel free to submit a PR to Seleno to allow that timespan to be passed in as an option to the various drivers (probably in the override that has capabilities passed in).
Hope it helps!
I want to block c# controlled Google ChromeDriver via Selenium
Is that possible?
This is how i start the selenium and navigate
var Driver = new ChromeDriver();
Driver.Navigate().GoToUrl(URL)
Lets say i want that particular Chrome instance to do not connect http://googleads.g.doubleclick.net:443
Is this possible?
When that chrome instance requests to connect http://googleads.g.doubleclick.net:443, it should be automatically blocked or maybe return 127.0.0.1 I do not know
I need some solution which will prevent driver waiting response
Ok after i saw the answer i think i couldnt clarify the question well enough
The thing i want is, i am navigating to a page, which makes a lot of unnecessary requests to other hosts. I want to block those hosts, so page load timings would be much better
I tried via editing hosts file in etc and now it is 10 times better
But this is system wide
I wonder same may apply to chromedriver or not
The WebDriver does not allow this level of control (on purpose), so the short answer is no, you can't do that through it.
There is a solution though - just proxy all the traffic, and blacklist the offending URIs; or whitelist the one you're working with. In any ways, make the proxy return a reasonable response in time (a pixel file, or 404) for any unneeded URIs.
Sample code (my C#is rusty :):
ChromeOptions options = new ChromeOptions();
options.AddArguments("--proxy-server=XXX.XXX.XXX.XXX");
var Driver = new ChromeDriver(options);
(for Firefox it's a bit different, it's set as capabilities there)
There are a lot of options for the proxy, from the squid legend, the tinyproxy I've used in the past for similar purposes, to whatever will suit your environment and needs.