I've been playing around with Selenium and PhantomJS in C# but I want to be able to fake my User Agent to be this:
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:26.0) Gecko/20100101 Firefox/26.0
Instead of:
Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534.34 (KHTML, like Gecko) PhantomJS/1.9.1 Safari/534.34
Is it possible to modify the HTTP headers of PhantomJS with Selenium to achieve this? If so, how?
Thanks in advance.
I found the answer:
PhantomJSOptions options = new PhantomJSOptions();
options.AddAdditionalCapability("phantomjs.page.settings.userAgent", "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:26.0) Gecko/20100101 Firefox/26.0");
IWebDriver driver = new PhantomJSDriver(options);
Thanks.
Related
I'm attempting to download image from Google Drive, but the image is not opening for some reason.
WebClient webClient = new WebClient();
webClient.Headers["User-Agent"] = "User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36";
webClient.DownloadFile(url, #"path_to_download.png");
The URL looks like https://drive.google.com/file/d/1231231313+yDnEx7upHEsM4ozjh/view from this URL I take the image source. It looks like https://lh3.googleusercontent.com/drive-viewer/ASAdd1d123sT1QX_ZYlcoHMTeDBeUJaNnwxqjJzFgC8e24i1HOylgu2MnqzO9iNb41exySXsp2sJWA=w1106-h772 and after that, I run the code upper. Everything works, but I cannot open the image.
I want to send a link request to any site in asp.net. In doing so, I want the outgoing connection to be perceived as a mobile device. If I want to improve it further, I want the sent link to be at the mobile connection level. For example, 3G or 4G. I'm waiting for your help. Thank you.
The device is determined by the UserAgent header.
To set the user agent header, see How do I set a default User Agent on an HttpClient?
Regarding network speed throttling, it is possible, see Bandwidth throttling in C#
Here are some user agents, from https://deviceatlas.com/blog/mobile-browser-user-agent-strings
Safari for iOS
Mozilla/5.0 (iPhone; CPU iPhone OS 10_3_1 like Mac OS X) AppleWebKit/603.1.30 (KHTML, like Gecko) Version/10.0 Mobile/14E304 Safari/602.1
Android Browser
Mozilla/5.0 (Linux; U; Android 4.4.2; en-us; SCH-I535 Build/KOT49H) AppleWebKit/534.30 (KHTML, like Gecko) Version/4.0 Mobile Safari/534.30
Chrome Mobile
Mozilla/5.0 (Linux; Android 7.0; SM-G930V Build/NRD90M) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.125 Mobile Safari/537.36
Opera Mobile (Blink rendering engine)
Mozilla/5.0 (Linux; Android 7.0; SM-A310F Build/NRD90M) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.91 Mobile Safari/537.36 OPR/42.7.2246.114996
Opera Mobile (Presto rendering engine)
Opera/9.80 (Android 4.1.2; Linux; Opera Mobi/ADR-1305251841) Presto/2.11.355 Version/12.10
Opera Mini
Opera/9.80 (J2ME/MIDP; Opera Mini/5.1.21214/28.2725; U; ru) Presto/2.8.119 Version/11.10
Opera Mini (iOS WebKit)
Mozilla/5.0 (iPhone; CPU iPhone OS 7_1_2 like Mac OS X) AppleWebKit/537.51.2 (KHTML, like Gecko) OPiOS/10.2.0.93022 Mobile/11D257 Safari/9537.53
Firefox for Android
Mozilla/5.0 (Android 7.0; Mobile; rv:54.0) Gecko/54.0 Firefox/54.0
Firefox for iOS
Mozilla/5.0 (iPhone; CPU iPhone OS 10_3_2 like Mac OS X) AppleWebKit/603.2.4 (KHTML, like Gecko) FxiOS/7.5b3349 Mobile/14F89 Safari/603.2.4
UC Browser
Mozilla/5.0 (Linux; U; Android 7.0; en-US; SM-G935F Build/NRD90M) AppleWebKit/534.30 (KHTML, like Gecko) Version/4.0 UCBrowser/11.3.8.976 U3/0.8.0 Mobile Safari/534.30"
Dolphin
Mozilla/5.0 (Linux; Android 6.0.1; SM-G920V Build/MMB29K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.98 Mobile Safari/537.36
Puffin for Android
Mozilla/5.0 (Linux; Android 5.1.1; SM-N750K Build/LMY47X; ko-kr) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.135 Mobile Safari/537.36 Puffin/6.0.8.15804AP
Puffin for iOS
Mozilla/5.0 (Linux; Android 5.1.1; SM-N750K Build/LMY47X; ko-kr) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.135 Mobile Safari/537.36 Puffin/6.0.8.15804AP
Samsung Browser
Mozilla/5.0 (Linux; Android 7.0; SAMSUNG SM-G955U Build/NRD90M) AppleWebKit/537.36 (KHTML, like Gecko) SamsungBrowser/5.4 Chrome/51.0.2704.106 Mobile Safari/537.36
Yandex Browser
Mozilla/5.0 (Linux; Android 6.0; Lenovo K50a40 Build/MRA58K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.137 YaBrowser/17.4.1.352.00 Mobile Safari/537.36
MIUI Browser
Mozilla/5.0 (Linux; U; Android 7.0; en-us; MI 5 Build/NRD90M) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/53.0.2785.146 Mobile Safari/537.36 XiaoMi/MiuiBrowser/9.0.3
IE Mobile
Mozilla/5.0 (compatible; MSIE 10.0; Windows Phone 8.0; Trident/6.0; IEMobile/10.0; ARM; Touch; Microsoft; Lumia 950)
Edge Mobile
Mozilla/5.0 (Windows Phone 10.0; Android 6.0.1; Microsoft; Lumia 950) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Mobile Safari/537.36 Edge/15.14977
BlackBerry Browser
Mozilla/5.0 (BB10; Kbd) AppleWebKit/537.35+ (KHTML, like Gecko) Version/10.3.3.2205 Mobile Safari/537.35+
I am trying to get the html source of a youtube video using cURL command line but I need it to be without https/ssl.
My problem is that I must use the compiled version of cURL with SSL/SSH.
I am using the following command:
curl --user-agent "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36" -L -x http://my.foo.proxy:8080 http://youtube.com/watch?v=youtubevideo > html.html
this works but a specific part of the html source is in https (look for a really long script string inside that file. some of the links there start with httpS)
curl --proto =http --proto-redir =http --user-agent "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36" -L -x http://my.foo.proxy:8080 http://youtube.com/watch?v=youtubevideo > html.html
this command causes an error:
protocol https not supported or disabled in libcurl.
which is really weird because the curl version I am using does have ssl and I dont even want https (see the -proto and -proto-redir args).
As a test I also tried using .NET Webclient class like:
public static void DownloadString (string address)
{
WebClient client = new WebClient ();
string reply = client.DownloadString (address);
Console.WriteLine (reply);
}
and in this case I get a html source file without https.
My question is, how do I get a html source file of a youtube video using cURL without https inside my html source file like when I use .NET/Webclient?
using an user agent without firefox fixes this issue when used inside a console:
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:24.0) Gecko/20100101
When used with a binding set SSL_VERIFYPEER to false and SSL_VERIFYHOST to 0. Allows man in the middle attack but if that is the only option...
In addition HTTPGET and FOLLOWLOCATION should also both be set to true.
i had created an empty c# web site with just one page with Request.Browser.Version & UserAgent output on it. Then hit it with different Chrome versions using "User-Agent Switcher" Chrome extension.
For time to time, though the Request.UserAgent is correct, Request.Browser.Version seems to return wrong value:
"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.16 Safari/537.36" Returned Request.Browser.Version:39
"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2272.16 Safari/537.36" Returned Request.Browser.Version:41
So yes, .net 4.5 caches the user agent by its first 64 chars. And that's just gets them before the version number. So the next user with the same browser but with a different version will get the wrong browser version and so.
To solve it just change the :browserCaps userAgentCacheKeyLength="...", as can be seen here:
.Net 4.0 website cannot identify some AppleWebKit based browsers
How isn't this stupid Microsoft bug on the headlines?
i'm using phantomjsdriver 1.8.1 for .net (C#)
http://www.nuget.org/packages/phantomjs.exe/ and wonder how to add user-agent firefox before loading the web content
Although Cybermax's answer is somewhat correct, it isn't correct in terms of what you are actually using - C#. To specify a user agent for the PhantomJSDriver in C#, you will need to give it as an "additional capability":
var options = new PhantomJSOptions();
options.AddAdditionalCapability("phantomjs.page.settings.userAgent", "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:25.0) Gecko/20100101 Firefox/25.0");
These options should be passed into the constructor used to create the driver:
var driver = new PhantomJSDriver(options);
To verify this has been set correctly, you can check against a website that tells you the user agent of your browser, something like WhatIsMyUserAgent.com or look closely at the PhantomJS console window, it'll have a "useragent" value there, and you should see it's been set to what you specified above.
In your script, you have to define property page.settings.userAgent before the first call to page.open.
var page = require('webpage').create();
page.settings.userAgent = 'Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:25.0) Gecko/20100101 Firefox/25.0'; //firefox 25
page.open('http://stackoverflow.com/', function (status) {
// do something
});
Note : the last version of PhantomJS is 1.9.2. Another package is available here.