I would like to know how and if it is possible to use puppeteer-extra-plugin-stealth inside of visual studio project.
As far as I read there is a project called: puppeteer sharp but they didn't showed how to use the plugin stealth, as this is best browser which is not detectable as a "bot".
In my other puppeteer .js file compiled by Docker I have this code to load installed puppeteer-extra-plugin-stealth:
Docker puppeteer code:
const puppeteer = require('puppeteer-extra');
const stealthPlugin = require('puppeteer-extra-plugin-stealth');
const { IS_PROD } = require('../utils/constants');
puppeteer.use(stealthPlugin());
// In order to run chromium processes in parallel. https://github.com/puppeteer/puppeteer/issues/594#issuecomment-325919885
process.setMaxListeners(Infinity);
const getBrowserInstance = async (port) => {
const browser = await puppeteer.launch({
args: IS_PROD ? ['--no-sandbox', `--proxy-server=socks5://127.0.0.1:${port}`] : ['--no-sandbox'],
devtools: !IS_PROD,
executablePath: IS_PROD ? '/usr/bin/chromium-browser' : undefined,
});
const incognitoBrowserContext = browser.createIncognitoBrowserContext();
incognitoBrowserContext.close = browser.close;
return incognitoBrowserContext;
};
module.exports = {
getBrowserInstance,
};
BUT THIS is my C# Form where I don't know how to use or load or implement puppeteer-extra-plugin-stealth
My VS Code:
using PuppeteerSharp;
using System;
using System.Collections.Generic;
using System.ComponentModel;
using System.Data;
using System.Drawing;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using System.Windows.Forms;
namespace pupe
{
public partial class Form1 : Form
{
public Form1()
{
InitializeComponent();
}
private async void button1_Click(object sender, EventArgs e)
{
await new BrowserFetcher().DownloadAsync(BrowserFetcher.DefaultRevision);
var browser = await Puppeteer.LaunchAsync(new LaunchOptions
{
Headless = true
});
var page = await browser.NewPageAsync();
await page.SetViewportAsync(new ViewPortOptions
{
Width = 1920,
Height = 1080
});
await page.GoToAsync("https://bot.sannysoft.com/");
await page.ScreenshotAsync("screens/test.png", new ScreenshotOptions { Type = ScreenshotType.Png });
MessageBox.Show("screen done!");
}
}
}
Adding the stealth plugin to puppeteer-sharp may not be as simple than on the NodeJS Version. A C# port of the puppeteer-extra which works with puppeteersharp exists. Check https://github.com/Overmiind/Puppeteer-sharp-extra.
Related
I am trying to add BLE functionality into a classic (WinForms?) C# desktop application, and have added references (Windows.winmd and System.Runtime.WindowsRuntime) to allow me to access the new BLE API recently introduced by Microsoft for Windows 10 UWP applications. I need to create a classic desktop application, as I need to use an older driver device wrapper (teVirtualMIDI) and want to create a .exe, not an app package.
I am referencing the aformentioned libraries from the following locations...
C:\Program Files (x86)\Windows Kits\10\UnionMetadata\Facade\Windows.WinMD
C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework.NETCore\v4.5\System.Runtime.WindowsRuntime.dll
C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework.NETCore\v4.5\System.Runtime.WindowsRuntime.UI.Xaml.dll
At this point, I simply want to be able to view connected services and characteristics in the debug output window, as is done in this blog post...
https://blogs.msdn.microsoft.com/cdndevs/2017/04/28/uwp-working-with-bluetooth-devices-part-1/
It seems that I am getting errors because the BLE API needs to perform async operations, but I am honestly at a loss. The code I have written so far is included below. Essentially, I am receiving errors when trying to call the "GetGattServicesAsync()" method, as Visual Studio says that class "BluetoothLEDevice" does not contain such a definition. That method is included in the online documentation though, and I am wondering why I am not able to access it.
I hope I have given sufficient information, and any help in solving this problem will be more than appreciated. Thank you all for all the helpful advice you give!
using System;
using System.Diagnostics;
using System.Collections.Generic;
using System.Collections.ObjectModel;
using System.ComponentModel;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using System.Runtime.InteropServices;
using Windows.Devices.Bluetooth;
using Windows.Devices.Midi;
using Windows.Devices.Bluetooth.Advertisement;
using Windows.Devices.Bluetooth.GenericAttributeProfile;
using Windows.Devices.Enumeration;
using Windows.Foundation;
using Windows.Foundation.Collections;
using Windows.Storage.Streams;
using Windows.UI.Core;
using Windows.UI.Xaml;
using Windows.UI.Xaml.Controls;
using Windows.UI.Xaml.Navigation;
namespace BDBMidiClient
{
public class BLEHandlingDiscovery : Page
{
//private ObservableCollection<BluetoothLEAttributeDisplay> ServiceCollection = new ObservableCollection<BluetoothLEAttributeDisplay>();
//private ObservableCollection<BluetoothLEAttributeDisplay> CharacteristicCollection = new ObservableCollection<BluetoothLEAttributeDisplay>();
public ObservableCollection<BluetoothLEDeviceDisplay> KnownDevices = new ObservableCollection<BluetoothLEDeviceDisplay>();
//private List<DeviceInformation> UnknownDevices = new List<DeviceInformation>();
//private DeviceWatcher deviceWatcher;
//private BluetoothLEDevice bluetoothLeDevice = null;
//private GattCharacteristic selectedCharacteristic;
private void StartBLEDeviceWatcher()
{
string[] requestedProperties = { "System.Devices.Aep.DeviceAddress", "System.Devices.Aep.IsConnected" };
DeviceWatcher deviceWatcher =
DeviceInformation.CreateWatcher(
BluetoothLEDevice.GetDeviceSelectorFromPairingState(false),
requestedProperties,
DeviceInformationKind.AssociationEndpoint);
/*
DeviceWatcher deviceWatcher =
DeviceInformation.CreateWatcher(
"System.ItemNameDisplay:~~\"BDB\"",
requestedProperties,
DeviceInformationKind.AssociationEndpoint);*/
deviceWatcher.Added += DeviceWatcher_Added;
deviceWatcher.Updated += DeviceWatcher_Updated;
deviceWatcher.Removed += DeviceWatcher_Removed;
deviceWatcher.Start();
//Debug.WriteLine(requestedProperties);
}
private async void DeviceWatcher_Added(DeviceWatcher sender, DeviceInformation deviceInfo)
{
Guid gattService = new Guid();
var device = await BluetoothLEDevice.FromIdAsync(deviceInfo.Id);
var services=await device.GetGattServicesAsync();
foreach (var service in services.Services)
{
Debug.WriteLine($"Service: {service.Uuid}");
var characteristics = await service.GetCharacteristicsAsync();
foreach (var character in characteristics.Characteristics)
{
Debug.WriteLine($"Characteristic: {character.Uuid}");
}
}
}
private void DeviceWatcher_Updated(DeviceWatcher sender, DeviceInformationUpdate deviceInfoUpdate)
{
}
private void DeviceWatcher_Removed(DeviceWatcher sender, DeviceInformationUpdate deviceInfoUpdate)
{
}
async void ConnectToBLEDevice(DeviceInformation deviceInformation)
{
BluetoothLEDevice bluetoothLeDevice = await BluetoothLEDevice.FromIdAsync("BDB");
}
private BluetoothLEDeviceDisplay FindBluetoothLEDeviceDisplay(string id)
{
foreach (BluetoothLEDeviceDisplay bleDeviceDisplay in KnownDevices)
{
if (bleDeviceDisplay.Id == id)
{
return bleDeviceDisplay;
}
}
return null;
}
}
The doc says the API belongs to "Windows 10 Creators Update (introduced v10.0.15063.0)". So please try to add the one from "C:\Program Files (x86)\Windows Kits\10\UnionMetadata\10.0.15063.0\Windows.winmd"
Here is the result from my project
You can see my code works well.
I am working on building a primitive and basic web browser on which my workplace would like to host some internal applications. I"m using cefSharp in a WinForms application written in C#. I've succeeded in building the browser to navigate the application, but I'm having trouble with the download handler. I would like to download files directly to the C:\Users\[username]\Downloads folder (all of our computers are Windows computers) without having to use the dialog.
Reading from Force CEFSharp to download without showing dialog suggests that using showDialog: false should work, but when I apply this, nothing downloads. Likewise, I've made no progress by studying any of the following:
WPF : download files through CefSharp
https://github.com/cefsharp/CefSharp/blob/cd934267c65f494ceb9ee75995cd2a1ca0954543/CefSharp.Example/DownloadHandler.cs
WPF : download files through CefSharp
https://groups.google.com/forum/?nomobile=true#!topic/cefsharp/bS8PhHRlSAc
https://groups.google.com/forum/#!topic/cefsharp/3cMUHSGxPDc
As a bonus, it'd be nice to have the option to open the file, such as in Google Chrome, but this isn't strictly necessary.
The code below runs smoothly and approximates what I am attempting. This example opens to a GitHub Gist. Clicking on the "Download Zip" button on the right opens the dialog to download and save the file.
using System;
using System.Collections.Generic;
using System.ComponentModel;
using System.Data;
using System.Drawing;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using System.Web.Script.Serialization;
using System.Windows.Forms;
using CefSharp;
using CefSharp.WinForms;
using System.IO;
namespace ShinyChrome
{
public partial class ShinyApp : Form
{
public class DownloadHandler : IDownloadHandler
{
public event EventHandler<DownloadItem> OnBeforeDownloadFired;
public event EventHandler<DownloadItem> OnDownloadUpdatedFired;
public void OnBeforeDownload(IBrowser browser, DownloadItem downloadItem, IBeforeDownloadCallback callback)
{
var handler = OnBeforeDownloadFired;
if (handler != null)
{
handler(this, downloadItem);
}
if (!callback.IsDisposed)
{
using (callback)
{
callback.Continue(downloadItem.SuggestedFileName, showDialog: true);
}
}
}
public void OnDownloadUpdated(IBrowser browser, DownloadItem downloadItem, IDownloadItemCallback callback)
{
var handler = OnDownloadUpdatedFired;
if (handler != null)
{
handler(this, downloadItem);
}
}
}
public ShinyApp()
{
InitializeComponent();
}
ChromiumWebBrowser chrome;
private void ShinyApp_Load(object sender, EventArgs e)
{
CefSettings settings = new CefSettings();
Cef.Initialize(settings);
chrome = new ChromiumWebBrowser("https://gist.github.com/nutterb/32992747c1a69aa7a8fdcc2b5347178f");
chrome.DownloadHandler = new DownloadHandler();
this.shinyContainer.Controls.Add(chrome);
}
}
}
On TEK's advice, I replaced the if(!callback.IsDisposed) block in the question with the code below.
if (!callback.IsDisposed)
{
using (callback)
{
callback.Continue(#"C:\Users\" +
System.Security.Principal.WindowsIdentity.GetCurrent().Name. +
#"\Downloads\" +
downloadItem.SuggestedFileName,
showDialog: false);
}
}
I've been working on my project from the last two months, everything works fine but I have ended up with a problem here. I have been running my script in c#:
using OpenQA.Selenium;
using OpenQA.Selenium.PhantomJS;
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
namespace console
{
class Program
{
static void Main(string[] args)
{
IWebDriver driver = new PhantomJSDriver();
var url = "http://example1.com";
driver.Navigate().GoToUrl(url);
var r = driver.FindElements(By.CssSelector("css selector of example1.com"));
var d = r.First().Text;
Console.WriteLine(d);
var phantomJSDriverService = PhantomJSDriverService.CreateDefaultService();
IWebDriver driver_p = new PhantomJSDriver(phantomJSDriverService);
driver_p.Manage().Timeouts().ImplicitlyWait(new TimeSpan(0, 0, 30));
var url_p = "http://example2.com";
driver.Navigate().GoToUrl(url_p);
var title = driver.FindElements(By.CssSelector("css selector of example2.com"));
var myFirstScrape = title[0].Text;
Console.WriteLine(myFirstScrape);
Console.ReadLine();
driver_p.Quit();
}
}
}
on Visual studio express for desktop in console application . I get wonderful result in 40 sec. Both gives 40 sec while the first one gives result in lessthan 12 seconds.
The problem is here, While running the script based on the above in asp.net project of visual studio for web,
The script in asp.net project of visual studio is:
using OpenQA.Selenium;
using OpenQA.Selenium.PhantomJS;
using System;
using System.Web;
using System.Web.UI;
using System.Web.UI.WebControls;
namespace WebScraperapp
{
public partial class _Default : Page
{
protected void Page_Load(object sender, EventArgs e)
{
var phantomJSDriverService = PhantomJSDriverService.CreateDefaultService();
IWebDriver driver = new PhantomJSDriver(phantomJSDriverService);
driver.Manage().Timeouts().ImplicitlyWait(new TimeSpan(0, 0, 30));
var url = "http://www.example1.com";
driver.Navigate().GoToUrl(url);
var title = driver.FindElements(By.CssSelector("css selector of example 1"));
var myFirstScrape = title[0].Text;
Label2.Text = myFirstScrape;
driver.Quit();
}
}
}
The result I got from the above asp.net project code is over 150s which is not fare for my business. I have been struggling in breaking this down to minimum of < 50 sec.
Note: I am very happy if I will get the final output in php script, I am not aware of phantomjs on web hosting server, hence preferred asp.net for this project.
After removing the implicitlywait function, I'm getting the same results!
Thanks in advance!
I'm compiling code on-the-fly using System.CodeDom.Compiler. Everything inside the compiled source works well, whatever I'm putting inside this source. I know how to call my functions:
o = results.CompiledAssembly.CreateInstance("Foo.Bar");
MethodInfo mi = o.GetType().GetMethod("SayHello");
mi.Invoke(o, null);
But let's say I'm using a WebClient to retrieve a string asynchronously using WebClient.DownloadStringAsync. Or any other context where I want my compiled source to tell to the host "Hey, I got a nice string ready for you." For the example, I've used a WebBrowser. Basically, I know how to deal with each of the two instances: My hosting program and the compiled program, but I want my compiled program to communicate with the host. By the way, I'm not a super-experimented programmer, so no obvious method comes to my mind.
What I've tried:
1 . I don't really need to try it because it would work, but I could use a timer reading a strings stack or tasks queue inside the compiled source, but the purpose of my application is to have +- 60 scripts able to execute ponctual tasks, not continuous background processes, so it wouldn't be efficient on the CPU.
2 . I've passed the handler to the compiled source like if it was in the hosting app:
//In the hosting app
MethodInfo mi2 = o.GetType().GetMethod("attachCallbackToHost");
mi2.Invoke(o2, new object[] { new WebBrowserNavigatedEventHandler (wb_navigated) });
//... And the handler
public static void wb_navigated(object sender, WebBrowserNavigatedEventArgs e)
{
string browserHtmlFromCompiledSource = ((WebBrowser)sender).DocumentText;
MessageBox.Show(browserHtmlFromCompiledSource);
}
// Plain text from the compiled source code
public void attachCallbackToHost(WebBrowserNavigatedEventHandler handlerFromTheHost)
{
wb.Navigated += handlerFromTheHost;
}
And it did nothing.
3 . Maybe I could share a class or variable by passing it to the compiled assembly?
So, the question is either this or the other:
How to watch efficiently for change inside a specific variable or property inside the compiled program?
How to attach a callback to the host?
Ok. I got it: In order to access the host from the compiled source, the only thing required is to add the host assembly to the refered assemblies in the compiler parameters:
compilerParams.ReferencedAssemblies.Add(Assembly.GetExecutingAssembly().Location);
So no need for any special callback or INotifier.
Here's the full code that strictly answers my question and nothing more:
using System;
using System.Collections.Generic;
using System.ComponentModel;
using System.Data;
using System.Drawing;
using System.Linq;
using System.Text;
using System.Windows.Forms;
using Microsoft.CSharp;
using System.CodeDom.Compiler;
using System.Reflection;
namespace MamaProgram
{
public partial class Form1 : Form
{
public Form1()
{
InitializeComponent();
}
private void Form1_Load(object sender, EventArgs e)
{
string source =
#"
using System;
using System.Collections.Generic;
using System.ComponentModel;
using System.Data;
using System.Drawing;
using System.Linq;
using System.Text;
using System.Windows.Forms;
using System.Net;
using MyMama = MamaProgram;
namespace Baby
{
public class Program
{
public WebBrowser wb = new WebBrowser();
public void navigateTo(string url)
{
wb.Navigated += wb_navigated;
wb.Navigate(url);
}
public void wb_navigated(object sender, WebBrowserNavigatedEventArgs e)
{
MyMama.Form1.getResult(wb.DocumentText);
}
}
}
";
Dictionary<string, string> providerOptions = new Dictionary<string, string>
{
{"CompilerVersion", "v3.5"}
};
CSharpCodeProvider provider = new CSharpCodeProvider(providerOptions);
CompilerParameters compilerParams = new CompilerParameters
{
GenerateInMemory = true,
GenerateExecutable = false,
TreatWarningsAsErrors = false
};
compilerParams.ReferencedAssemblies.Add(Assembly.GetExecutingAssembly().Location);
compilerParams.ReferencedAssemblies.Add("System.Data.dll");
compilerParams.ReferencedAssemblies.Add(typeof(System.Linq.Enumerable).Assembly.Location); // Trick to add assembly without knowing their name
compilerParams.ReferencedAssemblies.Add(typeof(System.ComponentModel.Component).Assembly.Location); // Trick to add assembly without knowing their name
compilerParams.ReferencedAssemblies.Add("System.Windows.Forms.dll");
CompilerResults results = provider.CompileAssemblyFromSource(compilerParams, source);
if (results.Errors.Count != 0)
throw new Exception("Compilation failed");
object o = results.CompiledAssembly.CreateInstance("Baby.Program");
MethodInfo mi2 = o.GetType().GetMethod("navigateTo");
mi2.Invoke(o, new object[] { "http://www.google.com" });
}
public static void getResult(string result)
{
MessageBox.Show(result);
}
}
}
i know i could search proccessId / name of running tasks and kill processes i need .
though till now i was not developing schedualed tasks / self executble Applications,
so i didn't need to know how to make the application close itself after execition
trying to close everything (including WebDriver) via Application.Exit + OR this.Close()
right after i have got what i was looking for. mission Complete .
please close ... no more work for you .
but mr . Program.cs still needs somthing from Form1.
saying somthing about
Cannot access a disposed object.
Object name: 'Form1'.
any combination of both was returning in some point an exeption error
(from program.cs ) even though mission complete . no more code was requested .(?) by me..atleast.
using System;
using System.Collections.Generic;
using System.ComponentModel;
using System.Data;
using System.Drawing;
using System.Linq;
using System.Text;
using System.Windows.Forms;
using OpenQA.Selenium;
using OpenQA.Selenium.IE;
using System.IO;
namespace HT_R_WbBrows2
{
public partial class Form1 : Form
{
public IeEnginGenerator Iengn = new IeEnginGenerator();
public Form1()
{
InitializeComponent();
//setLogView(View.Details);
string extractededVal = Iengn.ExtractPageValue(Iengn.itrfWebEng);
string flnm = #" the directory path to file --> \dolarRate.asp";
File.WriteAllText(fn, extractededVal);
this.Close();
Application.Exit();
}
public class IeEnginGenerator
{
private string directory = Environment.CurrentDirectory;///Path.GetDirectoryName(Assembly.GetExecutingAssembly().CodeBase);
public IWebDriver IwebEngine;
public List<string> ListElementsInnerHtml = new List<string>();
public HtmlAgilityPack.HtmlDocument Dnetdoc = new HtmlAgilityPack.HtmlDocument();
#region <<=========== setupDriver ============>>
public string ExtractPageValue(IWebDriver DDriver, string url="")
{
if(string.IsNullOrEmpty(url))
url = #"http://www.boi.org.il/he/Markets/ExchangeRates/Pages/Default.aspx";
var service = InternetExplorerDriverService.CreateDefaultService(directory);
service.LogFile = directory + #"\seleniumlog.txt";
service.LoggingLevel = InternetExplorerDriverLogLevel.Trace;
var options = new InternetExplorerOptions();
options.IntroduceInstabilityByIgnoringProtectedModeSettings = true;
DDriver = new InternetExplorerDriver(service, options, TimeSpan.FromSeconds(60));
DDriver.Navigate().GoToUrl(url);
Dnetdoc.LoadHtml(DDriver.PageSource);
string Target = Dnetdoc.DocumentNode.SelectNodes("//table//tr")[1].ChildNodes[7].InnerText;
//.Select(tr => tr.Elements("td").Select(td => td.InnerText).ToList())
//.ToList();
return Math.Round(Convert.ToDouble(Target), 2).ToString();
//return "";//Math.Round(Convert.ToDouble( TempTxt.Split(' ')[10]),2).ToString();
}
#endregion
}
}
}
Why use a winform application? A Console application would probably suffice for what you are doing. Once Main() ends your app will close as well. Main() never ends in a winform app because of the applications runloop.
Edit:
Here would be the correct way to do this. You need to register to the forms Load event and run your code there, not in the constructor. You can't close a winform from inside a constructor.
Edit 2: Put this code in the Form1() constructor. Somewhere after InitializeComponent();
this.Load += (sender,args)=>{ /*do all your work here*/
string extractededVal = Iengn.ExtractPageValue(Iengn.itrfWebEng);
string flnm = #" the directory path to file --> \dolarRate.asp";
File.WriteAllText(fn, extractededVal);
Application.Exit();
};