We Keep Coding

Cannot access same source HTML as a browser

I am coming back to work on a BOT that scraped data from a site once a day for my personal use.
However they have changed the code during COVID and now it seems they are loading in a lot of the content with Ajax/JavaScript.
I thought that if I did a WebRequest and obtained the response HTML from a URL, it should match the same content that I see in a browser (FF/Chrome) when I right click and "view source". I thought the actual DOM and generated source code would come later when those files were loaded as onload events fired, scripts lazily loaded and so on.
However the source HTML I obtain with my BOT is NOT the same as the HTML I see when viewing the source code. So my regular expressions that find certain links are not available to me.
Why am I seeing a difference between "view source" and a download of the HTML?
I can only think that when the page loads, SCRIPTS run that load other content into the page and that when I view source I am actually seeing a partial generated source rather than the original source code. Therefore is there a way I can call the page with my BOT, wait X seconds before obtaining the response to get this "onload" generated HTML?
Or even better a way for MY BOT (not using someone elses), to view generated source.
This BOT runs as a web service. I can find another site to scrape but it's just painful when I have all the regular expressions working on the source I see, except it's NOT the source my BOT obtains.
A bit confused at why my browser is showing me more content with a view source (not generated source), than my BOT gets when making a valid request.
Any help would be much appreciated this is almost an 8 year project that I have been doing on/off and this change has ruined one of the core parts of the system.

In response to OP's comment, here is the Java code for how to click at different parts on the screen to do this:
You could use Java's Robot class. I just learned about it a few days ago:
// Import
import java.awt.Robot;
// Code
void click(int x, int y, int btn) {
Robot robot = new Robot();
robot.mouseMove(x, y);
robot.mousePress(btn);
robot.mouseRelease(btn);
}
You would then run the click function with the x and y position to click, as well as the button (MouseEvent.BUTTON1, MouseEvent.BUTTON2, etc.)
After stringing together the right positions (this will vary depending on the screen) you could do just about anything.
To use shortcuts, just use the keyPress and keyRelease functions. Here is a good way to do this:
void key(int keyCode, boolean ctrl, boolean alt, boolean shift) {
if (ctrl)
robot.keyPress(KeyEvent.VK_CONTROL);
if (alt)
robot.keyPress(KeyEvent.VK_ALT);
if (shift)
robot.keyPress(KeyEvent.VK_SHIFT);
robot.keyPress(keyCode);
robot.keyRelease(keyCode);
if (ctrl)
robot.keyRelease(KeyEvent.VK_CONTROL);
if (alt)
robot.keyRelease(KeyEvent.VK_ALT);
if (shift)
robot.keyRelease(KeyEvent.VK_SHIFT);
}
Thus, something like Ctrl+Shift+I to open the inspect menu would look like this:
key(KeyEvent.VK_I, true, false, true);
Here are the steps to copy a website's code (from the inspector) with Google Chrome:
Ctrl + Shift + I
Right click the HTML tag
Select "Edit as HTML"
Ctrl + A
Ctrl + C
Then, you can use the technique from this StackOverflow to get the content from the clipboard:
Clipboard c = Toolkit.getDefaultToolkit().getSystemClipboard();
String text = (String) c.getData(DataFlavor.stringFlavor);
Using something like FileOutputStream to put the info into a file:
FileOutputStream output = new FileOutputStream(new File( PATH HERE ));
output.write(text.getBytes());
output.close();
I hope this helps!

I have seemed to have fixed it by just turning on the ability to store cookies in my custom HTTP (Bot/Scraper) class, that was being called from the class trying to obtain the data. Probably the site has a defense system to prevent visitors requesting pages and not the JS/CSS with a different session ID on each request.
However I would like to see some other examples because if it is just cookies then they could use JavaScript to test for JavaScript e.g an AJAX call to log if JS is actually on or some DOM manipulation to determine if you are really Human or not which would break it again.
Every site uses different methods to prevent scrapers, email harvesters, job rapists, link harvesters etc inc working out the standard time between requests for 100% verifiable humans and BOTS and then using those values to help determine spoofed user-agents etc. I wrote a whole system to stop BOTS at my last place of work and its a layered approach, just glad the cookies being enabled solved it on this site but it could easily be beefed up with other tricks to test for BOTS vs HUMANS.
I do know some Java, enough to work out what is going on anyway. My BOT is in C#.

How to reset a password via BAPI in a CUA environment?

I am currently developing a C# application with SAP NCo 3.
I am wondering if I could invoke BAPI into CUA and this BAPI would pass details to child system.
This field is available through Test Function Module (field "RFC target sys"), but it is unavailable directly in standard BAPIs when accessed from SAP NCo.
In ABAP, devs can use:
call function 'BAPI_USER_CHANGE' destination '<TARGET_SYS>'
Can I use something similar in NCo library?
IRfcFunction rfcs = rfcDest.Repository.CreateFunction("BAPI_USER_CHANGE");
Does anybody know how this could be achieved?
Main intent is to reset user passwords to initial ones through App(BAPI) --> CUA --> ChildSystem
Without direct access into child systems.

Hmm, it looks like you have not yet fully understood the meaning of "RFC target sys".
In SE37 "RFC target sys" you enter the name of an RFC destination, which provides details about in which SAP system you want to execute the function module. These details are then defined in SM59, where you can specify parameters like hostname, system number, client, user, password, language, etc.
In the NCo library you do the same via the Class RfcDestinationManager. Here you define the parameters (hostname, system number, client, user, password, language, etc.) of the target system in which you want to execute the function module.
So the line
"RFC target sys: TARGET_SYS"
in SE37 corresponds to a line like
RfcDestination myDest = RfcDestinationManager.GetDestination("TARGET_SYS");
in your .NET program.
And a line of ABAP code like
call function 'BAPI_USER_CHANGE' destination 'TARGET_SYS'
would then correspond to some .NET code like
RfcDestination targetSys = RfcDestinationManager.GetDestination("TARGET_SYS");
IRfcFunction bapiUserChange = targetSys.Repository.CreateFunction("BAPI_USER_CHANGE");
targetSys.Invoke(bapiUserChange);
Note: setting of the input values and error handling is omitted here.

Ok, so I found out that what I wanted to achieve is not possible with just sapnco.
But, in SAP I created function module, which calls function module and uses DESTINATION 'target_sys' to run in end system. This way I achieved what I wanted. By calling my Z_FUNC_MODULE from sapnco I pass variable target_sys and FN is called in child system of CUA.
Hope this helps to someone.

C# not able to get console Logs from PhantomJSDriver (Selenium)

I'm using Selenium in C# with the PhantomJS Driver.
I need to click specific coordinates on a website, that works with using Javascript (im using the ExecutePhantomJS(string script) function of the selenium phantomjs driver). I also need to capture the network traffic. I used browsermob earlier to do that, but for now i cant use it because i also need to use another proxy. So i solved it like that until now:
//Hide CMD of PhantomJS.exe
var driverService = PhantomJSDriverService.CreateDefaultService();
driverService.HideCommandPromptWindow = true;
//Initialize Driver and execute Network Script to capture traffic
driver = new PhantomJSDriver(driverService);
driver.ExecutePhantomJS(networkScript);
//Call URL
driver.Navigate().GoToUrl(url);
This is the networkScript:
string networkScript = "var page = this; page.onResourceRequested = function(req) { console.log('received: ' + JSON.stringify(res, undefined, 4)); }; page.onResourceReceived = function(res) { console.log('received: ' + JSON.stringify(res, undefined, 4)); };";
The good thing:
URL is called and all network traffic is logged into the console of the PhantomJS.exe.
But I dont know how I can get these console logs now in my C# code (I need to get specific things like URLs etc.. out of the network log).
I already read the whole afternoon but couldn't find a solution until now. Some of the things I already tried:
1) Tried to use PhantomJSOptions, there u can set LoggingPreferences and later i called driver.Manager().Logs.GetLog(LogType), but there were none of the console logs
2) Dont use console.log inside networkScript. I used require('system').stdout.write(...). It was also logged into console but I cant get the standard output stream of the phantomjs.exe from my C# code.
...
I really dont know how i could solve the problem.
One way would be to log into a .txt file instead of console, but it is very much text and later there will be many opened drivers, so I want to avoid that because then i will have very much and big .txt files

Get opened tabs information of Firefox?

I'm trying to write a simple program that should search in Firefox window for duplicated tabs (checking the url of the tab) and then close the found duplicated tabs.
The idea is simple, but the implementation seems a nightmare.
Doing a lot of researchs messing with WinAPI I've found nDde library, which could retrieve the url of the current tab easy like this example:
VB.NET
Imports NDde.Client
Using dde As New DdeClient("Firefox", "WWW_GetWindowInfo")
dde.Connect()
Dim Url As String = dde.Request("URL", Integer.MaxValue).
Trim({ControlChars.NullChar, ControlChars.Quote, ","c})
MessageBox.Show(Url)
dde.Disconnect()
End Using
C#:
using (DdeClient dde = new DdeClient("Firefox", "WWW_GetWindowInfo")) {
dde.Connect();
string Url = dde.Request("URL", int.MaxValue).Trim({
ControlChars.NullChar,
ControlChars.Quote,
','
});
MessageBox.Show(Url);
dde.Disconnect();
}
//=======================================================
//Service provided by Telerik (www.telerik.com)
//Conversion powered by NRefactory.
//Twitter: #telerik
//Facebook: facebook.com/telerik
//=======================================================
But my knowledges about this library or dde in general are zero, so what I'm doing by the moment is sending ctrl+Tab keys to Firefox to change between tabs to get the url of each tab and then close duplicated founds sending ctrl+w, but this way I have not a reference point to know which tab was the "starting point" to know when I need to stop the dup-tab searching 'cause the first checked url could have a duplicated tab too, and also I can't know the exact number of opened tabs to have an Index reference.
I'm lost.
My question is, this library (or another library related to dde, or another totally different way) could retrieve at least one of those things in a dynamic way?:
· The url of the first tab, I mean the tab that is at the top-left, the first of all opened tabs.
· The total amount count of opened tabs.
· The url of all tabs.

There is already a Firefox extension that:
https://addons.mozilla.org/en-US/firefox/addon/duplicate-tabs-closer/

Can I use a language other than VBScript to programmatically execute QTP Tests?

I have VBScript code which launches QuickTest Professional, executes a series of QTP tests, and emails the results. This works well, but I would prefer to use a language with better tools support (a good IDE for example). I am currently calling .Net libraries from the launch script, so I was wondering if it was possible to use a language like C# to accomplish the same task. If so, are there any good resources which address this? I could find very little on this topic via Google and there do not seem to be any other questions on SO about this topic.
For clarity, I have included the code for the routine that does the bulk of the work. This does not include the .Net declarations, but failedTestsList and allTestsList are instances of System.ArrayList.
EDIT: All the QTP documentation examples use VBScript, but as you can see, the code is just creating the QTP objects. I would assume these would be callable from another language which supported creation of these objects. It just seems from my Google failures that no one is doing it.
Sub ExecuteQTPTest(name)
Dim App, resultsPath
Dim testPath, testResults
testPath = name
allTestsList.Add(name)
Set App = CreateObject("QuickTest.Application")
App.Launch
App.Visible = False
App.Open testPath
SetQTPTestOptions(App)
SetQTPRunOptions(App)
SetQTPWebOptions(App)
App.Folders.RemoveAll
Dim qtpTest, qtpResultsOpt
Set qtpTest = App.Test
Set qtpResultsOpt = CreateObject("QuickTest.RunResultsOptions")
resultsPath = testPath & "\RES1"
qtpResultsOpt.ResultsLocation = resultsPath
qtpTest.Run qtpResultsOpt ''// Run the test
testResults = "Test Status: " & qtpTest.LastRunResults.Status & vbCrLf & _
"Last Error: " & qtpTest.LastRunResults.LastError & vbCrLf & _
"Detailed Results located at " & qtpTest.LastRunResults.Path & _
" can be viewed with the QTP Results Viewer Tool " & vbCrLf
If qtpTest.LastRunResults.Status <> "Passed" Then
g_testRunPassed = False
failureCount = failureCount + 1
failedTestsList.Add(name)
LogResults testResults, name
End If
qtpTest.Close
Set qtpResultsOpt = Nothing
Set qtpTest = Nothing
App.Quit
Set App = Nothing
End Sub

Apologies, but I don't have time to convert your full sample over to C#. I've thrown together a simple demo that should get you going. This just uses C# to open a QTP instance:
using System;
using QTObjectModelLib;
namespace QtpDemo
{
class QtpDriver
{
[STAThread]
static void Main(string[] args)
{
Application app = new Application();
app.Launch();
app.Visible = true;
}
}
}
You'll need to compile it linking to C:\Program Files\Mercury Interactive\QuickTest Professional\bin\QTObjectModelLib.dll (which is the .NET interop library for QTObjectModel.dll) and have that and QTObjectModel.dll in your app directory when you run it.
It shouldn't be that hard from here for you to convert any object declarations and function calls from VBScript to C#. Please ask if anything's unclear.
To your other point about samples on the internet - there are plenty of people out there doing more advanced stuff with QTP and QC, but I think any really clever solutions aren't shared. I, for example, would probably be prohibited from sharing such things by my employment contract, but I agree with you - there is a dearth of good QTP API samples out there, at least on Google. Having said that, I heartily recommend the SQA Forums for your QTP and QC needs.
Rich

YES, you can use anything that can "do" COM, and that includes C#.
Also VB.NET of course.
and Perl, Python, Javascript, and others.
With no help from google, you will have to follow your nose, on how to deal with the interface, but it's not that difficult when you have the existing example. Also your vendor, ideally, will have documented the COM interface for you.

if this still matters to you... QTP 11 allows you to script in C#

Please see the following answer as it will give you the exact information you are looking for to connect a C# application with QTP/UFT:
https://stackoverflow.com/a/19887866/2794121