We Keep Coding

Cannot access same source HTML as a browser

I am coming back to work on a BOT that scraped data from a site once a day for my personal use.
However they have changed the code during COVID and now it seems they are loading in a lot of the content with Ajax/JavaScript.
I thought that if I did a WebRequest and obtained the response HTML from a URL, it should match the same content that I see in a browser (FF/Chrome) when I right click and "view source". I thought the actual DOM and generated source code would come later when those files were loaded as onload events fired, scripts lazily loaded and so on.
However the source HTML I obtain with my BOT is NOT the same as the HTML I see when viewing the source code. So my regular expressions that find certain links are not available to me.
Why am I seeing a difference between "view source" and a download of the HTML?
I can only think that when the page loads, SCRIPTS run that load other content into the page and that when I view source I am actually seeing a partial generated source rather than the original source code. Therefore is there a way I can call the page with my BOT, wait X seconds before obtaining the response to get this "onload" generated HTML?
Or even better a way for MY BOT (not using someone elses), to view generated source.
This BOT runs as a web service. I can find another site to scrape but it's just painful when I have all the regular expressions working on the source I see, except it's NOT the source my BOT obtains.
A bit confused at why my browser is showing me more content with a view source (not generated source), than my BOT gets when making a valid request.
Any help would be much appreciated this is almost an 8 year project that I have been doing on/off and this change has ruined one of the core parts of the system.

In response to OP's comment, here is the Java code for how to click at different parts on the screen to do this:
You could use Java's Robot class. I just learned about it a few days ago:
// Import
import java.awt.Robot;
// Code
void click(int x, int y, int btn) {
Robot robot = new Robot();
robot.mouseMove(x, y);
robot.mousePress(btn);
robot.mouseRelease(btn);
}
You would then run the click function with the x and y position to click, as well as the button (MouseEvent.BUTTON1, MouseEvent.BUTTON2, etc.)
After stringing together the right positions (this will vary depending on the screen) you could do just about anything.
To use shortcuts, just use the keyPress and keyRelease functions. Here is a good way to do this:
void key(int keyCode, boolean ctrl, boolean alt, boolean shift) {
if (ctrl)
robot.keyPress(KeyEvent.VK_CONTROL);
if (alt)
robot.keyPress(KeyEvent.VK_ALT);
if (shift)
robot.keyPress(KeyEvent.VK_SHIFT);
robot.keyPress(keyCode);
robot.keyRelease(keyCode);
if (ctrl)
robot.keyRelease(KeyEvent.VK_CONTROL);
if (alt)
robot.keyRelease(KeyEvent.VK_ALT);
if (shift)
robot.keyRelease(KeyEvent.VK_SHIFT);
}
Thus, something like Ctrl+Shift+I to open the inspect menu would look like this:
key(KeyEvent.VK_I, true, false, true);
Here are the steps to copy a website's code (from the inspector) with Google Chrome:
Ctrl + Shift + I
Right click the HTML tag
Select "Edit as HTML"
Ctrl + A
Ctrl + C
Then, you can use the technique from this StackOverflow to get the content from the clipboard:
Clipboard c = Toolkit.getDefaultToolkit().getSystemClipboard();
String text = (String) c.getData(DataFlavor.stringFlavor);
Using something like FileOutputStream to put the info into a file:
FileOutputStream output = new FileOutputStream(new File( PATH HERE ));
output.write(text.getBytes());
output.close();
I hope this helps!

I have seemed to have fixed it by just turning on the ability to store cookies in my custom HTTP (Bot/Scraper) class, that was being called from the class trying to obtain the data. Probably the site has a defense system to prevent visitors requesting pages and not the JS/CSS with a different session ID on each request.
However I would like to see some other examples because if it is just cookies then they could use JavaScript to test for JavaScript e.g an AJAX call to log if JS is actually on or some DOM manipulation to determine if you are really Human or not which would break it again.
Every site uses different methods to prevent scrapers, email harvesters, job rapists, link harvesters etc inc working out the standard time between requests for 100% verifiable humans and BOTS and then using those values to help determine spoofed user-agents etc. I wrote a whole system to stop BOTS at my last place of work and its a layered approach, just glad the cookies being enabled solved it on this site but it could easily be beefed up with other tricks to test for BOTS vs HUMANS.
I do know some Java, enough to work out what is going on anyway. My BOT is in C#.

ASP.NET Process.Start work background in task Manger but not come on front on IIS8.5 (Windows 8.1)

i want to execute a .EXE file in mvc 3 application when we run this project locally on browser it work perfectly . Problem is that when we publish this project and host it on the IIS(8.5) server on window 8.1 when we click the button code executed and process start in Task Manger but app not shown on front .
For examples in this case we execute notepad.exe file .
Following is our code in html we make a one button when user click on that button controller method is called that run code .
when user click on button this is cod that is executed.
$("#Button11").click(function (event) {
$.post("/Account/Start_Appl", {}, function (data) {
});
event.preventDefault();
});
and start Start_Appl Method inside Account controller have following line of code .
public string Start_Appl()
{ string path="C:\\Windows\\System32\\notepad.exe";
ProcessStartInfo psi = new ProcessStartInfo();
psi.UseShellExecute = true;
psi.LoadUserProfile = true;
psi.WorkingDirectory = path;
psi.FileName = path;
Process.Start(psi);
return "ok";
}
i want to execute .EXE file with iis(8.5) any solution to this problem .
i check on the internet but can not find any proper solution for this problem.
i also check this link but not give any help .
ASP.NET Process.Start not working on IIS8 (Windows 8.1)

OK, first off, this is a very odd thing to do - there is going to be a better approach for whatever your problem is. That said, what I would do in this situation is:
create a database with a JOB_QUEUE table. exactly what is in this table is up to you, but I'd suggest an ID, a date_added, something to indicate what the job should consist of (e.g. the name of the exe to run?) and a flag to indicate the status of the job (pending/in progress/failed/completed)
modify controller to insert a record into this table with a 'pending' status (this is ALL it does)
write a separate application that runs in a user process (windows forms or console application). this application could run on the server which hosts IIS or a separate one - all it needs it access to the same database
Make this separate application periodically check the job queue table for jobs with a pending status.
When a new job is detected, the application updates the database so the job is 'in progress' and runs the job. if successful it updates the job to be 'completed' and if it fails it updates it to be 'failed'.
It then goes back to simply monitoring the table.
There are lots of other things you could use in place of the database for a queue, but this would be quick to write and easy to debug/test as well as making it easy to add multiple clients and persist historical info, and add controller methods to report on jobs requested/completed and progress.

For execution external .exe file in iis(8.5) and window 8.1 . we used
Custom URL Protocol for Invoking Application Techniques check this link
Custom URL Protocol
i have change my Start_Appl function is as .
public string Start_Appl()
{
try
{
string myAppPath = "C:\\Windows\\System32\\notepad.exe";
RegistryKey key = Registry.ClassesRoot.OpenSubKey("myAppa"); //open myApp protocol's subkey
if (key == null) //if the protocol is not registered yet...we register it
{
key = Registry.ClassesRoot.CreateSubKey("myAppa");
key.SetValue(string.Empty, "URL: myApp Protocol");
key.SetValue("URL Protocol", string.Empty);
key = key.CreateSubKey(#"shell\open\command");
key.SetValue(string.Empty, myAppPath + " " + "%1");
}
key.Close();
}
catch (Exception e) {
return e.Message.ToString();
}
return "ok";
}
we first register a key if it is not already create and set value and after that we make a button on my aspx page is as
<input type="submit" name="Launch" id="Launch" value="Launch Custom URL" onclick="LaunchURLScript()">
When user click on button the function LaunchURLScript is call and inside this function we write following code in it
function LaunchURLScript() {
$.post("/Account/Start_Appl", {}, function (data) {
if (data = "ok") {
alert("ok");
var url = "myAppa:"; window.open(url); self.focus();
}
});
}
for inside we first call controller method to make URl protocol if not already exist and finally we open new window that lunch my note pad exe file .
in this way we solve our problem.

Detect Document Reload has occured on server

I have an winform/OCX that consumes a qlikview document. We have gotten a patch from QV so that RefreshDocument works in the OCX as the RefreshDocument does in QV application. But the Application shows a nice enabled button when the document has been reload on the server.
Does anyone know what needs to be done to detect that. Either in C# or in macro code or ManagementAPI ?
This is the ReloadDocument Code.
private void button2_Click(object sender, EventArgs e)
{
var myBloodybookmarkHack = "dynaBookmark" + Guid.NewGuid().ToString().Replace("-","");
axQlikOCX1.ActiveDocument.CreateUserBookmark(myBloodybookmarkHack, true);
//axQlikOCX1.OpenDocument(#"qvp://qvSeverName/path/MyDocument.qvw?bookmark=Server\dynaBookmarkb5aa82ae467540fdb0d18bb499044ed9");
axQlikOCX1.RefreshDocument();
axQlikOCX1.ActiveDocument.RecallUserBookmark(myBloodybookmarkHack);
axQlikOCX1.ActiveDocument.RemoveUserBookmark(myBloodybookmarkHack);
}
By suppressing the paint event I get this to run pretty ok. Next patch will include that it keeps the selections (Will be fixed in 11.2 servicerelease 6).
You need to detect if CreateUserBookmark was successfull or not and not restore the bookmark if the creation failed.
This code works in QV 11.2 serviceRelease 5.

The filesystem reads a new modified time when the qvw file is rewritten after load. Assuming the data portion of this application is not broken out from the QVW file. Likely, you could come very close to accomplishing this by checking for new timestamps. Alternatively, if logging is enabled in the qvw document you could log read the text file* that QlikView generates to accomplish the same thing.
*The text file writes are delayed sometimes so your file might be refreshed a little bit before the log states that it is.

We ended up using the QV Management api to get the last task reload time
Download the Qv management api demo from QV
This code shows you how to get tasks on a document. Through that you get when "last document reload task" was finished.
private DateTime GetLastDocumentRun(string documentName)
{
string QMS = "http://MyQlikviewserver:4799/QMS/Service";
var client = new QMSClient("BasicHttpBinding_IQMS", QMS);
string key = client.GetTimeLimitedServiceKey();
ServiceKeyClientMessageInspector.ServiceKey = key;
var taskStatusFilter = new TaskStatusFilter();
var clientTaskStatuses = client.GetTaskStatuses(taskStatusFilter, TaskStatusScope.All);
foreach (var taskStatus in clientTaskStatuses)
{
Trace.WriteLine(taskStatus.General.TaskName);
if (taskStatus.General.TaskName.ToLower().Contains(documentName.ToLower()))
{
string fin = taskStatus.Extended.FinishedTime + "";
DateTime finishedTime;
if (DateTime.TryParse(fin, out finishedTime))
return finishedTime;
Logger.ErrMessage("QvManagementApi.GetLastDocumentRun",new Exception("Task finished time did not return a valid datetime value:" + fin));
return DateTime.MinValue;
}
}
return DateTime.MinValue;
}
This is slow, so you should run on a different thread.
Also this does not show if the task is successfully reloaded. We haven't fix that yet but on taskStatus.Extended you have the last log, which you can text parse to get if it was successfully reloaded or not.

If I understand correctly you want to know if a document has finished reloading on a QlikView server right?
I've you OCX application has a constant connection, you could evaluate the ReloadTime() function in the document which would tell you when the document was last reloaded. If you listen for the function and issuing a DocumentRefresh while doing this, then you would get a changed timestamp once the newly reloaded document becomes avaible on the server.
The code your posting, does not reload a QlikView document. At least not in QlikView lingo, it just open the documents on the server.
Please elaborate if I misunderstand you.
Regards Torber

Flajaxian Custom Folder Save Location

I see a lot of people coming up with some excessive ways to change the folder location on the fly with flajaxian multiple file upload control.
Was just wondering if the more experienced could take a look at the way I've come up with and let me know if there are any major issues I should be concerned about. (Assuming I have the proper error checking in place.)
I planned on initializing the control as seen below. :
<cc1:FileUploader ID="FileUploader1" runat="server" OnFileReceived="fileUploader_FileReceived" RequestAsPostBack="true">
</cc1:FileUploader>
(I RequestAsPostBack="true" as there are some other controls I need to check in my event handler)
I simply change the HttpFileCollection.SaveAs property in the fileUploader_FileReceived event. Since flajaxian does this one file upload at a time, we can expect that there is only 1 file in the collection (or else we could use a loop).
protected void fileUploader_FileReceived(object sender,
com.flajaxian.FileReceivedEventArgs e)
{
HttpFileCollection files = Request.Files;
// Change path to whichever folder I need
String TempFileName = "C:\\NEW\\PATH\\TO\\Folder\\" + files[0].FileName;
// Save the file.
files[0].SaveAs(TempFileName);
}
This implementation seems to work great as long as the folder is existing! I was just wondering if there is anything technically wrong with an implementation like this, again , assuming all error checking was in place.
Thanks!

A better way to do this would be to use an adapter, and over write the folder location in the
OnFileNameDetermining event. This way, we also get all the goodies with the adapter.
<cc1:FileUploader ID="FileUploader1" runat="server"` OnFileReceived="fileUploader_FileReceived" RequestAsPostBack="true">
<Adapters>
<cc1:FileSaverAdapter runat="server" FolderName="Ups" OnFileNameDetermining="fileUploader_FileDetermined" />
</Adapters>
</cc1:FileUploader>
In the file determined event, we can change the folder location programatically
protected void fileUploader_FileDetermined(object sender, com.flajaxian.FileNameDeterminingEventArgs e)
{
e.FileName = "C:\\NewFolder\\" + e.File.FileName;
}
We can use the FileReceived event to check if the folder exists, and if not, create it.
protected void fileUploader_FileReceived(object sender, com.flajaxian.FileReceivedEventArgs e)
{
int fileIndex = e.Index;
if (fileIndex == 0)
{
// We are on our first file, check if the new folder exists, if not, create it
}
}

What you are doing is fine, although, if you are saving files within the web site, consider using the MapPath method to create a physical folder from a virtual path within the web site
MapPath("/Images/User1")
This my mininal APSX implementation
<fjx:FileUploader ID="FileUploader1" runat="server" OnFileReceived="FileUploader2_FileReceived">
</fjx:FileUploader>
No adapters or folder is specified. When the FileRecevied event fires, I save files to a folder based on the Forms Authentication user name (names do not use characters not allowed in folder names).
Also note that the FileReceivedEventArgs has a reference to the (HTTP) file
e.File
The FileUploader control will show all files processed - you can even set the status code (e.g. 550) if there is an error, which is returned to the client.
Note that, the server call to the FileReceived event does not occur inside a nornal page postback, even if you specify
RequestAsPostBack="true"
So, a PagePreRender does not take place.
The only issue is, how do you perform any other processing at the client after the uploads complete (e.g. showing images uploaded).
Work I have in progress to this end is to use the client side event
FileStateChanged
When the last file is processed
if (file.state > Flajaxian.File_Uploading && isLast) {
I use JQuery to click a hidden submit button. The postback looks through session values stored when the files were saved, and renders back the images into a DIV.
However, an immediate submit causes issues with empty session inside the FileReceived event for some reason (I assume because the internal asynchronous call back has not completed). A pause of a few seconds before initiating the postback works OK.

Link object persisting across multiple pages within one browser instance using WatiN

I'm just getting started with WatiN and attempting to test a large number of pages with authentication. I've taken the approach of only creating a new instance of IE each time new login details are required. Once authenticated, the framework needs to navigate to 2 pages on the site and click a link on each to download a file (repeated multiple times within one authenticated session for different clients).
Navigating to the pages is fine and the download is working with IE9 using a combination of WatiN and SendKeys(). However, when it navigates to the second page and attempts to find the Link object by Text (which has the same text as on the previous page) it returns the download URL from the first page. This means that essentially whatever page I direct WatiN to, it still seems to be persisting the Link object from the first page.
The first method creates my browser object and returns it to the parent class:
public IE CreateBrowser(string email, string password, string loginUrl)
{
Settings.MakeNewIe8InstanceNoMerge = true;
Settings.AutoCloseDialogs = true;
IE ie = new IE(loginUrl);
ie.TextField(Find.ById("Email")).TypeText(email);
ie.TextField(Find.ById("Password")).TypeText(password);
ie.Button(Find.ById("btnLogin")).Click();
Thread.Sleep(1500);
return ie;
}
I then iterate through logins, passing the URL for each required page to the following:
public void DownloadFile(IE ie, string url)
{
//ie.NativeBrowser.NavigateTo(new Uri(url));
ie.GoTo(url);
Thread.Sleep(1000);
//TODO: Why is link holding on to old object?
Link lnk = null;
lnk = ie.Link(Find.ByText("Download file"));
lnk.WaitUntilExists();
lnk.Focus();
lnk.Click();
//Pause to allow IE file dialog box to show
Thread.Sleep(2000);
//Alt + S to save
SendKeys.SendWait("%(S)");
}
The calling method ties it all together like so (I've obfuscated some of the details):
for (int i = 0; i < loginCount; i++)
{
using (IE ie = HelperClass.CreateBrowser(lLogins[i].Email, lLogins[i].Password, ConfigurationManager.AppSettings["loginUrl"]))
{
...Gets list of clients we're wanting to check
for (int j = 0; j < clientCount; j++)
{
string url = "";
switch ()
{
case "Page1":
string startDate = "20110831";
string endDate = "20110901";
url = String.Format(page1BaseUrl, HttpUtility.UrlEncode(lClients[j].Name), startDate, endDate);
break;
case "Page2":
url = String.Format(page2BaseUrl, HttpUtility.UrlEncode(lClients[j].Name));
break;
}
HelperClass.DownloadFile(ie, url);
}
}
}
Does anyone have any idea what could be causing this or how to get around it? Do I need to create a new IE object for each request?

Okay, so I've managed to find out what was causing my Link object (and the parent Page object) to persist across multiple URLs.
It seems that because I'm clicking the Link which forces the "Save As" box in IE9, this keeps the Page object current, even as the browser runs through all the other URLs in the background. This seems to update the HTML rendered in the window but not release the existing Page object (or possibly creates additional Page objects in memory).
Because I'm using SendKeys() to hit the "Save" button, rather than a handled dialog in WatiN, the dialog stays open and persists the Page object.
From the looks of things, I need to find a different, handled way of performing my file downloads/saving.