We Keep Coding

Cannot access same source HTML as a browser

I am coming back to work on a BOT that scraped data from a site once a day for my personal use.
However they have changed the code during COVID and now it seems they are loading in a lot of the content with Ajax/JavaScript.
I thought that if I did a WebRequest and obtained the response HTML from a URL, it should match the same content that I see in a browser (FF/Chrome) when I right click and "view source". I thought the actual DOM and generated source code would come later when those files were loaded as onload events fired, scripts lazily loaded and so on.
However the source HTML I obtain with my BOT is NOT the same as the HTML I see when viewing the source code. So my regular expressions that find certain links are not available to me.
Why am I seeing a difference between "view source" and a download of the HTML?
I can only think that when the page loads, SCRIPTS run that load other content into the page and that when I view source I am actually seeing a partial generated source rather than the original source code. Therefore is there a way I can call the page with my BOT, wait X seconds before obtaining the response to get this "onload" generated HTML?
Or even better a way for MY BOT (not using someone elses), to view generated source.
This BOT runs as a web service. I can find another site to scrape but it's just painful when I have all the regular expressions working on the source I see, except it's NOT the source my BOT obtains.
A bit confused at why my browser is showing me more content with a view source (not generated source), than my BOT gets when making a valid request.
Any help would be much appreciated this is almost an 8 year project that I have been doing on/off and this change has ruined one of the core parts of the system.

In response to OP's comment, here is the Java code for how to click at different parts on the screen to do this:
You could use Java's Robot class. I just learned about it a few days ago:
// Import
import java.awt.Robot;
// Code
void click(int x, int y, int btn) {
Robot robot = new Robot();
robot.mouseMove(x, y);
robot.mousePress(btn);
robot.mouseRelease(btn);
}
You would then run the click function with the x and y position to click, as well as the button (MouseEvent.BUTTON1, MouseEvent.BUTTON2, etc.)
After stringing together the right positions (this will vary depending on the screen) you could do just about anything.
To use shortcuts, just use the keyPress and keyRelease functions. Here is a good way to do this:
void key(int keyCode, boolean ctrl, boolean alt, boolean shift) {
if (ctrl)
robot.keyPress(KeyEvent.VK_CONTROL);
if (alt)
robot.keyPress(KeyEvent.VK_ALT);
if (shift)
robot.keyPress(KeyEvent.VK_SHIFT);
robot.keyPress(keyCode);
robot.keyRelease(keyCode);
if (ctrl)
robot.keyRelease(KeyEvent.VK_CONTROL);
if (alt)
robot.keyRelease(KeyEvent.VK_ALT);
if (shift)
robot.keyRelease(KeyEvent.VK_SHIFT);
}
Thus, something like Ctrl+Shift+I to open the inspect menu would look like this:
key(KeyEvent.VK_I, true, false, true);
Here are the steps to copy a website's code (from the inspector) with Google Chrome:
Ctrl + Shift + I
Right click the HTML tag
Select "Edit as HTML"
Ctrl + A
Ctrl + C
Then, you can use the technique from this StackOverflow to get the content from the clipboard:
Clipboard c = Toolkit.getDefaultToolkit().getSystemClipboard();
String text = (String) c.getData(DataFlavor.stringFlavor);
Using something like FileOutputStream to put the info into a file:
FileOutputStream output = new FileOutputStream(new File( PATH HERE ));
output.write(text.getBytes());
output.close();
I hope this helps!

I have seemed to have fixed it by just turning on the ability to store cookies in my custom HTTP (Bot/Scraper) class, that was being called from the class trying to obtain the data. Probably the site has a defense system to prevent visitors requesting pages and not the JS/CSS with a different session ID on each request.
However I would like to see some other examples because if it is just cookies then they could use JavaScript to test for JavaScript e.g an AJAX call to log if JS is actually on or some DOM manipulation to determine if you are really Human or not which would break it again.
Every site uses different methods to prevent scrapers, email harvesters, job rapists, link harvesters etc inc working out the standard time between requests for 100% verifiable humans and BOTS and then using those values to help determine spoofed user-agents etc. I wrote a whole system to stop BOTS at my last place of work and its a layered approach, just glad the cookies being enabled solved it on this site but it could easily be beefed up with other tricks to test for BOTS vs HUMANS.
I do know some Java, enough to work out what is going on anyway. My BOT is in C#.

How to reset a password via BAPI in a CUA environment?

I am currently developing a C# application with SAP NCo 3.
I am wondering if I could invoke BAPI into CUA and this BAPI would pass details to child system.
This field is available through Test Function Module (field "RFC target sys"), but it is unavailable directly in standard BAPIs when accessed from SAP NCo.
In ABAP, devs can use:
call function 'BAPI_USER_CHANGE' destination '<TARGET_SYS>'
Can I use something similar in NCo library?
IRfcFunction rfcs = rfcDest.Repository.CreateFunction("BAPI_USER_CHANGE");
Does anybody know how this could be achieved?
Main intent is to reset user passwords to initial ones through App(BAPI) --> CUA --> ChildSystem
Without direct access into child systems.

Hmm, it looks like you have not yet fully understood the meaning of "RFC target sys".
In SE37 "RFC target sys" you enter the name of an RFC destination, which provides details about in which SAP system you want to execute the function module. These details are then defined in SM59, where you can specify parameters like hostname, system number, client, user, password, language, etc.
In the NCo library you do the same via the Class RfcDestinationManager. Here you define the parameters (hostname, system number, client, user, password, language, etc.) of the target system in which you want to execute the function module.
So the line
"RFC target sys: TARGET_SYS"
in SE37 corresponds to a line like
RfcDestination myDest = RfcDestinationManager.GetDestination("TARGET_SYS");
in your .NET program.
And a line of ABAP code like
call function 'BAPI_USER_CHANGE' destination 'TARGET_SYS'
would then correspond to some .NET code like
RfcDestination targetSys = RfcDestinationManager.GetDestination("TARGET_SYS");
IRfcFunction bapiUserChange = targetSys.Repository.CreateFunction("BAPI_USER_CHANGE");
targetSys.Invoke(bapiUserChange);
Note: setting of the input values and error handling is omitted here.

Ok, so I found out that what I wanted to achieve is not possible with just sapnco.
But, in SAP I created function module, which calls function module and uses DESTINATION 'target_sys' to run in end system. This way I achieved what I wanted. By calling my Z_FUNC_MODULE from sapnco I pass variable target_sys and FN is called in child system of CUA.
Hope this helps to someone.

load bloomberg page from c#

I have an old excel workbook that I am trying to replace with a c# application. The only bit of functionality that I have not been able to replicate is the code below.
So the code below takes a bloomberg ticker (i.e. "VOD LN") and then with DDEInitiate it loads the bloomberg page.
I have read that C# doesn't support DDE or even if it does it is best avoided. In which case how can I do this via C#?
Public Sub LoadBbergPage(string ticker)
' loads bberg page
Dim strExe As String
Dim channelGP As Long
channelGP = DDEInitiate("Winblp", "BBK")
strExe = "<blp-2><home>" & Strings.Trim(ticker) & "<EQUITY><GO>"
DDEExecute channelGP, strExe
DDETerminate channelGP
End Sub

If you're trying to make it easier for your users to launch data into the terminal, you can use 'B-links'. Access it like any other web link. Below is an example for "IBM US Equity" - replace spaces with %20
https://blinks.bloomberg.com/securities/[ticker]/[function]
https://blinks.bloomberg.com/securities/IBM%20US%20Equity/DES
It will ask the user the first time to allow / remember settings and then should launch to terminal. If there are issues, you can go to https://blinks.bloomberg.com/help. Documentation is available on terminal via DOCS BLINKS<GO> (tons more special syntax)
But if you're trying to do some kind of screen scraping etc via DDE, don't bother; just use the Reference Data API instead: https://www.bloomberg.com/professional/support/api-library/

Understanding and creating a protocol for Python

I have documentation on how to use a TCP/IP binary stream API. The attached images show the protocol. I have a working example of this in C#. I want to do this using python instead, as I don't know c# or windows.
I am assuming I would use python sockets, then I have to send the API messages, with payloads looking at this docs.
Could you point me in the right direction to get this going with python?
How would I set it to know this is the authentication message, 0x21 and compress the data etc?
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((HOST, PORT))
s.sendall(u"{'username':'username', 'password':'password'}".encode('utf8'))

in OSI model you are above layer 4 (transport, TCP/IP,...) on at least layer 5 (session). there are some examples of implementations of session layer protocols like http, ftp,... i.e. in http://github.com/python-git/python/blob/master/Lib/ftplib.py or http://github.com/python-git/python/blob/master/Lib/httplib.py
as your protocol includes headers maybe http is the better example
to use the TCP/IP API in http protocol see
http://www.stackoverflow.com/questions/8315209/sending-http-headers-with-python
import socket
sock = socket.socket()
sock.bind(('', 8080))
sock.listen(5)
client, adress = sock.accept()
print "Incoming:", adress
print client.recv(1024)
print
client.send('HTTP/1.0 200 OK\r\n')
client.send("Content-Type: text/html\r\n\r\n")
client.send('<html><body><h1>Hello World</body></html>')
client.close()
print "Answering ..."
print "Finished."
sock.close()
as far as i can see you skipped the headers (version, sequence, type, encoding, ...) in your code completely you have to add them whenever you send a frame
so try
self.socket.send(...headers...)
self.socket.send(u"{'username':'username', 'password':'password'}".encode('utf8')) // has to be send as JSON ???
see also http://www.stackoverflow.com/questions/22083359/send-tetx-http-over-python-socket
ftp example (no headers...)
# Internal: send one line to the server, appending CRLF
def putline(self, line):
line = line + CRLF
if self.debugging > 1: print '*put*', self.sanitize(line)
self.sock.sendall(line)
also see scapy

I try to use tcp protocol once. You can send authorization values(username ,password ) in all requests.And other data with it exp({'username':'username', 'password':'password','data':'value'}).With this there is no any current standart for data. Many of tcp clients send that data like this #A#:username;password;#D:value\n (A -authorization ,D -data), example

Local ReportViewer Fails at Unnecessary Login to Database

I have inherited an application that runs small reports locally using Microsoft Web ReportViewer. Our application allows you to "Preview/Print" a report by clicking on a specific button that routes the user to a URL that allows them to download the report as a PDF. We have recently received the requirement to save these PDFs to the document table in our database. I have been able to get this to work successfully on localhost; however, when I publish the application to our IIS server, I get the following error:
System.Data.SqlClient.SqlException: Login failed for user 'Domain\Servername$'.
I've reviewed all of the sites that I could find involving this error (including this one) - most point to adding the server account to the SQL database; however, this shouldn't be an issue, since the button to preview/print the document is still functional and works as expected when the application is published and all of the data is held in a local object, which was previously pulled from the database (the model parameter below). The button and the auto-generation feature use the same two methods to create the PDF document(see below).
Here's some code:
public static byte[] CreatePDFDocument(DocumentTemplateType template, Request model)
{
Warning[] warnings;
string[] streamIds;
string mimeType = string.Empty;
string encoding = string.Empty;
string extension = string.Empty;
ReportViewer viewer = new ReportViewer();
viewer.ProcessingMode = ProcessingMode.Local;
viewer.LocalReport.ReportEmbeddedResource = "Xxx.Xxx.Bll.ReportViewerRDLCs." + template.RdlcFilename;
switch ((DocumentType)template.DocumentTypeId)
{
case eDocumentType.Report1:
viewer.LocalReport.SetParameters(GetForm1Parameters(model));
break;
/**
* Several other reports are in this switch. All reports have the
* same issue - all but one are removed for brevity.
*/
}
byte[] bytes = viewer.LocalReport.Render("PDF", null, out mimeType, out encoding, out extension, out streamIds, out warnings);
return bytes;
//return new byte[5] {5,6,7,8,9}; - used for troubleshooting.
}
public static List<ReportParameter> GetReport1Parameters(Request model)
{
List<ReportParameter> rptParams = new List<ReportParameter>();
//Start comment
rptParams.Add(new ReportParameter("EmployeeFullName", string.Format("{0:NN}", model.Employee)));
rptParams.Add(new ReportParameter("EmployeePhoneNumber", string.Format("{0:(###) ###-####}", Convert.ToInt64(model.Employee.PhoneNumber))));
rptParams.Add(new ReportParameter("HrchyShortDesc", model.Employee.HrchyShortDesc));
rptParams.Add(new ReportParameter("RequestDate", model.RequestDate.ToShortDateString()));
rptParams.Add(new ReportParameter("RequestRequested", model.RequestRequestType));
rptParams.Add(new ReportParameter("ReasonForRequest", model.RequestRequestReason));
rptParams.Add(new ReportParameter("LogNumber", model.CaseId));
if (!string.IsNullOrWhiteSpace(model.TimeSensitiveReason)) rptParams.Add(new ReportParameter("TimeSensitiveReason", model.TimeSensitiveReason));
var lastAction = model.LastActionOfType(WorkflowStateActionType.EmployeeConfirmation);
if (lastAction != null)
{
rptParams.Add(new ReportParameter("TodaysDate", lastAction.ActionDate.ToShortDateString()));
rptParams.Add(new ReportParameter("EmpConfirmed", "true"));
}
else rptParams.Add(new ReportParameter("TodaysDate", DateTime.Now.ToShortDateString()));
//end comment
return rptParams;
}
Through a lot of commenting in and out and pushes to our server, I've deduced the following:
From what I can tell, the error occurs on calling GetReport1Parameters. In the code above, I included a start and end comment - I've commented out everything in between, leaving only the list initialization and return statement (of an empty list) and still received the error.
I've commented out the call to GetReport1Parameters and returned a nonsensical byte array and didn't receive an Exception.
All functionality works fine on localhost and when I step through the functions, all of the variables seem to appear normal.
Things I've tried to do to remedy the situation:
1. Removed connection strings from the app.config, so that the application has to go to the web.config to get the correct strings (even though they were the same).
2. Commented in and out different sections of code to determine the problem area.
3. Tried calling the GetReport1Parameters method and returning null, leading to a null reference exception.
4. Tried calling the GetReport1Parameters with an empty parameter list, leading to the error mentioned above.
5. Tried running the report with no parameters (not even a blank list), got a ReportProcessingException for missing params.
Some additional information:
We use a service account for the application using impersonate identity in the web.config. That line is commented out on localhost, but is running on IIS.
All of other database interaction works correctly.
All of our database interaction is done using LINQ to SQL - model is an object based off of a database table, with some additional information that is calculated dynamically.
My desired outcome is that both the autogenerated documents and the preview/print documents both work. I have a feeling that this may be something simple that I'm overlooking, but I've already spent several hours today trying to fix this.
I can't think of any other pertinent information, but if you have questions I'll be more than happy to answer them.
Edit: Additional attempts to find solution:
Tried setting LINQ Deferred Loading equal to false. This caused more problems than it solved.
Implemented IReportServerCredentials and assigned the ReportViewer's ServerReport.ReportServerCredentials with the correct database credentials.
Assigned all pertinent report parameters to a Dictionary, and then called .ToString() on every object to ensure that it is pulled from the database. Then assigned those strings from the dictionary to the report parameters, so that ReportViewer should be receiving the data from the string pool, as opposed to pulling it from the database.

Even though you are using an ObjectDataSource to pass data to your report, Report Viewer will still invoke the Select method, which in turn could cause database access to occur. So even though it may seem that the login is unnecessary, you would need to dig into the data access methods you supplied with your ObjectDataSource to know for sure.
The error you are getting is being caused by a bug in Report Viewer 2010 that is describe in the following Microsoft Connect article:
ReportViewer.LocalReport.Render and ReportViewer.LocalReport.SetParameters changes ImpersonationLevel to None
Although the article mentions this problem should be fixed in Service Pack 1, it does not appear to be the case. I have not verified if this problem is fixed in Report Viewer 2012.
I worked around the problem by changing my data access layer to compare the current identity against the one in my HttpContext and restore it if necessary using the following code snippet:
System.Security.Principal.IIdentity id = System.Web.HttpContext.Current.User.Identity
if (id.Name != System.Security.Principal.WindowsIdentity.GetCurrent().Name)
{
context = (id as System.Security.Principal.WindowsIdentity).Impersonate()
}
I do this right before I connect to the database and undo it as soon as the connection is open.
I am not exactly thrilled with this workaround, mainly because now my data access layer is referencing the UI layer (System.Web).