I've been tasked with creating a function app that will work with data in a Data Lake that uses parquet Synapse Delta Format files. I've done this with a Synapse notebook in the past, but getting a spark session up and running in a function app is posing some challenges. No matter how I try to use the builder, it keeps trying to connect to local host for debugging, but I need it to accept a different location. I've tried passing in a .Master("") location directly on the SparkSession call and I've tried building a SparkConf and setting the master location there. In both cases, it will try to connect to local host instead and, of course, immediately fail. In the case of SparkConf, it will fail on the SparkConf config = new SparkConf(bool) line and in the case of the SparkSession, it will fail on the line where I try to create the session. I've looked at all of the examples that I can find and they all deal with just setting up a local debuggable instance, which is not what I want or need. It needs to be self-contained to work as a function app and remotely connect to the spark pool I've provided it.
Related
I'm writing a WPF application.
Trying to use the normal method of getting a connection returns an error similar to: "The 'Microsoft.ACE.OLEDB.12.0' provider is not registered on the local machine."
ACE.OLEDB has never been installed on this machine so this error makes sense.
I'm trying to create this application in a way so that our users won't need to contact IT to have the application installed. Getting IT involved is a no go situation and the project will be abandoned.
Another team has an Access database (accdb) that I want my application to extract information (only read, no insert or update). I talked to the team and they won't convert this database back to an earlier version (mdb).
After my research I assume that installing ACE.OLEDB without using Admin privileges is impossible. Because of this and my application requirement of not requiring admin privileges I need to start looking for "Mutant"/Dirty solutions that don't involve ACE.OLEDB.
I tried using power-shell but I'm getting the same problems as I had with C# (requires IT to install ACE.OLEDB).
I have two potential solutions. One write a VBA script that opens up the database and dumps a query result into a file. My C# application would call this VB script and then parse the created file.
The second option is to create a new Access process using Process.Start(fullFilePath) and somehow pass the command to execute a query and somehow pass the results back to the executing application (either via a method return or first to a file).
How would you get the data out?
Is there a way for C# to duplicate the DB file and convert it from (accdb -> mdb)?
This is the second question I ask that is very similar.
C# Connecting to Access DB with no install
The difference between the two (to prevent this is a duplicate question) is that in the previous question I was looking for ways to install ACE.OLEDB without admin privileges while here I'm just looking for any other work around.
Found a workaround. It uses Microsoft.Office.Interop.Access found in NuGet.
var accApp = new Microsoft.Office.Interop.Access.Application();
accApp.OpenCurrentDatabase(#tests.DatabasePath);
Microsoft.Office.Interop.Access.Dao.Database cdb = accApp.CurrentDb();
Microsoft.Office.Interop.Access.Dao.Recordset rst =
cdb.OpenRecordset(
"SELECT * FROM Users",
Microsoft.Office.Interop.Access.Dao.RecordsetTypeEnum.dbOpenSnapshot);
while (!rst.EOF)
{
Console.WriteLine(rst.Fields["username"].Value);
rst.MoveNext();
}
rst.Close();
accApp.CloseCurrentDatabase();
accApp.Quit();
I have a web service that uses SSH.NET to call a shell script on a Unix box.
If I run the script normally, it works fine, does its work correctly on the Informix DB.
Just some background:
I call a script that executes a .4gl (cant show this as its business knowledge).
The g4l is giving the following error back in a log, when I execute it with SSH.NET:
fglgo: error while loading shared libraries: libiffgisql.so: cannot open shared object file: No such file or directory
file_load ended: 2017-09-21 15:37:01
C# code to execute SSH.NET script
sshclients = new SshClient(p, 22, username, password);
sshclients.Connect();
sshclients.KeepAliveInterval = new TimeSpan(0, 0, 1);
sshclients.RunCommand("sh " + Script_dir);
I added the KeepAliveInterval, to see, if it helps.
My question is the error I am getting from Unix/4gl.
Why is this happening and who can I get the script to execute correctly?
The SshClient.RunCommand uses SSH "exec" channel internally. It, by default, (rightfully) does not allocate a pseudo terminal (PTY) for the session. As a consequence a different set of startup scripts is (might be) sourced. And/or different branches in the scripts are taken, based on absence/presence of the TERM environment variable. So the environment might differ from the interactive session, you use with your SSH client.
So, in your case, the PATH is probably set differently; and consequently the shared object cannot be found.
To verify that this is the root cause, disable the pseudo terminal allocation in your SSH client. For example in PuTTY, it's Connection > SSH > TTY > Don't allocate a pseudo terminal. Then, go to Connection > SSH > Remote command and enter your g4l command. Check Session > Close window on exit > Never and open the session. You should get the same "No such file or directory" error.
Ways to fix this, in preference order:
Fix the scripts not to rely on a specific environment.
Fix your startup scripts to set the PATH the same for both interactive and non-interactive sessions.
If the command itself relies on a specific environment setup and you cannot fix the startup scripts, you can change the environment in the command itself. Syntax for that depends on the remote system and/or the shell. In common *nix systems, this works:
sshclients.RunCommand("PATH=\"$PATH;/path/to/g4l\" && sh ...");
Another (not recommended) approach is to force the pseudo terminal allocation for the "exec" channel.
Though SSH.NET does not support this. You would have to modify its code issue SendPseudoTerminalRequest request in .RunCommand implementation (I didn't test this).
You can also try to use "shell" channel using .CreateShell method. For it, SSH.NET does support pseudo terminal allocation.
Though, using the pseudo terminal to automate a command execution can bring you nasty side effects. See for example Is there a simple way to get rid of junk values that come when you SSH using Python's Paramiko library and fetch output from CLI of a remote machine?
For a similar issues, see
Renci SSH.NET - no result string returned for opmnctl
Certain Unix commands fail with "... not found", when executed through Java using JSch
Commands executed using JSch behaves differently than in SSH terminal (bypasses confirm prompt message of "yes/"no")
JSch: Is there a way to expose user environment variables to "exec" channel?
Have seen similar questions asked by Informix-4gl developers as they transition to FourJs Genero and using its Web Services functionality. The question I'll put to them is "who owns the fglgo/fglrun process that the Genero Application Server has launched, where is it running from, and what is its environment". If needed, I'll illustrate with a simple program that does something like ...
MAIN
RUN "env > /tmp/myname.txt"
RUN "who >> /tmp/myname.txt"
RUN "pwd >> /tmp/myname.txt"
END MAIN
... and say compare with when program is running from command line. It is normally a case like in the earlier answer of configuring so that the environment is set correctly before the 4gl program is executed.
So I'm making a small chat application like ricochet, but then in C# and I succesfully connect to the tor controlport and create a hidden service id and private key, however after that point i got stuck, I send ADD_ONION NEW:BEST Port=8946,127.0.0.1:8946\r\n to the tor control port and it answers with hidden service id and private key and code 250. But what should I do to make it automatically run the service? I tried googling it but couldnt find anything and all examples are python or c++ if someone could point me in the right direction that would be great. Also, im using Knapcode.TorSharp, so the tor installation is NOT persistent, the user has a profile file where the key etc are saved and it should start from there.
Thanks in advance
When you call ADD_ONION, the hidden service starts running immediately (accessible once it can publish the Hidden Service descriptors and establish circuits [usually within a minute or two]).
If you want those services to start again automatically on subsequent runs (for a non-persisting Tor installation), then you'll need to programatically make similar calls to ADD_ONION when your application restarts and detects that private keys are saved to the profile.
You can re-create hidden services using existing keys with syntax like:
ADD_ONION RSA1024:*PKEY_GOES_HERE* Flags=DiscardPK Port=8946,8946
When you call ADD_ONION the first time, the response should look something like:
250-ServiceID=abcdefg123456
250-PrivateKey=RSA1024:MIICXAIBAAKBgQC91z4mjpNF5ddRL6jm7rnmgwSiQ6dNXF1Fo8sz1wOsGqWKgE4C6Bd3KT+zgQgXJlioIJOCEP9D0b/qlPCvEiGG3/fPEn1+Zpf5N4oNRI+in7J2m3xihhgAinbscJ0vM+1vfnRLlMrtYdE9J5aKle+t+OC6ZoXTxzPZRZkmXtqVpp8QIDAQPXAoGBAK7oh8zChBJch5u3i6jpvsIRaM2QA68VMKKfHPOwYSPKkUcgm7+10xjpGlXqxmd93yVYjk/CFU6JDIe3nmHPFK82BtPgyEMRtmVmcunS262Ead/ffpzAErBSdihOF7zO/wGjGgIaMW9Bhy69aK5LcNUB30Iu9+MWG62xz8tTgcEhAkEA8QNKMyKdRUbgGc9Gv1n8JtMs0Af/a/OHozdn1ywvHxw7mzahF936gqHIdg67XLtIj5TaUSM/44OoEvvURnG7QJBAMmlVttRd8y+/FnA6dPkesQMpPw+ipHLNUrf7qPrX3py670vLbprWDNYCOn6oaxoRtl/iRXPI5CgjMXmnu356pUCQDnWD0VMJi+MvZSUACbZXwP2ApP1bHfla3I7Xaezh5oDxtoAd0PS4STh1+HQUPvQW4WfLUcSsz9UaMAg2NI+fFUCQc7D1PVW7sqSGBth3jXE+3+H6WY2iy8Z1Ji+l2KRdJ8IiIOWdfcgUpMNzZV8jc7Y9Cm5p5l2wy7kjfGADyYBCXkCQD9fnmVMlUO1xITfW8K+pAf6FPcvfo8J0rpWHEhG4CxjFw4s4s9Mzjme1e17YnfK21CNIOxd2bkqVI4j4o=
250 OK
You'll want to save what it gave back in PrivateKey, and use this value to restart the hidden services the next time you run your application.
I've been working to try and convert Microsoft's EWS Streaming Notification Example to a service
( MS source http://www.microsoft.com/en-us/download/details.aspx?id=27154).
I tested it as a console app. I then used a generic service template and got it to the point it would compile, install, and start. It stops after about 10 seconds with the ubiquitous "the service on local computer started and then stopped."
So I went back in and upgraded to C# 2013 express and used NLog to put a bunch of log trace commands to so I could see where it was when it exited.
The last place I can find it is in the example code, SynchronizationChanges function,
public static void SynchronizeChanges(FolderId folderId)
{
logger.Trace("Entering SynchronizeChanges");
bool moreChangesAvailable;
do
{
logger.Trace("Synchronizing changes...");
//Console.WriteLine("Synchronizing changes...");
// Get all changes since the last call. The synchronization cookie is stored in the
// _SynchronizationState field.
// Only the the ids are requested. Additional properties should be fetched via GetItem
//calls.
logger.Trace("Getting changes into var changes.");
var changes = _ExchangeService.SyncFolderItems(folderId, PropertySet.IdOnly, null, 512,
SyncFolderItemsScope.NormalItems,
_SynchronizationState);
// Update the synchronization cookie
logger.Trace("Updating _SynchronizationState");
the log file shows the trace message ""Getting changes into var changes." but not the "Updating _SynchronizationState" message.
so it never gets past var changes = _ExchangeService.SyncFolderItems
I cannot for the life figure out why its just exiting. There are many examples of EWS streaming notifications. I have 3 that compile and run just fine but nobody as far as I can tell has posted an example of it done as a service.
If you don't see the "Updating..." message it's likely the sync threw an exception. Wrap it in a try/catch.
OK, so now that I see the error, this looks like your garden-variety permissions problem. When you ran this as a console app, you likely presented the default credentials to Exchange, which were for your login ID. For a Windows service, if you're running the service with one of the built-in accounts (e.g. Local System), your default credentials will not have access to Exchange.
To rectify, either (1) run the service under the account you did the console app with, or (2) add those credentials to the Exchange Service object.
Today, for each customer, we deploy same SSRS reports folder and data source folder.
The difference between these folders are the name of each folder and the connection string of the data source.
We are using Report Server 2008 R2.
Is it possible to maintain only one reports and data source folder and change programmatically its connection string on server-side before the report been rendered?
If not, Is it something that can be achieved by changing some logic in reports?
Today we use "shared data source" option.
This is something we've done in our environment - we maintain one set of reports that can be deployed at any client with their own configuration.
You've got a couple of options here. Since you're using a Shared Data Source this makes things easier as you won't need to define a Data Source for each report.
1. Use the rs.exe utility and a script file
rs.exe at Books Online
This program allows you to create script files (in VB.NET) that can interact with a Report Server Web Service. You create a script file (e.g. Deploy.rss) and call the rs.exe program with various parameters, including any custom ones you define:
rs.exe -i DeployReports.rss -s http://server/ReportServer -v DatabaseInstance="SQL" -v DatabaseName="ReportDB" -v ReportFolder="ClientReports"
So this would call a script DeployReports.rss, connect to http://server/ReportServer, with three user defined parameters which could be used to create a data source and the report folder.
In the scipt file you could have something like this:
Public Sub Main()
rs.Credentials = System.Net.CredentialCache.DefaultCredentials
CreateFolder(reportFolder, "Report folder")
CreateFolder(datasourceFolder, "Data source folder")
CreateDataSource()
End Sub
Which can then make Web Service calls like:
rs.CreateFolder(folderName, "/", Nothing)
'Define the data source definition.
Dim definition As New DataSourceDefinition()
definition.CredentialRetrieval = CredentialRetrievalEnum.Integrated
definition.ConnectString = "data source=" + DatabaseInstance + ";initial catalog=" + DatabaseName
definition.Enabled = True
definition.EnabledSpecified = True
definition.Extension = "SQL"
definition.ImpersonateUser = False
definition.ImpersonateUserSpecified = True
'Use the default prompt string.
definition.Prompt = Nothing
definition.WindowsCredentials = False
Try
rs.CreateDataSource(datasource, datasourcePath, False, definition, Nothing)
Console.WriteLine("Data source {0} created successfully", datasource)
Catch e As Exception
Console.WriteLine(e.Message)
End Try
You haven't specified what version of Reporting Services you're using, so I'm assuming 2008. Please note that there are multiple endpoints that can be used, depending on SQL Server version. The 2005/2008 end point is deprecated in 2008R2 and above but is still usable. Just something to bear in mind when writing your script.
2. Call the SSRS Web Service through an application
Report Server Web Service overview
The same calls that are made from the script above can be made in any other application, too. So you'd just need to add a reference to a Report Server Web Service through WSDL and you can connect to a remote service and call its methods to deploy reports, data sources, etc.
So ultimately you're connecting to the Report Server Web Service, it's just the medium used that you need to think about.
Using a script is easier to get running as it's just running a program from the command line, but writing your own deployment application will certainly give greater flexibility. I would recommend getting the script going, so you understand the process, then migrate this to a bespoke application if required. Good luck!
You can use an Expression Based Connection String to select the correct database. You can base this on a parameter your application passes in, or the UserId global variable. I do believe you need to configure the unattended execution account for this to work.
Note: be careful about the security implications. Realize that if you would pass sensitive data (e.g. passwords) into a parameter, that (a) it will go over the wire, and (b) will be stored in the execution log tables for reporting services.