We Keep Coding

Connecting C# Application to Azure Databricks

I am currently working on a project where we have data stored on Azure Datalake. The Datalake is hooked to Azure Databricks.
The requirement asks that the Azure Databricks is to be connected to a C# application to be able to run queries and get the result all from the C# application. The way we are currently tackling the problem is that we have created a workspace on Databricks with a number of queries that need to be executed. We created a job that is linked to the mentioned workspace. From the C# application we are calling a number of API's listed here in this documentation to call an instance of the job and wait for it to be executed. However I have not been able to extract the result from any of the APIs listed in the documentation.
My question is this, are we taking the correct approach or is there something we are not seeing? If this is the way to go, what has been your experience in extracting the result from a successfully run job on Azure Databricks from a C# application.

Microsoft has a nice architecture reference solution that might help you get some more insights too.
I'm not sure using the REST API is the best way to go to get your job output from Azure DataBricks.
First of all the REST API has a rate limit per databrick instance. It's not that bad at 30 requests per second but it strongly depend on the scale of your application and other uses of the databrick instance if that is sufficient. It should be enough for creating a job but if you want to poll the job status for completion it might not be enough.
There is also a limited capacity in datatransfer via the REST API.
For example: As per the docs the output api will only returns the first 5MB of a run output. If you want larger results you'll have to store it somewhere else before getting it from your C# application.
Alternative retrieval method
In Short: Use Azure PaaS to your advantage with blobstorage and eventgrid.
This is in no way an exhaustive solution and I'm sure someone can come up with a better one, however this has worked for me in similar usecases.
What you can do is write the result from your job runs to some form of cloud storage connected to databricks and then get the result from that storage location later. There is a step in this tutorial that shows the basic concept for storing the results of a job with SQL data warehouse, but you can use any storage you like, for example Blob storage
Let's say you store the result in blobstorage. Each time a new job output is written to a blob, you can raise an event. You can subscribe to these events via Azure Eventgrid and consume them in your application. There is a .net SDK that will let you to do this. The event will contain a blob uri that you can use to get the data into your application.
Form the docs a blobcreated event will look something like this:
[{
"topic": "/subscriptions/{subscription-id}/resourceGroups/Storage/providers/Microsoft.Storage/storageAccounts/my-storage-account",
"subject": "/blobServices/default/containers/test-container/blobs/new-file.txt",
"eventType": "Microsoft.Storage.BlobCreated",
"eventTime": "2017-06-26T18:41:00.9584103Z",
"id": "831e1650-001e-001b-66ab-eeb76e069631",
"data": {
"api": "PutBlockList",
"clientRequestId": "6d79dbfb-0e37-4fc4-981f-442c9ca65760",
"requestId": "831e1650-001e-001b-66ab-eeb76e000000",
"eTag": "\"0x8D4BCC2E4835CD0\"",
"contentType": "text/plain",
"contentLength": 524288,
"blobType": "BlockBlob",
"url": "https://my-storage-account.blob.core.windows.net/testcontainer/new-file.txt",
"sequencer": "00000000000004420000000000028963",
"storageDiagnostics": {
"batchId": "b68529f3-68cd-4744-baa4-3c0498ec19f0"
}
},
"dataVersion": "",
"metadataVersion": "1"
}]
It will be important to name your blobs with the required information such as job Id and Run Id. You can also create custom events, which will increase the complexity of the solution but will allow you to add more details to your event.
Once you have the blob created event data in your app you can use the storage SDK to get the blobdata for use in your application. Depending on your application logic, you'll also have to manage the job ID and run Id's in the application otherwise you run the risk of having job output in your storage that is no longer attached to a process in your app.

Your use case is to use databricks as a compute engine (something similar to MySQL) and get output into C# application . So the best way is to create tables in databricks and run those queries via ODBC connection .
https://learn.microsoft.com/en-us/azure/databricks/integrations/bi/jdbc-odbc-bi
This way you have more control over sql query output.

Sync gateway is not syncing with Couchbase server in Xamarin

I am trying to use couchbase for online and offline database in my xamarin application .
the offline part is working good , but it doesn't sync .
I followed this tutorial :
https://docs.couchbase.com/userprofile-couchbase-mobile/sync/userprofile/xamarin/userprofile_sync.html
I have installed couchbase server , I created a bucket named : userprofile , and a user and I have enabled the Application Access and Read Only Admin roles.
also I installed sync gateway and I have configured it ,
here is my configuration .json file
{
"log": ["*"],
"databases": {
"userprofile": {
"server": "http://127.0.0.1:8091",
"bucket": "user-profile",
"username": "Maria",
"password": "123456",
"enable_shared_bucket_access": true,
"import_docs": true,
"num_index_replicas": 0,
"delta_sync" :{"enabled":true},
"users": {
"Maria": { "password": "123456"},
"GUEST": { "disabled": false, "admin_channels": ["*"] }
},
"sync": `function (doc, oldDoc) {
if (doc.sdk) {
channel(doc.sdk);
}
}`
}
}
}
and I used this command to configure the sync gateway :
C:\Program Files\Couchbase\Sync Gateway\sync_gateway sync-gateway-config-userprofile-walrus.json
also I have changed the sync url to ws://10.0.2.2:4984 because I am using android emulator .
but it did not sync between devices , can anyone help me please ?

1) How are you verifying if the sync is working (what steps did you follow for testing)? If you are following the instructions here, that won't work with the config you have above. Because the only valid user that you have configured in your config file is the one with credentials "username" and "password" and the steps in tutorial indicates a different username/password. So make sure you log into your app with the credentials you have in your config file and try out
2) You indicated that you have a Couchbase Server running that you have configured for Sync Gateway access. But your config file indicates that you are using "walrus" mode (Walrus mode is a memory only mode intended only for dev/test purposes). In other words, in walrus mode, you are not pointing to an actual couchbase server.
If you want to sync with a backend Couchbase Server(which you should), then replace the "walrus:" in the "server" property in the config file to point to the couchbase server.
3) The best resource to troubleshoot is the Sync Gateway logs. That should give sufficient hints as to what is going wrong with the sync. Post the errors you are seeing if any if the above tips don't work
4) Check out this tutorial for relevant background. Probably from this point

Timer-triggered Azure Function not triggered on scheduled time and failed with Azure storage timeout exception

I have a V-1 time triggered Azure Function created from VS used to get data back from a service with estimated time 9 mins via HTTPCLIENT, it runs well locally, and using storage emulator. But when I published to Azure portal with App Service Plan, two problems arise: 1> Monitor tab not showing all executions, as there are more failed executions I can find if I go to Application Insights, plus the triggered time in Application Insights are not the scheduled time frequency(every 6 hrs I set) 2> Most of them failed with below snapshot.
System.TimeoutException
Any help would be really appreciated!

There is a suggestion in one of the Github issues, and the solution was to disable dashboard logging in the code using the following snippet:
var config = new JobHostConfiguration();
config.DashboardConnectionString = null;
The full post can be viewed here

Azure database randomly stopped working after publishing UWP app

Ive made an app that uses Azure easy tables and connects using the following:
try
{
await ctv.combatDrillsTable.Initialization;
await ctv.combatDrillsTable.InitLocalStoreAsync();
await AddItemsAsync();
}
catch
{
var addError = new MessageDialog("Connection to your drills could not be established at this time, returning to " +
"main menu");
await addError.ShowAsync();
if (MainPage.MyFrame.CanGoBack)
{
MainPage.MyFrame.GoBack();
}
}
This was doing a GET request fine the last few weeks but now the connection error throws, I checked my Azure portal and the drills are there, the server overview shows the following:
HTTP Error stats
Im not really sure whats wrong and why Azure has just decided to stop working. I get the same results on bith my local machince is Visual Studio 2017 and the app installed on windows from the store.

Im not really sure whats wrong and why Azure has just decided to stop working.
Firstly, you need to enable logs in Azure, in that case you could access the information logged by Azure and check the logs to find the detail reason. Details for how to do please reference this article.
Since you're using easy table, the azure backend should be node.js. After enabled logs, you could access logs by address like: https://{your app name}.scm.azurewebsites.net/api/vfs/LogFiles/Application/index.html.
If you still cannot resolve your issues by checking the log details, you could upload the log for further looking.

How to create a database in Azure?

I'm creating a web-app following the next guide. I need to have a sql database hosted in Azure portal. In Aure hosting settings I choose existing sql server (with no database) then choose to create new database. But when I create new ASP.NET MVC project but failed. Azure portal says
Tracking Id: 73aabcd6-7cdc-4bf5-9590-a55eb0cfe279
Status: Conflict
Provisioning State: Failed
Timestamp: 4/24/2016, 12:36:23 PM
Duration: PT32.4858855S
Type: Microsoft.Sql/servers/databases
Resource Id: /subscriptions/a56b3a76-22b0-4d67-a3dd-f726672d2b2f/resourceGroups/Isolenta29ResourceGroup/providers/Microsoft.Sql/servers/isolenta29dbserver/databases/Isolenta29Database
StatusMessage: {
"status": "Failed",
"error": {
"code": "ResourceDeploymentFailure",
"message": "The resource operation completed with terminal provisioning state 'Failed'.",
"details": [
{
"code": "40827",
"message": "The operation is not supported for your subscription offer type."
}
]
}
}
Resource: isolenta29dbserver/Isolenta29Database
I have a DreamSpark Subscription and already know it provides only one free Database. So what went wrong? How to create a databas ehosted in Azure and use it in web apps?

Looks like Dreamspark already have the support of SQL DB (however, not long ago there was no such support), but i suspect that it can be some kind of intermittent problem. You should be able to use MySQL. That is the tutorial.

as states in the error message
"message": "The operation is not supported for your subscription offer type."
seems like your DreamSpark subscription is not allowed to create any SQL database.
either you upgrade your subscription or try to use file base database that can site together with your site.