Google DataProc API spark cluster with c# - c#

I have data in Big Query I want to run analytics on in a spark cluster. Per documentation if I instantiate a spark cluster it should come with a Big Query connector. I was looking for any sample code to do this, found one in pyspark. Could not find any c# examples. Also found some documentation on the functions in DataProc APIs nuget package.
Looking for a sample to start a spark cluster in Google cloud using c#.

After installing Google.Apis.Dataproc.v1 version 1.10.0.40 (or higher):
Below is a quick sample console app for creating a Dataproc cluster in C#:
using Google.Apis.Auth.OAuth2;
using Google.Apis.Services;
using Google.Apis.Dataproc.v1;
using Google.Apis.Dataproc.v1.Data;
using System;
using System.Threading;
namespace DataprocSample {
class Program
{
static void Main(string[] args)
{
string project = "YOUR PROJECT HERE";
string dataprocGlobalRegion = "global";
string zone = "us-east1-b";
string machineType = "n1-standard-4";
string clusterName = "sample-cluster";
int numWorkers = 2;
// See the docs for Application Default Credentials:
// https://developers.google.com/identity/protocols/application-default-credentials
// In general, a previous 'gcloud auth login' will suffice if running as yourself.
// If running from a VM, ensure the VM was started such that the service account has
// the CLOUD_PLATFORM scope.
GoogleCredential credential = GoogleCredential.GetApplicationDefaultAsync().Result;
if (credential.IsCreateScopedRequired)
{
credential = credential.CreateScoped(new[] { DataprocService.Scope.CloudPlatform });
}
DataprocService service = new DataprocService(
new BaseClientService.Initializer()
{
HttpClientInitializer = credential,
ApplicationName = "Dataproc Sample",
});
// Create a new cluster:
Cluster newCluster = new Cluster
{
ClusterName = clusterName,
Config = new ClusterConfig
{
GceClusterConfig = new GceClusterConfig
{
ZoneUri = String.Format(
"https://www.googleapis.com/compute/v1/projects/{0}/zones/{1}",
project, zone),
},
MasterConfig = new InstanceGroupConfig
{
NumInstances = 1,
MachineTypeUri = String.Format(
"https://www.googleapis.com/compute/v1/projects/{0}/zones/{1}/machineTypes/{2}",
project, zone, machineType),
},
WorkerConfig = new InstanceGroupConfig
{
NumInstances = numWorkers,
MachineTypeUri = String.Format(
"https://www.googleapis.com/compute/v1/projects/{0}/zones/{1}/machineTypes/{2}",
project, zone, machineType),
},
},
};
Operation createOperation =
service.Projects.Regions.Clusters.Create(newCluster, project, dataprocGlobalRegion).Execute();
// Poll the operation:
while (!IsDone(createOperation))
{
Console.WriteLine("Polling operation {0}", createOperation.Name);
createOperation =
service.Projects.Regions.Operations.Get(createOperation.Name).Execute();
Thread.Sleep(1000);
}
Console.WriteLine("Done creating cluster {0}", newCluster.ClusterName);
}
static bool IsDone(Operation op)
{
return op.Done ?? false;
}
}
}

Related

GRPC .net framework 4.6.2 problems connecting server hosted in AWS

)
We are trying to develop a .net application in wpf with which is consuming grpc services which are deployed on AWS.
The connection is done through GRPC.
Our code is as follows:
public static GrpcManagerFactory GetGrpcManagerFactory(ConnectionConfiguration configuration, string connectionString, DataAccess.Util.Enumerations.LogLevel clientLogLevel, bool errorMode)
{
Logger logger = new Logger(connectionString, clientLogLevel);
var caRootsPem = File.ReadAllText(Path.Combine(#"..", "cacert.pem"));
var caRootsKey = new KeyCertificatePair(caRootsPem, File.ReadAllText(Path.Combine(#"..", "AmazonRootCA1.key")));
ChannelCredentials channelCredentials = new SslCredentials(caRootsPem, caRootsKey, VerifyPeerContext);
var channel = new Channel(configuration.GrpcHost, configuration.GrpcPort, channelCredentials, new List<ChannelOption> { });
channel.ConnectAsync(DateTime.Now.AddSeconds(10).ToUniversalTime());
var provider = DataAccess.DataProvider.Grpc.GrpcFactory.GetGrpcFactory(channel, configuration, logger.LogManager.LogController);
var manFactory = new GrpcManagerFactory(provider, channel, connectionString, configuration.InterfaceRetries, clientLogLevel);
manFactory.ErrorMode = errorMode;
return manFactory;
}
When we try to either do a heartbeat or grpc request (.protos), we get either a DNS error, or that all connections fail....
Any idea?
This is our ConnectionConfiguration
public ConnectionConfiguration ConnectionConfiguration
{
#if DEBUG
get
{
return new ConnectionConfiguration
{
ClientId = _client != null ? _client.ClientId : 0,
InterfaceRetries = _clientPreferences != null ? (_clientPreferences.Depot != null ? _clientPreferences.Depot.InterfaceRetries : 1) : 1,
WenAccessTokenUrl = "token",
WenClientId = "..",
WenClientSecret = "..",
RealWebServices = true,
GrpcHost = "..",
GrpcPort = ..,
GrpcChannelCredentials = ChannelCredentials.Insecure
};
}
}
We tried to self signed our request but we get the same error.
ERROR:
DNS resolution failed for service: (AWS host)
Finally we got it working.
.Net Framework version 4.6.2
In the following way
List<ChannelOption> channelOptions = new List<ChannelOption>()
{
new ChannelOption("grpc.ssl_target_name_override", "services.hosted.in.the.cloud.com"),
};
var channelCreds = new SslCredentials(File.ReadAllText(AppDomain.CurrentDomain.BaseDirectory + #".\xxxxxxxxxxxxx.pem"));
var channel = new Channel("services.hosted.in.the.cloud.com", 443, channelCreds, channelOptions);
and at the System Environment Variables..
GRPC_DNS_RESOLVER: native
with this settings it worked for us, connecting the grpc client to the services deployed on aws cloud

How to authorize Google anlaytics data api with OAuth2

I am trying to connect to the new Google Analytics Data api using C# to request data from the new google analytics GA4. The only sample i can find is
Quickstart client libraries .net This does work but it uses a service account. The cloud .net client library google-cloud-dotnet only has examples for using a service account.
When i try to pass it desktop app credentials for using Oauth" authorization i get
Error creating credential from JSON. Unrecognized credential type.
using System;
using System.Threading;
using System.Threading.Tasks;
using Google.Analytics.Data.V1Beta;
namespace GoogleAnalyticsExamplesData
{
class Program
{
private const string PropertyId = "250796939";
private const string PathToCreds = #"C:\dev\ServiceAccountCred.json";
static async Task Main(string[] args)
{
Console.WriteLine("Hello World!");
// Check whether the environment variable exists.
var environmentVariable = Environment.GetEnvironmentVariable("GOOGLE_APPLICATION_CREDENTIALS");
// If necessary, create it.
if (environmentVariable == null)
Environment.SetEnvironmentVariable("GOOGLE_APPLICATION_CREDENTIALS", PathToCreds);
await SampleRunReport(PropertyId);
}
static async Task SampleRunReport(string propertyId = "YOUR-GA4-PROPERTY-ID")
{
// Using a default constructor instructs the client to use the credentials
// specified in GOOGLE_APPLICATION_CREDENTIALS environment variable.
var client = await BetaAnalyticsDataClient.CreateAsync(CancellationToken.None);
var request = new RunReportRequest
{
Property = "properties/" + PropertyId,
Dimensions = {new Dimension {Name = "date"},},
Metrics = {new Metric {Name = "totalUsers"}, new Metric {Name = "newUsers"}},
DateRanges = {new DateRange {StartDate = "2021-04-01", EndDate = "today"},},
};
var response = await client.RunReportAsync(request);
Console.WriteLine("Report result:");
foreach (var row in response.Rows)
{
Console.WriteLine(
$"{row.DimensionValues[0].Value}, {row.MetricValues[0].Value}, {row.MetricValues[1].Value}");
}
}
}
}
Links to Google.Analytics.Data.V1Beta Web client credentials, desktop credentials
After several hours of digging around i found that you can use ICredential using a builder. This works with a Desktop app credentials, for installed applications.
using System;
using System.IO;
using System.Threading;
using System.Threading.Tasks;
using Google.Analytics.Data.V1Beta;
using Google.Apis.Auth.OAuth2;
using Google.Apis.Util.Store;
namespace GoogleAnalyticsExamplesData
{
class Program
{
private const string PropertyId = "250796939";
private const string PathToCreds = #"C:\dev\credentials.json";
static async Task Main(string[] args)
{
Console.WriteLine("Hello World!");
await SampleRunReport(PropertyId);
}
static async Task SampleRunReport(string propertyId = "YOUR-GA4-PROPERTY-ID")
{
// Using a default constructor instructs the client to use the credentials
// specified in GOOGLE_APPLICATION_CREDENTIALS environment variable.
//var client = await BetaAnalyticsDataClient.CreateAsync(CancellationToken.None);
BetaAnalyticsDataClient client ;
await using (var stream = new FileStream(PathToCreds, FileMode.Open, FileAccess.Read))
{
// Requesting Authentication or loading previously stored authentication for userName
var credential = GoogleWebAuthorizationBroker.AuthorizeAsync(GoogleClientSecrets.Load(stream).Secrets,
new[] { "https://www.googleapis.com/auth/analytics.readonly"},
"userName",
CancellationToken.None,
new FileDataStore("credPath", true)).Result;
client = await new BetaAnalyticsDataClientBuilder
{
TokenAccessMethod = credential.GetAccessTokenForRequestAsync
}.BuildAsync();
}
var request = new RunReportRequest
{
Property = "properties/" + PropertyId,
Dimensions = {new Dimension {Name = "date"},},
Metrics = {new Metric {Name = "totalUsers"}, new Metric {Name = "newUsers"}},
DateRanges = {new DateRange {StartDate = "2021-04-01", EndDate = "today"},},
};
var response = await client.RunReportAsync(request);
Console.WriteLine("Report result:");
foreach (var row in response.Rows)
{
Console.WriteLine(
$"{row.DimensionValues[0].Value}, {row.MetricValues[0].Value}, {row.MetricValues[1].Value}");
}
}
}
}

Not working example for Dialogflow V2 api

Experienced problems with C# SDK documentation which can be found here:
http://googleapis.github.io/google-cloud-dotnet/docs/Google.Cloud.Dialogflow.V2/api/Google.Cloud.Dialogflow.V2.SessionsClient.html#Google_Cloud_Dialogflow_V2_SessionsClient_Create_Google_Api_Gax_Grpc_ServiceEndpoint_Google_Cloud_Dialogflow_V2_SessionsSettings_
No reference for method ToChannelCredentials().
We cannot connect the SDK to dialogflow, even with blank project. Is this method still existing or deprecated?
using Google.Cloud.Dialogflow.V2;
using Google.Apis.Auth.OAuth2;
using Grpc.Auth;
using Grpc.Core;
...
GoogleCredential cred = GoogleCredential.FromFile("/path/to/credentials.json");
Channel channel = new Channel(
SessionsClient.DefaultEndpoint.Host, SessionsClient.DefaultEndpoint.Port, cred.ToChannelCredentials());
SessionsClient client = SessionsClient.Create(channel);
...
// Shutdown the channel when it is no longer required.
channel.ShutdownAsync().Wait();
Have you tried connecting using the service account private key ? (Json file)
Follow these steps (working example in C#)
After you create a Dialogflow agent go to the agent's settings --> General --> click on the Service Account link
You will be sent to to google cloud platform where you can create a service account
After you create a service account, there will be an option to create a KEY, create it and download the (JSON) format of it
This key will be used to connect from your C# project to the Dialogflow agent
Install Google.Cloud.Dialogflow.V2 package in your project
Create for example a Dialogflow manager class (check below for an example)
public class DialogflowManager {
private string _userID;
private string _webRootPath;
private string _contentRootPath;
private string _projectId;
private SessionsClient _sessionsClient;
private SessionName _sessionName;
public DialogflowManager(string userID, string webRootPath, string contentRootPath, string projectId) {
_userID = userID;
_webRootPath = webRootPath;
_contentRootPath = contentRootPath;
_projectId = projectId;
SetEnvironmentVariable();
}
private void SetEnvironmentVariable() {
try {
Environment.SetEnvironmentVariable("GOOGLE_APPLICATION_CREDENTIALS", _contentRootPath + "\\Keys\\{THE_DOWNLOADED_JSON_FILE_HERE}.json");
} catch (ArgumentNullException) {
throw;
} catch (ArgumentException) {
throw;
} catch (SecurityException) {
throw;
}
}
private async Task CreateSession() {
// Create client
_sessionsClient = await SessionsClient.CreateAsync();
// Initialize request argument(s)
_sessionName = new SessionName(_projectId, _userID);
}
public async Task < QueryResult > CheckIntent(string userInput, string LanguageCode = "en") {
await CreateSession();
QueryInput queryInput = new QueryInput();
var queryText = new TextInput();
queryText.Text = userInput;
queryText.LanguageCode = LanguageCode;
queryInput.Text = queryText;
// Make the request
DetectIntentResponse response = await _sessionsClient.DetectIntentAsync(_sessionName, queryInput);
return response.QueryResult;
}
}
And then this can be called like this for example to get detect Intents
DialogflowManager dialogflow = new DialogflowManager("{INSERT_USER_ID}",
_hostingEnvironment.WebRootPath,
_hostingEnvironment.ContentRootPath,
"{INSERT_AGENT_ID");
var dialogflowQueryResult = await dialogflow.CheckIntent("{INSERT_USER_INPUT}");

New to C#. How to use google-admin-sdk (add users, add user to group a.s.o.)?

Trying to get the hang of how to use google-admin-sdk in C# (got a possible job-opening)
I've managed to create code for creating users and adding a user to a group in Python 2.7 as commandline-tools.
But the employer asked med if I could do the same in C#. I think I would get the hang of it, but would appreciate some help on how to start.
I have installed Visual Studio Express 2012 for Desktop and downloaded:
google-admin-directory_v1-rev6-csharpp-1.4.0-beta.zip
google-api-dotnet-client-1.4.0-beta-samples.zip
google-api-dotnet-client-1.4.0-beta.zip
But I can't find any (for me understandble) samples.
Any one care to give me any good pointers? Would be very much appreciated. :)
/Jonas
Edit : Adding my code so far!
using System;
using System.Diagnostics;
using System.Linq;
using DotNetOpenAuth.OAuth2;
using Google.Apis.Authentication.OAuth2;
using Google.Apis.Authentication.OAuth2.DotNetOpenAuth;
using Google.Apis.Samples.Helper;
using Google.Apis.Services;
using Google.Apis.Util;
using Google.Apis.Admin.directory_v1;
using Google.Apis.Admin.directory_v1.Data;
namespace Bergstedts.ListUsers
{
public class Program
{
static void Main(string[] args)
{
// Display the header and initialize the sample.
CommandLine.EnableExceptionHandling();
CommandLine.DisplayGoogleSampleHeader("Lists all Users");
// Register the authenticator.
var provider = new NativeApplicationClient(GoogleAuthenticationServer.Description)
{
ClientIdentifier = "my ID",
ClientSecret = "my secret"
};
var auth = new OAuth2Authenticator<NativeApplicationClient>(provider, GetAuthorization);
// Create the service.
var service = new DirectoryService(new BaseClientService.Initializer()
{
Authenticator = auth,
ApplicationName = "List Users",
});
service.Users.List().Domain = "mydomain.com";
Users results = service.Users.List().Execute();
Console.WriteLine("Users:");
foreach (User list in results.UsersValue)
{
Console.WriteLine("- " + list.Name);
}
Console.ReadKey();
}
private static IAuthorizationState GetAuthorization(NativeApplicationClient arg)
{
// Get the auth URL:
IAuthorizationState state = new AuthorizationState(new[] { DirectoryService.Scopes.AdminDirectoryUser.GetStringValue() });
state.Callback = new Uri(NativeApplicationClient.OutOfBandCallbackUrl);
Uri authUri = arg.RequestUserAuthorization(state);
// Request authorization from the user (by opening a browser window):
Process.Start(authUri.ToString());
Console.Write(" Authorization Code: ");
string authCode = Console.ReadLine();
Console.WriteLine();
// Retrieve the access token by using the authorization code:
return arg.ProcessUserAuthorization(authCode, state);
}
}
}
Edit : Found how to add the domain :
service.Users.List().Domain = "mydomain.com";
But I still get the same error message :
An error has occured:
Google.Apis.Requests.RequestError
Bad Request [400]
Errors [
Message[Bad Request] Location[ - ] Reason[badRequest] Domain[global]
]
This is fixed now!
split the list().Execute() like this! Got help from #peleyal
var listReq = service.Users.List();
listReq.Domain = domain;
Users results = listReq.Execute();
This is another way to get users from a Domain (just a little different)
String serviceAccountEmail = "xxxxxxx#developer.gserviceaccount.com";
var certificate = new X509Certificate2(#"xxxxx.p12", "notasecret", X509KeyStorageFlags.Exportable);
ServiceAccountCredential credential = new ServiceAccountCredential(
new ServiceAccountCredential.Initializer(serviceAccountEmail)
{
Scopes = new[] { DirectoryService.Scope.AdminDirectoryUser},
User = "your USER",
}.FromCertificate(certificate));
var service = new DirectoryService(new BaseClientService.Initializer()
{
HttpClientInitializer = credential,
ApplicationName = "name of your app",
});
var listReq = service.Users.List();
listReq.Domain = "your domain";
Users allUsers = listReq.Execute();
foreach(User myUser in allUsers.UsersValue){
Console.WriteLine("*" + myUser.PrimaryEmail);
}
Console.ReadKey();
For people who want more information, can visit Admin-SDK Users: list and the Directory API: Limits and Quotas

Have you used Google's Directory API?

I'm trying to use Google Directory API Library for .NET to maintain email addresses for a domain. The latest library is google-admin-directory_v1-rev6-csharp-1.4.0-beta. The best and farthest I've gotten so far is to receive a 403 error (Not Authorized to access this resource/api).
Has anyone out there successfully used it? If so, could you share some code, tips, or tricks?
I have used it and got some success in creating a console-application.
At the moment I'm trying to find a way to skip the copy/paste of the authorization code.
Don't forget to turn on API access in your APIs Console.
I'll show you a small sample:
using System;
using System.Diagnostics;
using System.Linq;
using System.Threading.Tasks;
using System.Security.Cryptography;
using System.Security.Cryptography.X509Certificates;
using DotNetOpenAuth.OAuth2;
using Google.Apis.Authentication.OAuth2;
using Google.Apis.Authentication.OAuth2.DotNetOpenAuth;
using Google.Apis.Samples.Helper;
using Google.Apis.Services;
using Google.Apis.Util;
using Google.Apis.Admin.directory_v1;
using Google.Apis.Admin.directory_v1.Data;
namespace Bergstedts.CreateNewUser
{
public class Program
{
static void Main(string[] args)
{
// Display the header and initialize the sample.
CommandLine.EnableExceptionHandling();
Console.WriteLine("Create users in a google apps domain!");
Console.WriteLine("by Jonas Bergstedt 2013");
// Get the user data and store in user object
Console.Write("Email: ");
string userId = Console.ReadLine();
Console.Write("Givenname: ");
string GivenName = Console.ReadLine();
Console.Write("Familyname: ");
string FamilyName = Console.ReadLine();
Console.Write("Password: ");
string Password = Console.ReadLine();
User newuserbody = new User();
UserName newusername = new UserName();
newuserbody.PrimaryEmail = userId;
newusername.GivenName = GivenName;
newusername.FamilyName = FamilyName;
newuserbody.Name = newusername;
newuserbody.Password = Password;
// Register the authenticator.
var provider = new NativeApplicationClient(GoogleAuthenticationServer.Description)
{
ClientIdentifier = "<your clientId from Google APIs Console>",
ClientSecret = "<your clientsecret from Google APIs Console>",
};
var auth = new OAuth2Authenticator<NativeApplicationClient>(provider, GetAuthorization);
// Create the service.
var service = new DirectoryService(new BaseClientService.Initializer()
{
Authenticator = auth,
ApplicationName = "Create User",
ApiKey = "<your API Key from Google APIs console> (not sure if needed)"
});
User results = service.Users.Insert(newuserbody).Execute();
Console.WriteLine("User :" + results.PrimaryEmail + " is created");
Console.WriteLine("Press any key to continue!");
Console.ReadKey();
}
private static IAuthorizationState GetAuthorization(NativeApplicationClient arg)
{
// Get the auth URL:
IAuthorizationState state = new AuthorizationState(new[] { DirectoryService.Scopes.AdminDirectoryUser.GetStringValue() });
state.Callback = new Uri(NativeApplicationClient.OutOfBandCallbackUrl);
Uri authUri = arg.RequestUserAuthorization(state);
// Request authorization from the user (by opening a browser window):
Process.Start(authUri.ToString());
Console.WriteLine();
Console.Write("Authorization Code: ");
string authCode = Console.ReadLine();
// Retrieve the access token by using the authorization code:
return arg.ProcessUserAuthorization(authCode, state);
}
}
}

Categories