When my AWS Credentials File (see docs) is updated by an external process the AmazonSQSClient doesn't re-read it, SendMessageAsync fails with a security/token error.
We use a custom powershell script to refresh the local AWS cred's file periodically. The script works fine, the file is refreshed prior to the credentials expiring on AWS. However, if my app is running when the file is refreshed the new credentials are not re-read from the file, the "client" will show that the previous credentials are still in use.
The AWS docs list several AWSCredential providers but none of them seem to be the correct choice...I think..
Restarting the app works, the new credentials are read correctly and messages are sent until the next time the cred's file is updated.
using (var client = new AmazonSQSClient(Amazon.RegionEndpoint.EUWest1))
{
return client.SendMessageAsync(request);
}
I don't think there is a way for a running app to pick up the default credentials being refreshed in credentials file. There is a solution for Node.js loading credentials from a JSON file. You can create a similar solution in C#. You can also run a local DB to store credentials so whenever credentials file is updated DB table or JSON file is also updated. You will need to use access key and secret key in your SQS client constructor as opposed to using default credentials.
// Load these from JSON file or DB.
var accessKey = "";
var secretKey = "";
using (var client = new AmazonSQSClient(accessKey, secretKey, Amazon.RegionEndpoint.EUWest1))
{
return client.SendMessageAsync(request);
}
The following works "ok" but I've only tested it with one profile and the file watcher is not as timely as you'd like so I'd recommend you wrap your usage inside a Retry mechanism.
// Usage..
var credentials = new AwsCredentialsFile();
using (var client = new AmazonSQSClient(credentials, Amazon.RegionEndpoint.EUWest1))
{
return client.SendMessageAsync(request);
}
public class AwsCredentialsFile : AWSCredentials
{
// https://docs.aws.amazon.com/sdk-for-net/v2/developer-guide/net-dg-config-creds.html#creds-file
private const string DefaultProfileName = "default";
private static ConcurrentDictionary<string, ImmutableCredentials> _credentials = new ConcurrentDictionary<string, ImmutableCredentials>(StringComparer.OrdinalIgnoreCase);
private static FileSystemWatcher _watcher = BuildFileSystemWatcher();
private readonly System.Text.Encoding _encoding;
private readonly string _profileName;
public AwsCredentialsFile()
: this(AwsCredentialsFile.DefaultProfileName, System.Text.Encoding.UTF8)
{
}
public AwsCredentialsFile(string profileName)
: this(profileName, System.Text.Encoding.UTF8)
{
}
public AwsCredentialsFile(string profileName, System.Text.Encoding encoding)
{
_profileName = profileName;
_encoding = encoding;
}
private static FileSystemWatcher BuildFileSystemWatcher()
{
var watcher = new FileSystemWatcher
{
Path = Path.GetDirectoryName(GetDefaultCredentialsFilePath()),
NotifyFilter = NotifyFilters.LastWrite,
Filter = "credentials"
};
watcher.Changed += (object source, FileSystemEventArgs e) => { _credentials?.Clear(); };
watcher.EnableRaisingEvents = true;
return watcher;
}
public static string GetDefaultCredentialsFilePath()
{
return System.Environment.ExpandEnvironmentVariables(#"C:\Users\%USERNAME%\.aws\credentials");
}
public static (string AccessKey, string SecretAccessKey, string Token) ReadCredentialsFromFile(string profileName, System.Text.Encoding encoding)
{
var profile = $"[{profileName}]";
string awsAccessKeyId = null;
string awsSecretAccessKey = null;
string token = null;
var lines = File.ReadAllLines(GetDefaultCredentialsFilePath(), encoding);
for (int i = 0; i < lines.Length; i++)
{
var text = lines[i];
if (text.Equals(profile, StringComparison.OrdinalIgnoreCase))
{
awsAccessKeyId = lines[i + 1].Replace("aws_access_key_id = ", string.Empty);
awsSecretAccessKey = lines[i + 2].Replace("aws_secret_access_key = ", string.Empty);
if (lines.Length >= i + 3)
{
token = lines[i + 3].Replace("aws_session_token = ", string.Empty);
}
break;
}
}
var result = (AccessKey: awsAccessKeyId, SecretAccessKey: awsSecretAccessKey, Token: token);
return result;
}
public override ImmutableCredentials GetCredentials()
{
if (_credentials.TryGetValue(_profileName, out ImmutableCredentials value))
{
return value;
}
else
{
var (AccessKey, SecretAccessKey, Token) = ReadCredentialsFromFile(_profileName, _encoding);
var credentials = new ImmutableCredentials(AccessKey, SecretAccessKey, Token);
_credentials.TryAdd(_profileName, credentials);
return credentials;
}
}
}
Related
I'm trying to implement Firebase 3 Authentication mechanism using Custom Tokens (as described at https:// firebase.google.com/docs/auth/server/create-custom-tokens).
My server is ASP.NET MVC Application.
So according to the instructions (https://firebase.google.com/docs/server/setup) I've created a service account for my Firebase application and generated a key in '.p12' format.
After that according to instructions here (https://firebase.google.com/docs/auth/server/create-custom-tokens#create_custom_tokens_using_a_third-party_jwt_library) I tried to generate a custom token and sign it using the key received on the previous step. For token generation I used SystemIdentityModel.Tokens.Jwt library from Microsoft, so the code looks like the following:
var now = DateTime.UtcNow;
var tokenHandler = new JwtSecurityTokenHandler();
var key = new X509AsymmetricSecurityKey(new X509Certificate2(p12path, p12pwd));
var signinCredentials = new SigningCredentials(key, "http://www.w3.org/2001/04/xmldsig-more#rsa-sha256", "http://www.w3.org/2001/04/xmlenc#rsa-sha256");
Int32 nowInUnixTimestamp = (Int32)(now.Subtract(new DateTime(1970, 1, 1))).TotalSeconds;
var token = tokenHandler.CreateToken(
issuer: serviceAccountEmail,
audience: "https://identitytoolkit.googleapis.com/google.identity.identitytoolkit.v1.IdentityToolkit",
signingCredentials: signinCredentials,
subject: new ClaimsIdentity(new Claim[]
{
new Claim("sub", serviceAccountEmail),
new Claim("iat", nowInUnixTimestamp.ToString()),
new Claim("exp", (nowInUnixTimestamp + (60*60)).ToString()),
new Claim("uid", uid)
})
);
var tokenString = tokenHandler.WriteToken(token);
Then tried to sign in user in React Native application using Firebase Javascript SDK, with the following code:
//omitting initialization code
firebase.auth().signInWithCustomToken(firebaseJWT).catch(function(error) {
console.log('Error authenticating Firebase user. Code: ' + error.code + ' Message: ' + error.message);
});
But got an error from Firebase saying:
Error authenticating Firebase user. Code: auth/invalid-custom-token Message: The custom token format is incorrect. Please check the documentation.
Experimenting with adding different claims for token expiration control didn't help either.
Also I tried to generate tokens with "dvsekhvalnov/jose-jwt" library but can't get it working with "RS256" algorithm.
So the question:
Any suggestion on what am I doing wrong?
This pure .NET solution works for me, using the Org.BouncyCastle (https://www.nuget.org/packages/BouncyCastle/) and Jose.JWT (https://www.nuget.org/packages/jose-jwt/) libraries.
I followed these steps:
In the Firebase console click the 'cog' icon which is top left, next to the project name, and click 'Permissions'.
At the IAM and Admin page, click 'Service Accounts' on the left
Click 'Create Service Account' at the top, enter a 'Service Account Name', select 'Project->Editor' in the Role selection, tick the 'Furnish a new private key' checkbox and select JSON
Click 'Create' and download the Service Account JSON file and keep it safe.
Open the Service Account JSON file in a suitable text editor and put the values into the following code:
// private_key from the Service Account JSON file
public static string firebasePrivateKey=#"-----BEGIN PRIVATE KEY-----\nMIIE...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n-----END PRIVATE KEY-----\n";
// Same for everyone
public static string firebasePayloadAUD="https://identitytoolkit.googleapis.com/google.identity.identitytoolkit.v1.IdentityToolkit";
// client_email from the Service Account JSON file
public static string firebasePayloadISS="serviceaccountname#projectname.iam.gserviceaccount.com";
public static string firebasePayloadSUB="serviceaccountname#projectname.iam.gserviceaccount.com";
// the token 'exp' - max 3600 seconds - see https://firebase.google.com/docs/auth/server/create-custom-tokens
public static int firebaseTokenExpirySecs=3600;
private static RsaPrivateCrtKeyParameters _rsaParams;
private static object _rsaParamsLocker=new object();
void Main() {
// Example with custom claims
var uid="myuserid";
var claims=new Dictionary<string, object> {
{"premium_account", true}
};
Console.WriteLine(EncodeToken(uid, claims));
}
public static string EncodeToken(string uid, Dictionary<string, object> claims) {
// Get the RsaPrivateCrtKeyParameters if we haven't already determined them
if (_rsaParams == null) {
lock (_rsaParamsLocker) {
if (_rsaParams == null) {
StreamReader sr = new StreamReader(GenerateStreamFromString(firebasePrivateKey.Replace(#"\n","\n")));
var pr = new Org.BouncyCastle.OpenSsl.PemReader(sr);
_rsaParams = (RsaPrivateCrtKeyParameters)pr.ReadObject();
}
}
}
var payload = new Dictionary<string, object> {
{"claims", claims}
,{"uid", uid}
,{"iat", secondsSinceEpoch(DateTime.UtcNow)}
,{"exp", secondsSinceEpoch(DateTime.UtcNow.AddSeconds(firebaseTokenExpirySecs))}
,{"aud", firebasePayloadAUD}
,{"iss", firebasePayloadISS}
,{"sub", firebasePayloadSUB}
};
return Jose.JWT.Encode(payload, Org.BouncyCastle.Security.DotNetUtilities.ToRSA(_rsaParams), JwsAlgorithm.RS256);
}
private static long secondsSinceEpoch(DateTime dt) {
TimeSpan t = dt - new DateTime(1970, 1, 1);
return (long)t.TotalSeconds;
}
private static Stream GenerateStreamFromString(string s) {
MemoryStream stream = new MemoryStream();
StreamWriter writer = new StreamWriter(stream);
writer.Write(s);
writer.Flush();
stream.Position = 0;
return stream;
}
To get this working in IIS I needed to change the application's pool identity and set the "load user profile" setting to true.
Haven't found a direct answer for the question so far, so for now ended up with the following solution:
Using instruction here generated a JSON file with service account details and created a basic Node.js server using Firebase server SDK that does generate correct custom tokens for Firebase with the following code:
var http = require('http');
var httpdispatcher = require('httpdispatcher');
var firebase = require('firebase');
var config = {
serviceAccount: {
projectId: "{projectId}",
clientEmail: "{projectServiceEmail}",
privateKey: "-----BEGIN PRIVATE KEY----- ... ---END PRIVATE KEY-----\n"
},
databaseURL: "https://{projectId}.firebaseio.com"
};
firebase.initializeApp(config);
const PORT=8080;
httpdispatcher.onGet("/firebaseCustomToken", function(req, res) {
var uid = req.params.uid;
if (uid) {
var customToken = firebase.auth().createCustomToken(uid);
res.writeHead(200, {'Content-Type': 'application/json'});
res.end(JSON.stringify({'firebaseJWT' : customToken}));
} else {
res.writeHead(400, {'Content-Type': 'text/plain'});
res.end('No uid parameter specified');
}
});
function handleRequest(request, response){
try {
//log the request on console
console.log(request.url);
//Disptach
httpdispatcher.dispatch(request, response);
} catch(err) {
console.log(err);
}
}
//create a server
var server = http.createServer(handleRequest);
//start our server
server.listen(PORT, function(){
console.log("Server listening on: http://localhost:%s", PORT);
});
Maybe someone will find this helpful.
#Elliveny's answer worked great for me. I am using it in a .NET Core 2.0 application and have built upon the accepted answer to turn this solution into a class that can be registered as a singleton dependency in the app services container, as well as have configuration passed in via constructor so that we can leverage .NET secrets for local development configuration and environment variables for production configuration.
I have also tidied up the stream handling a bit.
Note for .NET Core devs - you'll need to use Portable.BouncyCastle
You can test your encoded results by parsing the output JWT token with Jwt.IO
using Jose;
using Org.BouncyCastle.Crypto.Parameters;
using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
public class FirebaseTokenGenerator
{
// private_key from the Service Account JSON file
public static string firebasePrivateKey;
// Same for everyone
public static string firebasePayloadAUD = "https://identitytoolkit.googleapis.com/google.identity.identitytoolkit.v1.IdentityToolkit";
// client_email from the Service Account JSON file
public static string firebasePayloadISS;
public static string firebasePayloadSUB;
// the token 'exp' - max 3600 seconds - see https://firebase.google.com/docs/auth/server/create-custom-tokens
public static int firebaseTokenExpirySecs = 3600;
private static RsaPrivateCrtKeyParameters _rsaParams;
private static object _rsaParamsLocker = new object();
public FirebaseTokenGenerator(string privateKey, string clientEmail)
{
firebasePrivateKey = privateKey ?? throw new ArgumentNullException(nameof(privateKey));
firebasePayloadISS = clientEmail ?? throw new ArgumentNullException(nameof(clientEmail));
firebasePayloadSUB = clientEmail;
}
public static string EncodeToken(string uid)
{
return EncodeToken(uid, null);
}
public static string EncodeToken(string uid, Dictionary<string, object> claims)
{
// Get the RsaPrivateCrtKeyParameters if we haven't already determined them
if (_rsaParams == null)
{
lock (_rsaParamsLocker)
{
if (_rsaParams == null)
{
using (var streamWriter = WriteToStreamWithString(firebasePrivateKey.Replace(#"\n", "\n")))
{
using (var sr = new StreamReader(streamWriter.BaseStream))
{
var pr = new Org.BouncyCastle.OpenSsl.PemReader(sr);
_rsaParams = (RsaPrivateCrtKeyParameters)pr.ReadObject();
}
}
}
}
}
var payload = new Dictionary<string, object> {
{"uid", uid}
,{"iat", SecondsSinceEpoch(DateTime.UtcNow)}
,{"exp", SecondsSinceEpoch(DateTime.UtcNow.AddSeconds(firebaseTokenExpirySecs))}
,{"aud", firebasePayloadAUD}
,{"iss", firebasePayloadISS}
,{"sub", firebasePayloadSUB}
};
if (claims != null && claims.Any())
{
payload.Add("claims", claims);
}
return JWT.Encode(payload, Org.BouncyCastle.Security.DotNetUtilities.ToRSA(_rsaParams), JwsAlgorithm.RS256);
}
private static long SecondsSinceEpoch(DateTime dt)
{
TimeSpan t = dt - new DateTime(1970, 1, 1);
return (long) t.TotalSeconds;
}
private static StreamWriter WriteToStreamWithString(string s)
{
MemoryStream stream = new MemoryStream();
StreamWriter writer = new StreamWriter(stream);
writer.Write(s);
writer.Flush();
stream.Position = 0;
return writer;
}
}
The #Elliveny's code worked for me in locally but in azure throws an error : "The system cannot find the file specified". Due that I have changed a little bit the code and now works in both servers.
private string EncodeToken(string uid, Dictionary<string, object> claims)
{
string jwt = string.Empty;
RsaPrivateCrtKeyParameters _rsaParams;
using (StreamReader sr = new StreamReader(GenerateStreamFromString(private_key.Replace(#"\n", "\n"))))
{
var pr = new Org.BouncyCastle.OpenSsl.PemReader(sr);
_rsaParams = (RsaPrivateCrtKeyParameters)pr.ReadObject();
}
using (RSACryptoServiceProvider rsa = new RSACryptoServiceProvider())
{
Dictionary<string, object> payload = new Dictionary<string, object> {
{"claims", claims}
,{"uid", uid}
,{"iat", secondsSinceEpoch(DateTime.UtcNow)}
,{"exp", secondsSinceEpoch(DateTime.UtcNow.AddSeconds(firebaseTokenExpirySecs))}
,{"aud", firebasePayloadAUD}
,{"iss", client_email}
,{"sub", client_email}
};
RSAParameters rsaParams = DotNetUtilities.ToRSAParameters(_rsaParams);
rsa.ImportParameters(rsaParams);
jwt = JWT.Encode(payload, rsa, Jose.JwsAlgorithm.RS256);
}
return jwt;
}
Can someone please provide an example of how to use Google.Apis.Storage.v1 for uploading files to google cloud storage in c#?
I found that this basic operation is not as straight forward as you might expect. Google's documentation about it's Storage API is lacking in information about using it in C# (or any other .NET language). Searching for 'how to upload file to google cloud storage in c#' didn't exactly help me, so here is my working solution with some comments:
Preparation:
You need to create OAuth2 account in your Google Developers Console - go to Project/APIs & auth/Credentials.
Copy Client ID & Client Secret to your code. You will also need your Project name.
Code (it assumes that you've added Google.Apis.Storage.v1 via NuGet):
First, you need to authorize your requests:
var clientSecrets = new ClientSecrets();
clientSecrets.ClientId = clientId;
clientSecrets.ClientSecret = clientSecret;
//there are different scopes, which you can find here https://cloud.google.com/storage/docs/authentication
var scopes = new[] {#"https://www.googleapis.com/auth/devstorage.full_control"};
var cts = new CancellationTokenSource();
var userCredential = await GoogleWebAuthorizationBroker.AuthorizeAsync(clientSecrets,scopes, "yourGoogle#email", cts.Token);
Sometimes you might also want to refresh authorization token via:
await userCredential.RefreshTokenAsync(cts.Token);
You also need to create Storage Service:
var service = new Google.Apis.Storage.v1.StorageService();
Now you can make requests to Google Storage API.
Let's start with creating a new bucket:
var newBucket = new Google.Apis.Storage.v1.Data.Bucket()
{
Name = "your-bucket-name-1"
};
var newBucketQuery = service.Buckets.Insert(newBucket, projectName);
newBucketQuery.OauthToken = userCredential.Result.Token.AccessToken;
//you probably want to wrap this into try..catch block
newBucketQuery.Execute();
And it's done. Now, you can send a request to get list of all of your buckets:
var bucketsQuery = service.Buckets.List(projectName);
bucketsQuery.OauthToken = userCredential.Result.Token.AccessToken;
var buckets = bucketsQuery.Execute();
Last part is uploading new file:
//enter bucket name to which you want to upload file
var bucketToUpload = buckets.Items.FirstOrDefault().Name;
var newObject = new Object()
{
Bucket = bucketToUpload,
Name = "some-file-"+new Random().Next(1,666)
};
FileStream fileStream = null;
try
{
var dir = Directory.GetCurrentDirectory();
var path = Path.Combine(dir, "test.png");
fileStream = new FileStream(path, FileMode.Open);
var uploadRequest = new Google.Apis.Storage.v1.ObjectsResource.InsertMediaUpload(service, newObject,
bucketToUpload,fileStream,"image/png");
uploadRequest.OauthToken = userCredential.Result.Token.AccessToken;
await uploadRequest.UploadAsync();
}
catch (Exception ex)
{
Console.WriteLine(ex.Message);
}
finally
{
if (fileStream != null)
{
fileStream.Dispose();
}
}
And bam! New file will be visible in you Google Developers Console inside of selected bucket.
You can use Google Cloud APIs without SDK in the following ways:
Required api-key.json file
Install package Google.Apis.Auth.OAuth2 in order to authorize the
HTTP web request
You can set the default configuration for your application in this
way
I did the same using .NET core web API and details are given below:
Url details:
"GoogleCloudStorageBaseUrl": "https://www.googleapis.com/upload/storage/v1/b/",
"GoogleSpeechBaseUrl": "https://speech.googleapis.com/v1/operations/",
"GoogleLongRunningRecognizeBaseUrl": "https://speech.googleapis.com/v1/speech:longrunningrecognize",
"GoogleCloudScope": "https://www.googleapis.com/auth/cloud-platform",
public void GetConfiguration()
{
// Set global configuration
bucketName = _configuration.GetValue<string>("BucketName");
googleCloudStorageBaseUrl = _configuration.GetValue<string>("GoogleCloudStorageBaseUrl");
googleSpeechBaseUrl = _configuration.GetValue<string>("GoogleSpeechBaseUrl");
googleLongRunningRecognizeBaseUrl = _configuration.GetValue<string>("GoogleLongRunningRecognizeBaseUrl");
// Set google cloud credentials
string googleApplicationCredentialsPath = _configuration.GetValue<string>("GoogleCloudCredentialPath");
using (Stream stream = new FileStream(googleApplicationCredentialsPath, FileMode.Open, FileAccess.Read))
googleCredential = GoogleCredential.FromStream(stream).CreateScoped(_configuration.GetValue<string>("GoogleCloudScope"));
}
Get Oauth token:
public string GetOAuthToken()
{
return googleCredential.UnderlyingCredential.GetAccessTokenForRequestAsync("https://accounts.google.com/o/oauth2/v2/auth", CancellationToken.None).Result;
}
To upload file to cloud bucket:
public async Task<string> UploadMediaToCloud(string filePath, string objectName = null)
{
string bearerToken = GetOAuthToken();
byte[] fileBytes = File.ReadAllBytes(filePath);
objectName = objectName ?? Path.GetFileName(filePath);
var baseUrl = new Uri(string.Format(googleCloudStorageBaseUrl + "" + bucketName + "/o?uploadType=media&name=" + objectName + ""));
using (WebClient client = new WebClient())
{
client.Headers.Add(HttpRequestHeader.Authorization, "Bearer " + bearerToken);
client.Headers.Add(HttpRequestHeader.ContentType, "application/octet-stream");
byte[] response = await Task.Run(() => client.UploadData(baseUrl, "POST", fileBytes));
string responseInString = Encoding.UTF8.GetString(response);
return responseInString;
}
}
In order to perform any action to the cloud API, just need to make a HttpClient get/post request as per the requirement.
Thanks
This is for Google.Cloud.Storage.V1 (not Google.Apis.Storage.v1), but appears to be a bit simpler to perform an upload now. I started with the Client libraries "Getting Started" instructions to create a service account and bucket, then experimented to find out how to upload an image.
The process I followed was:
Sign up for Google Cloud free trial
Create a new project in Google Cloud (remember the project name\ID for later)
Create a Project Owner service account - this will result in a json file being downloaded that contains the service account credentials. Remember where you put that file.
The getting started docs get you to add the path to the JSON credentials file into an environment variable called GOOGLE_APPLICATION_CREDENTIALS - I couldn't get this to work through the provided instructions. Turns out it is not required, as you can just read the JSON file into a string and pass it to the client constructor.
I created an empty WPF project as a starting point, and a single ViewModel to house the application logic.
Install the Google.Cloud.Storage.V1 nuget package and it should pull in all the dependencies it needs.
Onto the code.
MainWindow.xaml
<StackPanel>
<Button
Margin="50"
Height="50"
Content="BEGIN UPLOAD"
Click="OnButtonClick" />
<ContentControl
Content="{Binding Path=ProgressBar}" />
</StackPanel>
MainWindow.xaml.cs
public partial class MainWindow
{
readonly ViewModel _viewModel;
public MainWindow()
{
_viewModel = new ViewModel(Dispatcher);
DataContext = _viewModel;
InitializeComponent();
}
void OnButtonClick(object sender, RoutedEventArgs args)
{
_viewModel.UploadAsync().ConfigureAwait(false);
}
}
ViewModel.cs
public class ViewModel
{
readonly Dispatcher _dispatcher;
public ViewModel(Dispatcher dispatcher)
{
_dispatcher = dispatcher;
ProgressBar = new ProgressBar {Height=30};
}
public async Task UploadAsync()
{
// Google Cloud Platform project ID.
const string projectId = "project-id-goes-here";
// The name for the new bucket.
const string bucketName = projectId + "-test-bucket";
// Path to the file to upload
const string filePath = #"C:\path\to\image.jpg";
var newObject = new Google.Apis.Storage.v1.Data.Object
{
Bucket = bucketName,
Name = System.IO.Path.GetFileNameWithoutExtension(filePath),
ContentType = "image/jpeg"
};
// read the JSON credential file saved when you created the service account
var credential = Google.Apis.Auth.OAuth2.GoogleCredential.FromJson(System.IO.File.ReadAllText(
#"c:\path\to\service-account-credentials.json"));
// Instantiates a client.
using (var storageClient = Google.Cloud.Storage.V1.StorageClient.Create(credential))
{
try
{
// Creates the new bucket. Only required the first time.
// You can also create buckets through the GCP cloud console web interface
storageClient.CreateBucket(projectId, bucketName);
System.Windows.MessageBox.Show($"Bucket {bucketName} created.");
// Open the image file filestream
using (var fileStream = new System.IO.FileStream(filePath, System.IO.FileMode.Open))
{
ProgressBar.Maximum = fileStream.Length;
// set minimum chunksize just to see progress updating
var uploadObjectOptions = new Google.Cloud.Storage.V1.UploadObjectOptions
{
ChunkSize = Google.Cloud.Storage.V1.UploadObjectOptions.MinimumChunkSize
};
// Hook up the progress callback
var progressReporter = new Progress<Google.Apis.Upload.IUploadProgress>(OnUploadProgress);
await storageClient.UploadObjectAsync(
newObject,
fileStream,
uploadObjectOptions,
progress: progressReporter)
.ConfigureAwait(false);
}
}
catch (Google.GoogleApiException e)
when (e.Error.Code == 409)
{
// When creating the bucket - The bucket already exists. That's fine.
System.Windows.MessageBox.Show(e.Error.Message);
}
catch (Exception e)
{
// other exception
System.Windows.MessageBox.Show(e.Message);
}
}
}
// Called when progress updates
void OnUploadProgress(Google.Apis.Upload.IUploadProgress progress)
{
switch (progress.Status)
{
case Google.Apis.Upload.UploadStatus.Starting:
ProgressBar.Minimum = 0;
ProgressBar.Value = 0;
break;
case Google.Apis.Upload.UploadStatus.Completed:
ProgressBar.Value = ProgressBar.Maximum;
System.Windows.MessageBox.Show("Upload completed");
break;
case Google.Apis.Upload.UploadStatus.Uploading:
UpdateProgressBar(progress.BytesSent);
break;
case Google.Apis.Upload.UploadStatus.Failed:
System.Windows.MessageBox.Show("Upload failed"
+ Environment.NewLine
+ progress.Exception);
break;
}
}
void UpdateProgressBar(long value)
{
_dispatcher.Invoke(() => { ProgressBar.Value = value; });
}
// probably better to expose progress value directly and bind to
// a ProgressBar in the XAML
public ProgressBar ProgressBar { get; }
}
Use of Google.Apis.Storage.v1 for uploading files using SDK to google cloud storage in c#:
Required api-key.json file
Install the package Google.Cloud.Storage.V1; and Google.Apis.Auth.OAuth2;
The code is given below to upload the file to the cloud
private string UploadFile(string localPath, string objectName = null)
{
string projectId = ((Google.Apis.Auth.OAuth2.ServiceAccountCredential)googleCredential.UnderlyingCredential).ProjectId;
try
{
// Creates the new bucket.
var objResult = storageClient.CreateBucket(projectId, bucketName);
if (!string.IsNullOrEmpty(objResult.Id))
{
// Upload file to google cloud server
using (var f = File.OpenRead(localPath))
{
objectName = objectName ?? Path.GetFileName(localPath);
var objFileUploadStatus1 = storageClient.UploadObject(bucketName, objectName, null, f);
}
}
}
catch (Google.GoogleApiException ex)
{
// Error code =409, means bucket already created/exist then upload file in the bucket
if (ex.Error.Code == 409)
{
// Upload file to google cloud server
using (var f = File.OpenRead(localPath))
{
objectName = objectName ?? Path.GetFileName(localPath);
var objFileUploadStatus2 = storageClient.UploadObject(bucketName, objectName, null, f);
}
}
}
catch (Exception ex)
{
MessageBox.Show(ex.ToString());
}
return objectName;
}
To set the credentials
private bool SetStorageCredentials()
{
bool status = true;
try
{
if (File.Exists(credential_path))
{
Environment.SetEnvironmentVariable("GOOGLE_APPLICATION_CREDENTIALS", credential_path);
using (Stream objStream = new FileStream(Environment.GetEnvironmentVariable("GOOGLE_APPLICATION_CREDENTIALS"), FileMode.Open, FileAccess.Read))
googleCredential = GoogleCredential.FromStream(objStream);
// Instantiates a client.
storageClient = StorageClient.Create();
channel = new Grpc.Core.Channel(SpeechClient.DefaultEndpoint.Host, googleCredential.ToChannelCredentials());
}
else
{
DialogResult result = MessageBox.Show("File " + Path.GetFileName(credential_path) + " does not exist. Please provide the correct path.");
if (result == System.Windows.Forms.DialogResult.OK)
{
status = false;
}
}
}
catch (Exception ex)
{
MessageBox.Show(ex.ToString());
status = false;
}
return status;
}
I used SDK in one of my window application. You can use the same code according to your needs/requirements.
You'll be happy to know it still works in 2016...
I was googling all over using fancy key words like "google gcp C# upload image", until I just plain asked the question: "How do I upload an image to google bucket using C#"... and here I am. I removed the .Result in the user credential, and this was the final edit that worked for me.
// ******
static string bucketForImage = ConfigurationManager.AppSettings["testStorageName"];
static string projectName = ConfigurationManager.AppSettings["GCPProjectName"];
string gcpPath = Path.Combine(Server.MapPath("~/Images/Gallery/"), uniqueGcpName + ext);
var clientSecrets = new ClientSecrets();
clientSecrets.ClientId = ConfigurationManager.AppSettings["GCPClientID"];
clientSecrets.ClientSecret = ConfigurationManager.AppSettings["GCPClientSc"];
var scopes = new[] { #"https://www.googleapis.com/auth/devstorage.full_control" };
var cts = new CancellationTokenSource();
var userCredential = await GoogleWebAuthorizationBroker.AuthorizeAsync(clientSecrets, scopes, ConfigurationManager.AppSettings["GCPAccountEmail"], cts.Token);
var service = new Google.Apis.Storage.v1.StorageService();
var bucketToUpload = bucketForImage;
var newObject = new Google.Apis.Storage.v1.Data.Object()
{
Bucket = bucketToUpload,
Name = bkFileName
};
FileStream fileStream = null;
try
{
fileStream = new FileStream(gcpPath, FileMode.Open);
var uploadRequest = new Google.Apis.Storage.v1.ObjectsResource.InsertMediaUpload(service, newObject,
bucketToUpload, fileStream, "image/"+ ext);
uploadRequest.OauthToken = userCredential.Token.AccessToken;
await uploadRequest.UploadAsync();
}
catch (Exception ex)
{
Console.WriteLine(ex.Message);
}
finally
{
if (fileStream != null)
{
fileStream.Dispose();
}
}
// ******
Here is the link to their official C# example of ".NET Bookshelf App" using Google Cloud storage.
https://cloud.google.com/dotnet/docs/getting-started/using-cloud-storage
Source on github:
https://github.com/GoogleCloudPlatform/getting-started-dotnet/blob/master/aspnet/3-binary-data/Services/ImageUploader.cs
https://github.com/GoogleCloudPlatform/getting-started-dotnet/tree/master/aspnet/3-binary-data
Nuget
https://www.nuget.org/packages/Google.Cloud.Storage.V1/
Here are 2 examples that helped me to upload files to a bucket in Google Cloud Storage with Google.Cloud.Storage.V1 (not Google.Apis.Storage.v1):
Upload files to Google cloud storage using c#
Uploading .csv Files to Google Cloud Storage using C# .Net
I got both working on a C# Console Application just for testing purposes.
#February 2021
string _projectId = "YOUR-PROJECT-ID-GCP"; //ProjectID also present in the json file
GoogleCredential _credential = GoogleCredential.FromFile("credential-cloud-file-123418c9e06c.json");
/// <summary>
/// UploadFile to GCS Bucket
/// </summary>
/// <param name="bucketName"></param>
/// <param name="localPath">my-local-path/my-file-name</param>
/// <param name="objectName">my-file-name</param>
public void UploadFile(string bucketName, string localPath, string objectName)
{
var storage = StorageClient.Create(_credential);
using var fileStream = File.OpenRead(localPath);
storage.UploadObject(bucketName, objectName, null, fileStream);
Console.WriteLine($"Uploaded {objectName}.");
}
You get the credentials JSON file from the google cloud portal where you create a bucket under your project..
Simple, with auth:
private void SaveFileToGoogleStorage(string path, string? fileName, string ext)
{
var filePath = Path.Combine(path, fileName + ext);
var gcCredentialsPath = Path.Combine(Environment.CurrentDirectory, "gc_sa_key.json");
Environment.SetEnvironmentVariable("GOOGLE_APPLICATION_CREDENTIALS", gcCredentialsPath);
var gcsStorage = StorageClient.Create();
using var f = File.OpenRead(filePath);
var objectName = Path.GetFileName(filePath);
gcsStorage.UploadObject(_bucketName, objectName, null, f);
Console.WriteLine($"Uploaded {objectName}.");
}
I'm trying to set up a WCF service protected by ADFS. I'm currently able to request a token and send it with the request using WIF and Thinktecture IdentityModel 4.5 with the following code:
static SecurityToken GetToken()
{
var factory = new WSTrustChannelFactory(
new UserNameWSTrustBinding(SecurityMode.TransportWithMessageCredential),
"https://fs2.server2012.local/adfs/services/trust/13/usernamemixed")
{
TrustVersion = TrustVersion.WSTrust13
};
if (factory.Credentials != null)
{
factory.Credentials.UserName.UserName = #"username";
factory.Credentials.UserName.Password = "password";
}
var rst = new RequestSecurityToken
{
RequestType = RequestTypes.Issue,
KeyType = KeyTypes.Symmetric,
AppliesTo = new EndpointReference(
"https://wcfservicecertificate/wcfservice/Service.svc/wstrust"),
};
var channel = factory.CreateChannel();
RequestSecurityTokenResponse rstr;
return channel.Issue(rst, out rstr);
}
With this I can call the WCF service by using ChannelFactory.CreateChannelWithIssuedToken:
var factory = new ChannelFactory<IService>(binding,
new EndpointAddress("https://wcfservicecertificate/wcfservice/Service.svc/wstrust"));
if (factory.Credentials != null)
{
factory.Credentials.SupportInteractive = false;
factory.Credentials.UseIdentityConfiguration = true;
}
var proxy = factory.CreateChannelWithIssuedToken(GetToken());
var result= proxy.GetData(2);
This works as expected but can only be used on (mobile) windows platforms. I would also like to be able to use the same principle on iOS and Android. Using this article I was able to request a security token from ADFS using the following code:
const string soapMessage =
#"<s:Envelope xmlns:s=""http://www.w3.org/2003/05/soap-envelope""
xmlns:a=""http://www.w3.org/2005/08/addressing""
xmlns:u=""http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd"">
<s:Header>
<a:Action s:mustUnderstand=""1"">http://docs.oasis-open.org/ws-sx/ws-trust/200512/RST/Issue</a:Action>
<a:To s:mustUnderstand=""1"">https://fs2.server2012.local/adfs/services/trust/13/UsernameMixed</a:To>
<o:Security s:mustUnderstand=""1"" xmlns:o=""http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd"">
<o:UsernameToken u:Id=""uuid-6a13a244-dac6-42c1-84c5-cbb345b0c4c4-1"">
<o:Username>username</o:Username>
<o:Password Type=""http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-username-token-profile-1.0#PasswordText"">password</o:Password>
</o:UsernameToken>
</o:Security>
</s:Header>
<s:Body>
<trust:RequestSecurityToken xmlns:trust=""http://docs.oasis-open.org/ws-sx/ws-trust/200512"">
<wsp:AppliesTo xmlns:wsp=""http://schemas.xmlsoap.org/ws/2004/09/policy"">
<a:EndpointReference>
<a:Address>https://wcfservicecertificate/wcfservice/Service.svc/wstrust</a:Address>
</a:EndpointReference>
</wsp:AppliesTo>
<trust:KeyType>http://docs.oasis-open.org/ws-sx/ws-trust/200512/SymmetricKey</trust:KeyType>
<trust:RequestType>http://docs.oasis-open.org/ws-sx/ws-trust/200512/Issue</trust:RequestType>
<trust:TokenType>urn:oasis:names:tc:SAML:2.0:assertion</trust:TokenType>
</trust:RequestSecurityToken>
</s:Body>
</s:Envelope>";
var webClient = new WebClient();
webClient.Headers.Add("Content-Type", "application/soap+xml; charset=utf-8");
var result = webClient.UploadString(
address: "https://fs2.server2012.local/adfs/services/trust/13/UsernameMixed",
method: "POST",
data: soapMessage);
This results in a SAML2.0 token which I would like to send in a request to our WCF service in order to authenticate. There are various sources (including the article mentioned earlier) which state that this should be possible but I've yet to find a solution.
Any help would be appreciated.
You can use one of hybrid solutions which use SAML with OAuth or other authorization technologies. This is more secure against phising techniques. For SAML only approach, you can refer to following link: How to pass security tokenfrom one wcf service to another wcf service. It is said that you need to enable saveBootstrapTokens property on webconfig.
This link can be useful too: Availability of Bootstrap Tokens
This can easily be done without using WIF. Lets completely avoid WIF and the .Net framework and do it in Java for illustration purposes. First make a call to the Security Token Service using the template approach like you have done. You then need to extract the SAML from the response, Base64 encode it and stuff it in the Autorization header of the subsequent request to your protected WCF service. You may also need to do the same with a ProofKey if you are coding for Non-Repudiation. Also I'm only showing authentication using username/password for brevity as Certificate Authentication involves much more work - you have to hash (SHA1 )part of the message then encrypt the hash with the private key of the cert and then add this as a xml element to the original message etc...
Here is the java helper code:
import java.io.*;
import java.io.BufferedReader;
import java.io.DataOutputStream;
import java.io.InputStreamReader;
import java.net.HttpURLConnection;
import java.net.URL;
import java.time.Instant;
import java.util.Map;
import java.util.Properties;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
import java.util.Base64;
import javax.crypto.Mac;
import javax.crypto.spec.SecretKeySpec;
import javax.net.ssl.HttpsURLConnection;
public class SecurityService {
private String _username;
private String _password;
private String _stsUrl;
private String _samlAssertion;
private String _samlEncoded;
private String _binarySecret;
private String _workingDirectory;
private String _platformUrl;
private String _soapBody;
private Integer _responseCode;
private Integer _plaformResponseCode;
private String _response;
private String _platformResponse;
private String _xproofSignature;
private Map<String, String> _headerDictionary;
public void setUsername(String username) {
this._username = username;
}
public void setPassword(String password) {
this._password = password;
}
public void setStsUrl(String stsUrl) {
this._stsUrl = stsUrl;
}
public String getStsUrl() {
return _stsUrl;
}
public void setplatformUrl(String platformUrl) {
this._platformUrl = platformUrl;
}
public String getSamlAssertion() {
return _samlAssertion;
}
public String getSamlEncoded() {
return _samlEncoded;
}
public String getSoapBody() {
return _soapBody;
}
public Integer getResponseCode() {
return _responseCode;
}
public Integer getPlatformResponseCode() {
return _plaformResponseCode;
}
public String getResponse() {
return _response;
}
public String getPlatformResponse() {
return _platformResponse;
}
public String getXProofSignature() {
return _xproofSignature;
}
public String getBinarySecret() {
return _binarySecret;
}
public String gePlatFormUrl() {
return _platformUrl;
}
public void setHeaderDictionary(Map<String, String> headerDictionary){
this._headerDictionary = headerDictionary;
}
public Map<String, String> getHeaderDictionary(){
return _headerDictionary;
}
public SecurityService() throws Exception {
}
public SecurityService(Boolean useConfig) throws Exception {
if (useConfig) {
this._workingDirectory = System.getProperty("user.dir") + "\\app.config";
this.getProperties();
}
}
public void sendAuthenticatedGet() throws Exception {
URL obj = new URL(_platformUrl);
HttpURLConnection con = (HttpURLConnection) obj.openConnection();
// optional default is GET
con.setRequestMethod("GET");
// Add request header
con.setRequestProperty("Authorization", "Saml " + _samlEncoded);
con.setRequestProperty("X-ProofSignature", _xproofSignature);
_plaformResponseCode = con.getResponseCode();
BufferedReader in = new BufferedReader(new InputStreamReader(con.getInputStream()));
String inputLine;
StringBuffer response = new StringBuffer();
while ((inputLine = in.readLine()) != null) {
response.append(inputLine);
}
in.close();
_platformResponse = response.toString();
}
public void sendAuthenticatedPost(String body) throws Exception {
URL obj = new URL(_platformUrl);
HttpsURLConnection con = (HttpsURLConnection) obj.openConnection();
//add request header
con.setRequestMethod("POST");
con.setRequestProperty("Content-Type", "application/json");
// Add request header
con.setRequestProperty("Authorization", "Saml " + _samlEncoded);
con.setRequestProperty("X-ProofSignature", _xproofSignature);
// Add Azure Subscription Key using generic Add Headers method
if (_headerDictionary != null) {
for (String key : _headerDictionary.keySet()) {
con.setRequestProperty(key, _headerDictionary.get(key));
}
}
_soapBody = body;
// Send post request
con.setDoOutput(true);
DataOutputStream wr = new DataOutputStream(con.getOutputStream());
//wr.writeBytes(urlParameters);
wr.writeBytes(_soapBody);
wr.flush();
wr.close();
_responseCode = con.getResponseCode();
BufferedReader in = new BufferedReader(new InputStreamReader(con.getInputStream()));
String inputLine;
StringBuffer response = new StringBuffer();
while ((inputLine = in.readLine()) != null) {
response.append(inputLine);
}
in.close();
_response = response.toString();
}
// HTTP POST request
public void sendPostToSts() throws Exception {
URL obj = new URL(_stsUrl);
HttpsURLConnection con = (HttpsURLConnection) obj.openConnection();
//add request header
con.setRequestMethod("POST");
con.setRequestProperty("Content-Type", "application/soap+xml");
String body = getTemplateCertificate();
_soapBody = (((body.replace("[Created]", Instant.now().toString())).replace("[Expires]", Instant.now()
.plusSeconds(300).toString())).replace("[username]", _username)).replace("[password]", _password).replace("[stsUrl]", _stsUrl);
// Send post request
con.setDoOutput(true);
DataOutputStream wr = new DataOutputStream(con.getOutputStream());
//wr.writeBytes(urlParameters);
wr.writeBytes(_soapBody);
wr.flush();
wr.close();
_responseCode = con.getResponseCode();
BufferedReader in = new BufferedReader(new InputStreamReader(con.getInputStream()));
String inputLine;
StringBuffer response = new StringBuffer();
while ((inputLine = in.readLine()) != null) {
response.append(inputLine);
}
in.close();
_response = response.toString();
// Get Binary Secret
// <trust:BinarySecret></trust:BinarySecret>
final Pattern patternBinarySecret = Pattern.compile("<trust:BinarySecret>(.+?)</trust:BinarySecret>");
final Matcher matcherBinarySecret = patternBinarySecret.matcher(response.toString());
matcherBinarySecret.find();
_binarySecret = matcherBinarySecret.group(1);
// Get the SAML Assertion
final Pattern patternEncryptedAssertion = Pattern.compile("<trust:RequestedSecurityToken>(.+?)</trust:RequestedSecurityToken>");
final Matcher matcherEncryptedAssertion = patternEncryptedAssertion.matcher(response.toString());
matcherEncryptedAssertion.find();
_samlAssertion = matcherEncryptedAssertion.group(1);
byte[] proofKeyBytes = _binarySecret.getBytes("UTF-8");
String encoded = Base64.getEncoder().encodeToString(proofKeyBytes);
byte[] decoded = Base64.getDecoder().decode(encoded);
// SAML Stuff - Works beautifully
byte[] samlBytes = _samlAssertion.getBytes("UTF-8");
_samlEncoded = Base64.getEncoder().encodeToString(samlBytes);
_xproofSignature = this.encode(_samlAssertion, _binarySecret);
}
private static String readFile( String file ) throws IOException {
BufferedReader reader = new BufferedReader( new FileReader(file));
String line = null;
StringBuilder stringBuilder = new StringBuilder();
String ls = System.getProperty("line.separator");
try {
while( ( line = reader.readLine() ) != null ) {
stringBuilder.append( line );
stringBuilder.append( ls );
}
return stringBuilder.toString();
} finally {
reader.close();
}
}
// Embedded WS-Trust template for username/password RST
private static String getTemplate () {
return "<s:Envelope xmlns:s=\"http://www.w3.org/2003/05/soap-envelope\" xmlns:a=\"http://www.w3.org/2005/08/addressing\" xmlns:u= \"http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd\"><s:Header><a:Action s:mustUnderstand= \"1\">http://docs.oasis-open.org/ws-sx/ws-trust/200512/RST/Issue</a:Action><a:MessageID>urn:uuid:cfea5555-248c-46c3-9b4d- 54936b7f815c</a:MessageID><a:ReplyTo><a:Address>http://www.w3.org/2005/08/addressing/anonymous</a:Address></a:ReplyTo><a:To s:mustUnderstand=\"1\">[stsUrl]</a:To><o:Security s:mustUnderstand=\"1\" xmlns:o=\"http://docs.oasis- open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd\"><u:Timestamp u:Id=\"_0\"><u:Created>[Created] </u:Created><u:Expires>[Expires]</u:Expires></u:Timestamp><o:UsernameToken u:Id=\"uuid-e273c018-1da7-466e-8671-86f6bfe7ce3c- 17\"><o:Username>[username]</o:Username><o:Password Type=\"http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-username- token-profile-1.0#PasswordText\">[password] </o:Password></o:UsernameToken></o:Security></s:Header><s:Body><trust:RequestSecurityToken xmlns:trust=\"http://docs.oasis- open.org/ws-sx/ws-trust/200512\"><wsp:AppliesTo xmlns:wsp=\"http://schemas.xmlsoap.org/ws/2004/09/policy \"><wsa:EndpointReference xmlns:wsa=\"http://www.w3.org/2005/08/addressing \"><wsa:Address>https://mbplatform/</wsa:Address></wsa:EndpointReference></wsp:AppliesTo><trust:RequestType>http://docs.oasis- open.org/ws-sx/ws-trust/200512/Issue</trust:RequestType><trust:TokenType>http://docs.oasis-open.org/wss/oasis-wss-saml-token- profile-1.1#SAMLV2.0</trust:TokenType></trust:RequestSecurityToken></s:Body></s:Envelope>";
}
private String encode(String key, String data) throws Exception {
Mac sha256_HMAC = Mac.getInstance("HmacSHA256");
SecretKeySpec secret_key = new SecretKeySpec(key.getBytes("UTF-8"), "HmacSHA256");
sha256_HMAC.init(secret_key);
return Base64.getEncoder().encodeToString(sha256_HMAC.doFinal(data.getBytes("UTF-8")));
}
private void getProperties() throws Exception {
Properties prop = new Properties();
String fileName = _workingDirectory;
InputStream is = new FileInputStream(fileName);
prop.load(is);
_username = prop.getProperty("app.username");
_password = prop.getProperty("app.password");
_platformUrl = prop.getProperty("app.platformUrl");
_stsUrl = prop.getProperty("app.stsUrl");
}
}
and here is example usage:
SecurityService mbss = new SecurityService(true);
mbss.sendPostToSts();
System.out.println("CONTACTING AZURE SECURITY TOKEN SERVICE");
System.out.println("\nSending 'POST' request to URL : " + mbss.getStsUrl());
System.out.println("\nPost parameters : \n" + mbss.getSoapBody());
System.out.println("\nResponse Code : " + mbss.getResponseCode());
System.out.println("\nHERE IS THE SAML RESPONSE\n");
System.out.println(mbss.getResponse());
System.out.println("\nHERE IS THE BINARY SECRET\n");
System.out.println(mbss.getBinarySecret());
System.out.println("\nHERE IS THE SAML ASSERTION\n");
System.out.println(mbss.getSamlAssertion());
System.out.println("\nHERE IS THE ENCODED SAML ASSERTION\n");
System.out.println(mbss.getSamlEncoded());
System.out.println("\nHERE IS THE X-PROOF SIGNATURE\n");
System.out.println(mbss.getXProofSignature());
System.out.println("\nNOW CONTACTING WCF SERVICES WITH SECURITY HEADER\n");
mbss.sendAuthenticatedGet();
System.out.println("\nSending 'GET' request to URL : " + mbss.gePlatFormUrl());
System.out.println("Response Code : " + mbss.getPlatformResponseCode());
System.out.println("\nHERE ARE THE RESULTS FOLKS...ENJOY\n");
System.out.println(mbss.getPlatformResponse());
I am trying to connect with c#.
Here is the class that submits hive queries successfully to my remote HDInsight cluster. what do i need to change here to connect to the local emulator
public class HadoopImporter : IImporter
{
public static readonly Logger log = LogManager.GetCurrentClassLogger();
public void Import(string _query)
{
try
{
log.Warn("Inside Hive submission method");
var store = new X509Store();
store.Open(OpenFlags.ReadOnly);
var cert =
store.Certificates.Cast<X509Certificate2>()
.First(item => item.Thumbprint == "MYCERTTUMBPRINT");
if (cert == null)
log.Error("no cert found");
log.Warn(cert.FriendlyName);
log.Warn("got the cert with thumbprint ", cert.Thumbprint.ToString())
;
log.Warn("trying to create credentials from cert");
var creds = new JobSubmissionCertificateCredential(new Guid("MYSUBSCRIPTIONID"),
cert, "MYSTORAGECONTAINER");
log.Warn("trying to connect with cert");
var jobClient = JobSubmissionClientFactory.Connect(creds);
log.Warn("Setting Hive job parameters");
var hiveJob = new HiveJobCreateParameters()
{
Query = _query,
StatusFolder = "/samplequeryoutput"
};
var jobResults = jobClient.CreateHiveJob(hiveJob);
log.Warn("Executing wait for jhive results");
WaitForJobCompletion(jobResults, jobClient);
using (var stream = jobClient.GetJobOutput(jobResults.JobId))
{
var reader = new StreamReader(stream);
var res = reader.ReadToEnd();
log.Warn("trying to get the job results " + res.ToString());
}
}
catch (Exception exp)
{
log.Error(exp);
}
}
private static void WaitForJobCompletion(JobCreationResults jobDetails, IJobSubmissionClient client)
{
var jobInProgress = client.GetJob(jobDetails.JobId);
while (jobInProgress.StatusCode != JobStatusCode.Completed && jobInProgress.StatusCode != JobStatusCode.Failed)
{
log.Warn("Inside the while loop waiting for hive job to complete");
jobInProgress = client.GetJob(jobInProgress.JobId);
Thread.Sleep(TimeSpan.FromSeconds(10));
}
log.Trace("HIVE Job has Imported " + jobDetails.JobId);
}
}
You should be able to connect to a local one-box using the REST implementation of the client.
You're looking for the WebHCatHttpClient interface. The code below runs a basic query against my local one-box.
var httpClient = new WebHCatHttpClient(new Uri("http://localhost:50111/"), "username", "password");
string outputDir = "basichivejob";
var task = httpClient.CreateHiveJob(#"select * from iris;", null, null, outputDir, null);
task.Wait();
var response = task.Result;
var output = response.Content.ReadAsAsync<JObject>();
output.Wait();
response.EnsureSuccessStatusCode();
string id = output.Result.GetValue("id").ToString();
httpClient.WaitForJobToCompleteAsync(id).Wait();
See the SDK docs for more info.
I'm coding an application in c# using EC4 SP2 SDK.
I want to publish my file to a media server publishing point. I've searched and found 2 examples regarding seting up and auth on publishing points, but either are from older sdk's or do not work (and are for console). basicly my application doesn't encode nothing, as if it had nothing to encode.
When in degub mode checkpont i can see the correct properties for the source file and for the server.
The encoding process takes 0secs to process. I checked the logs on the server events and i get a warning "the security system has received and auth request that could not be decoded". I just havo no knowledge to break up further than this. Any help would be appreciated.
this is the piece of code:
private void broadcastSourceFileToMediaServer2()
{
using (LiveJob job = new LiveJob())
{
String filetoencode = #"c:\temp\niceday.wmv";
LiveFileSource filesource = job.AddFileSource(filetoencode);
filesource.PlaybackMode = FileSourcePlaybackMode.Loop;
job.ActivateSource(filesource);
job.ApplyPreset(LivePresets.VC1Broadband4x3);
//don't know which one is good to use
job.AcquireCredentials += new EventHandler<AcquireCredentialsEventArgs>(job_AcquireCredentials);
_myUserName = "indes";
_pw = PullPW("indes");
Uri url = new Uri("http://192.168.1.74:8080/live");
PushBroadcastPublishFormat pubpoint = new PushBroadcastPublishFormat();
pubpoint.PublishingPoint = url;
pubpoint.UserName = _myUserName;
pubpoint.Password = _pw;
job.PublishFormats.Add(pubpoint);
job.PreConnectPublishingPoint();
job.StartEncoding();
statusBox.Text = job.NumberOfEncodedSamples.ToString();
job.StopEncoding();
job.Dispose();
}
}
public static string _myUserName { get; set; }
public static SecureString _pw { get; set; }
//codificação de Password a enviar
private static SecureString PullPW(string pw)
{
SecureString s = new SecureString();
foreach (char c in pw) s.AppendChar(c);
return s;
}
static void job_AcquireCredentials(object sender, AcquireCredentialsEventArgs e)
{
e.UserName = _myUserName;
e.Password = _pw;
e.Modes = AcquireCredentialModes.None;
}
Progresses:
I managed to authenticate (at least get a positive audit event) on the server.
I changed from this:
//don't know which one is good to use
job.AcquireCredentials += new EventHandler<AcquireCredentialsEventArgs>(job_AcquireCredentials);
_myUserName = "indes";
_pw = PullPW("indes");
Uri url = new Uri("http://192.168.1.74:8080/live");
PushBroadcastPublishFormat pubpoint = new PushBroadcastPublishFormat();
pubpoint.PublishingPoint = url;
pubpoint.UserName = _myUserName;
pubpoint.Password = _pw;
To this:
job.AcquireCredentials += new EventHandler<AcquireCredentialsEventArgs>(job_AcquireCredentials);
_myUserName = #"mediaservername\user";
_pw = PullPW("user_password");
Uri url = new Uri("http://192.168.1.74:8080/live");
PushBroadcastPublishFormat pubpoint = new PushBroadcastPublishFormat();
pubpoint.PublishingPoint = url;
If you see on one side if had to include the domain (either domain or computername) before username. this changed the failed audit events on the server, so i could eliminate the manual credentials pubpoint.username and pubpoint.Password.
Now I'm just dealing with a lack of output format exception. On to it.
How about using SMOOTH Streaming, I managed to get my project going but I didn't get much more beyond Look below, to the part that has the PUBLISH switch type. ignore the file portion
internal bool StartStream()
{
Busy = true;
// Instantiates a new job for encoding
//
//***************************************Live Stream Archive******************************
if (blnRecordFromFile)
{
// Sets up publishing format for file archival type
FileArchivePublishFormat fileOut = new FileArchivePublishFormat();
// job.ApplyPreset(LivePresets.VC1512kDSL16x9);
// Gets timestamp and edits it for filename
string timeStamp = DateTime.Now.ToString();
timeStamp = timeStamp.Replace("/", "-");
timeStamp = timeStamp.Replace(":", ".");
// Sets file path and name
string path = "C:\\output\\";
string filename = "Capture" + timeStamp + ".ismv";
if (!Directory.Exists(path))
Directory.CreateDirectory(path);
fileOut.OutputFileName = Path.Combine(path, filename);
// Adds the format to the job. You can add additional formats as well such as
// Publishing streams or broadcasting from a port
job.PublishFormats.Add(fileOut);
}
//******************************END OF Stream PORTION****************************************
////////////////////////////////////////////////////////////////////////////////////////////////////
//*************************************** Process Files or Live Stream******************************
if (blnRecordFromFile)
{
job.ApplyPreset(LivePresets.VC1IISSmoothStreaming720pWidescreen);
job = new LiveJob();
// Verifies all information is entered
if (string.IsNullOrWhiteSpace(sourcePath) || string.IsNullOrWhiteSpace(destinationPath))
return false;
job.Status += new EventHandler<EncodeStatusEventArgs>(StreamStatus);
LiveFileSource fileSource;
try
{
// Sets file to active source and checks if it is valid
fileSource = job.AddFileSource(sourcePath);
}
catch (InvalidMediaFileException)
{
return false;
}
// Sets to loop media for streaming
// fileSource.PlaybackMode = FileSourcePlaybackMode.Loop;
// Makes this file the active source. Multiple files can be added
// and cued to move to each other at their ends
job.ActivateSource(fileSource);
}
//******************************END OF FILE PORTION****************************************
// Sets up variable for fomat data
switch (publishType)
{
case Output.Archive:
// Verifies destination path exists and if not creates it
try
{
if (!Directory.Exists(destinationPath))
Directory.CreateDirectory(destinationPath);
}
catch (IOException)
{
return false;
}
FileArchivePublishFormat archiveFormat = new FileArchivePublishFormat();
// Gets the location of the old extention and removes it
string filename = Path.GetFileNameWithoutExtension(sourcePath);
// Sets the archive path and file name
archiveFormat.OutputFileName = Path.Combine(destinationPath, filename + ".ismv");
job.PublishFormats.Add(archiveFormat);
break;
case Output.Publish:
// Setups streaming of media to publishing point
job = new LiveJob();
// Aquires audio and video devices
Collection<EncoderDevice> devices = EncoderDevices.FindDevices(EncoderDeviceType.Video);
EncoderDevice video = devices.Count > 0 ? devices[0] : null;
for (int i = 0; i < devices.Count; ++i)
// devices[i].Dispose();
devices.Clear();
devices = EncoderDevices.FindDevices(EncoderDeviceType.Audio);
EncoderDevice audio = devices.Count > 0 ? devices[0] : null;
for (int i = 1; i < devices.Count; ++i)
devices[i].Dispose();
devices.Clear();
// Checks for a/v devices
if (video != null && audio != null)
{
//job.ApplyPreset(Preset.FromFile(#"C:\Tempura\LivePreset3.xml"));
job.ApplyPreset(LivePresets.H264IISSmoothStreamingLowBandwidthStandard);
job.OutputFormat.VideoProfile.SmoothStreaming = true;
deviceSource = job.AddDeviceSource(video, audio);
// Make this source the active one
job.ActivateSource(deviceSource);
}
else
{
error = true;
}
PushBroadcastPublishFormat publishFormat = new PushBroadcastPublishFormat();
try
{
// checks the path for a valid publishing point
publishFormat.PublishingPoint = new Uri(destinationPath);
}
catch (UriFormatException)
{
return false;
}
// Adds the publishing format to the job
try
{
// job.ApplyPreset(LivePresets.VC1IISSmoothStreaming480pWidescreen);
job.PublishFormats.Add(publishFormat);
job.PreConnectPublishingPoint();
}
catch (Exception e)
{
MessageBox.Show(e.StackTrace.ToString());
}
break;
default:
return false;
}
job.StartEncoding();
return true;
}
Sadly I dont have enough rep to comment, so I have to write it as an answer.
Due to you are starting a live job, in order to stream you should not call job.StopEncoding() right after StartEncoding. I think usually you would use an event to stop the encoding. If you start encoding and immediately stop it, it is only logical you have no, or only a very small output.
I changed your code to the following and it seems work well. I guess your problem is that you disposed the instance of LiveJob class. You have to keep the instance alive before it finished encoding the whole stream. So change the using part and remove the StopEncoding and Dispose will be OK.
private void broadcastSourceFileToMediaServer2()
{
LiveJob job = new LiveJob();
String filetoencode = #"c:\temp\niceday.wmv";
LiveFileSource filesource = job.AddFileSource(filetoencode);
filesource.PlaybackMode = FileSourcePlaybackMode.Loop;
job.ActivateSource(filesource);
job.ApplyPreset(LivePresets.VC1Broadband4x3);
//don't know which one is good to use
job.AcquireCredentials += new EventHandler<AcquireCredentialsEventArgs>(job_AcquireCredentials);
_myUserName = "indes";
_pw = PullPW("indes");
Uri url = new Uri("http://192.168.1.74:8080/live");
PushBroadcastPublishFormat pubpoint = new PushBroadcastPublishFormat();
pubpoint.PublishingPoint = url;
pubpoint.UserName = _myUserName;
pubpoint.Password = _pw;
job.PublishFormats.Add(pubpoint);
job.PreConnectPublishingPoint();
job.StartEncoding();
statusBox.Text = job.NumberOfEncodedSamples.ToString();
}
public static string _myUserName { get; set; }
public static SecureString _pw { get; set; }
//codificação de Password a enviar
private static SecureString PullPW(string pw)
{
SecureString s = new SecureString();
foreach (char c in pw) s.AppendChar(c);
return s;
}
static void job_AcquireCredentials(object sender, AcquireCredentialsEventArgs e)
{
e.UserName = _myUserName;
e.Password = _pw;
e.Modes = AcquireCredentialModes.None;
}