We Keep Coding

Upload file to Contabo Object Storage (S3 compatible)

I'm trying to upload a file to the contabo object storage from memory stream using AWSSDK.S3
This is my client configuration.
string accessKey = "xxxxxxxxxxxxxxxxxxxxxxxxxx";
string secretKey = "xxxxxxxxxxxxxxxxxxxxxxxxxx";
Amazon.S3.AmazonS3Config config = new Amazon.S3.AmazonS3Config();
Amazon.S3.AmazonS3Client s3Client;
public BookingsController()
{
config.ServiceURL = "https://eu2.contabostorage.com";
config.DisableHostPrefixInjection = true;
s3Client = new Amazon.S3.AmazonS3Client(
accessKey,
secretKey,
config
);
}
This is the method I'm using:
[HttpPost("/api/Bookings/AddFile")]
public async Task<ActionResult> AddBookingFile([FromForm] IFormFile file)
{
using (var newMemoryStream = new MemoryStream())
{
ListBucketsResponse response = await s3Client.ListBucketsAsync();
file.CopyTo(newMemoryStream);
Amazon.S3.Model.PutObjectRequest request = new Amazon.S3.Model.PutObjectRequest();
request.BucketName = "test-bucket";
request.Key = "recording.wav";
request.ContentType = "audio/wav";
request.InputStream = newMemoryStream;
await s3Client.PutObjectAsync(request);
}
return Ok();
}
The ListBucket-Method works properly. The PutObject Method throws the exception that the Host "Der angegebene Host ist unbekannt. (test-bucket.eu2.contabostorage.com:443)" cannot be found.
Referenced to the Contabo-docs is that correct, because Contabo doesn't support virtual hosted buckets (dns prefix). Reference to contabo docs
I thought, that the following configuration fix this, but that wasn't the solution.
config.DisableHostPrefixInjection = true;
Does anyone has any advise hot to prevent the prefixing of the url?

Get X509 Certificate WITH PRIVATE KEY from Azure Keyvault c#

I am currently working on an authentication server developed in C #, this one is hosted on an azure function app, and I use a KeyVault where my secrets are stored. My problem is the following, in my keyvault, I store a certificate (certificate + private key)
and when I retrieve it in my code, but the private key is not returned. if I test the following method: HasPrivateKey the code returns false ...
but if i use the same .pfx in localy the code return me true ...
my code:
var client = new CertificateClient(vaultUri: new Uri("https://diiage2p1g3chest.vault.azure.net/"),credential: new DefaultAzureCredential());
KeyVaultCertificate kcertificate = client.GetCertificate("try");
var cert_content = kcertificate.Cer;
X509Certificate2 certificate = new X509Certificate2(cert_content, "password", X509KeyStorageFlags.EphemeralKeySet);
any idea where the problem comes from?

CertificateClient has a method that returns a certificate with private key, but it's not obvious that's what it does.
From CertificateClient.DownloadCertificate:
Because Cer contains only the public key, this method attempts to download the managed secret that contains the full certificate. If you do not have permissions to get the secret, RequestFailedException will be thrown with an appropriate error response. If you want an X509Certificate2 with only the public key, instantiate it passing only the Cer property. This operation requires the certificates/get and secrets/get permissions.
So just refactor your code to use DownloadCertificate to get a cert with the private key.
var client = new CertificateClient(new Uri("https://diiage2p1g3chest.vault.azure.net/"), new DefaultAzureCredential());
X509Certificate2 certificate = client.DownloadCertificate("try");

The simplest way to get the full bytes of a certificate with its private information from keyvault is this. Please note you need permission to get secrets in your client id.
You need the following packages:
Azure.Identity
Azure.Security.KeyVault.Secrets
The following are deprecated:
Microsoft.IdentityModel.Clients.ActiveDirectory
Microsoft.Azure.KeyVault
Code:
using System;
using System.Threading.Tasks;
using Azure;
using Azure.Identity;
using System.Security.Cryptography.X509Certificates;
using Azure.Security.KeyVault.Secrets;
...
public async Task<X509Certificate2> GetCertificate(string certificateName,string clientId, string clientSecret, string keyVaultAddress, string tenantId)
{
ClientSecretCredential clientCredential = new ClientSecretCredential(tenantId, clientId, clientSecret);
var secretClient = new SecretClient(new Uri(keyVaultAddress), clientCredential);
var response = await secretClient.GetSecretAsync(certificateName);
var keyVaultSecret = response?.Value;
if(keyVaultSecret != null)
{
var privateKeyBytes = Convert.FromBase64String(keyVaultSecret.Value);
return new X509Certificate2(privateKeyBytes);
}
return null;
}

var _keyVaultName = $"VAULTURL";
var secretName = "CERTIFICATENAME";
var azureServiceTokenProvider = new AzureServiceTokenProvider();
var _client = new KeyVaultClient(new KeyVaultClient.AuthenticationCallback(azureServiceTokenProvider.KeyVaultTokenCallback));
var secret = _client.GetSecretAsync(_keyVaultName, secretName);
var privateKeyBytes = Convert.FromBase64String(secret.Result.Value);
certificate = new X509Certificate2(privateKeyBytes, string.Empty);
i solve my probleme like that :)

X509Certificate2 certificate;
ClientSecretCredential clientCredential = new ClientSecretCredential(TenantId,ClientId,ClientSecret);
var secretClient = new SecretClient(new Uri(KeyVaultUrl), clientCredential);
var response = await secretClient.GetSecretAsync(individualEnrollment.DeviceId.Replace("-", ""));
var keyVaultSecret = response?.Value;
var privateKeyBytes = Convert.FromBase64String(keyVaultSecret.Value);
certificate = new X509Certificate2(privateKeyBytes);
using (var security = new SecurityProviderX509Certificate(certificate))
using (var transport = new ProvisioningTransportHandlerAmqp(TransportFallbackType.TcpOnly))
{
ProvisioningDeviceClient provClient =
ProvisioningDeviceClient.Create(GlobalDeviceEndpoint, Scope, security, transport);
var deviceClient = await ProvisionX509Device(provClient, security);
}

How do you download object from AWS S3 bucket using AmazonS3EncryptionClientV2 client side encryption?

We were using AmazonS3EncryptionClient in our code to interact with S3 bucket using client side encryption. But on updating nuget package today, I noticed that AmazonS3EncryptionClient has been marked obsolete. Looks like we will need to use AmazonS3EncryptionClientV2 if we want to get continuous update going forward. I am having this issue while trying to migrate from AmazonS3EncryptionClient to AmazonS3EncryptionClientV2.
In our old code we were using AmazonS3EncryptionClient constructor that takes RegionEnpoint as a parameter. see image below. Looks like constructors that takes RegionEnpoint has been removed in AmazonS3EncryptionClientV2.
Old code that was working to GetObject from S3 bucket.
S3BucketConfiguration _s3BucketConfiguration = provider
.GetService<IOptionsSnapshot<S3BucketConfiguration>>()
.Value;
var credential = new BasicAWSCredentials(
_s3BucketConfiguration.AccessKey, _s3BucketConfiguration.SecurityKey);
RegionEndpoint bucketRegion =
RegionEndpoint.GetBySystemName(_s3BucketConfiguration.Region);
EncryptionMaterials encryptionMaterials = new EncryptionMaterials(_s3BucketConfiguration.KMSKeyId);
var client = new AmazonS3EncryptionClient(credential, bucketRegion, encryptionMaterials);
GetObjectResponse response = await _client.GetObjectAsync(new GetObjectRequest
{
BucketName = _s3BucketConfig.BucketName,
Key = filePath
});
I cannot pass in RegionEnpoint in AmazonS3EncryptionClientV2.
My Code so far.
S3BucketConfiguration _s3BucketConfiguration = provider
.GetService<IOptionsSnapshot<S3BucketConfiguration>>()
.Value;
var credential = new BasicAWSCredentials(
_s3BucketConfiguration.AccessKey, _s3BucketConfiguration.SecurityKey);
RegionEndpoint bucketRegion =
RegionEndpoint.GetBySystemName(_s3BucketConfiguration.Region);
var encryptionMaterials = new EncryptionMaterialsV2(
_s3BucketConfiguration.KMSKeyId,
KmsType.KmsContext,
new Dictionary<string, string>()
);
var config = new AmazonS3CryptoConfigurationV2(SecurityProfile.V2AndLegacy);
//If I add this line it will instantiate AmazonS3EncryptionClientV2 but, the GetObject call fails.
//If I do not add this line, it will give me same error while instiantiating AmazonS3EncryptionClientV2
//config.RegionEndpoint = bucketRegion;
vr client = new AmazonS3EncryptionClientV2(credential, config, encryptionMaterials);
GetObjectResponse response = client.GetObjectAsync(new GetObjectRequest
{
BucketName = _s3BucketConfig.BucketName,
Key = filePath,
}).GetAwaiter().GetResult();
Exception
No RegionEndpoint or ServiceURL configured

I can successfully encrypt with V1 and decrypt with V2 client along with passing the RegionEndpoint.
var configuration = new AmazonS3CryptoConfiguration()
{
RegionEndpoint = RegionEndpoint.USWest2
};
var material = new EncryptionMaterials(KmsKeyId);
var client = new AmazonS3EncryptionClient(configuration, material);
var putObjectResponse = await client.PutObjectAsync(new PutObjectRequest()
{
ContentBody = ContentBody,
BucketName = Bucket,
Key = Key
});
if (putObjectResponse.HttpStatusCode == System.Net.HttpStatusCode.OK)
{
var configurationV2 = new AmazonS3CryptoConfigurationV2(SecurityProfile.V2AndLegacy)
{
RegionEndpoint = RegionEndpoint.USWest2
};
var materialV2 = new EncryptionMaterialsV2(KmsKeyId, KmsType.KmsContext, new Dictionary<string, string>());
var clientV2 = new AmazonS3EncryptionClientV2(configurationV2, materialV2);
var getObjectResponse = await clientV2.GetObjectAsync(new GetObjectRequest()
{
BucketName = Bucket,
Key = Key
});
using (var reader = new StreamReader(getObjectResponse.ResponseStream))
{
Console.WriteLine(reader.ReadToEnd());
}
}
Can you make sure you are using the same RegionEndpoint during the encryption and decryption?

Accessing a bucket at the google S3 api

I am trying to read a bucket at storage.googleapis.com, using the Amazon Web Services .Net SDK in C#.
Can anyone provide a working example of a S3 endpoint Config setup for google, just using the Auth. key/secret pair and a bucket name? Or using any other method to get this working?
According to this tutorial this should be a simple matter, but I get all sorts of exceptions when trying to follow the instructions given. Here is an extract of my current attempt - which throws a TrustFailure exception:
The remote certificate is invalid.
AmazonS3Config conf = new AmazonS3Config();
// Set regionEndpoint to null, or else the serviceURL will be ignored
conf.RegionEndpoint = null;
conf.ServiceURL = "https://s3.storage.googleapis.com";
conf.UseHttp = false;
conf.AuthenticationRegion = null;
conf.UseAccelerateEndpoint = false;
conf.UseDualstackEndpoint = false;
AWSCredentials cred = new BasicAWSCredentials("GOOG3LFXXXXXXXXXXXXX", "BQ6VeMXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX");
IAmazonS3 client = new AmazonS3Client(cred, conf);
GetBucketVersioningRequest request = new GetBucketVersioningRequest { BucketName = "hisbucket" };
GetBucketVersioningResponse response = client.GetBucketVersioning(request);

I finally got the .NET SDK to upload to Google Cloud Storage with:
AWSConfigsS3.UseSignatureVersion4 = false;
AmazonS3Config config = new AmazonS3Config();
config.ServiceURL = "https://storage.googleapis.com";
config.SignatureVersion = "2";
AmazonS3Client client = new AmazonS3Client(accessKey, secretKey, config);
var transferUtilityConfig = new TransferUtilityConfig
{
ConcurrentServiceRequests = 1,
MinSizeBeforePartUpload = 6291456000,
};
var fileTransferUtilityRequest = new TransferUtilityUploadRequest
{
BucketName = bucketName,
FilePath = filePath,
PartSize = 6291456000,
Key = keyName,
};
TransferUtility fileTransferUtility = new TransferUtility(client, transferUtilityConfig);
fileTransferUtility.Upload(fileTransferUtilityRequest);
fileTransferUtility.Dispose();

You need a Amazon S3 service URL, an access key id, a secret access key id and the bucket name.
var s3Config = new AmazonS3Config
{
ServiceURL = Constants.AmazonS3ServiceUrl,
RegionEndpoint = Amazon.RegionEndpoint.EUWest1
};
string accessKeyId = Constants.AmazonAccessKeyId;
string secretAccessKey = Constants.AmazonSecretAccessKey;
var config = new AwsS3Config(){AmazonS3BucketName = Constants.AmazonS3BucketName};
var client = new AmazonS3Client(accessKeyId, secretAccessKey, s3Config);
Then, you should be able to make calls to the amazon client:
var request = new GetObjectRequest
{
BucketName = _bucketName,
Key = entity.Path
};
var response = _client.GetObjectAsync(request).Result;
The code above works on an S3 account, not particularly storage.googleapis.com, which is your case. Anyway, I hope this helps and answers your question.

Cannot decrypt encrypted file by AWS SDK with Minio server

I use :
minio server to store files
nginx as reverse proxy to be able to use https with minio server
.NET AWSSDK.S3 to communicate with Minio server through nginx
Since server side encryption does not work with minio server, I tried to use the client side encryption help with AWS. But decryption of the file does not work with minio server.
When I use the same code with AWS server account, encryption / decryption work well.
It seems that file loses its metadata when it is created in minio server.
When I try to get file I have an exception :
An unhandled exception of type 'Amazon.Runtime.AmazonServiceException' occurred in AWSSDK.Core.dll
Additional information: Unable to decrypt data for object file-stream-27e52c5f-05d1-4296-
Here is my code
static void Main()
{
string filePath = #"c:/tempPrivateKey.txt";
string privateKey = File.ReadAllText(filePath);
RSA rsaAlgorithm = RSA.Create();
rsaAlgorithm.FromXmlString(privateKey);
EncryptionMaterials encryptionMaterials = new EncryptionMaterials(rsaAlgorithm);
var credentials = new BasicAWSCredentials(AccessKey, SecretKey);
AmazonS3CryptoConfiguration cryptoConfig = new AmazonS3CryptoConfiguration
{
RegionEndpoint = RegionEndpoint.EUWest1,
StorageMode = CryptoStorageMode.ObjectMetadata,
ServiceURL = EndPointNginx,
UseHttp = false,
ForcePathStyle = true
};
_amazonS3Client = new AmazonS3EncryptionClient(credentials, cryptoConfig, encryptionMaterials);
string bucketName = "bucket-" + Guid.NewGuid();
string fileStreamKey = "file-stream-" + Guid.NewGuid();
Stream fileStream = CreateRandomFileOnStream();
CreateBucket(bucketName);
AddFileToBucket(fileStreamKey, fileStream, bucketName);
Stream fileStreamToRead = GetFile(fileStreamKey, bucketName);
using (var reader = new StreamReader(fileStreamToRead))
{
Console.Out.WriteLine(reader.ReadToEnd());
}
DeleteFile(fileStreamKey, bucketName);
DeleteBucket(bucketName);
Console.ReadKey();
}
private static void AddFileToBucket(string fileKey, Stream fileStream, string bucketName)
{
Console.Out.WriteLine();
Console.Out.WriteLine($"adding file {fileKey} to bucket {bucketName}.");
var objectToPut = new PutObjectRequest
{
BucketName = bucketName,
Key = fileKey,
InputStream = fileStream
};
_amazonS3Client.PutObject(objectToPut);
if (fileStream.CanRead)
fileStream.Dispose();
Console.Out.WriteLine("file added");
}
private static Stream GetFile(string fileKey, string bucketName)
{
// This line throw an exception.
GetObjectResponse response = _amazonS3Client.GetObject(new GetObjectRequest { BucketName = bucketName, Key = fileKey });
return response.ResponseStream;
}
At worst I will encrypt / decrypt the file manually but I would like to be sure if there is a solution to this problem.