Hi I am writing Lambda function in .Net core. My requirement is Post api will send employee data. When data is received I want to store it in S3 bucket. The approach I am following is whenever Api sends data to lambda, Empid is unique. Each time I want to create one json file and name of the file should be equal to emp id. I am trying to write my function as below.
public async Task<string> FunctionHandler(Employee input, ILambdaContext context)
{
string bucketname = "someunquebucket";
var client = new AmazonS3Client();
bool doesBucketExists = await AmazonS3Util.DoesS3BucketExistV2Async(client,bucketname);
if(!doesBucketExists)
{
var request = new PutBucketRequest
{
BucketName = "someunquebucket",
};
var response = await client.PutBucketAsync(request);
}
using (var stream = new MemoryStream(bin))
{
var request = new PutObjectRequest
{
BucketName = bucketname,
InputStream = stream,
ContentType = "application/json",
Key = input.emp_id
};
var response = await client.PutObjectAsync(request).ConfigureAwait(false);
}
}
In the above code, PutObjectRequest is used to write data to s3 bucket. I am adding few parameters like bucketname etc. In the function I am receiving Employee data. Now I want to create json file with emp id. I found above code but I am not sure what to pass in place of bin. Can someone help me to execute the above function. Any help would be appreciated. Thanks
I am new to .Net Core, and came across your post while trying to solve a similar problem.
Here is how I was able to upload a json response string to my s3 Bucket (Hope it helps)
String timeStamp = DateTime.Now.ToString("yyyyMMddHHmmssfff");
byte[] byteArray = Encoding.ASCII.GetBytes(jsonString);
var seekableStream = new MemoryStream(byteArray);
seekableStream.Position = 0;
var putRequest = new PutObjectRequest
{
BucketName = this.BucketName,
Key = timeStamp+"_"+reg + ".json",
InputStream = seekableStream
};
try
{
var response2 = await this.S3Client.PutObjectAsync(putRequest);}
Hope you sorted your issue.
This is the example code from AWS SDK Code Examples.
https://docs.aws.amazon.com/sdk-for-net/v3/developer-guide/welcome.html
https://github.com/awsdocs/aws-doc-sdk-examples/tree/main/dotnetv3/S3
https://github.com/awsdocs/aws-doc-sdk-examples/blob/main/dotnetv3/S3/S3_Basics/S3Bucket.cs#L45
/// <summary>
/// Shows how to upload a file from the local computer to an Amazon S3
/// bucket.
/// </summary>
/// <param name="client">An initialized Amazon S3 client object.</param>
/// <param name="bucketName">The Amazon S3 bucket to which the object
/// will be uploaded.</param>
/// <param name="objectName">The object to upload.</param>
/// <param name="filePath">The path, including file name, of the object
/// on the local computer to upload.</param>
/// <returns>A boolean value indicating the success or failure of the
/// upload procedure.</returns>
public static async Task<bool> UploadFileAsync(
IAmazonS3 client,
string bucketName,
string objectName,
string filePath)
{
var request = new PutObjectRequest
{
BucketName = bucketName,
Key = objectName,
FilePath = filePath,
};
var response = await client.PutObjectAsync(request);
if (response.HttpStatusCode == System.Net.HttpStatusCode.OK)
{
Console.WriteLine($"Successfully uploaded {objectName} to {bucketName}.");
return true;
}
else
{
Console.WriteLine($"Could not upload {objectName} to {bucketName}.");
return false;
}
}
This is a modified service I wrote to upload json strings as json files both with sync and async methods:
public class AwsS3Service
{
private readonly IAmazonS3 _client;
private readonly string _bucketName;
private readonly string _keyPrefix;
/// <param name="bucketName">The Amazon S3 bucket to which the object
/// will be uploaded.</param>
public AwsS3Service(string accessKey, string secretKey, string bucketName, string keyPrefix)
{
BasicAWSCredentials credentials = new BasicAWSCredentials(accessKey, secretKey);
_client = new AmazonS3Client(credentials);
_bucketName=bucketName;
_keyPrefix=keyPrefix;
}
public bool UploadJson(string objectName, string json, int migrationId, long orgId)
{
return Task.Run(() => UploadJsonAsync(objectName, json, migrationId, orgId)).GetAwaiter().GetResult();
}
public async Task<bool> UploadJsonAsync(string objectName, string json, int migrationId, long orgId)
{
var request = new PutObjectRequest
{
BucketName = $"{_bucketName}",
Key = $"{_keyPrefix}{migrationId}/{orgId}/{objectName}",
InputStream = new MemoryStream(Encoding.UTF8.GetBytes(json)),
};
var response = await _client.PutObjectAsync(request);
if (response.HttpStatusCode == System.Net.HttpStatusCode.OK)
{
return true;
}
else
{
return false;
}
}
}
Related
I'm trying to upload a file to the contabo object storage from memory stream using AWSSDK.S3
This is my client configuration.
string accessKey = "xxxxxxxxxxxxxxxxxxxxxxxxxx";
string secretKey = "xxxxxxxxxxxxxxxxxxxxxxxxxx";
Amazon.S3.AmazonS3Config config = new Amazon.S3.AmazonS3Config();
Amazon.S3.AmazonS3Client s3Client;
public BookingsController()
{
config.ServiceURL = "https://eu2.contabostorage.com";
config.DisableHostPrefixInjection = true;
s3Client = new Amazon.S3.AmazonS3Client(
accessKey,
secretKey,
config
);
}
This is the method I'm using:
[HttpPost("/api/Bookings/AddFile")]
public async Task<ActionResult> AddBookingFile([FromForm] IFormFile file)
{
using (var newMemoryStream = new MemoryStream())
{
ListBucketsResponse response = await s3Client.ListBucketsAsync();
file.CopyTo(newMemoryStream);
Amazon.S3.Model.PutObjectRequest request = new Amazon.S3.Model.PutObjectRequest();
request.BucketName = "test-bucket";
request.Key = "recording.wav";
request.ContentType = "audio/wav";
request.InputStream = newMemoryStream;
await s3Client.PutObjectAsync(request);
}
return Ok();
}
The ListBucket-Method works properly. The PutObject Method throws the exception that the Host "Der angegebene Host ist unbekannt. (test-bucket.eu2.contabostorage.com:443)" cannot be found.
Referenced to the Contabo-docs is that correct, because Contabo doesn't support virtual hosted buckets (dns prefix). Reference to contabo docs
I thought, that the following configuration fix this, but that wasn't the solution.
config.DisableHostPrefixInjection = true;
Does anyone has any advise hot to prevent the prefixing of the url?
So currently my code uploads an image to S3 but what I want it to now do is to get the URL of the image that has been uploaded, so this URL can be stored in the DB and used later.
I know it's possible, as it was shown in this question here: s3 file upload does not return response (but that is JavaScript and I'm struggling to convert it to c#)
This is my code, it works perfectly, I just need to get the URL of the uploaded object + is there a way to make the object public by default. I tried to console write the response but that was no help
public class AmazonS3Uploader
{
private string bucketName = "cartalkio-image-storage-dev";
private string keyName = DateTime.Now.ToString() + ".png";
public async void UploadFile()
{
byte[] bytes = System.Convert.FromBase64String("iVBORw0KGgoAAAANSUhEUgAAAAUAAAAFCAYAAACNbyblAAAAHElEQVQI12P4//8/w38GIAXDIBKE0DHxgljNBAAO9TXL0Y4OHwAAAABJRU5ErkJggg==");
Stream stream = new MemoryStream(bytes);
var myAwsAccesskey = "*************";
var myAwsSecret = "**************************";
var client = new AmazonS3Client(myAwsAccesskey, myAwsSecret, Amazon.RegionEndpoint.EUWest2);
try
{
PutObjectRequest putRequest = new PutObjectRequest
{
BucketName = bucketName,
Key = keyName,
ContentType = "image/png",
InputStream = stream
};
PutObjectResponse response = await client.PutObjectAsync(putRequest);
// Console.Write(response);
}
catch (AmazonS3Exception amazonS3Exception)
{
if (amazonS3Exception.ErrorCode != null &&
(amazonS3Exception.ErrorCode.Equals("InvalidAccessKeyId")
||
amazonS3Exception.ErrorCode.Equals("InvalidSecurity")))
{
throw new Exception("Check the provided AWS Credentials.");
}
else
{
throw new Exception("Error occurred: " + amazonS3Exception.Message);
}
}
}
I've searched the documentation and various websites but this was all I found () - I guess there just isn't a URL that is returned. What I've done is changed my code around a bit because you can always predict what the object URL will be based on the name of the object you're uploading. i.e if you're uploading an image called 'test.png', the URL will be this:
https://[Your-Bucket-Name].s3.[S3-Region].amazonaws.com/[test.png]
I tried dependency injection but AWS didn't like that so I've changed my code to this:
(in this example I'm receiving a base64 string, turning it into a BYTE and then into a SYSTEM.IO.Stream)
This is the request I'm sending up:
{ "image":"iVBORw0KGgoAAAANSUhEUgAAAAUAAAAFCAYAAACNbyblAAAAHElEQVQI12P4//8/w38GIAXDIBKE0DHxgljNBAAO9TXL0Y4OHwAAAABJRU5ErkJggg=="
}
Controller:
public async Task<ActionResult> Index(string image)
{
AmazonS3Uploader amazonS3 = new AmazonS3Uploader();
// This bit creates a random string that will be used as part of the URL
var chars = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789";
var stringChars = new char[8];
var random = new Random();
for (int i = 0; i < stringChars.Length; i++)
{
stringChars[i] = chars[random.Next(chars.Length)];
}
var finalString = new String(stringChars);
// This bit adds the '.png' as its an image
var keyName = finalString + ".png";
// This uploads the file to S3, passing through the keyname (which is the end of the URL) and the image string
amazonS3.UploadFile(keyName, image);
// This is what the final URL of the object will be, so you can use this variable later or save it in your database
var itemUrl = "https://[your-bucket-name].s3.[S3-Region].amazonaws.com/" + keyName;
return Ok();
}
AmazonS3Uploader.cs
private string bucketName = "your-bucket-name";
public async void UploadFile(string keyName, string image)
{
byte[] bytes = System.Convert.FromBase64String(image);
Stream stream = new MemoryStream(bytes);
var myAwsAccesskey = "*************";
var myAwsSecret = "**************************";
var client = new AmazonS3Client(myAwsAccesskey, myAwsSecret, Amazon.RegionEndpoint.[S3-Region]);
try
{
PutObjectRequest putRequest = new PutObjectRequest
{
BucketName = bucketName,
Key = keyName,
ContentType = "image/png",
InputStream = stream
};
PutObjectResponse response = await client.PutObjectAsync(putRequest);
}
catch (AmazonS3Exception amazonS3Exception)
{
if (amazonS3Exception.ErrorCode != null &&
(amazonS3Exception.ErrorCode.Equals("InvalidAccessKeyId")
||
amazonS3Exception.ErrorCode.Equals("InvalidSecurity")))
{
throw new Exception("Check the provided AWS Credentials.");
}
else
{
throw new Exception("Error occurred: " + amazonS3Exception.Message);
}
}
}
The only problem with this is if the file upload to S3 Fails, you might still get the URL, and if you're saving it to your database - you'll save the URL to the database but the object wont exist in the S3 bucket, so it won't lead anywhere. If you implement dependency injection - this shouldn't be an issue (dependency injection not implemented in this example)
I got to upload file & streams to S3 periodically and the file size usually under 50MB. What am seeing is TransferUtility.UploadAsync method returns success but the file was never created. This occurs sporadically. Below is the code we use. Any suggestions around this?
public TransferUtilityUploadRequest CreateRequest(string bucket, string path)
{
var request = new TransferUtilityUploadRequest
{
BucketName = bucket,
StorageClass = S3StorageClass.Standard,
Key = path,
AutoCloseStream = true,
CannedACL = S3CannedACL.AuthenticatedRead,
ServerSideEncryptionMethod = ServerSideEncryptionMethod.AES256
};
return request;
}
public async Task CreateS3ObjectFromStreamAsync(MemoryStream memeoryStream, string bucket, string filePath)
{
var ftu = new TransferUtility(client);
var request = CreateRequest(bucket, filePath);
request.InputStream = memeoryStream;
await ftu.UploadAsync(request);
}
public async Task CreateS3ObjectFromFileAsync(string sourceFilePath, string bucketName, string destpath)
{
var request = CreateRequest(bucketName, destpath);
request.FilePath = sourceFilePath;
var ftu = new TransferUtility(client);
await ftu.UploadAsync(request);
}
Please use Upload or UploadAsync.Wait. It worked for me!
I'm trying to get Wopi host implementation in an ASP.NET MVC application.
Using this project
https://code.msdn.microsoft.com/office/Building-an-Office-Web-f98650d6
I don't get any calls hitting my API Controler
Discovery URL
<action name="view"
ext="docx"
default="true"
urlsrc="http://word-edit.officeapps.live.com/wv/wordviewerframe.aspx?<ui=UI_LLCC&><rs=DC_LLCC&><showpagestats=PERFSTATS&>" />
URL generated by my application
http://word-edit.officeapps.live.com/we/wordeditorframe.aspx?WOPISrc=http%3a%2f%2flocalhost%3a32876%2fapi%2fwopi%2ffiles%2ftest.docx&access_token=XskWxXK0Nro%3dhwYoiCFehrYAx85XQduYQHYQE9EEPC6EVgqaMiCp4%2bg%3d
I am using Local Host for testing purpose
Controller Route
[RoutePrefix("api/wopi")]
public class FilesController : ApiController
[Route("files/{name}/")]
public CheckFileInfo GetFileInfo(string name, string access_token)
{
Validate(name, access_token);
var fileInfo = _fileHelper.GetFileInfo(name);
bool updateEnabled = false;
if (bool.TryParse(WebConfigurationManager.AppSettings["updateEnabled"].ToString(), out updateEnabled))
{
fileInfo.SupportsUpdate = updateEnabled;
fileInfo.UserCanWrite = updateEnabled;
fileInfo.SupportsLocks = updateEnabled;
}
return fileInfo;
}
// GET api/<controller>/5
/// <summary>
/// Get a single file contents
/// </summary>
/// <param name="name">filename</param>
/// <returns>a file stream</returns>
[Route("files/{name}/contents")]
public HttpResponseMessage Get(string name, string access_token)
{
try
{
Validate(name, access_token);
var file = HostingEnvironment.MapPath("~/App_Data/" + name);
var responseMessage = new HttpResponseMessage(HttpStatusCode.OK);
var stream = new FileStream(file, FileMode.Open, FileAccess.Read);
responseMessage.Content = new StreamContent(stream);
responseMessage.Content.Headers.ContentType = new MediaTypeHeaderValue("application/octet-stream");
return responseMessage;
}
catch (Exception ex)
{
var errorResponseMessage = new HttpResponseMessage(HttpStatusCode.InternalServerError);
var stream = new MemoryStream(UTF8Encoding.Default.GetBytes(ex.Message ?? ""));
errorResponseMessage.Content = new StreamContent(stream);
return errorResponseMessage;
}
}
It is not hitting to the API URL
WOPISrc cannot be 'localhost', it must be a link that office application server can access,
While you use office online application, then you need to provide a public ip or domain name of your wopi server instead of 'localhost'
I want to know that how to generate signurl using google cloud storage classes in .net
I have created string as per the requirement
GET
1388534400
/bucket/objectname
but I now want to sign this url with p12 key and then want to make it url friendly
This library doesn't show specific function for it -> https://developers.google.com/resources/api-libraries/documentation/storage/v1/csharp/latest/annotated.html
So, basically I need .net alternate of Google_Signer_P12 class of php
$signer = new Google_Signer_P12(file_get_contents(__DIR__.'/'."final.p12"), "notasecret");
$signature = $signer->sign($to_sign);
Now there is a UrlSigner in the pre-release package Google.Cloud.Storage.V1 that can be used to to provide read-only access to existing objects:
// Create a signed URL which can be used to get a specific object for one hour.
UrlSigner urlSigner = UrlSigner.FromServiceAccountCredential(credential);
string url = urlSigner.Sign(
bucketName,
objectName,
TimeSpan.FromHours(1),
HttpMethod.Get);
Or write-only access to put specific object content into a bucket:
// Create a signed URL which allows the requester to PUT data with the text/plain content-type.
UrlSigner urlSigner = UrlSigner.FromServiceAccountCredential(credential);
var destination = "places/world.txt";
string url = urlSigner.Sign(
bucketName,
destination,
TimeSpan.FromHours(1),
HttpMethod.Put,
contentHeaders: new Dictionary<string, IEnumerable<string>> {
{ "Content-Type", new[] { "text/plain" } }
});
// Upload the content into the bucket using the signed URL.
string source = "world.txt";
ByteArrayContent content;
using (FileStream stream = File.OpenRead(source))
{
byte[] data = new byte[stream.Length];
stream.Read(data, 0, data.Length);
content = new ByteArrayContent(data)
{
Headers = { ContentType = new MediaTypeHeaderValue("text/plain") }
};
}
HttpResponseMessage response = await httpClient.PutAsync(url, content);
I know the question was for P12, but Google lead me here when I was looking to do this for the newer, preferred JSON method. I pieced this together with other samples and sites I found. Hope this help save some time.
public string GetSignedURL()
{
var myObj = "theObject";
var scopes = new string[] { "https://www.googleapis.com/auth/devstorage.read_write" };
var myBucket = "theBucket";
ServiceAccountCredential cred;
using ( var stream = new FileStream(#"\path to\private-key.json", FileMode.Open, FileAccess.Read) )
{
cred = GoogleCredential.FromStream(stream)
.CreateScoped(scopes)
.UnderlyingCredential as ServiceAccountCredential;
}
var urlSigner = UrlSigner.FromServiceAccountCredential(cred);
return urlSigner.Sign(myBucket, myObj, TimeSpan.FromHours(1), HttpMethod.Get);
}
A list of Scopes can be found here
The .NET client doesn't support signing URLs (it is an XML-only API), so you will need to either make a callout to a tool like gsutil or generate an RSA signature internal to your application (Signing and verifying signatures with RSA C#)
This is my google signer code, One can make it more dynamic as per their needs
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Security.Cryptography;
using System.Web;
using System.Security.Cryptography.X509Certificates;
namespace HHAFSGoogle
{
static class GoogleSigner
{
private static string hashAlgo = "SHA256";
public static string ServiceAccountEmail
{
get
{
return "XXXXXXXXXXXXX-YYYYYYYYYYYYYYYYYYYYYYYY#developer.gserviceaccount.com";
}
}
public static string GoogleSecreat
{
get
{
return "notasecret";
}
}
public static string GoogleBucketDir
{
get
{
return "MyBucketDirectory";
}
}
public static string GoogleBucketName
{
get
{
return "MyBucket";
}
}
public static string CertiFilelocation
{
get
{
return System.Web.HttpContext.Current.Server.MapPath("p12file.p12");
}
}
/// <summary>
/// Get URL signature
/// </summary>
/// <param name="base64EncryptedData"></param>
/// <param name="certiFilelocation"></param>
/// <returns></returns>
public static string GetSignature(string base64EncryptedData, string certiFilelocation)
{
X509Certificate2 certificate = new X509Certificate2(certiFilelocation, GoogleSecreat, X509KeyStorageFlags.Exportable);
RSACryptoServiceProvider csp = (RSACryptoServiceProvider)certificate.PrivateKey;
RSACryptoServiceProvider privateKey1 = new RSACryptoServiceProvider();
privateKey1.ImportParameters(csp.ExportParameters(true));
csp.ImportParameters(privateKey1.ExportParameters(true));
byte[] data = Encoding.UTF8.GetBytes(base64EncryptedData.Replace("\r", ""));
byte[] signature = privateKey1.SignData(data, hashAlgo);
bool isValid = privateKey1.VerifyData(data, hashAlgo, signature);
if (isValid)
{
return Convert.ToBase64String(signature);
}
else
{
return string.Empty;
}
}
/// <summary>
/// Get signed URL by Signature
/// </summary>
/// <param name="fileName"></param>
/// <param name="method"></param>
/// <param name="content_type"></param>
/// <param name="duration"></param>
/// <returns></returns>
public static string GetSignedURL(string fileName, string method = "GET", string content_type = "", int duration = 10)
{
TimeSpan span = (DateTime.UtcNow.AddMinutes(10) - new DateTime(1970, 1, 1, 0, 0, 0, 0, DateTimeKind.Utc));
var expires = Math.Round(span.TotalSeconds, 0);
// Encode filename, so URL characters like %20 for space could be handled properly in signature
fileName = HttpUtility.UrlPathEncode(fileName);
// Generate a string to sign
StringBuilder sbFileParam = new StringBuilder();
sbFileParam.AppendLine(method); //Could be GET, PUT, DELETE, POST
// /* Content-MD5 */ "\n" .
sbFileParam.AppendLine();
sbFileParam.AppendLine(content_type); // Type of content you would upload e.g. image/jpeg
sbFileParam.AppendLine(expires.ToString()); // Time when link should expire and shouldn't work longer
sbFileParam.Append("/" + GoogleBucketName + "/" + fileName);
var signature = System.Web.HttpContext.Current.Server.UrlEncode(GetSignature(sbFileParam.ToString(), CertiFilelocation));
return ("https://storage.googleapis.com/MyBucket/" + fileName +
"?response-content-disposition=attachment;&GoogleAccessId=" + ServiceAccountEmail +
"&Expires=" + expires + "&Signature=" + signature);
}
}
}
and to download file call above class to get signed url
GoogleSigner.GetSignedURL(bucketFileName)
and to upload file call above class to get signed url for upload url
GoogleSigner.GetSignedURL(fileName, "PUT", type);