We Keep Coding

get url of uploaded s3 object .net

So currently my code uploads an image to S3 but what I want it to now do is to get the URL of the image that has been uploaded, so this URL can be stored in the DB and used later.
I know it's possible, as it was shown in this question here: s3 file upload does not return response (but that is JavaScript and I'm struggling to convert it to c#)
This is my code, it works perfectly, I just need to get the URL of the uploaded object + is there a way to make the object public by default. I tried to console write the response but that was no help
public class AmazonS3Uploader
{
private string bucketName = "cartalkio-image-storage-dev";
private string keyName = DateTime.Now.ToString() + ".png";
public async void UploadFile()
{
byte[] bytes = System.Convert.FromBase64String("iVBORw0KGgoAAAANSUhEUgAAAAUAAAAFCAYAAACNbyblAAAAHElEQVQI12P4//8/w38GIAXDIBKE0DHxgljNBAAO9TXL0Y4OHwAAAABJRU5ErkJggg==");
Stream stream = new MemoryStream(bytes);
var myAwsAccesskey = "*************";
var myAwsSecret = "**************************";
var client = new AmazonS3Client(myAwsAccesskey, myAwsSecret, Amazon.RegionEndpoint.EUWest2);
try
{
PutObjectRequest putRequest = new PutObjectRequest
{
BucketName = bucketName,
Key = keyName,
ContentType = "image/png",
InputStream = stream
};
PutObjectResponse response = await client.PutObjectAsync(putRequest);
// Console.Write(response);
}
catch (AmazonS3Exception amazonS3Exception)
{
if (amazonS3Exception.ErrorCode != null &&
(amazonS3Exception.ErrorCode.Equals("InvalidAccessKeyId")
||
amazonS3Exception.ErrorCode.Equals("InvalidSecurity")))
{
throw new Exception("Check the provided AWS Credentials.");
}
else
{
throw new Exception("Error occurred: " + amazonS3Exception.Message);
}
}
}

I've searched the documentation and various websites but this was all I found () - I guess there just isn't a URL that is returned. What I've done is changed my code around a bit because you can always predict what the object URL will be based on the name of the object you're uploading. i.e if you're uploading an image called 'test.png', the URL will be this:
https://[Your-Bucket-Name].s3.[S3-Region].amazonaws.com/[test.png]
I tried dependency injection but AWS didn't like that so I've changed my code to this:
(in this example I'm receiving a base64 string, turning it into a BYTE and then into a SYSTEM.IO.Stream)
This is the request I'm sending up:
{ "image":"iVBORw0KGgoAAAANSUhEUgAAAAUAAAAFCAYAAACNbyblAAAAHElEQVQI12P4//8/w38GIAXDIBKE0DHxgljNBAAO9TXL0Y4OHwAAAABJRU5ErkJggg=="
}
Controller:
public async Task<ActionResult> Index(string image)
{
AmazonS3Uploader amazonS3 = new AmazonS3Uploader();
// This bit creates a random string that will be used as part of the URL
var chars = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789";
var stringChars = new char[8];
var random = new Random();
for (int i = 0; i < stringChars.Length; i++)
{
stringChars[i] = chars[random.Next(chars.Length)];
}
var finalString = new String(stringChars);
// This bit adds the '.png' as its an image
var keyName = finalString + ".png";
// This uploads the file to S3, passing through the keyname (which is the end of the URL) and the image string
amazonS3.UploadFile(keyName, image);
// This is what the final URL of the object will be, so you can use this variable later or save it in your database
var itemUrl = "https://[your-bucket-name].s3.[S3-Region].amazonaws.com/" + keyName;
return Ok();
}
AmazonS3Uploader.cs
private string bucketName = "your-bucket-name";
public async void UploadFile(string keyName, string image)
{
byte[] bytes = System.Convert.FromBase64String(image);
Stream stream = new MemoryStream(bytes);
var myAwsAccesskey = "*************";
var myAwsSecret = "**************************";
var client = new AmazonS3Client(myAwsAccesskey, myAwsSecret, Amazon.RegionEndpoint.[S3-Region]);
try
{
PutObjectRequest putRequest = new PutObjectRequest
{
BucketName = bucketName,
Key = keyName,
ContentType = "image/png",
InputStream = stream
};
PutObjectResponse response = await client.PutObjectAsync(putRequest);
}
catch (AmazonS3Exception amazonS3Exception)
{
if (amazonS3Exception.ErrorCode != null &&
(amazonS3Exception.ErrorCode.Equals("InvalidAccessKeyId")
||
amazonS3Exception.ErrorCode.Equals("InvalidSecurity")))
{
throw new Exception("Check the provided AWS Credentials.");
}
else
{
throw new Exception("Error occurred: " + amazonS3Exception.Message);
}
}
}
The only problem with this is if the file upload to S3 Fails, you might still get the URL, and if you're saving it to your database - you'll save the URL to the database but the object wont exist in the S3 bucket, so it won't lead anywhere. If you implement dependency injection - this shouldn't be an issue (dependency injection not implemented in this example)

Write content to a file on azure storage with using PUT request in C#

I am trying to make a put request to Azure storage file, where I want to add some simple contents. I change the URL and add ?comp=range at the end of the url but I get 403 error in response. I have created a basic console application in .net.
My Header is :
const string requestMethod = "PUT";
string urlPath = strShareName + "/" + "rahila.csv?comp=range";//+ "?comp=range HTTP/1.1";
String canonicalizedResource = String.Format("/{0}/{1}/{2}", StorageAccountName, strShareName, strFileName);
try
{
//GetWebRequest(requestMethod, urlPath, canonicalizedResource, "CreateFile");
HttpWebRequest request = null;
try
{
const string type = "file";
string MethodType = "CreateFile";
const string msVersion = "2015-04-05";
String dateInRfc1123Format = DateTime.UtcNow.ToString("R", CultureInfo.InvariantCulture);
String canonicalizedHeaders = "";
string data = "rahila sted";
canonicalizedHeaders = String.Format("x-ms-date:{0}\nx-ms-version:{1}", dateInRfc1123Format, msVersion);
if (MethodType == "CreateFile")
{
canonicalizedHeaders = String.Format("x-ms-content-length:65536\nx-ms-date:{0}\nx-ms-type:file\nx-ms-version:{1}", dateInRfc1123Format, msVersion);
}
String stringToSign = "";
stringToSign = String.Format("{0}\n\n\n\n\n\n\n\n\n\n\n\n{1}\n{2}", requestMethod, canonicalizedHeaders, canonicalizedResource);
if (String.IsNullOrEmpty(stringToSign))
{
throw new ArgumentNullException("canonicalizedString");
}
String signature;
if (String.IsNullOrEmpty(stringToSign))
{
throw new ArgumentNullException("unsignedString");
}
if (Convert.FromBase64String(StorageKey) == null)
{
throw new ArgumentNullException("key");
}
Byte[] dataToHmac = System.Text.Encoding.UTF8.GetBytes(stringToSign);
using (HMACSHA256 hmacSha256 = new HMACSHA256(Convert.FromBase64String(StorageKey)))
{
signature = Convert.ToBase64String(hmacSha256.ComputeHash(dataToHmac));
}
String authorizationHeader = String.Format(CultureInfo.InvariantCulture, "{0} {1}:{2}",
StorageScheme,
StorageAccountName, signature);
Uri uri = new Uri(FileEndPoint + urlPath);
request = (HttpWebRequest)WebRequest.Create(uri);
if (requestMethod != "Get")
{
request.ContentLength = data.Length;
}
// string data = "Hello testing";
//int a= ((data.Length) + 1);
request.Method = "PUT";//requestMethod;
request.Headers.Add("x-ms-write", "update");
request.Headers.Add("x-ms-date", dateInRfc1123Format);
request.Headers.Add("x-ms-version", msVersion);
request.Headers.Add("x-ms-range", "bytes=0-65535"); // + ((data.Length) - 1));
request.Headers.Add("Authorization", authorizationHeader);
the line where i get the exception is in the bold format.
HttpWebResponse response = null;
response = (HttpWebResponse)request.GetResponse();
string returnString = response.StatusCode.ToString();
Can anyone help me to resolve this issue or just guide me how to write content to a simple file on azure storage without using the azure client API.

update 12/19:
When using Put Range to upload content to azure file, you can follow the following code(I assume you have already created a file on the azure file share, and it's content length is larger than the content being uploaded):
static void UploadText()
{
string Account = "xxxx";
string Key = "xxxx";
string FileShare = "test1";
string FileName = "11.txt";
string apiversion = "2019-02-02";
//the string to be uploaded to azure file, note that the length of the uploaded string should less than the azure file length
string upload_text = "bbbbbbbbbbbbbbbbbbbbbbbb.";
Console.WriteLine("the string length: " + upload_text.Length);
DateTime dt = DateTime.UtcNow;
string StringToSign = String.Format("PUT\n"
+ "\n" // content encoding
+ "\n" // content language
+ upload_text.Length + "\n" // content length
+ "\n" // content md5
+ "\n" // content type
+ "\n" // date
+ "\n" // if modified since
+ "\n" // if match
+ "\n" // if none match
+ "\n" // if unmodified since
+ "\n"//+ "bytes=0-" + (upload_text.Length - 1) + "\n" // range
+"x-ms-date:" + dt.ToString("R") + "\nx-ms-range:bytes=0-"+(upload_text.Length-1) + "\nx-ms-version:" + apiversion + "\nx-ms-write:update\n" // headers
+ "/{0}/{1}/{2}\ncomp:range", Account, FileShare, FileName);
string auth = SignThis(StringToSign, Key, Account);
string method = "PUT";
string urlPath = string.Format("https://{0}.file.core.windows.net/{1}/{2}?comp=range", Account, FileShare,FileName);
Uri uri = new Uri(urlPath);
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(uri);
request.Method = method;
request.ContentLength = upload_text.Length;
request.Headers.Add("x-ms-range", "bytes=0-"+(upload_text.Length-1));
request.Headers.Add("x-ms-write", "update");
request.Headers.Add("x-ms-date", dt.ToString("R"));
request.Headers.Add("x-ms-version", apiversion);
request.Headers.Add("Authorization", auth);
//request.Headers.Add("Content-Length", upload_text.Length.ToString());
var bytes = System.Text.Encoding.ASCII.GetBytes(upload_text);
using (var requestStream = request.GetRequestStream())
{
requestStream.Write(bytes, 0, bytes.Length);
}
using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
{
//read the content
Console.WriteLine("the response is:" + response.StatusCode);
}
}
private static String SignThis(String StringToSign, string Key, string Account)
{
String signature = string.Empty;
byte[] unicodeKey = Convert.FromBase64String(Key);
using (HMACSHA256 hmacSha256 = new HMACSHA256(unicodeKey))
{
Byte[] dataToHmac = System.Text.Encoding.UTF8.GetBytes(StringToSign);
signature = Convert.ToBase64String(hmacSha256.ComputeHash(dataToHmac));
}
String authorizationHeader = String.Format(
CultureInfo.InvariantCulture,
"{0} {1}:{2}",
"SharedKey",
Account,
signature);
return authorizationHeader;
}
Then in the Main() method, you can call UploadText() method, it works at my side.
old:
guide me how to write content to a simple file on azure storage
without using the azure client API.
For this, you can directly use Azure File Storage SDK Microsoft.Azure.Storage.File, version 11.1.1. And we always recommend using SDK instead of using rest api, because the SDK is easy to use.
Here is an example of using this SDK.
First, create a console project of .NET framework in visual studio. Then install this nuget package Microsoft.Azure.Storage.File, version 11.1.1.
The code:
using Microsoft.Azure.Storage;
using Microsoft.Azure.Storage.Auth;
using Microsoft.Azure.Storage.File;
using System;
namespace AzureFileTest2
{
class Program
{
static void Main(string[] args)
{
string accountName = "xxx";
string accountKey = "xxx";
CloudStorageAccount storageAccount = new CloudStorageAccount(new StorageCredentials(accountName, accountKey), true);
CloudFileClient cloudFileClient = storageAccount.CreateCloudFileClient();
//make sure the file share named test1 exists.
CloudFileShare fileShare = cloudFileClient.GetShareReference("test1");
CloudFileDirectory fileDirectory = fileShare.GetRootDirectoryReference();
CloudFile myfile = fileDirectory.GetFileReference("test123.txt");
if (!myfile.Exists())
{
//if the file does not exists, then create the file and set the file max size to 100kb.
myfile.Create(100 * 1024 * 1024);
}
//upload text to the file
//Besides using UploadText() method to directly upload text, you can also use UploadFromFile() / UploadFromByteArray() / UploadFromStream() methods as per your need.
myfile.UploadText("hello, it is using azure storage SDK");
Console.WriteLine("**completed**");
Console.ReadLine();
}
}
}

Download Objects from S3 Bucket using c#

Im trying to download object from S3 bucket facing below issue
The Security token included in the request is Invalid .
Please check and correct where is the mistake.
Below is my code
1. Get Temporary credentails:
main()
{
string path = "http://XXX.XXX.XXX./latest/meta-data/iam/security-credentials/EC2_WLMA_Permissions";
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(path);
request.Method = "GET";
request.ContentType = "application/json";
HttpWebResponse response = request.GetResponse() as HttpWebResponse;
string result = string.Empty;
using (StreamReader reader = new StreamReader(response.GetResponseStream()))
{
result = reader.ReadToEnd();
dynamic metaData = JsonConvert.DeserializeObject(result);
_awsAccessKeyId = metaData.AccessKeyId;
_awsSecretAccessKey = metaData.SecretAccessKey;
}
}
Create SessionAWSCredentials instance:
SessionAWSCredentials tempCredentials =
GetTemporaryCredentials(_awsAccessKeyId, _awsSecretAccessKey);
//GetTemporaryCredentials method:
private static SessionAWSCredentials GetTemporaryCredentials(
string accessKeyId, string secretAccessKeyId)
{
AmazonSecurityTokenServiceClient stsClient =
new AmazonSecurityTokenServiceClient(accessKeyId,
secretAccessKeyId);
Console.WriteLine(stsClient.ToString());
GetSessionTokenRequest getSessionTokenRequest =
new GetSessionTokenRequest();
getSessionTokenRequest.DurationSeconds = 7200; // seconds
GetSessionTokenResponse sessionTokenResponse =
stsClient.GetSessionToken(getSessionTokenRequest);
Console.WriteLine(sessionTokenResponse.ToString());
Credentials credentials = sessionTokenResponse.Credentials;
Console.WriteLine(credentials.ToString());
SessionAWSCredentials sessionCredentials =
new SessionAWSCredentials(credentials.AccessKeyId,
credentials.SecretAccessKey,
credentials.SessionToken);
return sessionCredentials;
}
Get files from S3 using AmazonS3Client:
using (IAmazonS3 client = new AmazonS3Client(tempCredentials,RegionEndpoint.USEast1))
{
GetObjectRequest request = new GetObjectRequest();
request.BucketName = "bucketName" + #"/" + "foldername";
request.Key = "Terms.docx";
GetObjectResponse response = client.GetObject(request);
response.WriteResponseStreamToFile("C:\\MyFile.docx");
}

We do something a little simpler for interfacing with S3 (downloads and uploads)
It looks like you went the more complex approach. You should try just using the TransferUtility instead:
TransferUtility fileTransferUtility =
new TransferUtility(
new AmazonS3Client("ACCESS-KEY-ID", "SECRET-ACCESS-KEY", Amazon.RegionEndpoint.CACentral1));
// Note the 'fileName' is the 'key' of the object in S3 (which is usually just the file name)
fileTransferUtility.Download(filePath, "my-bucket-name", fileName);
NOTE: TransferUtility.Download() returns void because it downloads the file to the path specified in the filePath argument. This may be a little different than what you were expecting but you can still open a FileStream to that path afterwards and manipulate the file all you want. For example:
using (FileStream fileDownloaded = new FileStream(filePath, FileMode.Open, FileAccess.Read))
{
// Do stuff with our newly downloaded file
}

Bucketname, Accesskey and secretkey, I took from web config. You could type manually.
public void DownloadObject(string imagename)
{
RegionEndpoint bucketRegion = RegionEndpoint.USEast1;
IAmazonS3 client = new AmazonS3Client(bucketRegion);
string accessKey = System.Configuration.ConfigurationManager.AppSettings["AWSAccessKey"];
string secretKey = System.Configuration.ConfigurationManager.AppSettings["AWSSecretKey"];
AmazonS3Client s3Client = new AmazonS3Client(new BasicAWSCredentials(accessKey, secretKey), Amazon.RegionEndpoint.USEast1);
string objectKey = "EMR" + "/" + imagename;
//EMR is folder name of the image inside the bucket
GetObjectRequest request = new GetObjectRequest();
request.BucketName = System.Configuration.ConfigurationManager.AppSettings["bucketname"];
request.Key = objectKey;
GetObjectResponse response = s3Client.GetObject(request);
response.WriteResponseStreamToFile("D:\\Test\\"+ imagename);
}
//> D:\Test\ is local file path.

Following is how I download it. I need to download only .zip files in this case. Restricting to only required file types (.zip in my case), helped me to avoid errors related to (416) Requested Range Not Satisfiable
public static class MyAWS_S3_Helper
{
static string _S3Key = System.Configuration.ConfigurationManager.ConnectionStrings["S3BucketKey"].ConnectionString;
static string _S3SecretKey = System.Configuration.ConfigurationManager.ConnectionStrings["S3BucketSecretKey"].ConnectionString;
public static void S3Download(string bucketName, string _ObjectKey, string downloadPath)
{
IAmazonS3 _client = new AmazonS3Client(_S3Key, _S3SecretKey, Amazon.RegionEndpoint.USEast1);
TransferUtility fileTransferUtility = new TransferUtility(_client);
fileTransferUtility.Download(downloadPath + "\\" + _ObjectKey, bucketName, _ObjectKey);
_client.Dispose();
}
public static async Task AsyncDownload(string bucketName, string downloadPath, string requiredSunFolder)
{
var bucketRegion = RegionEndpoint.USEast1; //Change it
var credentials = new BasicAWSCredentials(_S3Key, _S3SecretKey);
var client = new AmazonS3Client(credentials, bucketRegion);
var request = new ListObjectsV2Request
{
BucketName = bucketName,
MaxKeys = 1000
};
var response = await client.ListObjectsV2Async(request);
var utility = new TransferUtility(client);
foreach (var obj in response.S3Objects)
{
string currentKey = obj.Key;
double sizeCheck = Convert.ToDouble(obj.Size);
int fileNameLength = currentKey.Length;
Console.WriteLine(currentKey + "---" + fileNameLength.ToString());
if (currentKey.Contains(requiredSunFolder))
{
if (currentKey.Contains(".zip")) //This helps to avoid errors related to (416) Requested Range Not Satisfiable
{
try
{
S3Download(bucketName, currentKey, downloadPath);
}
catch (Exception exTest)
{
string messageTest = currentKey + "-" + exTest;
}
}
}
}
}
}
Here is how it is called
static void Main(string[] args)
{
string downloadPath = #"C:\SourceFiles\TestDownload";
Task awsTask = MyAWS_S3_Helper.AsyncDownload("my-files", downloadPath, "mysubfolder");
awsTask.Wait();
}

Here is what I have done to download the files from S3 bucket,
var AwsImportFilePathParcel = "TEST/TEMP"
IAmazonS3 client = new AmazonS3Client(AwsAccessKey,AwsSecretKey);
S3DirectoryInfo info = new S3DirectoryInfo(client, S3BucketName, AwsImportFilePathParcel);
S3FileInfo[] s3Files = info.GetFiles(pattenForParcel);
now in s3Files, you have all the files which are on provided location, using for each you can save all files in to your system
foreach (var fileInfo in s3Files)
{
var localPath = Path.Combine("C:\TEST\", fileInfo.Name);
var file = fileInfo.CopyToLocal(localPath);
}

Convert API, new API version docx to PDF

I'm using Convert API to convert docx to PDF. With the old API version everything works good, but I'm trying to migrate to the new API version and when I open the PDF is not a valid document and it will not open. Not sure what I am doing wrong, maybe something about the encoding?
The response that I get from Convert API is a JSON with the File Name, File Size and File Data. Maybe this File Data needs to be processed to create a valid PDF file? if I just write that data in a file it does not work.
public string ConvertReportToPDF(string fileName)
{
string resultFileName = "";
key = "xxxxx";
var requestContent = new MultipartFormDataContent();
var fileStream = System.IO.File.OpenRead(fileName);
var stream = new StreamContent(fileStream);
requestContent.Add(stream, "File", fileStream.Name);
var response = new HttpClient().PostAsync("https://v2.convertapi.com/docx/to/pdf?Secret=" + key, requestContent).Result;
FileReportResponse responseDeserialized = JsonConvert.DeserializeObject<FileReportResponse>(response.Content.ReadAsStringAsync().Result);
var path = SERVER_TEMP_PATH + "\\" + responseDeserialized.Files.First().FileName;
System.IO.File.WriteAllText(path, responseDeserialized.Files.First().FileData);
return responseDeserialized.Files.First().FileName;
}

File data in JSON is Base64 encoded, decode it before writing to a file.
public string ConvertReportToPDF(string fileName)
{
string resultFileName = "";
key = "xxxxx";
var requestContent = new MultipartFormDataContent();
var fileStream = System.IO.File.OpenRead(fileName);
var stream = new StreamContent(fileStream);
requestContent.Add(stream, "File", fileStream.Name);
var response = new HttpClient().PostAsync("https://v2.convertapi.com/docx/to/pdf?Secret=" + key, requestContent).Result;
FileReportResponse responseDeserialized = JsonConvert.DeserializeObject<FileReportResponse>(response.Content.ReadAsStringAsync().Result);
var path = SERVER_TEMP_PATH + "\\" + responseDeserialized.Files.First().FileName;
System.IO.File.WriteAllText(path, Convert.FromBase64String(responseDeserialized.Files.First().FileData));
return responseDeserialized.Files.First().FileName;
}

Why to use JSON response in C# when you can use binary response instead. A response will be smaller, no need to decode. To change response type you need to add accept=application/octet-stream header to request to ask for binary response from server. The whole code will look like
using System;
using System.Net;
using System.IO;
class MainClass {
public static void Main (string[] args) {
const string fileToConvert = "test.docx";
const string fileToSave = "test.pdf";
const string Secret="";
if (string.IsNullOrEmpty(Secret))
Console.WriteLine("The secret is missing, get one for free at https://www.convertapi.com/a");
else
try
{
Console.WriteLine("Please wait, converting!");
using (var client = new WebClient())
{
client.Headers.Add("accept", "application/octet-stream");
var resultFile = client.UploadFile(new Uri("http://v2.convertapi.com/docx/to/pdf?Secret=" + Secret), fileToConvert);
File.WriteAllBytes(fileToSave, resultFile );
Console.WriteLine("File converted successfully");
}
}
catch (WebException e)
{
Console.WriteLine("Status Code : {0}", ((HttpWebResponse)e.Response).StatusCode);
Console.WriteLine("Status Description : {0}", ((HttpWebResponse)e.Response).StatusDescription);
Console.WriteLine("Body : {0}", new StreamReader(e.Response.GetResponseStream()).ReadToEnd());
}
}
}

How to download a file from azure blob in Asp.Net core?

I'm trying to download an image from azure blob in Asp.Net Core using the below code. But i am getting
"The given path's format is not supported"
My Code:
StorageCredentials creds = new StorageCredentials(accountName, accountKey);
CloudStorageAccount account = new CloudStorageAccount(creds, useHttps: true);
CloudBlobClient client = account.CreateCloudBlobClient();
container = client.GetContainerReference(blobName);
// For download
public async void DownloadAsync(string path, string[] names)
{
string MyPath = path.Replace("https://browsercontent.blob.core.windows.net/blob1/", "");
CloudBlockBlob blockBlob = container.GetBlockBlobReference(MyPath+names[0]);
path = path + names[0];
await blockBlob.DownloadToFileAsync(path, FileMode.Create);
}
How can I resolve this issue?

The first argument DownloadToFileAsync to is the path to the file on disk, that should be a local path like "C:\Temp\3.png" for example. See example below:
public async void DownloadAsync(string path, string[] names)
{
string MyPath = path.Replace("https://browsercontent.blob.core.windows.net/blob1/", "");
CloudBlockBlob blockBlob = container.GetBlockBlobReference(MyPath+names[0]);
var localPath = #"C:\Temp" + names[0];
await blockBlob.DownloadToFileAsync(localPath , FileMode.Create);
}
Some other tips:
Next time when you ask a question add the location where the exception occurs
Avoid having async void. Return a Task like public async Task DownloadAsync(string path, string[] names) so you can await that method