We Keep Coding

How to create an empty folder in S3 using C#? [duplicate]

I'm trying to determine if a folder exists on my Amazon S3 Bucket and if it doesn't I want to create it.
At the moment I can create the folder using the .NET SDK as follows:
public void CreateFolder(string bucketName, string folderName)
{
var folderKey = folderName + "/"; //end the folder name with "/"
var request = new PutObjectRequest();
request.WithBucketName(bucketName);
request.StorageClass = S3StorageClass.Standard;
request.ServerSideEncryptionMethod = ServerSideEncryptionMethod.None;
//request.CannedACL = S3CannedACL.BucketOwnerFullControl;
request.WithKey(folderKey);
request.WithContentBody(string.Empty);
S3Response response = m_S3Client.PutObject(request);
}
Now when I try to see if the folder exists using this code:
public bool DoesFolderExist(string key, string bucketName)
{
try
{
S3Response response = m_S3Client.GetObjectMetadata(new GetObjectMetadataRequest()
.WithBucketName(bucketName)
.WithKey(key));
return true;
}
catch (Amazon.S3.AmazonS3Exception ex)
{
if (ex.StatusCode == System.Net.HttpStatusCode.NotFound)
return false;
//status wasn't not found, so throw the exception
throw;
}
}
It cannot find the folder. The strange thing is if I create the folder using the AWS Management Console, the 'DoesFolderExist' method can see it.
I'm not sure if it's an ACL/IAM thing but am not sure how to resolve this.

Your code actually works for me, but there are a few things you need to be aware off.
As I understand it, Amazon S3 does not have a concept of folders, but individual clients may display the S3 objects as if they did. So if you create an object called A/B , then the client may display it as if it was an object called B inside a folder called A. This is intuitive and seems to have become a standard, but simulating an empty folder does not appear to have a standard.
For example, I used your method to create a folder called Test, then actually end up creating an object called Test/. But I created a folder called Test2 in AWS Explorer (ie the addon to Visual Studio) and it ended up creating an object called Test2/Test2_$folder$
(AWS Explorer will display both Test and Test2 as folders)
Once of the things that this means is that you don't need to create the 'folder' before you can use it, which may mean that you don't need a DoesFolderExist method.
As I mention I tried your code and it works and finds the Test folder it created, but the key had to be tweaked to find the folder created by AWS Explorer , ie
DoesFolderExist("Test/" , bucketName); // Returns true
DoesFolderExist("Test2/" , bucketName); // Returns false
DoesFolderExist("Test2/Test2_$folder$", bucketName); // Returns true
So if you do still want to have a DoesFolderExist method, then it might be safer to just look for any objects that start with folderName + "/" , ie something like
ListObjectsRequest request = new ListObjectsRequest();
request.BucketName = bucketName ;
request.WithPrefix(folderName + "/");
request.MaxKeys = 1;
using (ListObjectsResponse response = m_S3Client.ListObjects(request))
{
return (response.S3Objects.Count > 0);
}

Just refactored above codes to on async method with version 2 of AWS .Net SDK:
public async Task CreateFoldersAsync(string bucketName, string path)
{
path = path.EnsureEndsWith('/');
IAmazonS3 client =
new AmazonS3Client(YOUR.AccessKeyId, YOUR.SecretAccessKey,
RegionEndpoint.EUWest1);
var findFolderRequest = new ListObjectsV2Request();
findFolderRequest.BucketName = bucketName;
findFolderRequest.Prefix = path;
findFolderRequest.MaxKeys = 1;
ListObjectsV2Response findFolderResponse =
await client.ListObjectsV2Async(findFolderRequest);
if (findFolderResponse.S3Objects.Any())
{
return;
}
PutObjectRequest request = new PutObjectRequest()
{
BucketName = bucketName,
StorageClass = S3StorageClass.Standard,
ServerSideEncryptionMethod = ServerSideEncryptionMethod.None,
Key = path,
ContentBody = string.Empty
};
// add try catch in case you have exceptions shield/handling here
PutObjectResponse response = await client.PutObjectAsync(request);
}

ListObjectsRequest findFolderRequest = new ListObjectsRequest();
findFolderRequest.BucketName = bucketName;
findFolderRequest.Prefix = path;
ListObjectsResponse findFolderResponse = s3Client.ListObjects(findFolderRequest);
Boolean folderExists = findFolderResponse.S3Objects.Any();
path can be something like "images/40/".
Using the above code can check if a so-called folder "images/40/" under bucket exists or not.
But Amazon S3 data model does not have the concept of folders. When you try to copy a image or file to certain path, if this co-called folder does not exist it will be created automatically as part of key name of this file or image. Therefore, you actually do not need to check if this folder exists or not.
Very important information from docs.aws.amazon.com : The Amazon S3 data model is a flat structure: you create a bucket, and the bucket stores objects. There is no hierarchy of subbuckets or subfolders; however, you can infer logical hierarchy using key name prefixes and delimiters as the Amazon S3 console does.
http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingMetadata.html

Why is my download from Azure storage empty?

I can connect to the Azure Storage account and can even upload a file, but when I go to download the file using DownloadToFileAsync() I get a 0kb file as a result.
I have checked and the "CloudFileDirectory" and the "CloudFile" fields are all correct, which means the connection with Azure is solid. I can even write the output from the file to the console, but I cannot seem to save it as a file.
public static string PullFromAzureStorage(string azureFileConn, string remoteFileName, string clientID)
{
var localDirectory = #"C:\cod\clients\" + clientID + #"\ftp\";
var localFileName = clientID + "_xxx_" + remoteFileName;
//Retrieve storage account from connection string
var storageAccount = CloudStorageAccount.Parse(azureFileConn);
var client = storageAccount.CreateCloudFileClient();
var share = client.GetShareReference("testing");
// Get a reference to the root directory for the share
CloudFileDirectory rootDir = share.GetRootDirectoryReference();
//Get a ref to client folder
CloudFileDirectory cloudFileDirectory = rootDir.GetDirectoryReference(clientID);
// Get a reference to the directory we created previously
CloudFileDirectory unprocessed = cloudFileDirectory.GetDirectoryReference("Unprocessed");
// Get a reference to the file
CloudFile sourceFile = unprocessed.GetFileReference(remoteFileName);
//write to console and log
Console.WriteLine("Downloading file: " + remoteFileName);
LogWriter.LogWrite("Downloading file: " + remoteFileName);
//Console.WriteLine(sourceFile.DownloadTextAsync().Result);
sourceFile.DownloadToFileAsync(Path.Combine(localDirectory, localFileName), FileMode.Create);
//write to console and log
Console.WriteLine("Download Successful!");
LogWriter.LogWrite("Download Successful!");
//delete remote file after download
//sftp.DeleteFile(remoteDirectory + remoteFileName);
return localFileName;
}

In the commented out line of code where you write the output to the Console, you explicitly use .Result because you're calling an async method in a synchronous one. You should either also do so while downloading the file as well, or make the entire method around it async.
The first solution would look something like this:
sourceFile.DownloadToFileAsync(Path.Combine(localDirectory, localFileName), FileMode.Create).Result();
EDIT:
As far as the difference with the comment, that uses GetAwaiter().GetResult(), goes: .Result wraps any exception that might occur in an AggregateException, while GetAwaiter().GetResult() won't. Anyhow: if there's any possibility you can refactor the method to be async so you can use await: please do so.

Not getting the Uploaded filepath in asp.net

I want to save file to a specific location with some folder creation based on my requirement. So I wrote the below code.
public string CreateFilePath(string addedFolderName)
{
string folderPath = ConfigurationManager.AppSettings["DocDirectory"].ToString();
string FileUplPath = folderPath + "\\" + addedFolderName + "\\";
if (!Directory.Exists(FileUplPath))
{
Directory.CreateDirectory(FileUplPath);
}
flUploadDocs.SaveAs(FileUplPath + Path.GetFileName(flUploadDocs.FileName));
return folderPath;
}
But I am unable to get the filepath here. I am getting it as null
getting null at
Path.GetFileName(flUploadDocs.FileName)
<asp:FileUpload ID="flUploadDocs" runat="server" />
Please suggest what is wrong here.

Path.GetFileName() returns the file name and extension of the specified path string
if im correct this only fills in the file name and not the directory + name.
Path.GetFileName(flUploadDocs.FileName)
possible solution
Path.GetFileName(FileUplPath+flUploadDocs.FileName)
eventough im confused why you try to retrieve the path again after just having saved it?

The issue is that the webservice does not have the fileupload data. Here is the full code from our extended conversation:
[WebMethod]
public static string InsertUpdateMWSiteData(MWInsertUpdateFields MWInsertUpdateFields)
{
string strInsertUpdateMWInfo = "";
try
{
Dashboard dshb = new Dashboard();
dshb.CreateFilePath(MWInsertUpdateFields.SapID + "_" + MWInsertUpdateFields.CandidateID);
strInsertUpdateMWInfo = CommonDB.InsertUpdateMWSiteInfo(MWInsertUpdateFields);
}
catch (Exception)
{
throw;
}
return strInsertUpdateMWInfo;
}
public string CreateFilePath(string addedFolderName)
{
string folderPath = ConfigurationManager.AppSettings["DocDirectory"].ToString();
string FileUplPath = folderPath + "\\" + addedFolderName + "\\";
if (!Directory.Exists(FileUplPath))
{
Directory.CreateDirectory(FileUplPath);
}
if (flUploadDoc.HasFile == true)
{
string strFilename = Path.GetFileName(flUploadDoc.FileName);
flUploadDoc.SaveAs(FileUplPath + Path.GetFileName(flUploadDoc.PostedFile.FileName));
}
return folderPath;
}
The problem is that after uploading a file, a request is sent to a webmethod which is being hosted in another instance of the program. This Webmethod checks its own instance for the fileupload control and data, and doesn't find it because it is in a different instance. This is why your fileupload control is returning null even on a sanity check of .HasFile().
One solution is to pass the data to the Webservice. You could for example pass the data to your webmethod as a byte[], and then on the webservice side reconvert it back into its original file type. After completing this process, save the file to your local filesystem. To do this you may need to pass the extension type and file name.
You may also want to add some validation to limit the file types accepted to only the most common file types like images, .doc, .excel, and whatever you have the library to support the conversion of.
If you want to save files directly to your filesystem using the upload control, you can do so but you will have to exclude the webservice step.
Please also see the discussion in chat for details.

Get all files inside a specific folder in a library with UWP

I'm trying to get all the videos in a specific folder inside the Videos library using UWP, right now I can get all videos inside the Videos library, but I'd like to reduce my results to only those inside the specified folder. My code is this:
Windows.Storage.Search.QueryOptions queryOption = new QueryOptions(CommonFileQuery.OrderByTitle, new string[] {".mp4"});
queryOption.FolderDepth = FolderDepth.Deep;
var files = await KnownFolders.VideosLibrary.CreateFileQueryWithOptions(queryOption).GetFilesAsync();
StorageFile videoToPlay = (files[new Random().Next(0, files.Count)] as StorageFile);
var stream = await videoToPlay.OpenAsync(Windows.Storage.FileAccessMode.Read);
Player.SetSource(stream, videoToPlay.ContentType);
Debug.WriteLine(Player.Source);
How could I access a subfolder named "Videos to Play" and then get all the videos inside that folder? I tried accesing it by using a path like:
string localfolder = Windows.Storage.ApplicationData.Current.LocalFolder.Path;
var array = localfolder.Split('\\');
var username = array[2];
string[] allVideos = System.IO.Directory.GetFiles("C:/Users/" + username + "/Videos/Videos to Play");
But I get access denied even though I already requested access to the Videos library (and the fact that the first example works shows that I actually have access to it).

try
{
var folder = await KnownFolders.VideosLibrary.GetFolderAsync("Videos to Play");
}
catch (FileNotFoundException exc)
{
// TODO: Handle the case when the folder wasn't found on the user's machine.
}
In the folder variable you'll have the reference to the desired folder. Then it's the very same stuff that you already do, but instead of KnownFolders.VideosLibrary folder use this one!

Uploading a file to Sitecore keeping the extension in the DisplayName

As the title says, I'm trying to upload a file to somewhere in my Sitecore tree from the web and I want to keep the file's extension in the DisplayName of the item.
I tried using MediaCreator, but it requires a specific filepath which I do not know when I let people select a file using asp:FileUpload. So that's no good.
I'm now using the asp:FileUpload in combination with a pipeline. When I use Sitecore.Pipelines.Upload.Save from Sitecore.Kernel it works except it removes the extension of the file from the DisplayName. So I created a custom pipeline method that looks like this:
using System;
using System.IO;
using Sitecore.Diagnostics;
using Sitecore.IO;
namespace Sitecore.Pipelines.Upload
{
public class CustomSave
{
public void Process(UploadArgs args)
{
foreach (string key in args.Files)
{
var file = args.Files[key];
if (file != null && file.FileName.Length > 0 && file.ContentLength > 0)
{
var filename = FileUtil.MakePath(args.Folder, Path.GetFileName(file.FileName), '/');
try
{
if (!args.Overwrite)
{
filename = FileUtil.GetUniqueFilename(filename);
}
file.SaveAs(filename);
EventDispatcher.DispatchTrace("File has been uploaded: " + filename);
}
catch (Exception ex)
{
Log.Error("Could not save posted file: " + filename, ex, this);
}
}
}
}
}
}
The problem I now face is that it's trying to save to my C:\ disk instead of the Sitecore database.
Any suggestions on how to write to the correct destination?
The end goal, as explained, is to use the client friendly asp:FileUpload to select a file and upload it to a specific path in Sitecore retaining the file's extension in its DisplayName. If you have another way to do that I'd like to hear about it too.

There is a setting that appends file extension to the item name
"Media.IncludeExtensionsInItemNames". If you can live with that, just turn it on.
Otherwise, I'd add a custom processor into "uiUpload" after "Save" one. At that moment you would already have a list of created media items from "args.UploadedItems". You can do your simple manipulations with items in there.