AWS S3 - move folder with all files problem - c#

I am trying to move data for example;
Source = "Uploads/Photos/" to Destination="Uploads/mytest/"
I am getting error like that but but when i give a specific file this works.
Basically, I want to move folder with all files.
My code is below;
public async Task<MoveResponse> MoveObject(MoveRequest moveRequest)
{
MoveResponse moveResponse = new MoveResponse();
CopyObjectRequest copyObjectRequest = new CopyObjectRequest
{
SourceBucket = moveRequest.BucketName,
DestinationBucket = moveRequest.BucketName + "/" + moveRequest.Destination,
SourceKey = moveRequest.Source,
DestinationKey = moveRequest.Source,
};
var response1 = await client.CopyObjectAsync(copyObjectRequest).ConfigureAwait(false);
if (response1.HttpStatusCode != System.Net.HttpStatusCode.OK)
{
moveResponse.IsError = true;
moveResponse.ErrorMessage = "Files could not moved to destination!";
return moveResponse;
}
return moveResponse;
}

I hope, you are using high level S3 API's.
Check out this sample code
private void uploadFolderToolStripMenuItem_Click(object sender, EventArgs e)
{
string directoryPath = textBoxBasePath.Text + listBoxFolder.SelectedItem.ToString().Replace("[", "").Replace("]", "");
string bucketName = comboBoxBucketNames.Text;
string FolderName = listBoxFolder.SelectedItem.ToString().Replace("[", "").Replace("]", "");
try
{
TransferUtility directoryTransferUtility = new TransferUtility(new AmazonS3Client(AwsAccessKeyID, AwsSecretAccessKey, RegionEndpoint.USEast1));
TransferUtilityUploadDirectoryRequest request = new TransferUtilityUploadDirectoryRequest
{
BucketName = bucketName,
KeyPrefix = FolderName,
StorageClass = S3StorageClass.StandardInfrequentAccess,
ServerSideEncryptionMethod = ServerSideEncryptionMethod.AES256,
Directory = directoryPath,
SearchOption = SearchOption.AllDirectories,
SearchPattern = "*.*",
CannedACL = S3CannedACL.AuthenticatedRead
};
ListMultipartUploadsRequest req1 = new ListMultipartUploadsRequest
{
BucketName = bucketName
};
var t = Task.Factory.FromAsync(directoryTransferUtility.BeginUploadDirectory, directoryTransferUtility.EndUploadDirectory, request, null);
t.Wait();
MessageBox.Show(string.Format("The Directory '{0}' is successfully uploaded", FolderName));
}
catch (Exception ex)
{
MessageBox.Show(ex.Message);
}
finally
{ }
}

Related

Google.Cloud.Dialogflow.V2 Detect Intent not working

I am running this code below but get no answer from the API at the DetectIntent() line. I also tested the DetectIntentsAsync() method instead, but have the same problem.
I also call the bot from nodejs fulfillment and all responses were OK.
Code is below:
private void Button_Click(object sender, RoutedEventArgs e)
{
messageList.Items.Add("User: " + messageText.Text);
try
{
var query = new QueryInput
{
Text = new TextInput
{
Text = messageText.Text,
LanguageCode = "en-us"
}
};
var sessionId = Guid.NewGuid().ToString();
var agent = "...";
var parameters = new JsonCredentialParameters
{
Type = JsonCredentialParameters.ServiceAccountCredentialType,
ClientEmail = "...",
PrivateKey = "..."
};
string json = JsonConvert.SerializeObject(parameters);
var creds = GoogleCredential.FromJson(json);
var channel = new Grpc.Core.Channel(SessionsClient.DefaultEndpoint.Host, creds.ToChannelCredentials());
var client = SessionsClient.Create(channel);
var sessionName = new SessionName(agent, sessionId);
var dialogFlow = client.DetectIntent(
sessionName,
query
);
channel.ShutdownAsync();
messageList.Items.Add("Bot: " + dialogFlow);
}
catch (Exception ex)
{
messageList.Items.Add("error: " + ex.Message);
}
}

Upload file in asp.net and angular 7 null

I upload image from angular to .net core but the file input I receive always null , with the postman , it's receive exactly. I don't know why ?
Can you help me how to solve it?
Thank you so much !
[HttpPost("upload-file")]
public async Task<IActionResult> UploadFile(IFormFile file)
{
try
{
//var file = Request.Form.Files[0];
var rnd = FileHelper.RandomNumber();
//var file = file;
var folder = "Folder";
var index = file.FileName.LastIndexOf('.');
var onlyName = file.FileName.Substring(0, index);
var fileName = onlyName + rnd;
var extension = Path.GetExtension(file.FileName);
var folderThumb = folder + "Thumbnail";
var fileNameThumb = fileName + "Thumbnail";
var t1 = Task.Run(() => FileHelper.SaveFile(folder, fileName, file));
await Task.WhenAll(t1);
var path = new MediaResponseModel()
{
Url = Url.Action("GetFile", "Media", new { folder = folder, fileName = string.Format("{0}{1}", fileName, extension) }, Request.Scheme),
UrlThumbnail = Url.Action("GetFile", "Media", new { folder = folderThumb, fileName = string.Format("{0}{1}", fileNameThumb, extension) }, Request.Scheme),
Title = onlyName
};
return Ok(path);
}
catch (System.Exception e)
{
throw e;
}
}
handleFileInput(files: FileList) {
const file = files.item(0);
this.fileToUpload = {};
this.fileToUpload.data = files.item(0);
this.fileToUpload.fileName = files.item(0).name;
}
onUpload() {
const formData = new FormData();
formData.append('file', this.fileToUpload.data, this.fileToUpload.name);
this.http.post('http://localhost:56000/api/media/upload-file', formData).subscribe(res => console.log(res));
}

How to fix Amazon.Rekognition.AmazonRekognitionException?

I am getting AmazonRekognitionException as below when trying to run CompareFacesResponse, I am stuck, what should I do or check?
Amazon.Rekognition.AmazonRekognitionException: The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details. ---> Amazon.Runtime.Internal.HttpErrorResponseException: Exception of type 'Amazon.Runtime.Internal.HttpErrorResponseException' was thrown
AWS credentials access key and secret are checked and correct
public static async Task<Tuple<bool, string>> Rekognition_Compare_Faces(string _source, string _target, string _bucketName)
{
const string HOSTNAME = "https://rekognition.ap-southeast-1.amazonaws.com/";
const string ACCESS_KEY = "my_access_key";
const string ACCESS_SECRET = "my_secret_key";
float _similarityThreshold = 70F;
bool _ret = false;
string _confidence = string.Empty;
try
{
AmazonRekognitionConfig _config = new AmazonRekognitionConfig();
_config.ServiceURL = HOSTNAME + _bucketName;
AmazonRekognitionClient _rekognitionClient = new AmazonRekognitionClient(ACCESS_KEY, ACCESS_SECRET, _config);
Amazon.Rekognition.Model.Image _imageSource = new Amazon.Rekognition.Model.Image();
Amazon.Rekognition.Model.Image _imageTarget = new Amazon.Rekognition.Model.Image();
Amazon.Rekognition.Model.S3Object _s3_source = new Amazon.Rekognition.Model.S3Object { Bucket = _bucketName, Name = _source };
Amazon.Rekognition.Model.S3Object _s3_target = new Amazon.Rekognition.Model.S3Object { Bucket = _bucketName, Name = _target };
CompareFacesRequest _compareFacesRequest = new CompareFacesRequest()
{
SourceImage = new Amazon.Rekognition.Model.Image
{
S3Object = new Amazon.Rekognition.Model.S3Object
{
Bucket = HOSTNAME + _bucketName,
Name = _source
}
},
TargetImage = new Amazon.Rekognition.Model.Image
{
S3Object = new Amazon.Rekognition.Model.S3Object
{
Bucket = HOSTNAME + _bucketName,
Name = _target
}
},
SimilarityThreshold = _similarityThreshold
};
// IT THROWN HERE!!
CompareFacesResponse _compareFacesResponse = await _rekognitionClient.CompareFacesAsync(_compareFacesRequest);
// Display results
foreach (CompareFacesMatch match in _compareFacesResponse.FaceMatches)
{
ComparedFace face = match.Face;
BoundingBox position = face.BoundingBox;
_confidence = match.Similarity.ToString(AppSettings.Decimal_Number_Format) + "%";
_ret = true;
}
}
catch (Exception ex) { await ClsMain.SaveLog("AWS.Compare_Faces: " + ex.ToString()); }
finally { }
return await Task.FromResult(new Tuple<bool, string>(_ret, _confidence));
}
has anybody experience on this?
thanks a lot in advance
Regards
Don
I had the same error.
I tried adding RegionEndpoint = RegionEndpoint.EUWest1 to my AmazonRekognitionConfig so it now looks like this:
var config = new AmazonRekognitionConfig
{
ServiceURL = $"https://rekognition.ap-southeast-1.amazonaws.com/{_awsSettings.BucketName}",
RegionEndpoint = RegionEndpoint.EUWest1
};
var client = new AmazonRekognitionClient(_awsSettings.AccessKey, _awsSettings.Secret, config);
This fixed the problem for me.

How can i run .cmd file from http trigger azure function

I want to run .cmd file using azure function. I want to run this in background process instead of main process of azure function.
I have already saved the cmd file on azure function platform using Kudu Editor. I can run this locally but after deploying its not working at all (I am also not getting any error).
string cmdFileLocation = #"D:\home\jobs.cmd";
Process proc = new Process();
ProcessStartInfo info = new ProcessStartInfo();
try
{
info.FileName = cmdFileLocation;
info.Arguments = name;
info.WindowStyle = ProcessWindowStyle.Minimized;
info.UseShellExecute = false;
proc.StartInfo = info;
info.RedirectStandardOutput = true;
info.RedirectStandardError = true;
proc.Start();
proc.WaitForExit();
}
catch (Exception ex)
{
log.LogInformation("Exception Occurred :{0},{1}", ex.Message, ex.StackTrace.ToString());
}
For testing there is curl command in cmd file. The curl will trigger on local machine using azure function as I can see the request coming (https://webhook.site) but after deplying nothing happens.
Here is an easy way of getting any .exe/.cmd running as a web service. You just specify the input parameters to your exe/cmd in a configuration file. You can use binary files as inputs to the exe by specifying a URL to download it from.
Here's what the configuration file looks like
{
"name": "consoleAppToFunctions",
"input": {
"command": "ffmpeg.exe -i {inputFile} {output1}",
"arguments": {
"inputFile": {
"url": "https://1drv.ms/v/<link-to-file>"
//binary file input
},
"output1": {
"localfile": "out.mp3"
//stored in a temp folder
}
}
},
"output": {
"folder": "outputFolder",
"binaryFile": {
"returnFile": "*.mp3",
"returnFileName": "yourFile.mp3"
}
}
}
Here is the AZURE FUNCTION CODE for the same:
#r "Newtonsoft.Json"
using System.Net;
using Newtonsoft.Json;
using System.IO;
using System.Diagnostics;
//Code from https://github.com/Azure-Samples/functions-dotnet-migrating-console-apps/edit/master/code/run.csx
//Written by Ambrose http://github.com/efficientHacks and Murali http://github.com/muralivp
public class ExeArg
{
public string Name { get; set; }
public string Type { get; set; }
public string Value { get; set; }
}
public static async Task<HttpResponseMessage> Run(HttpRequestMessage req, TraceWriter log)
{
log.Info("C# HTTP trigger function processed a request.");
string localPath = req.RequestUri.LocalPath;
string functionName = localPath.Substring(localPath.LastIndexOf('/')+1);
var json = File.ReadAllText(string.Format(#"D:\home\site\wwwroot\{0}\FunctionConfig.json",functionName));
var config = JsonConvert.DeserializeObject<dynamic>(json);
var functionArguments = config.input.arguments;
var localOutputFolder = Path.Combine(#"d:\home\data", config.output.folder.Value, Path.GetFileNameWithoutExtension(Path.GetTempFileName()));
Directory.CreateDirectory(localOutputFolder);
var workingDirectory = Path.Combine(#"d:\home\site\wwwroot", functionName + "\\bin");
Directory.SetCurrentDirectory(workingDirectory);//fun fact - the default working directory is d:\windows\system32
var command = config.input.command.Value;
var argList = new List<ExeArg>();
//Parse the config file's arguments
//handle file system, local file etc. and construct the input params for the actual calling of the .exe
foreach (var arg in functionArguments)
{
var exeArg = new ExeArg();
exeArg.Name = arg.Name;
var value = (Newtonsoft.Json.Linq.JObject)arg.Value;
var property = (Newtonsoft.Json.Linq.JProperty)value.First;
exeArg.Type = property.Name;
exeArg.Value = property.Value.ToString();
var valueFromQueryString = await getValueFromQuery(req, exeArg.Name);
log.Info("valueFromQueryString name=" + exeArg.Name);
log.Info("valueFromQueryString val=" + valueFromQueryString);
if(!string.IsNullOrEmpty(valueFromQueryString))
{
exeArg.Value = valueFromQueryString;
log.Info(exeArg.Name + " " + valueFromQueryString);
}
if(exeArg.Type.ToLower() == "localfile" || exeArg.Type.ToLower() == "localfolder")
{
exeArg.Value = Path.Combine(localOutputFolder, exeArg.Value);
exeArg.Type = "string";
}
if(string.IsNullOrEmpty(exeArg.Value))
{
//throw exception here
}
argList.Add(exeArg);
}
//call the exe
Dictionary<string, string> paramList = ProcessParameters(argList, localOutputFolder);
foreach (string parameter in paramList.Keys)
{
command = command.Replace(parameter, paramList[parameter]);
}
string commandName = command.Split(' ')[0];
string arguments = command.Split(new char[] { ' ' }, 2)[1];
log.Info("the command is " + command);
log.Info("the working dir is " + workingDirectory);
Process p = new Process();
p.StartInfo.UseShellExecute = false;
p.StartInfo.RedirectStandardOutput = true;
p.StartInfo.FileName = commandName;
p.StartInfo.Arguments = arguments;
p.Start();
string output = p.StandardOutput.ReadToEnd();
p.WaitForExit();
File.WriteAllText(localOutputFolder+"\\out.txt",output);
//handle return file
log.Info("handling return file localOutputFolder=" + localOutputFolder);
string outputFile = config.output.binaryFile.returnFile.Value;
string outputFileName = config.output.binaryFile.returnFileName.Value;
var path = Directory.GetFiles(localOutputFolder, outputFile)[0];
log.Info("returning this file " + path);
var result = new FileHttpResponseMessage(localOutputFolder);
var stream = new FileStream(path, FileMode.Open, FileAccess.Read);
result.Content = new StreamContent(stream);
result.Content.Headers.ContentType = new System.Net.Http.Headers.MediaTypeHeaderValue("application/octet-stream");
result.Content.Headers.ContentDisposition = new System.Net.Http.Headers.ContentDispositionHeaderValue("attachment")
{
FileName = outputFileName
};
return result;
}
private static Dictionary<string, string> ProcessParameters(List<ExeArg> arguments, string outputFolder)
{
Dictionary<string, string> paramList = new Dictionary<string, string>();
foreach (var arg in arguments)
{
switch (arg.Type)
{
case "url":
string downloadedFileName = ProcessUrlType((string)arg.Value, outputFolder);
paramList.Add("{" + arg.Name + "}", downloadedFileName);
break;
case "string":
paramList.Add("{" + arg.Name + "}", arg.Value.ToString());
break;
default:
break;
}
}
return paramList;
}
//you can modify this method to handle different URLs if necessary
private static string ProcessUrlType(string url, string outputFolder)
{
Directory.CreateDirectory(outputFolder);
string downloadedFile = Path.Combine(outputFolder, Path.GetFileName(Path.GetTempFileName()));
//for oneDrive links
HttpWebRequest webRequest = (HttpWebRequest)HttpWebRequest.Create(url);
webRequest.AllowAutoRedirect = false;
WebResponse webResp = webRequest.GetResponse();
webRequest = (HttpWebRequest)HttpWebRequest.Create(webResp.Headers["Location"].Replace("redir", "download"));
webResp = webRequest.GetResponse();
string fileUrl = webResp.Headers["Content-Location"];
WebClient webClient = new WebClient();
webClient.DownloadFile(fileUrl, downloadedFile);
return downloadedFile;
}
private async static Task<string> getValueFromQuery(HttpRequestMessage req, string name)
{
// parse query parameter
string value = req.GetQueryNameValuePairs()
.FirstOrDefault(q => string.Compare(q.Key, name, true) == 0)
.Value;
//if not found in query string, look for it in the body (json)
// Get request body
dynamic data = await req.Content.ReadAsAsync<object>();
// Set name to query string or body data
value = value ?? data?[name];
return value;
}
//this is to delete the folder after the response
//thanks to: https://stackoverflow.com/a/30522890/2205372
public class FileHttpResponseMessage : HttpResponseMessage
{
private string filePath;
public FileHttpResponseMessage(string filePath)
{
this.filePath = filePath;
}
protected override void Dispose(bool disposing)
{
base.Dispose(disposing);
Content.Dispose();
Directory.Delete(filePath,true);
}
}
Here you can find more on this. Hope it helps.

c# sync only new files to amazon s3

I'm uploading all my local files to s3 using this code:
static string bucketName = "s3bucket";
static string directoryPath = #"C:\data\";
private void btnUpload_Click(object sender, EventArgs e) {
try {
TransferUtility directoryTransferUtility =
new TransferUtility(new AmazonS3Client(Amazon.RegionEndpoint.USWest2));
TransferUtilityUploadDirectoryRequest request =
new TransferUtilityUploadDirectoryRequest {
BucketName = bucketName,
Directory = directoryPath,
SearchPattern = "*.xml"
};
directoryTransferUtility.UploadDirectory(request);
MessageBox.Show("Upload completed");
} catch (AmazonS3Exception ex) {
MessageBox.Show(ex.Message);
}
}
If I run the code again all files are re-uploaded to s3 and that's a bad idea if let's say we have 1000+ files in our local directory.
I know I can compare file by file because aws stores the md5 of each file. So my question is can I do this with a command that comes preinstalled? Do I have to recursively compare file by file? If the sync command exists on awscli bundle (aws s3 sync ./sourceDir s3://bucketname/) does it exists on c# as well?
All the files will be replaced. S3 docs , but if you still want to check files you can use some function like this
ListObjectsRequest request = new ListObjectsRequest()
{
BucketName = "Your Bucket name",
Delimiter = "/",
Prefix = "location"
};
public bool CheckFile(ListObjectsRequest request)
{
bool res = false;
try
{
ListObjectsResponse response = s3client.ListObjects(request);
if (response.S3Objects != null && response.S3Objects.Count > 0)
{
S3Object o = response.S3Objects.Where(x => x.Size != 0 && x.LastModified > DateTime.Now.AddHours(-24)).FirstOrDefault();
if (o != null)
{
//you can use thes fields
// o.Key; //o.Size, //o.LastModified
res = true;
}
}
else
{
res = false;
}
}
catch (Exception)
{
throw;
}
return res;
}

Categories