We Keep Coding

Store temp file in .net lambda and then publish to s3Bucket

I am generating a .csv file for further storing in a s3 bucket using .net c# Lambda function.
This is the process i follow:
Generate the .csv and store it in /tmp/ folder of the lambda function execution.
In this step im not sure if it is really saving the file in that path.
//filepath = #"/tmp/test.csv"
try
{
using (System.IO.StreamWriter file = new System.IO.StreamWriter(#filepath, true))
{
file.WriteLine(ID + "," + workItemType + "," + title + "," + assignedTo + "," + state + "," + iterationPath);
Console.WriteLine("Successfully added");
}
}
catch (Exception ex)
{
throw new ApplicationException(" somethings wrong: ", ex);
}
Upload the file to s3 bucket.
try
{
await client.PutObjectAsync(new Amazon.S3.Model.PutObjectRequest
{
BucketName = "mys3bucket",
Key = "test.csv",
ContentType = #"/tmp/test.csv"
});
await Task.CompletedTask;
}
catch (Exception ex)
{
Console.WriteLine("Exception in PutS3Object:" + ex.Message); ;
}
In this last step i get this error message:
Exception in PutS3Object:The format of value '\tmp\test.csv' is invalid.
What i am doing wrong?

You need to send the data to include in the csv file:
await client.PutObjectAsync(new Amazon.S3.Model.PutObjectRequest
{
BucketName = "mys3bucket",
Key = "test.csv",
ContentBody = DATAINSTRINGFORMAT,
ContentType = #"text/csv"
});
or as filepath to send:
await client.PutObjectAsync(new Amazon.S3.Model.PutObjectRequest
{
BucketName = "mys3bucket",
Key = "test.csv",
FilePath = FILEPATHONYOURTEMPFOLDER,
ContentType = #"text/csv"
});

C# code fetching cached version of data from FTP

In my project I am downloading few files from a ftp created over IIS7 and also over linux server and saving it to my Appdata/Roamingfolder. Problem is coming when either I modify the content of the csv file or simply deleting the old file and replacing it with new file with same name but modified content.
Every time i have to rename that file and downloading the renamed file works. This indicates its downloading some cached image of the file which i am unable to locate either on my local system as well as over ftp server.
public static bool FTPFileDownload(string strFolderName, string
pathToStore, bool blIsSingleFile = true, string strFileType = "")
{
try
{
if (!Directory.Exists(pathToStore))
{
// Try to create the directory.
DirectoryInfo di = Directory.CreateDirectory(pathToStore);
}
FtpWebRequest request = (FtpWebRequest)WebRequest.Create(ConfigurationManager.AppSettings["FTPUrl"].ToString() + strFolderName);
request.Credentials = new NetworkCredential(ConfigurationManager.AppSettings["FTPUser"].ToString(), ConfigurationManager.AppSettings["FTPPassword"].ToString());
request.Method = WebRequestMethods.Ftp.ListDirectory;
request.Proxy = null;
FtpWebResponse response = (FtpWebResponse)request.GetResponse();
StreamReader streamReader = new StreamReader(response.GetResponseStream());
System.Collections.Generic.List<string> directories = new System.Collections.Generic.List<string>();
string line = streamReader.ReadLine();
while (!string.IsNullOrEmpty(line))
{
//If extension is available match with extension and add.
bool blAddFile = false;
if (!String.IsNullOrEmpty(strFileType))
{
string strExt = Path.GetExtension(ConfigurationManager.AppSettings["FTPUrl"].ToString() + line).Remove(0, 1);
if (strExt.ToLower() == strFileType.ToLower())
blAddFile = true;
}
else
blAddFile = true;
if (blAddFile)
{
directories.Add(line);
}
line = streamReader.ReadLine();
}
streamReader.Close();
using (WebClient ftpClient = new WebClient())
{
ftpClient.Credentials = new System.Net.NetworkCredential(ConfigurationManager.AppSettings["FTPUser"].ToString(), ConfigurationManager.AppSettings["FTPPassword"].ToString());
for (int i = 0; i <= directories.Count - 1; i++)
{
if (directories[i].Contains("."))
{
string path = ConfigurationManager.AppSettings["FTPUrl"].ToString() + strFolderName
+ (blIsSingleFile ? "" : "/" + directories[i].ToString());
string trnsfrpth = pathToStore + directories[i].ToString();
ftpClient.DownloadFile(path, trnsfrpth);
}
}
return true;
}
}
catch (Exception ex)
{
FileLogger.logMessage(ex.Message);
if (FileLogger.IsDebuggingLogEnabled)
{
FileLogger.HandleError("FTPFileDownload", ex, "Common Helper Error 4:");
}
return false;
}
}
I don't know what is going wrong with it. Either my code is wrong or the settings or environment over ftp server.
Please suggest.

Failed to upload files to google drive intermittently

Using the Google Drive API to upload files. Multiple same console app is running at the same time to upload different parts of files in the folder and their files don't overlap. This hits the quota limit. Then a while-try-catch is implemented to re-execute the query whenever it throws the exception because of the quota limit. The list and create directory method works well but not the upload (i.e. create) method. Some files are missing when i checked from the Google Drive site
Tried using FileStream instead of MemoryStream but it seems not related.
public static Google.Apis.Drive.v3.Data.File uploadFile(DriveService _service, string _uploadFile)
{
bool again = true;
string[] p = _uploadFile.Split('/');
if (System.IO.File.Exists("C:/"+_uploadFile))
{
Google.Apis.Drive.v3.Data.File body = new Google.Apis.Drive.v3.Data.File();
body.Name = System.IO.Path.GetFileName(p[p.Length-1]);
body.Description = "";
body.MimeType = GetMimeType("C:/"+_uploadFile);
body.Parents = new List<string>() { ID };
// File's content.
System.IO.FileStream stream = new System.IO.FileStream("C:/" + _uploadFile, FileMode.Open);
try
{
FilesResource.CreateMediaUpload request = _service.Files.Create(body, stream, GetMimeType("C:/" + _uploadFile));
while (again)
{
try
{
request.Upload();
again = false;
}
catch (Exception e)
{
Console.WriteLine("uploadFile: "+p[p.Length-1]);
}
}
return body;
}
catch (Exception e)
{
Console.WriteLine("An error occurred: " + e.Message);
return null;
}
}
else
{
Console.WriteLine("File does not exist: " + _uploadFile);
return null;
}
}

Return File in net Core is corrupted when using FileStreamResult

I have a method that returns a FileStreamResult to download a file created by NPOI, an Excel file. In the method i use a sample to save locally the file in wwwroot with FileStream and it works, but when i need to return the file and download in front end, the file downloaded is corrupted.
The file is open, and contains the columns that i write.
The file created locally is open normally.
This is the method:
public IActionResult DescargarRepEstacion()
{
if (!ModelState.IsValid) {
return StatusCode(400, ModelState);
}
try
{
var fullPath = Path.Combine(_hostingEnvironment.WebRootPath, "Formatos", "Formato Estacion.xls");
var workBook = new HSSFWorkbook();
var sheet = workBook.CreateSheet("Formato Estación");
var rowIndex = 0;
var row = sheet.CreateRow(rowIndex);
row.CreateCell(0).SetCellValue("ID");
row.CreateCell(1).SetCellValue("DESTINO");
row.CreateCell(2).SetCellValue("PERMISO");
row.CreateCell(3).SetCellValue("TERMINAL");
rowIndex++;
var ms = new MemoryStream();
workBook.Write(ms);
using (var file = new FileStream(fullPath, FileMode.Create)) {
workBook.Write(file);
}
ms.Position = 0;
return new FileStreamResult(ms, "application/vnd.ms-excel");
}
catch (Exception ex) {
return StatusCode(500, ex.Message.ToString());
}
}
In the front end:
axios.get("Formatos/DescargarEstacion", {
headers: {
Authorization: "Bearer " + localStorage.getItem("token"),
'Content-Type': 'application/json',
'Accept':'application/vnd.ms-excel',
}
}).then(response => {
this.loadingDownloadEstacion = false;
download(response.data, 'Formato Estacion.xls');
console.log(response.data);
// const url = window.URL.createObjectURL(new Blob([response.data]));
// const link = document.createElement('a');
// link.href = url;
// link.setAttribute('download', 'Formato Estaciones.xls');
// document.body.appendChild(link);
// link.click();
}).catch(error => {
this.loadingDownloadEstacion = false;
console.log(error.response);
this.snackbar = true;
this.textSnackbar = error.response.data;
})
What am i doing wrong?
UPDATE WHIT SCREENSHOT:

GoogleDrive C# SDK: Too much items in FileList

When I retrieve all Files (and Folders) of my GoogleDrive Account I should get something like 1500 List elements back, but I get a bit more than 3000 back. I looked into the List and found that some files are 2-3 times in it. Why is that?
Here is the code I use to retrieve the files:
public async Task<List<File>> RetrieveAllFilesAsList(DriveService service, string query = null)
{
List<File> result = new List<File>();
FilesResource.ListRequest request = service.Files.List();
if (query != null)
{
request.Q = query;
}
do
{
try
{
FileList files = await request.ExecuteAsync();
result.AddRange(files.Items);
request.PageToken = files.NextPageToken;
}
catch (Exception e)
{
Console.WriteLine("An error occurred (from RetrieveAllFilesAsList): " + e.Message);
request.PageToken = null;
}
}
while (!String.IsNullOrEmpty(request.PageToken));
return result;
}
Update1:
public async Task<List<File>> RetrieveAllFilesAsList(DriveService service, string query = null)
{
List<File> result = new List<File>();
FilesResource.ListRequest request = service.Files.List();
request.MaxResults = 1000;
if (query != null)
{
request.Q = query + " AND trashed=false";
}
else
{
request.Q = "trashed=false";
}
do
{
try
{
FileList files = await request.ExecuteAsync();
result.AddRange(files.Items);
request.PageToken = files.NextPageToken;
}
catch (Exception e)
{
Console.WriteLine("An error occurred (from RetrieveAllFilesAsList): " + e.Message);
request.PageToken = null;
}
}
while (!String.IsNullOrEmpty(request.PageToken));
int i;
for (i = 0; i < result.Count; i++ )
{
System.IO.File.AppendAllText(#"C:\Users\carl\Desktop\log.txt", result[i].Id + "\t" + result[i].Title + "\t" + result[i].ExplicitlyTrashed.ToString() + "\r\n");
}
// prints 3120 Lines
System.IO.File.AppendAllText(#"C:\Users\carl\Desktop\log.txt", "" + i + Environment.NewLine);
//Count = 3120
System.IO.File.AppendAllText(#"C:\Users\carl\Desktop\log.txt", "" + result.Count);
return result;
}
Word failed to give me the right the linecount, so I did it over my Function.
But I can find the FileId 2-3 times in the File.

I cannot write on the comments yet, so according to the API
from google
"Note: This method returns all files by default. This includes files with trashed=true in the results. Use the trashed=false query parameter to filter these from the results."
So can you check what url of the rest api is actually being called? It seems you need to put some filters on the List method.