Question:
How can I tell my backup tool to download all the files it recorded in fileids?
The method I'm using is C#/.NET https://developers.google.com/drive/v3/web/manage-downloads#examples
I'll spare the boring details and say that part of my program logs in Once as each user (well, using the Apps Service API), grabs all their files' fileIds and records them to a flat text file. My program then opens that flat text file and begins downloading each fileId recorded for that user, but the problem is: it's soooo slow because it opens a new connection for a file, waits for the file to finish, then gets a new fileid and starts the whole process over again. It's not very efficient.
Google's example, which I copied pretty much Verbatim (I modified the vars a little bit by immediately grabbing and exporting their mimetype, so the first 3 lines are moot):
var fileId = "0BwwA4oUTeiV1UVNwOHItT0xfa2M";
var request = driveService.Files.Get(fileId);
var stream = new System.IO.MemoryStream();
// Add a handler which will be notified on progress changes.
// It will notify on each chunk download and when the
// download is completed or failed.
request.MediaDownloader.ProgressChanged +=
(IDownloadProgress progress) =>
{
switch(progress.Status)
{
case DownloadStatus.Downloading:
{
Console.WriteLine(progress.BytesDownloaded);
break;
}
case DownloadStatus.Completed:
{
Console.WriteLine("Download complete.");
break;
}
case DownloadStatus.Failed:
{
Console.WriteLine("Download failed.");
break;
}
}
};
request.Download(stream);
Is there Any way I can streamline this so that my program can download all the files it knows for the user in one big handshake, vs reading a fileid individually, then opening a session, exporting, downloading, closing, then doing the same exact thing for the next file? Hope this makes sense.
Thank you for any help ahead of time!
--Mike
---EDIT---
I wanted to add more details so that hopefully what I'm looking to do makes more sense:
So what's happening in the following code is: I am creating a "request" that will let me export the filetype (which I have from the flat text file as the fileId[0], and the "mimetype" which is in the array as fileId[1].)
What's killing the speed of the program is having to build the "BuildService" request each time for each file.
foreach (var file in deltafiles)
{
try
{
if (bgW.CancellationPending)
{
stripLabel.Text = "Backup canceled!";
e.Cancel = true;
break;
}
DateTime dt = DateTime.Now;
string[] foldervalues = File.ReadAllLines(savelocation + "folderlog.txt");
cnttototal++;
bgW.ReportProgress(cnttototal);
// Our file is a CSV. Column 1 = file ID, Column 2 = File name
var values = file.Split(',');
string fileId = values[0];
string fileName = values[1];
string mimetype = values[2];
mimetype = mimetype.Replace(",", "_");
string folder = values[3];
int foundmatch = 0;
int folderfilelen = foldervalues.Count();
fileName = fileName.Replace('\\', '_').Replace('/', '_').Replace(':', '_').Replace('!', '_').Replace('\'', '_').Replace('*', '_').Replace('#', '_').Replace('[', '_').Replace(']', '_');
var request = CreateService.BuildService(user).Files.Export(fileId, mimetype);
//Default extensions for files. Not sure what this should be, so we'll null it for now.
string ext = null;
// Things get sloppy here. The reason we're checking MimeTypes
// is because we have to export the files from Google's format
// to a format that is readable by a desktop computer program
// So for example, the google-apps.spreadsheet will become an MS Excel file.
if (mimetype == mimeSheet || mimetype == mimeSheetRitz || mimetype == mimeSheetml)
{
request = CreateService.BuildService(user).Files.Export(fileId, exportSheet);
ext = ".xls";
}
if (mimetype == mimeDoc || mimetype == mimeDocKix || mimetype == mimeDocWord)
{
request = CreateService.BuildService(user).Files.Export(fileId, exportDoc);
ext = ".docx";
}
if (mimetype == mimePres || mimetype == mimePresPunch)
{
request = CreateService.BuildService(user).Files.Export(fileId, exportPres);
ext = ".ppt";
}
if (mimetype == mimeForm || mimetype == mimeFormfb || mimetype == mimeFormDrawing)
{
request = CreateService.BuildService(user).Files.Export(fileId, exportForm);
ext = ".docx";
}
// Any other file type, assume as know what it is (which in our case, will be a txt file)
// apply the mime type and carry on.
string dest = Path.Combine(savelocation, fileName + ext);
var stream = new System.IO.FileStream(dest, FileMode.Create, FileAccess.ReadWrite);
int oops = 0;
// Add a handler which will be notified on progress changes.
// It will notify on each chunk download and when the
// download is completed or failed.
request.MediaDownloader.ProgressChanged +=
(IDownloadProgress progress) =>
{
switch (progress.Status)
{
case DownloadStatus.Downloading:
{
throw new Exception("File may be corrupted.");
break;
}
case DownloadStatus.Completed:
{
Console.WriteLine("Download complete.");
break;
}
case DownloadStatus.Failed:
{
oops = 1;
logFile.WriteLine(fileName + " could not be downloaded. Possible Google draw/form OR bad name.\n");
break;
}
}
};
request.Download(stream);
stream.Close();
stream.Dispose();
Is there any way I could streamline this process so I don't have to build the drive service Every time I want to download a file? The flat text file the program reads looks similar to
FILEID,ACTUAL FILE NAME,MIMETYPE
So is there any way I could cut out the middle man and feed the request.Download method without constantly reminding the "foreach" statement to export the file type as a file system-readable file? (good grief, sorry, I know this sounds like a lot.)
Any pointers would be great!!
You might want to try the tutorial : Google Drive API with C# .net – Download. This is a much simpler code to download a file. Also there are other factors like intermittent internet connect that may affect the ETA of downloading the file.
Code Sample :
/// Download a file
/// Documentation: https://developers.google.com/drive/v2/reference/files/get
///
/// a Valid authenticated DriveService
/// File resource of the file to download
/// location of where to save the file including the file name to save it as.
///
public static Boolean downloadFile(DriveService _service, File _fileResource, string _saveTo)
{
if (!String.IsNullOrEmpty(_fileResource.DownloadUrl))
{
try
{
var x = _service.HttpClient.GetByteArrayAsync(_fileResource.DownloadUrl );
byte[] arrBytes = x.Result;
System.IO.File.WriteAllBytes(_saveTo, arrBytes);
return true;
}
catch (Exception e)
{
Console.WriteLine("An error occurred: " + e.Message);
return false;
}
}
else
{
// The file doesn't have any content stored on Drive.
return false;
}
}
Using _service.HttpClient.GetByteArrayAsync we can pass it the download url of the file we would like to download. Once the file is download its a simple matter of wright the file to the disk.
Hope this helps!
This isn't an answer as much as it is a work around, even then it's only half the answer (for right now.) I threw my gloves off and played dirty.
First, I updated my nuget google api packages to the latest version available today inside my VS project, then went to https://github.com/google/google-api-dotnet-client, forked/cloned it, changed the Google.Apis.Drive.v3.cs file (which compiles to google.apis.drive.v3.dll) so that the mimetype is no longer read only (it can do get; and set;, when by default, it only allowed get).
Because I already knew the mime types, I am able to force assign the mime type now to the request and go on with my life, instead of having to build the client service, connect, only to export the file type that I already know it is.
It's not pretty, not how it should be done, but this was really bothering me!
Going back to #Mr.Rebot, I thank you again for your help and research! :-)
Related
I have a requirement where
when uploading the files to the pick-up folder, files will be uploaded
with a .tmp (or)._ (or) .filepart extensions and after successful
upload files will be renamed to the original file name.
This is required to avoid any partial pick-up of .xml files by settings on SFTP folder side.
For eg. Upload with .xml.tmp and after successful upload, rename the files to .xml
Any idea on how to achieve this in MVC, C#.
I prefer to do this in a separate folder entirely. And then do a move to the pickup folder.
Then renaming is not required.
private bool IsFileLocked()
{
try
{
FileStream fs = File.OpenWrite(FilePath);
fs.Close();
return false;
}
catch (Exception)
{
Console.WriteLine("File locked: " + FileName);
return true;
}
}
To check if the file is locked prior to attempting to send, might also work, or in conjunction.
I was talking about generating a local file first, once its completely done being written, simply use the File.Move() method, so you move the newly generated file from its "safe" folder, into the pickup folder, that the SFTP is continually looking for files in.
If it is picking up a file you are receiving, then it's just the check prior to attempting to do anything with it.
First of all, once you receive the file stream from the post, the upload is "already" successful (most likely). Therefore, the moment you have the data from the post, you should already be good to write it. The only point I can remotely see here is that, the remote process either checks .xml files constantly so let's say if the .xml file is quite large, and let's assume (which wont be the case) it takes a while for you to write the stream to the remote destination, they do not want to check just part of the xml, they need all of it. If that is the case, something like the following should work (modify it for your needs);
[HttpPost]
public ActionResult Upload()
{
if (Request.Files.Count < 1)
{
ViewBag.Result = "No files were provided";
return PartialView("Error");
}
foreach (string F in Request.Files)
{
var FInfo = Request.Files[F];
var TemporaryFileName = $"{FInfo.FileName}.tmp";
try
{
using (var FStream = new FileStream(TemporaryFileName, FileMode.Create, FileAccess.Write))
{
FInfo.InputStream.CopyTo(FStream);
}
}
catch (Exception e)
{
ViewBag.Result = e.Message;
return PartialView("Error");
}
finally
{
System.IO.File.Move(TemporaryFileName, $"{FInfo.FileName}");
}
}
ViewBag.Result = "Files have been uploaded";
return View();
}
I am trying to create a file for the further write to and read from.
I use Directory.CreateDirectory and File.Create but neither path nor file are being created.
On the Page which part I show here below I check if File exists, and if not, I create a File. On Second Page (that I dont show here) I add new lines to the file using StreamWrite. When saved, first Page comes to focus again and lists the content of the File(only one row in this study).
Here is my code for the part in question:
public async Task ReadFileAsync()
{
string directoryName = Path.GetDirectoryName(#"C:\Users\...\DataBase\");
Task.Run(async() => Directory.CreateDirectory(directoryName));
Task.Run(async() => File.Create(directoryName + "ProductsDatabase.txt"));
//code for reading from file
string path = (directoryName + "ProductsDatabase.txt");
using (StreamReader ProductsDatabaseRead = new StreamReader(File.OpenRead(path)))
{
ProductOneTextBlock.Text = ProductsDatabaseRead.ReadLine();
}
if (ProductOneTextBlock.Text == "")
{
ProductOneTextBlock.Text = "Nothing to show";
}
}
The file and folder are not being created.
I don't get any error.
I tried also different folders on the drive in case if there was READ ONLY folder in solution folder. No difference.
Anyone could help?
(I found many threads about this problem but here I cannot resolve it with none of the solutions.
Physical file is not being created.
When I attempt to write to it (from another page) I get error that the file could not be found(because it is not there indeed).
It seems that program loses itself somewhere between
Task.Run(async() => Directory.CreateDirectory(directoryName));
Task.Run(async() => File.Create(directoryName + "ProductsDatabase.txt"));
and:
using (StreamReader ProductsDatabaseRead = new StreamReader(File.OpenRead(path)))
{
ProductOneTextBlock.Text = ProductsDatabaseRead.ReadLine();
}
, as TextBlock is not being updated even if ProductsDatabaseRead is null.
If I put
ProductOneTextBlock.Text = "Nothing to show";
a the begining of the method, TextBlock gets updated.
SO, why the
using (StreamReader ProductsDatabaseRead = new StreamReader(File.OpenRead(path)))
does not work?
You're not waiting for Task.Run to complete. Your directory creation, file creation and attempt to open a "as you think newly created file" are out of order. That's why you're probably not able to open a file (it still does not exist at this point).
Task.Run returns a task that will be completed when the work is done. You need to wait for completion.
public void ReadFile()
{
string folderPath = #"C:\Users\patri\source\repos\DietMate\";
string fileName = "ProductsDatabase.txt";
string fullPath = Path.Combine(folderPath, fileName);
//insert code to check whether file exists.
// use Exists()
if (!Directory.Exists(folderPath))
{
Directory.CreateDirectory(folderPath);
File.Create(fullPath);
}
//if yes, follow with code below
//insert code for reading from file
using (StreamReader ProductsDatabaseRead = new StreamReader(fullPath))
{
ProductTest.Text = ProductsDatabaseRead.ReadLine();
}
}
**I created one excel file and change the extension type=”.docx”.I try to upload.it is uploading but if content and extension type both are different. then i need to block that one i tried few codes to avoid this with magic number and mime type but magic number is same for word and excel and zip also if i change the extension then mime type also changed. So i need to identify this content with any unique identifier
**Here I have my sample code for finding the mime type and Magic number****
string FileExtension = FileName.Substring((FileName.LastIndexOf(".") + 1)).ToUpper();
string MagicNumber = HexaDecimalData.Substring(0, 47);
string HexaDecimalData=string.Empty;
string MimeType = MimeMapping.GetMimeMapping(_File.FileName);
using (var fileStream = File.OpenRead(FileName))
{
BinaryReader Reader = new BinaryReader(fileStream);
Reader.BaseStream.Position = 0x0; // The offset you are reading the data from
byte[] data = Reader.ReadBytes(0x10); // Read 16 bytes into an array
HexaDecimalData = BitConverter.ToString(data);
Reader.Close();
} switch (FileExtension)
{
case "XLS":
if ((!MimeType.Equals("application/excel", StringComparison.OrdinalIgnoreCase) &&
!MimeType.Equals("application/vnd.ms-excel", StringComparison.OrdinalIgnoreCase) &&
!MimeType.Equals("application/x-excel", StringComparison.OrdinalIgnoreCase) &&
!MimeType.Equals("application/x-msexcel", StringComparison.OrdinalIgnoreCase)) ||
(!MagicNumber.StartsWith("D0-CF-11-E0-A1-B1-1A-E1") ))
ErrorMessage = "Invalid File";
break;
case "XLSX":
if ((!MimeType.Equals("application/vnd.openxmlformats-officedocument.spreadsheetml.sheet", StringComparison.OrdinalIgnoreCase)) ||
(!MagicNumber.StartsWith("50-4B-03-04") && !MagicNumber.StartsWith("50-4B-05-06") && !MagicNumber.StartsWith("50-4B-07-08") ))
ErrorMessage = "Invalid File";
break;
case "PDF":
if ((!MimeType.Equals("application/pdf", StringComparison.OrdinalIgnoreCase)) || (!MagicNumber.StartsWith("25-50-44-46") ))
ErrorMessage = "Invalid File";
break;
case "DOCX":
if ((!MimeType.Equals("application/vnd.openxmlformats-officedocument.wordprocessingml.document", StringComparison.OrdinalIgnoreCase)) ||
(!MagicNumber.StartsWith("50-4B-03-04") && !MagicNumber.StartsWith("50-4B-05-06") && !MagicNumber.StartsWith("50-4B-07-08") ))
ErrorMessage = "Invalid File";
break;
case "DOC":
if ((!MimeType.Equals("application/msword", StringComparison.OrdinalIgnoreCase)) || (!MagicNumber.StartsWith("D0-CF-11-E0-A1-B1-1A-E1") ))
ErrorMessage = "Invalid File";
break;
}
Please look at https://stackoverflow.com/a/14276391/911804
from above link:
However, note that all Office 2007 file formats ending in "x" are zip
compressed XML documents. So this approach is not 100% safe but at
least it might help you to filter out some invalid.
Updated:
If you want solid solution then try the following, but it will be time consuming.
In you switch case statement, try to open the uploaded file using Office DLL's, if opened properly then it's the correct format.
Add assembly reference and use Ms-Word assembly DLL's function.
It may be time consuming, but this will give 100% solution.
You have to get your hands dirty for a proper format validation. You have to try opening the file with the Open XML SDK.
Do a try-catch. If the try block ends successfuly, then it's a Word document - the catch block will tell you that either it is not a Word document, or something about it is broken.
For example:
try
{
// "path" is a string with the path to the file.
// The "false" parameter indicates we don't intend do edit the file.
using (WordprocessingDocument doc = WordprocessingDocument.Open(path, false))
{
// Seems like we got an ok file.
}
}
catch
{
// Let your user know that they should upload a proper Word file.
}
But do try a MIME validation prior to that, it'll save you the overhead of trying to open something you already know not to be a valid document.
Problem is now solved. Mistake by me that I hadn't seen before.
I am pretty new to coding in general and am very new to C# so I am probably missing something simple. I wrote a program to pull data from a login website and save that data to files on the local hard drive. The data is power and energy data for solar modules and each module has its own file. On my main workstation I am running Windows Vista and the program works just fine. When I run the program on the machine running Server 2003, instead of the new data being appended to the files, it just overwrites the data originally in the file.
The data I am downloading is csv format text over a span of 7 days at a time. I run the program once a day to pull the new day's data and append it to the local file. Every time I run the program, the local file is a copy of the newly downloaded data with none of the old data. Since the data on the web site is only updated once a day, I have been testing by removing the last day's data in the local file and/or the first day's data in the local file. Any time I change the file and run the program, the file contains the downloaded data and nothing else.
I just tried something new to test why it wasn't working and think I have found the source of the error. When I ran on my local machine, the "filePath" variable was set to "". On the server and now on my local machine I have changed the "filePath" to #"C:\Solar Yard Data\" and on both machines it catches the file not found exception and creates a new file in the same directory which overwrites the original. Anyone have an idea as to why this happens?
The code is the section that download's each data set and appends any new data to the local file.
int i = 0;
string filePath = "C:/Solar Yard Data/";
string[] filenamesPower = new string[]
{
"inverter121201321745_power",
"inverter121201325108_power",
"inverter121201326383_power",
"inverter121201326218_power",
"inverter121201323111_power",
"inverter121201324916_power",
"inverter121201326328_power",
"inverter121201326031_power",
"inverter121201325003_power",
"inverter121201326714_power",
"inverter121201326351_power",
"inverter121201323205_power",
"inverter121201325349_power",
"inverter121201324856_power",
"inverter121201325047_power",
"inverter121201324954_power",
};
// download and save every module's power data
foreach (string url in modulesPower)
{
// create web request and download data
HttpWebRequest req_csv = (HttpWebRequest)HttpWebRequest.Create(String.Format(url, auth_token));
req_csv.CookieContainer = cookie_container;
HttpWebResponse res_csv = (HttpWebResponse)req_csv.GetResponse();
// save the data to files
using (StreamReader sr = new StreamReader(res_csv.GetResponseStream()))
{
string response = sr.ReadToEnd();
string fileName = filenamesPower[i] + ".csv";
// save the new data to file
try
{
int startIndex = 0; // start index for substring to append to file
int searchResultIndex = 0; // index returned when searching downloaded data for last entry of data on file
string lastEntry; // will hold the last entry in the current data
//open existing file and find last entry
using (StreamReader sr2 = new StreamReader(fileName))
{
//get last line of existing data
string fileContents = sr2.ReadToEnd();
string nl = System.Environment.NewLine; // newline string
int nllen = nl.Length; // length of a newline
if (fileContents.LastIndexOf(nl) == fileContents.Length - nllen)
{
lastEntry = fileContents.Substring(0, fileContents.Length - nllen).Substring(fileContents.Substring(0, fileContents.Length - nllen).LastIndexOf(nl) + nllen);
}
else
{
lastEntry = fileContents.Substring(fileContents.LastIndexOf(nl) + 2);
}
// search the new data for the last existing line
searchResultIndex = response.LastIndexOf(lastEntry);
}
// if the downloaded data contains the last record on file, append the new data
if (searchResultIndex != -1)
{
startIndex = searchResultIndex + lastEntry.Length;
File.AppendAllText(filePath + fileName, response.Substring(startIndex+1));
}
// else append all the data
else
{
Console.WriteLine("The last entry of the existing data was not found\nin the downloaded data. Appending all data.");
File.AppendAllText(filePath + fileName, response.Substring(109)); // the 109 index removes the file header from the new data
}
}
// if there is no file for this module, create the first one
catch (FileNotFoundException e)
{
// write data to file
Console.WriteLine("File does not exist, creating new data file.");
File.WriteAllText(filePath + fileName, response);
//Debug.WriteLine(response);
}
}
Console.WriteLine("Power file " + (i + 1) + " finished.");
//Debug.WriteLine("File " + (i + 1) + " finished.");
i++;
}
Console.WriteLine("\nPower data finished!\n");
Couple of suggestions wich I think will probably resolve the issue
First change your filePath string
string filePath = #"C:\Solar Yard Data\";
create a string with the full path
String fullFilePath = filePath + fileName;
then check to see if it exists and create it if it doesnt
if (!File.Exists(fullFilePath ))
File.Create(fullFilePath );
put the full path to the file in your streamReader
using (StreamReader sr2 = new StreamReader(fullFilePath))
I want to create a WCF service (working like windows service). This service will read a PDF file from a specific path, extract pages, create a new PDF file and return it to the caller.
How can I do this ? I use QuickPDF to process on PDF files, I can extract and create new PDF file. How can use this in a WCF service ?
Waiting your helps...
This is only sample code :
public Stream ExtractPdf(string PathOfOriginalPdfFile, int StartPage,int PageCount)
{
PDFLibrary qp = new PDFLibrary();
Stream Stream_ = null;
if (qp.UnlockKey(".................") == 0)
{
string fileName = #"..\..\Test Files\sample1.pdf";
string OutputFile = #"..\..\Test Files\sample1_extracted.pdf";
if (qp.Unlocked() == 1)
{
int docID = qp.LoadFromFile(fileName, "");
int extractPageSuccess = qp.ExtractPages(StartPage, PageCount);
if (extractPageSuccess == 0)
{
// error
}
else
{
qp.SaveToFile(OutputFile);
}
}
}
//
// Codes here
//
return Stream_;
}
I edited it :
public byte[] ExtractPdf(string PathOfOriginalPdfFile, int StartPage,int PageCount)
{
QuickPDFDLL0815.PDFLibrary qp = new QuickPDFDLL0815.PDFLibrary(#"C:\Program Files (x86)\Quick PDF Library\DLL\QuickPDFDLL0815.dll");
string fileName = #"..\..\Test Files\sample1.pdf";
byte[] binFile = null;
if (qp.UnlockKey("...................") == 0)
{
if (qp.Unlocked() == 1)
{
int docID = qp.LoadFromFile(fileName, "");
int extractPageSuccess = qp.ExtractPages(StartPage, PageCount);
if (extractPageSuccess == 0)
{
// error
}
else
{
binFile = qp.SaveToString();
}
}
}
return binFile;
}
You could send the file as a Stream, see How to: Enable Streaming, then on the client save off the file and have the shell execute it. The MSDN article includes a sample GetStream method as well as a whole section on Writing a custom stream.
If you would like fuller sample code the forum post Streamed file transfer using WCF starts with some, however, note that the author posted it there because they were encountering issues running it.
As to byte[] or stream see Uploading Image Blobs–Stream vs Byte Array and Stream vs Raw Bytes. The second states
Streams will perform better for large files since not all of it needs to be read into memory at one time