Upload multiple files by Multi threading Using Azure.Storage in C# - c#

Hello i was able to upload multiple files in multiple threads when i was using WindowsAzure.Storage 2.0.4.0. but i have recently upgraded my library to 9.3.3.
Now i am facing error in setting my multiple threads to upload my file. Please have a look at my code and tell me that where i am missing. Although i have searched to set the parallel threads but its not setting the threads of the blob as it was setting before.
public void UploadBlobAsync(Microsoft.WindowsAzure.StorageClient.CloudBlob
blob, string LocalFile)
{
Microsoft.WindowsAzure.StorageCredentialsAccountAndKey account = blob.ServiceClient.Credentials as Microsoft.WindowsAzure.StorageCredentialsAccountAndKey;
ICloudBlob blob2 = new CloudBlockBlob(blob.Attributes.Uri, new Microsoft.WindowsAzure.Storage.Auth.StorageCredentials(blob.ServiceClient.Credentials.AccountName, account.Credentials.ExportBase64EncodedKey()));
UploadBlobAsync(blob2, LocalFile);
}
public void UploadBlobAsync(ICloudBlob blob, string LocalFile)
{
// The class currently stores state in class level variables so calling UploadBlobAsync or DownloadBlobAsync a second time will cause problems.
// A better long term solution would be to better encapsulate the state, but the current solution works for the needs of my primary client.
// Throw an exception if UploadBlobAsync or DownloadBlobAsync has already been called.
lock (WorkingLock)
{
if (!Working)
Working = true;
else
throw new Exception("BlobTransfer already initiated. Create new BlobTransfer object to initiate a new file transfer.");
}
// Attempt to open the file first so that we throw an exception before getting into the async work
using (FileStream fstemp = new FileStream(LocalFile, FileMode.Open, FileAccess.Read)) { }
// Create an async op in order to raise the events back to the client on the correct thread.
asyncOp = AsyncOperationManager.CreateOperation(blob);
TransferType = TransferTypeEnum.Upload;
m_Blob = blob;
m_FileName = LocalFile;
var file = new FileInfo(m_FileName);
long fileSize = file.Length;
FileStream fs = new FileStream(m_FileName, FileMode.Open, FileAccess.Read, FileShare.Read);
ProgressStream pstream = new ProgressStream(fs);
pstream.ProgressChanged += pstream_ProgressChanged;
pstream.SetLength(fileSize);
m_Blob.ServiceClient.ParallelOperationThreadCount = 10; //This Line is giving an error that is does not contain the definition.
m_Blob.StreamWriteSizeInBytes = GetBlockSize(fileSize);
asyncresult = m_Blob.BeginUploadFromStream(pstream, BlobTransferCompletedCallback, new BlobTransferAsyncState(m_Blob, pstream));
}
m_Blob.ServiceClient.ParallelOperationThreadCount = 10; is giving the error that it does not contain the definition. As i tried to find the work around but couldn't. I fount the code on Microsoft forum but it didn't help much.

Updated code of uploading azure blob storage multiple files in multi-threaded way here is the snippet of updated code which can be integrated in my previous code.
//Replace
m_Blob.ServiceClient.ParallelOperationThreadCount = 10
//with
BlobRequestOptions options = new BlobRequestOptions
{
ParallelOperationThreadCount = 8,
DisableContentMD5Validation = true,
StoreBlobContentMD5 = false
};
//Replace
asyncresult = m_Blob.BeginUploadFromStream(pstream, BlobTransferCompletedCallback, new BlobTransferAsyncState(m_Blob, pstream));
//with
asyncresult = m_Blob.BeginUploadFromStream(pstream,null,options,null,BlobTransferCompletedCallback, new BlobTransferAsyncState(m_Blob, pstream));

Related

getting an error that 'the process cannot access the file because it is being used by another process' in my dot net program

I tried 'using' but it says that the method is not Idisposable. I checked for running processes in Task Manager, nothing there. My goal is to upload a file from local directory to the Rich Text editor in my website. Please help me resolve this issue. Thanks in Advance
public void OnPostUploadDocument()
{
var projectRootPath = Path.Combine(_hostingEnvironment.ContentRootPath, "UploadedDocuments");
var filePath = Path.Combine(projectRootPath, UploadedDocument.FileName);
UploadedDocument.CopyTo(new FileStream(filePath, FileMode.Create));
// Retain the path of uploaded document between sessions.
UploadedDocumentPath = filePath;
ShowDocumentContentInTextEditor();
}
private void ShowDocumentContentInTextEditor()
{
WordProcessingLoadOptions loadOptions = new WordProcessingLoadOptions();
Editor editor = new Editor(UploadedDocumentPath, delegate { return loadOptions; }); //passing path and load options (via delegate) to the constructor
EditableDocument document = editor.Edit(new WordProcessingEditOptions()); //opening document for editing with format-specific edit options
DocumentContent = document.GetBodyContent(); //document.GetContent();
Console.WriteLine("HTMLContent: " + DocumentContent);
//string embeddedHtmlContent = document.GetEmbeddedHtml();```
//Console.WriteLine("EmbeddedHTMLContent: " + embeddedHtmlContent);
}
FileStream is disposable, so you can use using on it:
using (var stream = new FileStream(filePath, FileMode.Create)
{
UploadedDocument.CopyTo(stream);
}

UWP - Read a file and be able to cancel the playback/opening

Note : I use a translation app. Sorry if it's not always very understandable.
I'm developing a UWP application, and I'm having a problem with managing a file type, the CBZ extension.
Some files open without a problem, others the file never opens and blocks the Task.
Here's the code I use :
Task loadEbookTask = Task.Factory.StartNew(() =>
{
Stream streamEbook = WindowsRuntimeStorageExtensions.OpenStreamForReadAsync(ebookFile).Result;
Encoding.RegisterProvider(CodePagesEncodingProvider.Instance);
Content = new ZipArchive(streamEbook, ZipArchiveMode.Read, false);
// Pour charque archive, prendre que des extensions valident.
foreach (var file in Content.Entries)
{
string extension = Path.GetExtension(file.Name).ToLower();
bool isFileExtensionOk = EbooksManager.AvailableExtensionsImage.Contains(extension);
if (isFileExtensionOk)
{
ArchivesExploitable.Add(file);
}
}
TotalPage = Convert.ToUInt32(ArchivesExploitable.Count());
});
if (loadEbookTask.Wait(4000))
{
EbookCbz.LoadEbook = EbookLoad.Ok;
}
else
{
EbookCbz.LoadEbook = EbookLoad.Timeout;
}
It's looping on :
Stream streamEbook = WindowsRuntimeStorageExtensions.OpenStreamForReadAsync(ebookFile).Result;
In Visual Studio, memory doesn't go up any more, but the Garbage Collector keeps being called.
With the Task.Wait(4000), it does not stop the Task, so it does not stop turning in background.
And if I open another file, a new task is created, and will turn into a background task.
My question is:
- Is there a method that open a file, and that it is possible to cancel if it exceeds a certain time.
It is this method that is problematic.
Stream streamEbook = WindowsRuntimeStorageExtensions.OpenStreamForReadAsync(ebookFsile).Result;
I change my code to :
byte[] buffer = await ebookFile.ReadBytesAsync();
Stream stream = new MemoryStream(buffer);
Encoding.RegisterProvider(CodePagesEncodingProvider.Instance);
Content = new ZipArchive(stream, ZipArchiveMode.Read, false);
It's fast and if the file is corrupted, there's an exception. It's no longer in memory.
Thanks for your help, I learned a new concept.

How to monitor a logfile that seems to be open all the time (much like notepad++ does)?

I'm trying to build a small program to monitor my pfirewall.log, but I can't seem to open it.
I found quite many (simple) answers, that all kinda say
// use FilesystemWatcher
// open FileStream
// read from last position to end
// output new lines
The problem here is: The file seems to always be opened by another process already. I guess that's the windows process writing to the file, since it's getting written to all the time, as Notepad++ shows me.
Which means, Notepad++ can for some reason do what I can not: Read the file despite it being opened already.
I initialize my monitor in the constructor:
public FirewallLogMonitor(string path)
{
if (!File.Exists(path))
throw new FileNotFoundException("Logfile not found");
this.file = path;
this.lastPosition = 0;
this.monitor = new FileSystemWatcher(Path.GetDirectoryName(path), Path.GetFileName(path));
this.monitor.NotifyFilter = NotifyFilters.Size;
}
And try to read the file on monitor.Changed event:
private void LogFileChanged(object sender, FileSystemEventArgs e)
{
using (FileStream stream = new FileStream(e.FullPath, FileMode.Open, FileAccess.Read, FileShare.Read))
using (StreamReader reader = new StreamReader(stream))
{
stream.Seek(this.lastPosition, SeekOrigin.Begin);
var newLines = reader.ReadToEnd();
this.lastPosition = stream.Length;
var filteredLines = filterLines(newLines);
if (filteredLines.Count > 0)
NewLinesAvailable(this, filteredLines);
}
}
It always throws the IOException on new FileStream(...) to tell me the file is already in use.
Since Notepad++ does it, there has to be a way I can do it too, right?
**Edit: ** A button does this:
public void StartLogging()
{
this.IsRunning = true;
this.monitor.Changed += LogFileChanged;
this.monitor.EnableRaisingEvents = true;
}
**Edit2: ** This is not a duplicate of FileMode and FileAccess and IOException: The process cannot access the file 'filename' because it is being used by another process, since that one assumes I have control over the writing process. Will try the other suggestions, and report back with results.
If i understand your question you can use the notepad++ itself with a plugin to monitor you need to go to:
plugins -> Document Moniter -> Start to monitor
if you dont have this plugin you can download it here:
http://sourceforge.net/projects/npp-plugins/files/DocMonitor/

Streaming files from amazon s3 with seek possibility in C#

I need to work with huge files in Amazon S3. How can I get part of huge file from S3? Best way would be get stream with the seek possibility.
Unfortunately, CanSeek property of response.ResponseStream is false:
GetObjectRequest request = new GetObjectRequest();
request.BucketName = BUCKET_NAME;
request.Key = NumIdToAmazonKey(numID);
GetObjectResponse response = client.GetObject(request);
You could do following to read a certain part of your file
GetObjectRequest request = new GetObjectRequest
{
BucketName = bucketName,
Key = keyName,
ByteRange = new ByteRange(0, 10)
};
See the documentation
I know this isn't exactly what OP is asking for but I needed a seekable s3 stream so I could read Parquet files without downloading them so I gave this a shot here: https://github.com/mukunku/RandomHelpers/blob/master/SeekableS3Stream.cs
Performance wasn't as bad as I expected. You can use the TimeWastedSeeking property to see how much time is being wasted by allowing Seek() on an s3 stream.
Here's an example on how to use it:
using (var client = new AmazonS3Client(credentials, Amazon.RegionEndpoint.USEast1))
{
using (var stream = SeekableS3Stream.OpenFile(client, "myBucket", "path/to/myfile.txt", true))
{
//stream is seekable!
}
}
After a frustrating afternoon with the same problem I found the static class AmazonS3Util
https://docs.aws.amazon.com/sdkfornet/v3/apidocs/items/S3/TS3Util.html
Which has a MakeStreamSeekable method.
Way late for the OP, but I've just posted an article and code demonstration of a SeekableS3Stream that performs reasonably well in real-world use cases.
https://github.com/mlhpdx/seekable-s3-stream
Specifically, I demonstrate reading a single small file from a much larger ISO disk image using the DiscUtils library unmodified by implementing a random-access stream that uses Range requests to pull sections of the file as-needed and maintains them in an MRU list to prevent re-downloading ranges for hot data structures in the file (e.g. zip central directory records).
The use is similarly simple:
using System;
using System.IO;
using System.Threading.Tasks;
using Amazon.S3;
using DiscUtils.Iso9660;
namespace Seekable_S3_Stream
{
class Program
{
const string BUCKET = "rds.nsrl.nist.gov";
const string KEY = "RDS/current/RDS_ios.iso"; // "RDS/current/RDS_modern.iso";
const string FILENAME = "READ_ME.TXT";
static async Task Main(string[] args)
{
var s3 = new AmazonS3Client();
using var stream = new Cppl.Utilities.AWS.SeekableS3Stream(s3, BUCKET, KEY, 1 * 1024 * 1024, 4);
using var iso = new CDReader(stream, true);
using var file = iso.OpenFile(FILENAME, FileMode.Open, FileAccess.Read);
using var reader = new StreamReader(file);
var content = await reader.ReadToEndAsync();
await Console.Out.WriteLineAsync($"{stream.TotalRead / (float)stream.Length * 100}% read, {stream.TotalLoaded / (float)stream.Length * 100}% loaded");
}
}
}

Getting current log file content in Enterprise Library Logging

I've set up the Enterprise Library Logging Application Block to log in a file called 'app.log' residing in the execution path of my application. This application is a Windows service which runs a configuration website on top of it, where I now want to show the contents of the log file.
Getting the log file was a rather easy task:
var config = ConfigurationManager.OpenExeConfiguration(ConfigurationUserLevel.None);
var logSection = config.GetSection("loggingConfiguration") as LoggingSettings;
var lookup = logSection.TraceListeners
.Where(x => x is RollingFlatFileTraceListenerData).FirstOrDefault() as RollingFlatFileTraceListenerData;
if(lookup != null) {
var filePath = Path.Combine(AppDomain.CurrentDomain.BaseDirectory, lookup.FileName);
return File.ReadAllText(_logFilePath);
}
However, the RollingFlatFileTraceListener I've set up constantly BLOCKS the file I want to read from. Is there any possibility to access it?
Check this answer. That this is not the default behavior for File.ReadAllText is beyond me...
using (var logFileStream = new FileStream(filePath, FileMode.Open, FileAccess.Read, FileShare.ReadWrite))
using (var logFileReader = new StreamReader(logFileStream))
{
return logFileReader.ReadToEnd();
}
Also note that you are mixing filePath and _logFilePath.

Categories