I'm having a problem with inclomplete blobs being downloaded from Azure storage. The files that are stored are an images. Almost every file that's downloaded ends up missing several lines on the bottom. I've checked the blobs and they were uploaded correctly.
I'm using the following code for downloading a blob from the Azure service:
private async Task Download(CloudBlobClient client)
{
try
{
_media = await _directory.CreateFileAsync(ResourceName, CreationCollisionOption.FailIfExists);
}
catch (Exception)
{
return;
}
using (var stream = await _media.OpenAsync(FileAccessMode.ReadWrite))
{
var blob = await GetBlob(client);
await blob.DownloadToStreamAsync(stream);
_category.NotifyAzureProgress();
await stream.FlushAsync();
}
}
The method GetBlob() looks like this:
private async Task<CloudBlockBlob> GetBlob(CloudBlobClient client)
{
CloudBlobContainer container = client.GetContainerReference(ContainerName);
await container.CreateIfNotExistsAsync();
var blob = container.GetBlockBlobReference(ResourceName);
return blob;
}
Upload code:
private async Task UploadAsync(CloudBlobClient client)
{
_media = await _directory.GetFileAsync(ResourceName);
using (var stream = await _media.OpenAsync(FileAccessMode.Read))
{
var blob = await GetBlob(client);
await blob.UploadFromStreamAsync(stream);
_category.NotifyAzureProgress();
}
}
Thanks for any help!
Edit: I've realized I've missed out one detail - the downloaded image has correct dimensions, but several lines from the bottom are black - it doesn't has the same pixels as the source image. I've checked the MD5 hashes and while they match, when I download the image through an external app, they don't match when I download them with the code above.
Edit2: after inspecting the properties of CloudBlob and the output stream, I've noticed, that even though the blob gives correct length after download, the stream usually says something a little lower. I've tried downloading throught range, but to no avail
Ok, so I've managed to download the images afterall, by partially using the WinRT Azure library combined with a standard .NET HttpClient.
I used the Azure Lib establish the initial connection and then to get only the Blob reference, because the BlockBlobReference has a method to create Shared Access Signature (and I really didn't want to try to construct it myself). Then I created the HttpClient, made a download URL using the SAS and issued a GET request to the URL, which finally worked and downloaded all the images intact.
I think there might be some weird bug in the official library, since using my download method instead of theirs solved everything.
Code sample:
internal async Task Download(CloudBlobClient client)
{
try
{
_media = await _directory.CreateFileAsync(ResourceName, CreationCollisionOption.FailIfExists);
}
catch (Exception)
{
return;
}
try
{
var blob = await GetBlob(client);
HttpClient httpClient = new HttpClient();
var date = DateTime.UtcNow;
var policy = new SharedAccessBlobPolicy();
policy.Permissions = SharedAccessBlobPermissions.Read;
policy.SharedAccessStartTime = new DateTimeOffset(date);
policy.SharedAccessExpiryTime = new DateTimeOffset(date.AddDays(1));
var signature = blob.GetSharedAccessSignature(policy);
var uriString = string.Format("{0}{1}", blob.Uri.ToString(), signature);
var data = await httpClient.GetByteArrayAsync(uriString);
var buf = new Windows.Storage.Streams.Buffer((uint)data.Length);
await FileIO.WriteBytesAsync(_media, data);
_category.NotifyAzureProgress();
}
catch (Exception e)
{
_media.DeleteAsync();
throw e;
}
}
Related
I have been working on this for a week. I have done a lot of searching and a lot of tests for different methods.
When I use HttpClient to download a file, no errors are generated but the files do not show up in the folder until after the program exits. I have a synchronous method (before someone asks - no I cannot change it to asynchronous) that calls an asynchronous method to download language files for my Tesseract OCR (I test with language == "eng"):
if (!Directory.Exists(folderName))
Directory.CreateDirectory(folderName);
Task.Run(async () => await HelperMethods.LoadLanguage(folderName, language));
Task.Run(async () => await HelperMethods.LoadLanguage(folderName, "osd"));
And here is the method that is being awaited:
public static async Task LoadLanguage(string folderName, string language)
{
string dest = Path.GetFullPath(Path.Combine(folderName, $"{language}.traineddata"));
if (!File.Exists(dest))
{
// Now we know that we need network - start it up if it isn't already.
if (httpClient == null)
httpClient = new HttpClient();
Uri uri = new Uri($"https://github.com/tesseract-ocr/tessdata/raw/main/{language}.traineddata");
HttpResponseMessage response = await httpClient.GetAsync(uri);
using (FileStream fs = new FileStream(dest, FileMode.Create, FileAccess.Write))
{
await response.Content.CopyToAsync(fs);
await fs.FlushAsync();
fs.Close();
}
}
}
I added the Flush and Close as part of the testing, but it did not make a difference.
This is supposed to download the language files and allow the next lines to perform an OCR using those languages. The files are downloaded and are written to the folder (only show up) after the program exits.
If I run the program a second time, it works - because it does not need to download new files.
How do I get this to download the files and save them immediately so the files can be used in subsequent operations?
I also tried this method:
var request = new HttpRequestMessage(HttpMethod.Get, uri);
var sendTask = httpClient.SendAsync(request, HttpCompletionOption.ResponseHeadersRead);
var response = sendTask.Result.EnsureSuccessStatusCode();
var httpStream = await response.Content.ReadAsStreamAsync();
using (var fileStream = File.Create(dest))
{
using (var reader = new StreamReader(httpStream))
{
httpStream.CopyTo(fileStream);
fileStream.Flush();
}
}
No difference. And quite a few other methods of working with the stream.
This is .NET framework 4.8 (not able to update to .NET6).
Your post states
I have a synchronous method [...] that calls an asynchronous method to download language files
If the caller is synchronous anyway, why not make the downloader synchronous, too?
public void LoadLanguage(string folderName, string language)
{
Enabled = false;
try
{
Uri uri = new Uri($"https://github.com/tesseract-ocr/tessdata/raw/main/{language}.traineddata");
using (var client = new HttpClient())
{
using (var response =
client
.GetAsync(uri)
.GetAwaiter()
.GetResult())
{
var bytes =
response
.Content
.ReadAsByteArrayAsync()
.GetAwaiter()
.GetResult();
File.WriteAllBytes(
Path.Combine(
folderName,
$"{language}.traineddata"),
bytes
);
}
}
}
finally
{
Enabled = true;
}
}
I tested this and it seems to work [clone]. Does this get you any closer?
I'm trying to upload a file using Microsoft's Graph SDK but have hit a problem.
I have pretty much copied verbatim the C# example from here, commented-out the progress part and updated the using statement for C# 8, and here's what I have...
public async Task<bool> UploadFileAsync(string parentFolderId, string filename, byte[] bytes)
{
var graphClient = GetGraphClient();
// Declare the variable outside the `using` statement to get around a little C# problem - https://www.tabsoverspaces.com/233779-using-await-using-iasyncdisposable-with-configureawait
var memoryStream = new MemoryStream(bytes);
await using (memoryStream.ConfigureAwait(false))
{
// Use properties to specify the conflict behavior.
// - in this case, replace.
var uploadProps = new DriveItemUploadableProperties
{
AdditionalData = new Dictionary<string, object> {{"#microsoft.graph.conflictBehavior", "replace"}},
ODataType = null
};
try
{
// Create the upload session.
// - itemPath does not need to be a path to an existing item.
var uploadSession = await graphClient.Drives[_driveId]
.Items[parentFolderId]
.ItemWithPath(filename)
.CreateUploadSession(uploadProps)
.Request()
.PostAsync()
.ConfigureAwait(false);
// Max slice size must be a multiple of 320KB.
const int maxSliceSize = 320 * 1024;
var fileUploadTask = new LargeFileUploadTask<DriveItem>(uploadSession, memoryStream, maxSliceSize);
// Upload the file.
var uploadResult = await fileUploadTask.UploadAsync().ConfigureAwait(false);
if (uploadResult.UploadSucceeded)
{
// The ItemResponse object in the result represents the created item.
return true;
}
return false;
}
catch (ServiceException exception)
{
// ...
}
}
}
However the line...
var uploadSession = await graphClient.Drives[_driveId]
.Items[parentFolderId]
...
...throws an exception:
Microsoft.Graph.ServiceException: Code: BadRequest
Message: Multiple action overloads were found with the same binding parameter for 'microsoft.graph.createUploadSession'.
Can anyone help?
I figured out the cause of the problem - the filename that I was using contained invalid symbols (in my case I was stringifying a DateTime and that contained :).
It's frustrating that this exception doesn't bubble-up correctly and instead I got that "Multiple action overloads" message.
i have this two methods for writting and reading from the file.
public static async Task WriteDataToFileAsync(string fileName, string content)
{
byte[] data = Encoding.Unicode.GetBytes(content);
var folder = ApplicationData.Current.LocalFolder;
var file = await folder.CreateFileAsync(fileName, CreationCollisionOption.OpenIfExists);
using (var s = await file.OpenStreamForWriteAsync())
{
await s.WriteAsync(data, 0, data.Length);
}
}
public async static Task<string> ReadFileContentsAsync()
{
var folder = ApplicationData.Current.LocalFolder;
try
{
var file = await folder.OpenStreamForReadAsync("MenuData.json");
using (var streamReader = new StreamReader(file))
{
Debug.WriteLine(streamReader.ReadToEnd());
return streamReader.ReadToEnd();
}
}
catch (Exception)
{
return string.Empty;
}
}
which are then used in this two methods
public static async void ApiToFileRestaurants()
{
HttpClient client = new HttpClient();
HttpResponseMessage response = client.GetAsync("http://bonar.si/api/restaurants").Result;
response.EnsureSuccessStatusCode();
string responseBody = response.Content.ReadAsStringAsync().Result;
await Restaurant.WriteDataToFileAsync("MenuData.json", responseBody);
}
public async static Task<List<Restaurant>> FileToRestaurantList()
{
var responseBody = await Restaurant.ReadFileContentsAsync();
List<Restaurant> parsedRestaurants = (List<Restaurant>)Newtonsoft.Json.JsonConvert.DeserializeObject(responseBody, typeof(List<Restaurant>));
return parsedRestaurants;
}
now my problem here is that ReadFileAsync doesn't return the results which i know are saved in MenuData.json file but instead returns empty string.
I was mostly getting source code for this from msdn
documentation.
Location of the file in my wp power tools looks like that.
I'm a novice programer so i might overlooked something else
Can you try to read the data from file asyncronously by using ReadToEndAsync which basically parses the complete data and sends response as one string.
var file = await folder.OpenStreamForReadAsync("MenuData.json");
using (var streamReader = new StreamReader(file))
{
return await streamReader.ReadToEndAsync();
}
Hope this helps!
i got the solution from one other forum
You're calling streamReader.ReadToEnd() twice. The first time you log
it to the Debug stream, the second is what you actually use as a
result. The method moves the file pointer to the end everytime it's
called and by the second time there's nothing to read.
so removing that debug line almost fixed my problem. I did get the string i wanted to but there was an error somewhere in it so Newtonsoft.Json had a hard time parsing it. So i tried #asitis solution and changed .json to .text and it worked
i'm trying to create an app that downloads a file and then edits this file.
The Problem Im having is once the file is downloaded it doesn't seem to let go of that file, i can download the file to its local storage, i have gotten the file manually from the Iso and its fine. if i use the app to proceed after downloading the file i get the System.UnauthorizedAccessException error, but if i close and open the app and then just edit the file saved in iso it works, like i said its like something is still using the downloaded file.
public async void DownloadTrack(Uri SongUri)
{
var httpClient = new HttpClient();
var data = await httpClient.GetByteArrayAsync(SongUri);
var file = await ApplicationData.Current.LocalFolder.CreateFileAsync("Test.mp3", CreationCollisionOption.ReplaceExisting);
var targetStream = await file.OpenAsync(FileAccessMode.ReadWrite);
await targetStream.AsStreamForWrite().WriteAsync(data, 0, data.Length);
await targetStream.FlushAsync();
}
this code works fine to download the mp3, as ive tested the download file. I have seen if examples where the code ends with
targetStream.Close();
but it doesnt give me that, is there another way to close the download
thanks.
Instead of calling Close() or Dispose() I really like to use using which does the job automatically. So your method could look like this:
public async void DownloadTrack(Uri SongUri)
{
using (HttpClient httpClient = new HttpClient())
{
var data = await httpClient.GetByteArrayAsync(SongUri);
var file = await ApplicationData.Current.LocalFolder.CreateFileAsync("Test.mp3", CreationCollisionOption.ReplaceExisting);
using (var targetStream = await file.OpenAsync(FileAccessMode.ReadWrite))
{
await targetStream.AsStreamForWrite().WriteAsync(data, 0, data.Length);
await targetStream.FlushAsync();
}
}
}
I am working on a metro app. I used background uploader to upload file but my question is how I get response value after uploading it. I coded like this:
BackgroundUploader uploader = new BackgroundUploader();
uploader.SetRequestHeader("Content-Disposition", "form-data");
uploader.SetRequestHeader("name", "userfile");
uploader.SetRequestHeader("filename", App.ViewModel.DeviceId + ".png");
uploader.SetRequestHeader("Content-Type", "multipart/form-data");
UploadOperation upload = uploader.CreateUpload(uri, file);
await upload.StartAsync();
I came up with the following after noticing there were BytesReceived in my upload progress object.
async private Task<string> GetUploadResponseBody(UploadOperation operation)
{
string responseBody = string.Empty;
using (var response = operation.GetResultStreamAt(0))
{
uint size = (uint)operation.Progress.BytesReceived;
IBuffer buffer = new Windows.Storage.Streams.Buffer(size);
var f = await response.ReadAsync(buffer, size, InputStreamOptions.None);
using (var dr = DataReader.FromBuffer(f))
{
responseBody = dr.ReadString(dr.UnconsumedBufferLength);
}
}
return responseBody;
}
upload.StartAsync().Completed = UploadCompletedHandler;
...
void UploadCompletedHandler(IAsyncOperationWithProgress<TResult, TProgress> asyncInfo,
AsyncStatus asyncStatus)
{
// get a response body from an asyncInfo using the asyncInfo.GetResults() method
}
Follow this resources:
UploadOperation.StartAsync | startAsync Method (Windows)
IAsyncOperationWithProgress Interface (Windows)
AsyncOperationWithProgressCompletedHandler Delegate (Windows)
I've been looking for the same thing for the last few days and no luck. Finally discovered that you can not do this. You can get the "headers" of the response but there is no way of getting the "body" of the response from a BackgroundTransfer getResponseInformation() method.
Currently it's a limitation of the windows API. Hope they'll add it soon.
http://msdn.microsoft.com/en-us/library/windows/apps/windows.networking.backgroundtransfer.responseinformation.aspx#properties
The workaround is you can add your custom header in the response. For this you need to modify your server side script. But if you don't have any control on your server side script then use a proxy script that will do the communication between your app and remote server. For my case I created a proxy script in php that communicates with the remote server and after getting the response I'm adding it into a custom header key.
Then in the app in your complete method use this:
function complete(e){
var mykey = e.getResponseInformation().headers.lookup("mykey");
}
Hope that'll help.