We Keep Coding

Cannot get link from file sent up to dropbox getting message WaitingForActivation

I'm attempting to upload a file and then get a link for it once it has been uploaded.
I seem to be sending up the file without a problem but when I go to obtain a link to the file, it sends back WaitingForActivation.
Here's my code:
private async Task ChunkUpload2(DropboxClient client, string folder, string fileName)
{
try
{
Console.WriteLine("Starting file upload...");
var path = $"{folder}/{Path.GetFileName(fileName)}";
// Chunk size is 128KB.
const int chunkSize = 4096 * 1024;
if (!File.Exists(fileName))
{
Console.WriteLine();
Console.WriteLine($"File does not exist: {fileName}");
Console.WriteLine();
return;
}
using (var stream = new MemoryStream(File.ReadAllBytes(fileName)))
{
int numChunks = (int) Math.Ceiling((double) stream.Length / chunkSize);
byte[] buffer = new byte[chunkSize];
string sessionId = null;
if (numChunks == 1)
{
using (var memStream = new MemoryStream(buffer, 0, chunkSize))
{
Console.WriteLine($"Sending file: {path}");
var tst = await client.Files.UploadAsync(path, WriteMode.Overwrite.Instance, body: memStream);
//Console.WriteLine(updated.Result);
}
}
else
{
for (var idx = 0; idx < numChunks; idx++)
{
Console.WriteLine($"Uploading chunk {idx + 1} / {numChunks}.");
var byteRead = stream.Read(buffer, 0, chunkSize);
using (MemoryStream memStream = new MemoryStream(buffer, 0, byteRead))
{
var cursor = new UploadSessionCursor(sessionId, (ulong) (chunkSize * idx));
if (idx == numChunks - 1)
{
Console.WriteLine($"Finalizing file: {path}");
var response = await client.Files.UploadSessionFinishAsync(cursor, new CommitInfo(path), memStream);
}
else
{
await client.Files.UploadSessionAppendV2Async(cursor, body: memStream);
}
}
}
}
var url = string.Empty;
var link = await client.Sharing.ListSharedLinksAsync(path);
if (link.Links.Count == 0)
{
var result = client.Sharing.CreateSharedLinkWithSettingsAsync(path);
url = result.Result.Url;
}
else
{
url = link.Links[0].Url;
}
Console.WriteLine();
Console.WriteLine("Dropbox Download Link:");
Console.WriteLine(url);
Console.WriteLine();
//Console.ReadKey();
}
}
catch (Exception e)
{
Console.WriteLine(e);
throw;
}
}
The last portion, at CreateSharedLinkWithSettingsAsync is where it's giving me back no share urls. Instead it returns the code above.
I'm getting this message:
Any suggestions on how to 'Active' this file? I'm assuming I'm missing something?

OneDrive REST API - Requested Range Error While Uploading Chunk

I've been trying to upload files to my OneDrive via HTTP Requests following this document (https://learn.microsoft.com/en-us/onedrive/developer/rest-api/api/driveitem_createuploadsession?view=odsp-graph-online) without success. I have the following steps rounded up (Authentication, folder creation for the file, create an upload session) but when I try the last step, byte upload to the created session, I get this error in the second PUT request:
Requested Range Not Satisfiable {"error":{"code":"invalidRange","message":"Optimistic concurrency failure during fragmented upload"}}
This is my code:
//Get File Data
byte[] FileByteArray = File.ReadAllBytes(FilePath);
//Create Upload Session
OutlookEndpoint = $"{AppSettings.DriveSettings.OneDriveSettings.Endpoint}/me/drive/items/{FolderId}:/{Name}:/createuploadsession";
OutlookResponseMessage = await OutlookClient.PostAsync(OutlookEndpoint, new StringContent("{}", Encoding.UTF8, "application/json"));
OutlookResponseContent = await OutlookResponseMessage.Content.ReadAsStringAsync();
if (OutlookResponseMessage.IsSuccessStatusCode)
{
OutlookUpload OutlookUpload = JsonConvert.DeserializeObject<OutlookUpload>(OutlookResponseContent);
//Check the Created URL
if (!string.IsNullOrEmpty(OutlookUpload.UploadUrl))
{
//Chunk Calculation
int TotalSize = FileByteArray.Length;
int AcumulativeSize = 0;
int ChunkSize = 327680;
int ChunkBuffer = ChunkSize;
int ChunkNumber = TotalSize / ChunkSize;
int ChunkLeftover = TotalSize - ChunkSize * ChunkNumber;
int ChunkCounter = 0;
while (true)
{
if (ChunkNumber == ChunkCounter)
{
ChunkSize = ChunkLeftover;
}
byte[] ChunkData = FileByteArray.Skip(ChunkBuffer * ChunkCounter).Take(ChunkSize).ToArray();
AcumulativeSize += ChunkData.Length;
//PUT Upload of Chunk
string UploadEndpoint = OutlookUpload.UploadUrl;
string BytesHeader = $"bytes {AcumulativeSize - ChunkSize}-{AcumulativeSize - 1}/{TotalSize}";
OutlookClient.DefaultRequestHeaders.Clear();
OutlookClient.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Bearer", AccessToken);
OutlookClient.DefaultRequestHeaders.TryAddWithoutValidation("Content-Length", ChunkSize.ToString());
OutlookClient.DefaultRequestHeaders.TryAddWithoutValidation("Content-Range", BytesHeader);
OutlookResponseMessage = await OutlookClient.PutAsync(UploadEndpoint, new ByteArrayContent(ChunkData));
OutlookResponseContent = await OutlookResponseMessage.Content.ReadAsStringAsync();
if (OutlookResponseMessage.IsSuccessStatusCode)
{
Console.WriteLine("SUCCESS");
}
else
{
Console.WriteLine(OutlookResponseMessage.ReasonPhrase);
}
if (ChunkNumber == ChunkCounter)
{
break;
}
ChunkCounter++;
}
}
}
Perhaps I'm missing something. I only get a SUCCESS message in the first PUT request, the others always give me the error described above. Here's an image of the error with the headers I send. Image
I'd appreciate any help, thanks for reading this far.
EDIT:
Got it working after modifying the the header configuration for the request and changing the way chunks are created.
//Get File Data
byte[] FileByteArray = File.ReadAllBytes(FilePath);
//Create Upload Session
OutlookEndpoint = $"{AppSettings.DriveSettings.OneDriveSettings.Endpoint}/me/drive/items/{FolderId}:/{Name}:/createuploadsession";
OutlookResponseMessage = await OutlookClient.PostAsync(OutlookEndpoint, new StringContent("{}", Encoding.UTF8, "application/json"));
OutlookResponseContent = await OutlookResponseMessage.Content.ReadAsStringAsync();
if (OutlookResponseMessage.IsSuccessStatusCode)
{
OutlookUpload OutlookUpload = JsonConvert.DeserializeObject<OutlookUpload>(OutlookResponseContent);
//Check the Created URL
if (!string.IsNullOrEmpty(OutlookUpload.UploadUrl))
{
using MemoryStream FileStream = new MemoryStream(FileByteArray);
//Chunk Calculation
int ChunkSize = 320 * 1024;
int ChunkRemaining = 0;
byte[] ByteBuffer = new byte[ChunkSize];
int BytesRead = 0;
while ((BytesRead = FileStream.Read(ByteBuffer, 0, ByteBuffer.Length)) > 0)
{
if (BytesRead < ChunkSize)
{
byte[] LastBuffer = new byte[BytesRead];
Buffer.BlockCopy(ByteBuffer, 0, LastBuffer, 0, BytesRead);
ByteBuffer = new byte[BytesRead];
ByteBuffer = LastBuffer;
}
try
{
OutlookClient.DefaultRequestHeaders.Clear();
string UploadEndpoint = OutlookUpload.UploadUrl;
string BytesHeader = $"bytes {ChunkRemaining}-{ChunkRemaining + ByteBuffer.Length - 1}/{FileByteArray.Length}";
HttpRequestMessage MicrosoftResponseMessage = new HttpRequestMessage()
{
Content = new ByteArrayContent(ByteBuffer),
RequestUri = new Uri(UploadEndpoint),
Method = HttpMethod.Put,
};
MicrosoftResponseMessage.Content.Headers.Add("Content-Length", ByteBuffer.Length.ToString());
MicrosoftResponseMessage.Content.Headers.Add("Content-Range", BytesHeader);
OutlookResponseMessage = await OutlookClient.SendAsync(MicrosoftResponseMessage);
OutlookResponseContent = await OutlookResponseMessage.Content.ReadAsStringAsync();
if (OutlookResponseMessage.IsSuccessStatusCode)
{
Console.WriteLine("SUCCESS");
ChunkRemaining += ByteBuffer.Length;
if (ChunkRemaining == FileByteArray.Length)
{
Console.WriteLine("COMPLETED");
}
}
else
{
Console.WriteLine(OutlookResponseMessage.ReasonPhrase);
}
}
catch (Exception Exception)
{
Console.WriteLine(Exception.Message);
break;
}
}
}
}

Please note that on failures when the client sent a fragment the server had already received, the server will respond with HTTP 416 Requested Range Not Satisfiable. You can request upload status to get a more detailed list of missing ranges. Apparently the content-range and content-length were the problem. You changed the header configuration from the HttpClient to a HttpRequestMessage and it worked perfectly now.

C# gRPC file streaming, original file smaller than the streamed one

I am having some problems with setting up a request-stream type gRPC architecture. The code below is just for testing purposes and it is missing various validation checks, but the main issue is that the original file is always smaller than the received one.
Could the cause here be encoding? It doesn't matter what the file type is, the end result is always that the file sizes are different.
Protobuf inteface:
syntax = "proto3";
package FileTransfer;
option csharp_namespace = "FileTransferProto";
service FileTransferService {
rpc DownloadFile(FileRequest) returns (stream ChunkMsg);
}
message ChunkMsg {
string FileName = 1;
int64 FileSize = 2;
bytes Chunk = 3;
}
message FileRequest {
string FilePath = 1;
}
Server side (sending):
public override async Task DownloadFile(FileRequest request, IServerStreamWriter<ChunkMsg> responseStream, ServerCallContext context)
{
string filePath = request.FilePath;
if (!File.Exists(filePath)) { return; }
FileInfo fileInfo = new FileInfo(filePath);
ChunkMsg chunk = new ChunkMsg();
chunk.FileName = Path.GetFileName(filePath);
chunk.FileSize = fileInfo.Length;
int fileChunkSize = 64 * 1024;
byte[] fileByteArray = File.ReadAllBytes(filePath);
byte[] fileChunk = new byte[fileChunkSize];
int fileOffset = 0;
while (fileOffset < fileByteArray.Length && !context.CancellationToken.IsCancellationRequested)
{
int length = Math.Min(fileChunkSize, fileByteArray.Length - fileOffset);
Buffer.BlockCopy(fileByteArray, fileOffset, fileChunk, 0, length);
fileOffset += length;
ByteString byteString = ByteString.CopyFrom(fileChunk);
chunk.Chunk = byteString;
await responseStream.WriteAsync(chunk).ConfigureAwait(false);
}
}
Client side (receiving):
public static async Task GetFile(string filePath)
{
var channel = Grpc.Net.Client.GrpcChannel.ForAddress("https://localhost:5001/", new GrpcChannelOptions
{
MaxReceiveMessageSize = 5 * 1024 * 1024, // 5 MB
MaxSendMessageSize = 5 * 1024 * 1024, // 5 MB
});
var client = new FileTransferProto.FileTransferService.FileTransferServiceClient(channel);
var request = new FileRequest { FilePath = filePath };
string tempFileName = $"temp_{DateTime.UtcNow.ToString("yyyyMMdd_HHmmss")}.tmp";
string finalFileName = tempFileName;
using (var call = client.DownloadFile(request))
{
await using (Stream fs = File.OpenWrite(tempFileName))
{
await foreach (ChunkMsg chunkMsg in call.ResponseStream.ReadAllAsync().ConfigureAwait(false))
{
Int64 totalSize = chunkMsg.FileSize;
string tempFinalFilePath = chunkMsg.FileName;
if (!string.IsNullOrEmpty(tempFinalFilePath))
{
finalFileName = chunkMsg.FileName;
}
fs.Write(chunkMsg.Chunk.ToByteArray());
}
}
}
if (finalFileName != tempFileName)
{
File.Move(tempFileName, finalFileName);
}
}

To add to Marc's answer, I feel like you can simplify your code a little bit.
using var fs = File.Open(filePath, System.IO.FileMode.Open);
int bytesRead;
var buffer = new byte[fileChunkSize];
while ((bytesRead = await fs.ReadAsync(buffer)) > 0)
{
await call.RequestStream.WriteAsync(new ChunkMsg
{
// Here the correct number of bytes must be sent which is starting from
// index 0 up to the number of read bytes from the file stream.
// If you solely pass 'buffer' here, the same bug would be present.
Chunk = ByteString.CopyFrom(buffer[0..bytesRead]),
});
}
I've used the array range operator from C# 8.0 which makes this cleaner or you can also use the overload of ByteString.CopyFrom which takes in an offset and count of how many bytes to include.

In your write loop, the chunk you actually send is for the oversized buffer, not accounting for length. This means that the last segment includes some garbage and is oversized. The received payload will be oversized by this same amount. So: make sure you account for length when constructing the chunk to send.

I tested the code and modified it to transfer the correct size.
The complete code is available at the following URL: https://github.com/lisa3907/grpc.fileTransfer
server-side-code
while (_offset < _file_bytes.Length)
{
if (context.CancellationToken.IsCancellationRequested)
break;
var _length = Math.Min(_chunk_size, _file_bytes.Length - _offset);
Buffer.BlockCopy(_file_bytes, _offset, _file_chunk, 0, _length);
_offset += _length;
_chunk.ChunkSize = _length;
_chunk.Chunk = ByteString.CopyFrom(_file_chunk);
await responseStream.WriteAsync(_chunk).ConfigureAwait(false);
}
client-side-code
await foreach (var _chunk in _call.ResponseStream.ReadAllAsync().ConfigureAwait(false))
{
var _total_size = _chunk.FileSize;
if (!String.IsNullOrEmpty(_chunk.FileName))
{
_final_file = _chunk.FileName;
}
if (_chunk.Chunk.Length == _chunk.ChunkSize)
_fs.Write(_chunk.Chunk.ToByteArray());
else
{
_fs.Write(_chunk.Chunk.ToByteArray(), 0, _chunk.ChunkSize);
Console.WriteLine($"final chunk size: {_chunk.ChunkSize}");
}
}

Await Stream WriteAsync method is locking/freezing

I try stream audios with PushStreamContent and WebApi.
For this I coded something. Main code;
[HttpGet]
public HttpResponseMessage StreamCall(long callId,int playSpeed)
{
var audio = new AudioStreamHelper();
var response = Request.CreateResponse();
response.Content = new PushStreamContent((Action<Stream, HttpContent, TransportContext>) audio.WriteToStream, new MediaTypeHeaderValue("audio/wav"));
return response;
}
AudioStreamHelper code;
public class AudioStreamHelper
{
private readonly string _filename;
protected static readonly ILogger Logger = LogManager.GetLogger(MethodBase.GetCurrentMethod().DeclaringType.Name);
public AudioStreamHelper()
{
_filename = #"C:/195545.mp3";
}
public async void WriteToStream(Stream outputStream, HttpContent content, TransportContext context)
{
try
{
var buffer = new byte[12365536];
using (var audio = File.Open(_filename, FileMode.Open, FileAccess.Read,FileShare.ReadWrite))
{
var length = (int)audio.Length;
var bytesRead = 1;
while (length > 0 && bytesRead > 0)
{
bytesRead = audio.Read(buffer, 0, Math.Min(length, buffer.Length));
await outputStream.WriteAsync(buffer, 0, bytesRead);
length -= bytesRead;
}
}
}
catch (HttpException ex)
{
Logger.Error(ex.Message);
throw new UserFriendlyException(AppTexts.AudioStreamError);
}
finally
{
outputStream.Close();
}
}
}
When i debug this code; In AudioStreamHelper class WriteToStream method and await outputStream.WriteAsync(buffer, 0, bytesRead);line, my thread is locking/freezing and gives no response.I check current thread Id. It's same.(In Api and Helper class) Also, this code works another demo application with System.Net.Http.Formattion(version 4.0.0.0)
My project use System.Net.Http.Formattion(version 5.1.0.0)
Is associated with this condition ? I don't understand this situation.

Retrieving image from HTTPRuntime Cache

I am trying to save and retrieve an Image from HTTPRuntime Cache but I am getting an exception. I am able to save a stream to cache but when I try to retrieve it I get an exception saying:
the request was aborted. The connection was closed unexpectedly
Here is my code:
public void ProcessRequest(HttpContext context)
{
string courseKey = context.Request.QueryString["ck"];
string objKey = context.Request.QueryString["file"];
if(HttpRuntime.Cache[objKey] !=null)
{
using (Stream stream = (Stream)HttpRuntime.Cache[objKey]) // here is where I get an exception
{
var buffer = new byte[8000];
var bytesRead = -1;
while ((bytesRead = stream.Read(buffer, 0, buffer.Length)) > 0)
{
context.Response.OutputStream.Write(buffer, 0, bytesRead);
}
}
return;
}
var response = Gets3Response(objKey, courseKey, context);
if (response != null)
{
using (response)
{
var MIMEtype = response.ContentType;
context.Response.ContentType = MIMEtype;
var cacheControl = context.Response.CacheControl;
HttpRuntime.Cache.Insert(objKey, response.ResponseStream, null, DateTime.UtcNow.AddMinutes(20), Cache.NoSlidingExpiration);
using (Stream responseStream = response.ResponseStream)
{
var buffer = new byte[8000];
var bytesRead = -1;
while ((bytesRead = responseStream.Read(buffer, 0, buffer.Length)) > 0)
{
context.Response.OutputStream.Write(buffer, 0, bytesRead);
}
}
}
}
}
And here is the exception I am getting

This code is pretty confusing. First, you are caching response.ResponseStream, but that has been wrapped in a using block. So by the time you get to HttpRuntime.Cache.Insert, response.ResponseStream is already disposed and closed. Hence the error.
You should not be caching a stream. For one thing, once you put a distributed cache service in place, your approach will be impossible. You need to refactor this. Consider:
public class CacheAsset
{
public string FileName { get; set; }
public string ContentType { get; set; }
public byte[] Content { get; set; }
}
CacheAsset GetAsset(HttpContext context)
{
string courseKey = context.Request.QueryString["ck"];
string objKey = context.Request.QueryString["file"];
var asset = context.Cache[objKey] as CacheAsset;
if (asset != null) return asset;
using (var response = Gets3Response(objKey, courseKey, context))
using (var stream = new MemoryStream())
{
var buffer = new byte[8000];
var read = 0;
while ((read = response.ReponseStream.Read(buffer, 0, buffer.Length)) > 0)
{
stream.Write(buffer, 0, read);
}
asset = new CacheAsset
{
FileName = objKey,
ContentType = reponse.ContentType,
Content = stream.ToArray()
};
context.Cache.Insert(objKey, asset, null, DateTime.UtcNow.AddMinutes(20), Cache.NoSlidingExpiration);
}
return asset;
}
public void ProcessRequest(HttpContext context)
{
var asset = GetAsset(context);
context.Response.ContentType = asset.ContentType;
context.Response.BinaryWrite(asset.Content);
}