I am writing an ASP.NET Core Web API with .NET 5.0 as an exercise.
In MyController.cs there is the method DownloadZip(). Here, it should be possible for the client to download a zip file. By the way, I create a zip file because I did not achieve to transfer multiple pictures. That is the actual goal. Provisionally, the zip file is still stored in the picture folder. Of course, that should not happen either. I simply still have difficulties with web services and transferring zip files via them.
Anyway, in the line return File(fullName, "text/plain"); I get the following error message:
System.InvalidOperationException: No file provider has been configured to process the supplied file.
I found several threads on StackOverflow last Friday about how to transfer a zip file using a memory stream. When I do it this way, the browser shows the individual bytes, but no finished file has been downloaded.
Postings is a list(of post) with
using System;
using System.Collections.Generic;
namespace ImageRepository
{
public sealed class Posting
{
public DateTime CreationTime { get; set; }
public List<ImageProperties> Imageproperties { get; }
public Posting(DateTime creationTime, List<ImageProperties> imPr)
{
CreationTime = creationTime;
Imageproperties = imPr;
}
}
}
And Imageproperties is the following:
namespace ImageRepository
{
public sealed class ImageProperties
{
public string FullName { get; set; }
public string _Name { get; set; }
public byte[] DataBytes { get; set; }
public ImageProperties(string FullName, string Name, byte[] dataBytes)
{
this.FullName = FullName;
this._Name = Name;
this.DataBytes = dataBytes;
}
}
}
MyController.cs
using Microsoft.AspNetCore.Mvc;
using System.Collections.Generic;
using ImageRepository;
using System.IO.Compression;
namespace WebApp2.Controllers
{
[Route("api/[controller]")]
[ApiController]
public class MyController : ControllerBase
{
private readonly IImageTransferRepository imageRepository;
private readonly System.Globalization.CultureInfo Deu = new System.Globalization.CultureInfo("de-DE");
public MyController(IImageTransferRepository imageTransferRepository)
{
this.imageRepository = imageTransferRepository;
}
//––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––
[HttpGet("WhatAreTheNamesOfTheLatestPictures")] // Route will be https://localhost:44355/api/My/WhatAreTheNamesOfTheLatestPictures/
public ActionResult GetNamesOfNewestPosting()
{
List<string> imageNames = this.imageRepository.GetImageNames();
if (imageNames.Count == 0)
{
return NoContent();
}
return Ok(imageNames);
}
//––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––
[HttpGet("ImagesOfLatestPost")] //route will be https://localhost:44355/api/My/ImagesOfLatestPost
public ActionResult DownloadZip()
{
List<Posting> Postings = this.imageRepository.GetImages();
if (Postings is null || Postings.Count == 0)
{
return NoContent();
}
System.DateTime now = System.DateTime.Now;
string now_as_string = now.ToString("G", Deu).Replace(':', '-');
string folderPath = System.Environment.GetFolderPath(System.Environment.SpecialFolder.MyPictures);
string fullName = $"{folderPath}\\{now_as_string}.zip";
using (ZipArchive newFile = ZipFile.Open(fullName, ZipArchiveMode.Create))
{
for (int i = 0; i < Postings[0].Imageproperties.Count; i++)
{
newFile.CreateEntryFromFile(Postings[0].Imageproperties[i].FullName,
Postings[0].Imageproperties[i]._Name);
}
}
return File(fullName, "text/plain");
}
}
}
Edit June 20, 2022, 4:16 pm
Based on Bagus Tesa's comment, I wrote the following:
byte[] zip_as_ByteArray = System.IO.File.ReadAllBytes(fullName);
return File(zip_as_ByteArray, "application/zip");
The automatic download takes place, but I still have to rename the file by attaching (a) .zip so that Windows recognises it as a zip file.
Furthermore, there is still the problem that I am still creating the zip file on the hard disk (using (ZipArchive newFile = ZipFile.Open(fullName, ZipArchiveMode.Create))). How can I change this?
Thanks to the thread linked by Bagus Tesa, I can now answer my question. I have adapted a few things to my needs, see for-loop, because I have several images.
[HttpGet("ImagesOfLatestPost")] //route will be https://localhost:44355/api/My/ImagesOfLatestPost
public ActionResult DownloadZip()
{
List<Posting> Postings = this.imageRepository.GetImages();
if (Postings is null || Postings.Count == 0)
{
return NoContent();
}
byte[] compressedBytes;
using (var outStream = new System.IO.MemoryStream())
{
using (var archive = new ZipArchive(outStream, ZipArchiveMode.Create, true))
{
for (int i = 0; i < Postings[0].Imageproperties.Count; i++)
{
ZipArchiveEntry fileInArchive = archive.CreateEntry(Postings[0].Imageproperties[i]._Name, CompressionLevel.Optimal);
using System.IO.Stream entryStream = fileInArchive.Open();
using System.IO.MemoryStream fileToCompressStream = new System.IO.MemoryStream(Postings[0].Imageproperties[i].DataBytes);
fileToCompressStream.CopyTo(entryStream);
}
}
compressedBytes = outStream.ToArray();
}
return File(compressedBytes, "application/zip", $"Export_{System.DateTime.Now:yyyyMMddhhmmss}.zip");
}
This question already has answers here:
My console app shutdown prematurely when using async / await?
(4 answers)
Program exits upon calling await
(3 answers)
Closed 2 years ago.
Before you all go on a rampage about how this is a duplicate question, I have spent two days working on this issue, watching youtube tutorials on asynchronous programming, surfing similar stackoverflow posts etc, and I cannot for the life of me figure out how to apply Asynchronous Parallel Downloading of files into my project.
First things first, some background:
I am creating a program that, when given a query input via the user, will make a call to the twitch API and download clips.
My program is two parts
1- A web scraper that generates a .json file with all details needed to download files and
2 - A downloader.
Part 1 works perfectly fine and generates the .json files no trouble.
My Downloader contains reference to a Data class that is a handler for common properties and methods like my ClientID, Authentication, OutputPath, JsonFile, QueryURL. It also contains methods to give values to these properties.
Here are the two methods of my FileDownloader.cs that are the problem:
public async static void DownloadAllFiles(Data clientData)
{
data = clientData;
data.OutputFolderExists();
// Deserialize .json file and get ClipInfo list
List<ClipInfo> clips = JsonConvert.DeserializeObject<List<ClipInfo>>(File.ReadAllText(data.JsonFile));
tasks = new List<Task>();
foreach(ClipInfo clip in clips)
{
tasks.Add(DownloadFilesAsync(clip));
}
await Task.WhenAll(tasks);
}
private async static Task DownloadFilesAsync(ClipInfo clip)
{
WebClient client = new WebClient();
string url = GetClipURL(clip);
string filepath = data.OutputPath + clip.id + ".mp4";
await client.DownloadFileTaskAsync(new Uri(url), filepath);
}
This is only one of my many attempts of downloading files, one which I got the idea from this post:
stackoverflow_link
I have also tried methods like the following from a YouTube video by IAmTimCorey:
video_link
I have spent many an hour tackling this problem, and I honestly can't figure out why it won't work with any of my attempts. I would vastly appreciate your help.
Thanks,
Ben
Below is the entirety of my code, should anyone need it for any reason.
Code Structure:
The only external libraries I have downloaded is Newtonsoft.Json
ClipInfo.cs
using System;
using System.Collections.Generic;
using System.Text;
namespace Downloader
{
public class ClipInfo
{
public string id { get; set; }
public string url { get; set; }
public string embed_url { get; set; }
public string broadcaster_id { get; set; }
public string broadcaster_name { get; set; }
public string creator_id { get; set; }
public string creator_name { get; set; }
public string video_id { get; set; }
public string game_id { get; set; }
public string language { get; set; }
public string title { get; set; }
public int view_count { get; set; }
public DateTime created_at { get; set; }
public string thumbnail_url { get; set; }
}
}
Pagination.cs
namespace Downloader
{
public class Pagination
{
public string cursor { get; set; }
}
}
Root.cs
using System.Collections.Generic;
namespace Downloader
{
public class Root
{
public List<ClipInfo> data { get; set; }
public Pagination pagination { get; set; }
}
}
Data.cs
using System;
using System.IO;
namespace Downloader
{
public class Data
{
private static string directory = Directory.GetCurrentDirectory();
private readonly static string defaultJsonFile = directory + #"\clips.json";
private readonly static string defaultOutputPath = directory + #"\Clips\";
private readonly static string clipsLink = "https://api.twitch.tv/helix/clips?";
public string OutputPath { get; set; }
public string JsonFile { get; set; }
public string ClientID { get; private set; }
public string Authentication { get; private set; }
public string QueryURL { get; private set; }
public Data()
{
OutputPath = defaultOutputPath;
JsonFile = defaultJsonFile;
}
public Data(string clientID, string authentication)
{
ClientID = clientID;
Authentication = authentication;
OutputPath = defaultOutputPath;
JsonFile = defaultJsonFile;
}
public Data(string clientID, string authentication, string outputPath)
{
ClientID = clientID;
Authentication = authentication;
OutputPath = directory + #"\" + outputPath + #"\";
JsonFile = OutputPath + outputPath + ".json";
}
public void GetQuery()
{
Console.Write("Please enter your query: ");
QueryURL = clipsLink + Console.ReadLine();
}
public void GetClientID()
{
Console.WriteLine("Enter your client ID");
ClientID = Console.ReadLine();
}
public void GetAuthentication()
{
Console.WriteLine("Enter your Authentication");
Authentication = Console.ReadLine();
}
public void OutputFolderExists()
{
if (!Directory.Exists(OutputPath))
{
Directory.CreateDirectory(OutputPath);
}
}
}
}
JsonGenerator.cs
using System;
using System.IO;
using System.Net.Http.Headers;
using System.Net.Http;
using System.Threading.Tasks;
using Newtonsoft.Json;
using System.Linq;
namespace Downloader
{
public static class JsonGenerator
{
// This class has no constructor.
// You call the Generate methods, passing in all required data.
// The file will then be generated.
private static Data data;
public static async Task Generate(Data clientData)
{
data = clientData;
string responseContent = null;
// Loop that runs until the api request goes through
bool authError = true;
while (authError)
{
authError = false;
try
{
responseContent = await GetHttpResponse();
}
catch (HttpRequestException)
{
Console.WriteLine("Invalid authentication, please enter client-ID and authentication again!");
data.GetClientID();
data.GetAuthentication();
authError = true;
}
catch (Exception e)
{
Console.WriteLine(e.Message);
authError = true;
}
}
data.OutputFolderExists();
GenerateJson(responseContent);
}
// Returns the contents of the resopnse to the api call as a string
private static async Task<string> GetHttpResponse()
{
// Creating client
HttpClient client = new HttpClient();
if (data.QueryURL == null)
{
data.GetQuery();
}
// Setting up request
HttpRequestMessage requestMessage = new HttpRequestMessage(HttpMethod.Get, data.QueryURL);
// Adding Headers to request
requestMessage.Headers.Add("client-id", data.ClientID);
requestMessage.Headers.Authorization = new AuthenticationHeaderValue("Bearer", data.Authentication);
// Receiving response to the request
HttpResponseMessage responseMessage = await client.SendAsync(requestMessage);
// Gets the content of the response as a string
string responseContent = await responseMessage.Content.ReadAsStringAsync();
return responseContent;
}
// Generates or adds to the .json file that contains data on each clip
private static void GenerateJson(string responseContent)
{
// Parses the data from the response to the api request
Root responseResult = JsonConvert.DeserializeObject<Root>(responseContent);
// If the file doesn't exist, we need to create it and add a '[' at the start
if (!File.Exists(data.JsonFile))
{
FileStream file = File.Create(data.JsonFile);
file.Close();
// The array of json objects needs to be wrapped inside []
File.AppendAllText(data.JsonFile, "[\n");
}
else
{
// For a pre-existing .json file, The last object won't have a comma at the
// end of it so we need to add it now, before we add more objects
string[] jsonLines = File.ReadAllLines(data.JsonFile);
File.WriteAllLines(data.JsonFile, jsonLines.Take(jsonLines.Length - 1).ToArray());
File.AppendAllText(data.JsonFile, ",");
}
// If the file already exists, but there was no [ at the start for whatever reason,
// we need to add it
if (File.ReadAllText(data.JsonFile).Length == 0 || File.ReadAllText(data.JsonFile)[0] != '[')
{
File.WriteAllText(data.JsonFile, "[\n" + File.ReadAllText(data.JsonFile));
}
string json;
// Loops through each ClipInfo object that the api returned
for (int i = 0; i < responseResult.data.Count; i++)
{
// Serializes the ClipInfo object into a json style string
json = JsonConvert.SerializeObject(responseResult.data[i]);
// Adds the serialized contents of ClipInfo to the .json file
File.AppendAllText(data.JsonFile, json);
if (i != responseResult.data.Count - 1)
{
// All objects except the last require a comma at the end of the
// object in order to correctly format the array of json objects
File.AppendAllText(data.JsonFile, ",");
}
// Adds new line after object entry
File.AppendAllText(data.JsonFile, "\n");
}
// Adds the ] at the end of the file to close off the json objects array
File.AppendAllText(data.JsonFile, "]");
}
}
}
FileDownloader.cs
using Newtonsoft.Json;
using System;
using System.Collections.Generic;
using System.IO;
using System.Net;
using System.Threading.Tasks;
namespace Downloader
{
public class FileDownloader
{
private static Data data;
private static List<Task> tasks;
public async static void DownloadAllFiles(Data clientData)
{
data = clientData;
data.OutputFolderExists();
// Deserialize .json file and get ClipInfo list
List<ClipInfo> clips = JsonConvert.DeserializeObject<List<ClipInfo>>(File.ReadAllText(data.JsonFile));
tasks = new List<Task>();
foreach (ClipInfo clip in clips)
{
tasks.Add(DownloadFilesAsync(clip));
}
await Task.WhenAll(tasks);
}
private static void GetData()
{
if (data.ClientID == null)
{
data.GetClientID();
}
if (data.Authentication == null)
{
data.GetAuthentication();
}
if (data.QueryURL == null)
{
data.GetQuery();
}
}
private static string GetClipURL(ClipInfo clip)
{
// Example thumbnail URL:
// https://clips-media-assets2.twitch.tv/AT-cm%7C902106752-preview-480x272.jpg
// You can get the URL of the location of clip.mp4
// by removing the -preview.... from the thumbnail url */
string url = clip.thumbnail_url;
url = url.Substring(0, url.IndexOf("-preview")) + ".mp4";
return url;
}
private async static Task DownloadFilesAsync(ClipInfo clip)
{
WebClient client = new WebClient();
string url = GetClipURL(clip);
string filepath = data.OutputPath + clip.id + ".mp4";
await client.DownloadFileTaskAsync(new Uri(url), filepath);
}
private static void FileDownloadComplete(object sender, System.ComponentModel.AsyncCompletedEventArgs e)
{
tasks.Remove((Task)sender);
}
}
}
Program.cs
using System;
using System.Threading.Tasks;
using Downloader;
namespace ClipDownloader
{
class Program
{
private static string clientID = "{your_client_id}";
private static string authentication = "{your_authentication}";
async static Task Main(string[] args)
{
Console.WriteLine("Enter your output path");
string outputPath = Console.ReadLine();
Data data = new Data(clientID, authentication, outputPath);
Console.WriteLine(data.OutputPath);
//await JsonGenerator.Generate(data);
FileDownloader.DownloadAllFiles(data);
}
}
}
The example query I usually type in is "game_id=510218"
async void is your problem
Change
public static async void DownloadAllFiles(Data clientData)
To
public static async Task DownloadAllFiles(Data clientData)
Then you can await it
await FileDownloader.DownloadAllFiles(data);
The longer story:
async void runs unobserved (fire and forget). You can't wait for them to finish. In essence as soon as your program starts the task, it finishes, and tears down the App Domain and all your sub tasks, leading you to believe nothing is working.
I'm trying to stay on topic here as best as I can, but when using JsonConvert.DeserializeObject{T}, isn't T suppose to be an encapsulating root object type? I have never used it the way you're using it, so I'm just curious if that might be your bug. I could be completely wrong, and spare me if i am, but JSON is key:value based. Deserializing directly to a List doesn't really make sense. Unless there is a special case in the deserializer? List would be a file that's purely an array of ClipInfo values being deserialized into the members of List{T}(private T[] _items, private int _size, etc.) It needs a parent root object.
// current JSON file format implication(which i dont think is valid JSON?(correct me please)
clips:
[
// clip 1
{ "id": "", "url": "" },
// clip N
{ "id": "", "url": "" },
]
// correct(?) JSON file format
{ // { } is the outer encasing object
clips:
[
// clip 1
{ "id": "", "url": "" },
// clip N
{ "id": "", "url": "" },
]
}
class ClipInfoJSONFile
{
public List<ClipInfo> Info { get; set; }
}
var clipInfoList = JsonConverter.DeserializeObject<ClipInfoJSONFile>(...);
I'm writing a desktop application and wish to give users ability to verify network traffic, so they know they are not being abused. My application establishes a TLS connection to servers using .NET's SslStream with the AuthenticateAsClient method. Wireshark users can decode TLS traffic using NSS key logs. I can see that both Firefox and Chrome have options to logging encryption keys. How can I do the same in my .NET application? i.e. How can I extract the session key from SslStream, programatically?
As of this writing, there is no way of doing that with dotnet's SslStream. Here's how to export the session keys using BouncyCastle:
internal static class BouncyCastleTls
{
public static Stream WrapWithTls(Stream stream)
{
var client = new MyTlsClient();
var tlsClientProtocol = new TlsClientProtocol(stream, new SecureRandom());
tlsClientProtocol.Connect(client);
return tlsClientProtocol.Stream;
}
}
internal sealed class MyTlsClient : DefaultTlsClient
{
public override TlsAuthentication GetAuthentication()
{
return new MyTlsAuthentication();
}
public override void NotifyHandshakeComplete()
{
var clientRandom = mContext.SecurityParameters.ClientRandom;
var masterSecret = mContext.SecurityParameters.MasterSecret;
Console.WriteLine("CLIENT_RANDOM {0} {1}", ToHex(clientRandom), ToHex(masterSecret));
}
private static string ToHex(byte[] bytes)
{
var sb = new StringBuilder(bytes.Length * 2);
for (var i = 0; i < bytes.Length; ++i)
sb.Append($"{bytes[i]:x2}");
return sb.ToString();
}
}
internal sealed class MyTlsAuthentication : TlsAuthentication
{
public void NotifyServerCertificate(Certificate serverCertificate)
{
}
public TlsCredentials GetClientCredentials(CertificateRequest certificateRequest)
{
return null;
}
}
Using Azure storage, I'm writing to a blob using a stream. I have a method something like this:
public async Task<BlobSteamContainer> GetBlobStreamAsync(string filename, string contentType = "text/csv")
{
var blob = container.GetBlockBlobReference($"{filename}--{Guid.NewGuid().ToString()}.csv");
blob.Properties.ContentType = contentType;
return new BlobSteamContainer(blob.Uri.ToString(), await blob.OpenWriteAsync());
}
Where BlobStreamContainer is just a simple object so I can keep track of the filename and the stream together:
public class BlobSteamContainer : IDisposable
{
public CloudBlobStream Stream { get; private set; }
public string Filename { get; private set; }
public BlobSteamContainer(string filename, CloudBlobStream stream)
{
Stream = stream;
Filename = filename;
}
public void Dispose()
{
Stream.Close();
Stream?.Dispose();
}
}
And then I use it something like this:
using (var blobStream = await GetBlobStreamAsync(filename))
using (var outputStream = new StreamWriter(blobStream.Stream))
using (var someInputStream = ...)
{
try
{
outputStream.WriteLine("write some stuff...");
//....processing
if (someCondition) {
throw new MyException("can't write the file");
}
//....more processing
outputStream.Flush();
}
catch(MyException e)
{
// what to do here? I want to stop writing
// and remove any trace of the file in azure
throw; // let the higher ups handle this
}
}
Where somecondition is something that I know before hand (obviously there's more going on involving processing an input stream and writing out as I go). If everything is fine, then this works great. My problem is on figuring out the best way to handle the case where an exception is thrown during writing.
I tried just deleting the file in the catch like this:
DeleteBlob(blobStream.Filename);
where:
public void DeleteBlob(string filename)
{
var blob = container.GetBlobReference(filename);
blob.Delete();
}
But the problem is that the file might not have been created yet and so this will throw a Microsoft.WindowsAzure.Storage.StorageException telling me the file wasn't found (and then the file will end up getting created anyway!).
So what would be the cleanest way to handle this?
I have code:
public Upload.UploadResponse Post(Upload.UploadRequest request)
{
Stream str = request.RequestStream; // RequestStream is System.Web.HttpInputStream
byte[] result;
using (var streamReader = new MemoryStream())
{
str.CopyTo(streamReader);
result = streamReader.ToArray();
}
return new Upload.UploadResponse() { Successed = 1 };
}
Is there any way to get file name ( with extension) from MemoryStream or stream or System.Web.HttpInputStream (part of Upload.UploadRequest request) without saving the file? I need to recognize the file without knowing what is sent to me. I've tried to cast it to FileStream but it was null. Service framework that I am using is service stack ServiceStack
edit: Maybe I need to send file info with request?
p.s sorry for my poor English any corrections are welcome
EDIT:
this is UploadClass that I am using for code above
public class Upload
{
[Route("/upload")]
public class UploadRequest : IRequiresRequestStream
{
public System.IO.Stream RequestStream { set; get; }
}
public class UploadResponse
{
public int Successed { set; get; }
}
}
You cannot extract file name from stream.
You need to add FileName property to your request.