I am using asp net web api. I need to create an update request to my client. Example i need my client to retreive all necessery files for making an update om his application.
So i have create a get method witch converts all files from server to bytes, then i am place them inside a list and i send it to my client. There are 15 total files. Total size is 150mbytes and after compression is 70mbytes.
public class DataFiles
{
public string FileName { get; set; }
public byte[] Data { get; set; }
}
public IHttpActionResult Get(string version)
{
var mItems = new List<DataFiles>();
var path = "C:\\Versions\\" + version;
if (Directory.Exists(path))
{
DirectoryInfo d = new DirectoryInfo(path);//Assuming Test is your Folder
FileInfo[] Files = d.GetFiles("*"); //Getting all files
foreach (FileInfo file in Files)
{
mItems.Add(new DataFiles
{
FileName = file.Name,
Data = Compress(File.ReadAllBytes(file.FullName))
});
}
}
return Ok(mItems);
}
public static byte[] Compress(byte[] data)
{
var output = new MemoryStream();
using (var dstream = new DeflateStream(output, CompressionLevel.Optimal))
{
dstream.Write(data, 0, data.Length);
}
return output.ToArray();
}
Is this method right? Shoud i send each file one by one? Should i use memorystream?
This question already has answers here:
My console app shutdown prematurely when using async / await?
(4 answers)
Program exits upon calling await
(3 answers)
Closed 2 years ago.
Before you all go on a rampage about how this is a duplicate question, I have spent two days working on this issue, watching youtube tutorials on asynchronous programming, surfing similar stackoverflow posts etc, and I cannot for the life of me figure out how to apply Asynchronous Parallel Downloading of files into my project.
First things first, some background:
I am creating a program that, when given a query input via the user, will make a call to the twitch API and download clips.
My program is two parts
1- A web scraper that generates a .json file with all details needed to download files and
2 - A downloader.
Part 1 works perfectly fine and generates the .json files no trouble.
My Downloader contains reference to a Data class that is a handler for common properties and methods like my ClientID, Authentication, OutputPath, JsonFile, QueryURL. It also contains methods to give values to these properties.
Here are the two methods of my FileDownloader.cs that are the problem:
public async static void DownloadAllFiles(Data clientData)
{
data = clientData;
data.OutputFolderExists();
// Deserialize .json file and get ClipInfo list
List<ClipInfo> clips = JsonConvert.DeserializeObject<List<ClipInfo>>(File.ReadAllText(data.JsonFile));
tasks = new List<Task>();
foreach(ClipInfo clip in clips)
{
tasks.Add(DownloadFilesAsync(clip));
}
await Task.WhenAll(tasks);
}
private async static Task DownloadFilesAsync(ClipInfo clip)
{
WebClient client = new WebClient();
string url = GetClipURL(clip);
string filepath = data.OutputPath + clip.id + ".mp4";
await client.DownloadFileTaskAsync(new Uri(url), filepath);
}
This is only one of my many attempts of downloading files, one which I got the idea from this post:
stackoverflow_link
I have also tried methods like the following from a YouTube video by IAmTimCorey:
video_link
I have spent many an hour tackling this problem, and I honestly can't figure out why it won't work with any of my attempts. I would vastly appreciate your help.
Thanks,
Ben
Below is the entirety of my code, should anyone need it for any reason.
Code Structure:
The only external libraries I have downloaded is Newtonsoft.Json
ClipInfo.cs
using System;
using System.Collections.Generic;
using System.Text;
namespace Downloader
{
public class ClipInfo
{
public string id { get; set; }
public string url { get; set; }
public string embed_url { get; set; }
public string broadcaster_id { get; set; }
public string broadcaster_name { get; set; }
public string creator_id { get; set; }
public string creator_name { get; set; }
public string video_id { get; set; }
public string game_id { get; set; }
public string language { get; set; }
public string title { get; set; }
public int view_count { get; set; }
public DateTime created_at { get; set; }
public string thumbnail_url { get; set; }
}
}
Pagination.cs
namespace Downloader
{
public class Pagination
{
public string cursor { get; set; }
}
}
Root.cs
using System.Collections.Generic;
namespace Downloader
{
public class Root
{
public List<ClipInfo> data { get; set; }
public Pagination pagination { get; set; }
}
}
Data.cs
using System;
using System.IO;
namespace Downloader
{
public class Data
{
private static string directory = Directory.GetCurrentDirectory();
private readonly static string defaultJsonFile = directory + #"\clips.json";
private readonly static string defaultOutputPath = directory + #"\Clips\";
private readonly static string clipsLink = "https://api.twitch.tv/helix/clips?";
public string OutputPath { get; set; }
public string JsonFile { get; set; }
public string ClientID { get; private set; }
public string Authentication { get; private set; }
public string QueryURL { get; private set; }
public Data()
{
OutputPath = defaultOutputPath;
JsonFile = defaultJsonFile;
}
public Data(string clientID, string authentication)
{
ClientID = clientID;
Authentication = authentication;
OutputPath = defaultOutputPath;
JsonFile = defaultJsonFile;
}
public Data(string clientID, string authentication, string outputPath)
{
ClientID = clientID;
Authentication = authentication;
OutputPath = directory + #"\" + outputPath + #"\";
JsonFile = OutputPath + outputPath + ".json";
}
public void GetQuery()
{
Console.Write("Please enter your query: ");
QueryURL = clipsLink + Console.ReadLine();
}
public void GetClientID()
{
Console.WriteLine("Enter your client ID");
ClientID = Console.ReadLine();
}
public void GetAuthentication()
{
Console.WriteLine("Enter your Authentication");
Authentication = Console.ReadLine();
}
public void OutputFolderExists()
{
if (!Directory.Exists(OutputPath))
{
Directory.CreateDirectory(OutputPath);
}
}
}
}
JsonGenerator.cs
using System;
using System.IO;
using System.Net.Http.Headers;
using System.Net.Http;
using System.Threading.Tasks;
using Newtonsoft.Json;
using System.Linq;
namespace Downloader
{
public static class JsonGenerator
{
// This class has no constructor.
// You call the Generate methods, passing in all required data.
// The file will then be generated.
private static Data data;
public static async Task Generate(Data clientData)
{
data = clientData;
string responseContent = null;
// Loop that runs until the api request goes through
bool authError = true;
while (authError)
{
authError = false;
try
{
responseContent = await GetHttpResponse();
}
catch (HttpRequestException)
{
Console.WriteLine("Invalid authentication, please enter client-ID and authentication again!");
data.GetClientID();
data.GetAuthentication();
authError = true;
}
catch (Exception e)
{
Console.WriteLine(e.Message);
authError = true;
}
}
data.OutputFolderExists();
GenerateJson(responseContent);
}
// Returns the contents of the resopnse to the api call as a string
private static async Task<string> GetHttpResponse()
{
// Creating client
HttpClient client = new HttpClient();
if (data.QueryURL == null)
{
data.GetQuery();
}
// Setting up request
HttpRequestMessage requestMessage = new HttpRequestMessage(HttpMethod.Get, data.QueryURL);
// Adding Headers to request
requestMessage.Headers.Add("client-id", data.ClientID);
requestMessage.Headers.Authorization = new AuthenticationHeaderValue("Bearer", data.Authentication);
// Receiving response to the request
HttpResponseMessage responseMessage = await client.SendAsync(requestMessage);
// Gets the content of the response as a string
string responseContent = await responseMessage.Content.ReadAsStringAsync();
return responseContent;
}
// Generates or adds to the .json file that contains data on each clip
private static void GenerateJson(string responseContent)
{
// Parses the data from the response to the api request
Root responseResult = JsonConvert.DeserializeObject<Root>(responseContent);
// If the file doesn't exist, we need to create it and add a '[' at the start
if (!File.Exists(data.JsonFile))
{
FileStream file = File.Create(data.JsonFile);
file.Close();
// The array of json objects needs to be wrapped inside []
File.AppendAllText(data.JsonFile, "[\n");
}
else
{
// For a pre-existing .json file, The last object won't have a comma at the
// end of it so we need to add it now, before we add more objects
string[] jsonLines = File.ReadAllLines(data.JsonFile);
File.WriteAllLines(data.JsonFile, jsonLines.Take(jsonLines.Length - 1).ToArray());
File.AppendAllText(data.JsonFile, ",");
}
// If the file already exists, but there was no [ at the start for whatever reason,
// we need to add it
if (File.ReadAllText(data.JsonFile).Length == 0 || File.ReadAllText(data.JsonFile)[0] != '[')
{
File.WriteAllText(data.JsonFile, "[\n" + File.ReadAllText(data.JsonFile));
}
string json;
// Loops through each ClipInfo object that the api returned
for (int i = 0; i < responseResult.data.Count; i++)
{
// Serializes the ClipInfo object into a json style string
json = JsonConvert.SerializeObject(responseResult.data[i]);
// Adds the serialized contents of ClipInfo to the .json file
File.AppendAllText(data.JsonFile, json);
if (i != responseResult.data.Count - 1)
{
// All objects except the last require a comma at the end of the
// object in order to correctly format the array of json objects
File.AppendAllText(data.JsonFile, ",");
}
// Adds new line after object entry
File.AppendAllText(data.JsonFile, "\n");
}
// Adds the ] at the end of the file to close off the json objects array
File.AppendAllText(data.JsonFile, "]");
}
}
}
FileDownloader.cs
using Newtonsoft.Json;
using System;
using System.Collections.Generic;
using System.IO;
using System.Net;
using System.Threading.Tasks;
namespace Downloader
{
public class FileDownloader
{
private static Data data;
private static List<Task> tasks;
public async static void DownloadAllFiles(Data clientData)
{
data = clientData;
data.OutputFolderExists();
// Deserialize .json file and get ClipInfo list
List<ClipInfo> clips = JsonConvert.DeserializeObject<List<ClipInfo>>(File.ReadAllText(data.JsonFile));
tasks = new List<Task>();
foreach (ClipInfo clip in clips)
{
tasks.Add(DownloadFilesAsync(clip));
}
await Task.WhenAll(tasks);
}
private static void GetData()
{
if (data.ClientID == null)
{
data.GetClientID();
}
if (data.Authentication == null)
{
data.GetAuthentication();
}
if (data.QueryURL == null)
{
data.GetQuery();
}
}
private static string GetClipURL(ClipInfo clip)
{
// Example thumbnail URL:
// https://clips-media-assets2.twitch.tv/AT-cm%7C902106752-preview-480x272.jpg
// You can get the URL of the location of clip.mp4
// by removing the -preview.... from the thumbnail url */
string url = clip.thumbnail_url;
url = url.Substring(0, url.IndexOf("-preview")) + ".mp4";
return url;
}
private async static Task DownloadFilesAsync(ClipInfo clip)
{
WebClient client = new WebClient();
string url = GetClipURL(clip);
string filepath = data.OutputPath + clip.id + ".mp4";
await client.DownloadFileTaskAsync(new Uri(url), filepath);
}
private static void FileDownloadComplete(object sender, System.ComponentModel.AsyncCompletedEventArgs e)
{
tasks.Remove((Task)sender);
}
}
}
Program.cs
using System;
using System.Threading.Tasks;
using Downloader;
namespace ClipDownloader
{
class Program
{
private static string clientID = "{your_client_id}";
private static string authentication = "{your_authentication}";
async static Task Main(string[] args)
{
Console.WriteLine("Enter your output path");
string outputPath = Console.ReadLine();
Data data = new Data(clientID, authentication, outputPath);
Console.WriteLine(data.OutputPath);
//await JsonGenerator.Generate(data);
FileDownloader.DownloadAllFiles(data);
}
}
}
The example query I usually type in is "game_id=510218"
async void is your problem
Change
public static async void DownloadAllFiles(Data clientData)
To
public static async Task DownloadAllFiles(Data clientData)
Then you can await it
await FileDownloader.DownloadAllFiles(data);
The longer story:
async void runs unobserved (fire and forget). You can't wait for them to finish. In essence as soon as your program starts the task, it finishes, and tears down the App Domain and all your sub tasks, leading you to believe nothing is working.
I'm trying to stay on topic here as best as I can, but when using JsonConvert.DeserializeObject{T}, isn't T suppose to be an encapsulating root object type? I have never used it the way you're using it, so I'm just curious if that might be your bug. I could be completely wrong, and spare me if i am, but JSON is key:value based. Deserializing directly to a List doesn't really make sense. Unless there is a special case in the deserializer? List would be a file that's purely an array of ClipInfo values being deserialized into the members of List{T}(private T[] _items, private int _size, etc.) It needs a parent root object.
// current JSON file format implication(which i dont think is valid JSON?(correct me please)
clips:
[
// clip 1
{ "id": "", "url": "" },
// clip N
{ "id": "", "url": "" },
]
// correct(?) JSON file format
{ // { } is the outer encasing object
clips:
[
// clip 1
{ "id": "", "url": "" },
// clip N
{ "id": "", "url": "" },
]
}
class ClipInfoJSONFile
{
public List<ClipInfo> Info { get; set; }
}
var clipInfoList = JsonConverter.DeserializeObject<ClipInfoJSONFile>(...);
Good day,
There is MVC backend - How solution should be designed for this type of problem?
So FE send this array of files:
FormData:[
file0: (binary)
file1: (binary)
file2: (binary)
]
Then on BE Side there is Controller:
Currently solutions store those files to filesystem.
[AllowAnonymous]
[HttpPost("file")]
public async Task<ActionResult> PostAsync([FromForm] List<IFormFile> files)
{
long size = files.Sum(f => f.Length);
Console.WriteLine(files);
Console.WriteLine(size);
foreach (var formFile in files)
{
if (formFile.Length > 0)
{
var filePath = Path.Combine(Directory.GetCurrentDirectory(), "wwwroot", formFile.FileName);
using var stream = System.IO.File.Create(filePath);
await formFile.CopyToAsync(stream);
}
}
// Process uploaded files
// Don't rely on or trust the FileName property without validation.
return Ok(new { count = files.Count, size });
}
There is a model with that Files (it should have more properties in future)
namespace WebApi.Models
{
public class Model
{
public string Name { get; set; }
public byte[] Files { get; set; }
}
}
The question is: How map this array of binary object to Example model and store this Materials data in database(postgress) ?
It sounds like your problem can be distilled down to two different questions:
How do I get a byte[] from IFormFile?
foreach (var formFile in files)
{
if (formFile.Length > 0)
{
using var memoryStream = new MemoryStream();
await formFile.CopyToAsync(memoryStream);
var model = new Model
{
Name = formFile.Name,
Files = memoryStream.ToArray()
};
// add model to database
}
}
How do I save a byte[] in PostgreSQL?
Using Npgsql, you can directly map a byte[] in C# to a bytea column in PostgreSQL.
I have tried alot of solution but no one is working in my case, my question is simple but i cant find any answer specially for windows build. I have tried to load json form persistent , streaming and resource all are working good in android but no any solution work for windows build. here is my code please help me.
public GameData gameData;
private void LoadGameData()
{
string path = "ItemsData";
TextAsset targetFile = Resources.Load<TextAsset>(path);
string json = targetFile.text;
gameData = ResourceHelper.DecodeObject<GameData>(json);
// gameData = JsonUtility.FromJson<GameData>(json);
print(gameData.Items.Count);
}
here is my data class
[Serializable]
public class GameData
{
[SerializeField]
public List<Item> Items;
public GameData()
{
Items = new List<Item>();
}
}
public class Item
{
public string Id;
public string[] TexturesArray;
public bool Active;
public Item()
{
}
public Item(string _id, string[] _textureArray , bool _active = true)
{
Id = _id;
TexturesArray = _textureArray;
Active = _active;
}
}
In order to be (de)serialized Item should be [Serializable]
using System;
...
[Serializable]
public class Item
{
...
}
Then you could simply use Unity's built-in JsonUtility.FromJson:
public GameData gameData;
private void LoadGameData()
{
string path = "ItemsData";
TextAsset targetFile = Resources.Load<TextAsset>(path);
string json = targetFile.text;
gameData = JsonUtility.FromJson<GameData>(json);
print(gameData.Items.Count);
}
For loading something from e.g. persistentDataPath I always use something like
var path = Path.Combine(Application.persistentDataPath, "FileName.txt")
using (var fs = File.Open(path, FileMode.Open, FileAccess.Read, FileShare.Read))
{
using (var sr = new StreamReader(fs))
{
var json = sr.ReadToEnd();
...
}
}
For development I actually place my file into StreamingAssets (streamingAssetsPath) while running the code in the Unity Editor.
Then on runtime I read from persistentFilePath. If the file is not there I first copy it from the streamingassetsPath the first time.
Here I wrote more about this approach
How can I read an Excel spreadsheet that was just posted to my server?
I searched for something but I only found how to read an Excel spreadsheet with the file name path which is not my case.
I need something like that:
public ActionResult Import(HttpPostedFileBase file)
{
var excel = new ExcelQueryFactory(file); //using linq to excel
}
I was running into your same issue but I didn't want to switch to a paid service so this is what I did.
public class DataImportHelper : IDisposable
{
private readonly string _fileName;
private readonly string _tempFilePath;
public DataImportHelper(HttpPostedFileBase file, string tempFilePath)
{
_fileName = file.FileName;
_tempFilePath = Path.Combine(tempFilePath, _fileName);
(new FileInfo(_tempFilePath)).Directory.Create();
file.SaveAs(_tempFilePath);
}
public IQueryable<T> All<T>(string sheetName = "")
{
if (string.IsNullOrEmpty(sheetName))
{
sheetName = (typeof (T)).Name;
}
var excelSheet = new ExcelQueryFactory(_tempFilePath);
return from t in excelSheet.Worksheet<T>(sheetName)
select t;
}
public void Dispose()
{
File.Delete(_tempFilePath);
}
}
Here is a Test
[Fact]
public void AcceptsAMemoryStream()
{
MemoryFile file;
using (var f = File.OpenRead("SampleData.xlsx"))
{
file = new MemoryFile(f, "multipart/form-data", "SampleData.xlsx");
using (var importer = new DataImportHelper(file, "Temp/"))
{
var products = importer.All<Product>();
Assert.NotEmpty(products);
}
}
}
Here is MemoryFile.cs. This file is only used for testing. It is just an implementation of HttpPostedFileBase so you can test your controllers and my little helper. This was borrowed from another post.
public class MemoryFile : HttpPostedFileBase
{
Stream stream;
string contentType;
string fileName;
public MemoryFile(Stream stream, string contentType, string fileName)
{
this.stream = stream;
this.contentType = contentType;
this.fileName = fileName;
}
public override int ContentLength
{
get { return (int)stream.Length; }
}
public override string ContentType
{
get { return contentType; }
}
public override string FileName
{
get { return fileName; }
}
public override Stream InputStream
{
get { return stream; }
}
public override void SaveAs(string filename)
{
using (var file = File.Open(filename, FileMode.Create))
stream.CopyTo(file);
}
}
Unfortunately it's not possible to read a spreadsheet from a stream with LinqToExcel.
That's because it uses OLEDB to read from the spreadsheets and it can't read from a stream.
You can use the InputStream property of HttpPostedFileBase to read the excel spreadsheet in memory.
I use ClosedXML nuget package to read excel content from stream which is available in your case. It has a simple overload which takes stream pointing to stream for the excel file (aka workbook).
imported namespaces at the top of the code file:
using ClosedXML.Excel;
Source code:
public ActionResult Import(HttpPostedFileBase file)
{
//HttpPostedFileBase directly is of no use so commented your code
//var excel = new ExcelQueryFactory(file); //using linq to excel
var stream = file.InputStream;
if (stream.Length != 0)
{
//handle the stream here
using (XLWorkbook excelWorkbook = new XLWorkbook(stream))
{
var name = excelWorkbook.Worksheet(1).Name;
//do more things whatever you like as you now have a handle to the entire workbook.
var firstRow = excelWorkbook.Worksheet(1).Row(1);
}
}
}
You need Office Interops assemblies. Check the Excel Object Model for reference.