By downloading a file with UnityEngine.WWW, I get the error
OverflowException: Number overflow.
I found out the error is caused form the structure itself, because the byte-array has more bytes than int.MaxValue can allocate (~2GB).
The error is fired by returning the array with www.bytes, which means, that the framework probably stores the array in on other way.
How can I access the downloaded data in another way or is there an alternative for bigger files?
public IEnumerator downloadFile()
{
WWW www = new WWW(filesource);
while(!www.isDone)
{
progress = www.progress;
yield return null;
}
if(string.IsNullOrEmpty(www.error))
{
data = www.bytes; // <- Errormessage fired here
}
}
New answer (Unity 2017.2 and above)
Use UnityWebRequest with DownloadHandlerFile. The DownloadHandlerFile class is new and is used to download and save file directly while preventing high memory usage.
IEnumerator Start()
{
string url = "http://dl3.webmfiles.org/big-buck-bunny_trailer.webm";
string vidSavePath = Path.Combine(Application.persistentDataPath, "Videos");
vidSavePath = Path.Combine(vidSavePath, "MyVideo.webm");
//Create Directory if it does not exist
if (!Directory.Exists(Path.GetDirectoryName(vidSavePath)))
{
Directory.CreateDirectory(Path.GetDirectoryName(vidSavePath));
}
var uwr = new UnityWebRequest(url);
uwr.method = UnityWebRequest.kHttpVerbGET;
var dh = new DownloadHandlerFile(vidSavePath);
dh.removeFileOnAbort = true;
uwr.downloadHandler = dh;
yield return uwr.SendWebRequest();
if (uwr.isNetworkError || uwr.isHttpError)
Debug.Log(uwr.error);
else
Debug.Log("Download saved to: " + vidSavePath.Replace("/", "\\") + "\r\n" + uwr.error);
}
OLD answer (Unity 2017.1 and below) Use if you want to access each bytes while the file is downloading)
A problem like this is why Unity's UnityWebRequest was made but it won't work directly because WWW API is now implemented on top of the UnityWebRequest API in the newest version of Unity which means that if you get error with the WWW API, you will also likely get that same error with UnityWebRequest. Even if it works, you'll likely have have issues on mobile devices with the small ram like Android.
What to do is use UnityWebRequest's DownloadHandlerScript feature which allows you to download data in chunks. By downloading data in chunks, you can prevent causing the overflow error. The WWW API did not implement this feature so UnityWebRequest and DownloadHandlerScript must be used to download the data in chunks. You can read how this works here.
While this should solve your current issue, you may run into another memory issue when trying to save that large data with File.WriteAllBytes. Use FileStream to do the saving part and close it only when the download has finished.
Create a custom UnityWebRequest for downloading data in chunks as below:
using System;
using System.IO;
using UnityEngine;
using UnityEngine.Networking;
public class CustomWebRequest : DownloadHandlerScript
{
// Standard scripted download handler - will allocate memory on each ReceiveData callback
public CustomWebRequest()
: base()
{
}
// Pre-allocated scripted download handler
// Will reuse the supplied byte array to deliver data.
// Eliminates memory allocation.
public CustomWebRequest(byte[] buffer)
: base(buffer)
{
Init();
}
// Required by DownloadHandler base class. Called when you address the 'bytes' property.
protected override byte[] GetData() { return null; }
// Called once per frame when data has been received from the network.
protected override bool ReceiveData(byte[] byteFromServer, int dataLength)
{
if (byteFromServer == null || byteFromServer.Length < 1)
{
Debug.Log("CustomWebRequest :: ReceiveData - received a null/empty buffer");
return false;
}
//Write the current data chunk to file
AppendFile(byteFromServer, dataLength);
return true;
}
//Where to save the video file
string vidSavePath;
//The FileStream to save the file
FileStream fileStream = null;
//Used to determine if there was an error while opening or saving the file
bool success;
void Init()
{
vidSavePath = Path.Combine(Application.persistentDataPath, "Videos");
vidSavePath = Path.Combine(vidSavePath, "MyVideo.webm");
//Create Directory if it does not exist
if (!Directory.Exists(Path.GetDirectoryName(vidSavePath)))
{
Directory.CreateDirectory(Path.GetDirectoryName(vidSavePath));
}
try
{
//Open the current file to write to
fileStream = new FileStream(vidSavePath, FileMode.OpenOrCreate, FileAccess.ReadWrite);
Debug.Log("File Successfully opened at" + vidSavePath.Replace("/", "\\"));
success = true;
}
catch (Exception e)
{
success = false;
Debug.LogError("Failed to Open File at Dir: " + vidSavePath.Replace("/", "\\") + "\r\n" + e.Message);
}
}
void AppendFile(byte[] buffer, int length)
{
if (success)
{
try
{
//Write the current data to the file
fileStream.Write(buffer, 0, length);
Debug.Log("Written data chunk to: " + vidSavePath.Replace("/", "\\"));
}
catch (Exception e)
{
success = false;
}
}
}
// Called when all data has been received from the server and delivered via ReceiveData
protected override void CompleteContent()
{
if (success)
Debug.Log("Done! Saved File to: " + vidSavePath.Replace("/", "\\"));
else
Debug.LogError("Failed to Save File to: " + vidSavePath.Replace("/", "\\"));
//Close filestream
fileStream.Close();
}
// Called when a Content-Length header is received from the server.
protected override void ReceiveContentLength(int contentLength)
{
//Debug.Log(string.Format("CustomWebRequest :: ReceiveContentLength - length {0}", contentLength));
}
}
How to use:
UnityWebRequest webRequest;
//Pre-allocate memory so that this is not done each time data is received
byte[] bytes = new byte[2000];
IEnumerator Start()
{
string url = "http://dl3.webmfiles.org/big-buck-bunny_trailer.webm";
webRequest = new UnityWebRequest(url);
webRequest.downloadHandler = new CustomWebRequest(bytes);
webRequest.SendWebRequest();
yield return webRequest;
}
Related
I am trying to figure out how to stop UnityWebRequest download immediately with a button in android application.
Stopping coroutine with StopCoroutine("downLoadFromServer"); does not stop UnityWebRequest.
I tried using UnityWebRequest.Dispose(); or UnityWebRequest.Abort(); with no success.
Below is the download i'm trying to stop.
private IEnumerator downLoadFromServer()
{
var url = "https://example.com/app.apk";
var savePath = Path.Combine(Application.persistentDataPath, "data", "app.apk");
using (var uwr = UnityWebRequest.Get(url))
{
uwr.SetRequestHeader("Authorization", "Basic " + System.Convert.ToBase64String(System.Text.Encoding.ASCII.GetBytes("test:test")));
uwr.SendWebRequest();
while (!uwr.isDone)
{
progressText.text = $"Progress: {uwr.downloadProgress:P}";
yield return null;
}
var yourBytes = uwr.downloadHandler.data;
progressText.text = $"Done downloading. Size: {yourBytes.Length}";
//Create Directory if it does not exist
var directoryName = Path.GetDirectoryName(savePath);
if (!Directory.Exists(directoryName))
{
Directory.CreateDirectory(directoryName);
progressText.text = "Created Dir";
}
try
{
//Now Save it
System.IO.File.WriteAllBytes(savePath, yourBytes);
Debug.Log("Saved Data to: " + savePath.Replace("/", "\\"));
progressText.text = "Saved Data";
}
catch (Exception e)
{
Debug.LogWarning("Failed To Save Data to: " + savePath.Replace("/", "\\"));
Debug.LogWarning("Error: " + e.Message);
progressText.text = "Error Saving Data";
}
}
//Install APK
installApp(savePath);
}
Thank you,
Chris
You are right: Stopping the Coroutine does not cancel the UnityWebRequest, you are just not keeping track of it anymore.
Actually UnityWebRequest.Abort should do exactly that
If in progress, halts the UnityWebRequest as soon as possible.
This method may be called at any time. If the UnityWebRequest has not already completed, the UnityWebRequest will halt uploading or downloading data as soon as possible. Aborted UnityWebRequests are considered to have encountered a system error. Either the isNetworkError or the isHttpError property will return true and the error property will be "User Aborted".
However this makes ofcourse only sense if you actually check the result of the request which currently you aren't doing and you should do anyway!
A way to use it would be to make the uwr a class wide field so you can access it from somewhere else and as said actually add a check if the download succeeded:
private UnityWebRequest uwr;
private IEnumerator downLoadFromServer()
{
...
// NOTE: Remove the var so the class field uwr is assigned to, not only
// a local variable
using(uwr = UnityWebRequest.Get(...))
{
...
// Always check if a request was actually successful before continuing
if(uwr.isHttpError || uwr.isNetworkError || !string.IsNullOrWhiteSpace(uwr.error))
{
Debug.LogWarning($"Download Failed with {uwr.responseCode}, reason: {uwr.error}", this);
progressText.text = $"Download Failed: {uwr.error}";
// Cancel this Coroutine
yield break;
}
var yourBytes = uwr.downloadHandler.data;
...
}
//Install APK
installApp(savePath);
}
Then you can simply do
public void AbortDownload()
{
if(uwr != null && !uwr.isDone)
{
Debug.Log("Aborting download ...", this);
uwr.Abort();
}
else
{
Debug.Log("Not downloading, nothing to do ...", this);
}
}
which should make the download "fail" and thereby also finish the Coroutine
I am trying to upload a file into the server which is a .json file.I am sending the Binary data along with file name and Mimetype but in the backend it does not show which file extension it is.
How does this thing work?What happens if I have same filename but different extensions to upload into the server.Wht will happen if I want to upload and later deserialize it as two different extensions?
IEnumerator UploadFileData()
{
string mapnamedl = "123";
Debug.Log("Mapname local = " + mapnamedl);
string locapath = "file://" + Application.persistentDataPath + "/" + mapnamedl + ".json";
Debug.Log("local path = " + locapath);
WWW localFile = new WWW(locapath);
yield return localFile;
if (localFile.error == null)
{
Debug.Log("Local file found successfully");
}
else
{
Debug.Log("Open file error: " + localFile.error);
yield break; // stop the coroutine here
}
Debug.Log("local file text" + localFile.text);
WWWForm postForm = new WWWForm();
string mimetypes = "application/json";
postForm.AddBinaryData("Jsondata", localFile.bytes, mapnamedl, mimetypes);
UnityWebRequest www = UnityWebRequest.Post("https://test.com/car/hariupload/save.php", postForm);
yield return www.SendWebRequest();
if (www.isNetworkError || www.isHttpError)
{
Debug.Log(www.error);
}
else
{
string JSONDATAstring = www.downloadHandler.text;
Debug.Log("Json String is = " + JSONDATAstring);
JSONNode JNode = SimpleJSON.JSON.Parse(JSONDATAstring);
string login = (JNode["upload"][0]["success"]).ToString();
Debug.Log("login is = " + login);
if (login == "1")
{
Debug.Log("Form upload complete!");
}
else if (login == "0")
{
Debug.Log("Failed ");
}
}
}
+if you have binary (non-alphanumeric) data (or a significantly sized payload) to transmit, use multipart/form-data. Otherwise, use application/x-www-form-urlencoded.
-if you have to send non-ASCII text or large binary data, the form-data is for that.
You can use Raw if you want to send plain text or JSON or any other kind of string. Like the name suggests, Postman sends your raw string data as it is without modifications. The type of data that you are sending can be set by using the content-type header from the drop down.
Binary can be used when you want to attach non-textual data to the request, e.g. a video/audio file, images, or any other binary data file.
more details on : documentation
Using the Google Drive API to upload files. Multiple same console app is running at the same time to upload different parts of files in the folder and their files don't overlap. This hits the quota limit. Then a while-try-catch is implemented to re-execute the query whenever it throws the exception because of the quota limit. The list and create directory method works well but not the upload (i.e. create) method. Some files are missing when i checked from the Google Drive site
Tried using FileStream instead of MemoryStream but it seems not related.
public static Google.Apis.Drive.v3.Data.File uploadFile(DriveService _service, string _uploadFile)
{
bool again = true;
string[] p = _uploadFile.Split('/');
if (System.IO.File.Exists("C:/"+_uploadFile))
{
Google.Apis.Drive.v3.Data.File body = new Google.Apis.Drive.v3.Data.File();
body.Name = System.IO.Path.GetFileName(p[p.Length-1]);
body.Description = "";
body.MimeType = GetMimeType("C:/"+_uploadFile);
body.Parents = new List<string>() { ID };
// File's content.
System.IO.FileStream stream = new System.IO.FileStream("C:/" + _uploadFile, FileMode.Open);
try
{
FilesResource.CreateMediaUpload request = _service.Files.Create(body, stream, GetMimeType("C:/" + _uploadFile));
while (again)
{
try
{
request.Upload();
again = false;
}
catch (Exception e)
{
Console.WriteLine("uploadFile: "+p[p.Length-1]);
}
}
return body;
}
catch (Exception e)
{
Console.WriteLine("An error occurred: " + e.Message);
return null;
}
}
else
{
Console.WriteLine("File does not exist: " + _uploadFile);
return null;
}
}
I am transfering files through sockets. When I try to save files to a custom directory, I get this error using BinaryWrite function.
private void downloadFromServer()
{
try
{
byte[] buffer = new byte[5000 * 1024];
byte[] incomingFile = new byte[5000 * 1024];
buffer = Encoding.Default.GetBytes(getUserName.Text + "Download"
+ getFileName.Text + "end");
clientSocket.Send(buffer);
activityLog.AppendText("Preparing to download... \n");
while (incomingFile != null)
{
clientSocket.Receive(incomingFile);
int receivedBytesLen = incomingFile.Length;
int fileNameLen = BitConverter.ToInt32(incomingFile, 0);
File.WriteAllBytes(fileDir, incomingFile);
}
activityLog.AppendText("File saved to " + fileDir + "\n");
}
catch (Exception ex)
{
MessageBox.Show(ex.Message);
}
}
File.WriteAllBytes(fileDir, incomingFile); requires file name. From variable name it looks like you are using folder name. I.e. should be #"e:\tempfile.bin" instead of #"e:\".
File.WriteAllBytes
Given a byte array and a file path, this method opens the specified file, writes the contents of the byte array to the file, and then closes the file.
Note if fileDir means file name than you should looks for other not-so-truthful names throughout your code...
I am working on a project that downloads some images and put them in a arrarList to be processed later. The following portion of code is where the problem is. It works with first download, but somehow the file images were saving to is locked up after the first download. I can't seems to find a way to unlock it. File.Delete("BufferImg"); is giving error saying the file is been used by another process when "BufferImg" was not used anywhere else of the program. What am I doing wrong?
int attempcnt=0;
if (ok)
{
System.Net.WebClient myWebClient = new System.Net.WebClient();
try
{
myWebClient.DownloadFile(pth, "BufferImg");
lock (IMRequest) { IMRequest.RemoveAt(0); }
attempcnt = 0;
}
catch // will attempcnt 3 time before it remove the request from the queue
{
attempcnt++;
myWebClient.Dispose();
myWebClient = null;
if(attempcnt >2)
{
lock (IMRequest) { IMRequest.RemoveAt(0); }
attempcnt = 0;
}
goto endofWhile;
}
myWebClient.Dispose();
myWebClient = null;
using (Image img = Image.FromFile("BufferImg"))
{
lock (IMBuffer)
{
IMBuffer.Add(img.Clone());
MessageBox.Show("worker filled: " + IMBuffer.Count.ToString() + ": " + pth);
}
img.Dispose();
}
}
endofWhile:
File.Delete("BufferImg");
continue;
}
The following line is why the image is not being released:
IMBuffer.Add(img.Clone());
When you clone something loaded through a resource (file), the file is still attached to the cloned object. You will have to use a FileStream, like so:
FileStream fs = new FileStream("BufferImg", FileMode.Open, FileAccess.Read);
using (Image img = Image.FromStream(fs))
{
lock (IMBuffer)
{
IMBuffer.Add(img);
MessageBox.Show("worker filled: " + IMBuffer.Count.ToString() + ": " + pth);
}
}
fs.Close();
This should release the file after you've loaded it in the buffer.