We Keep Coding

.Net upload xml files up to 1gb end in error

I have a application, where the user can upload xml files. Everything under 25mb is no problem, but when i try to upload my test file (117 mb), I get an error when the application is hosted.
Since I prepared the application like described in 1000 other posts, its working locally up to 2gb and also when hosted. But after upload i get "HTTP Error 503.0 - Service Unavailable".
When i log in again, the file is there but the error is inconvenienced.
web.config:
<security>
<requestFiltering>
<requestLimits maxAllowedContentLength="1073741824" />
</requestFiltering>
</security>
upload function:
[HttpPost]
[RequestFormLimits(MultipartBodyLengthLimit = 1073741824)]
public IActionResult Upload(IFormFile file){
if(file == null) return RedirectToAction("OnixIndex");
string completePath = app_resources_path + file.FileName;
using(FileStream fs = System.IO.File.Create(completePath))
{
file.CopyTo(fs);
fs.Flush();
}
startup.cs
services.Configure<FormOptions>(options =>
{
options.ValueLengthLimit = int.MaxValue;
options.MultipartBodyLengthLimit = int.MaxValue;
options.MultipartHeadersLengthLimit = int.MaxValue;
});
Should i use a different way to upload the file? I tryed a stream, but without improvement. I there a different technology or nuget bib?

could you please list system specs (Ram,Storage Type,Cpu Cores, .net version),
this error is mostly related to system specs not being able to handle the 1 gb file and you need to scale your system as mentioned in the following thread.
503 Server Unavailable after user uploads many files. Is this an httpRuntime setting?

ASP.NET Core web application - How to upload large files

Problem
I'm trying to create an ASP.NET Core (3.1) web application that accepts file uploads and then breaks it into chunks to send to Sharepoint via MS Graph API. There are a few other posts here that address the similar questions but they assume a certain level of .NET knowledge that I don't have just yet. So I'm hoping someone can help me cobble something together.
Configure Web server & app to Accept Large Files
I have done the following to allow IIS Express to upload up to 2GB files:
a) created a web.config file with the following code:
<?xml version="1.0" encoding="utf-8"?>
<configuration>
<location path="Home/UploadFile">
<system.webServer>
<handlers>
<add name="aspNetCore" path="*" verb="*" modules="AspNetCoreModuleV2" resourceType="Unspecified" />
</handlers>
<security>
<requestFiltering>
<!--unit is bytes => 2GB-->
<requestLimits maxAllowedContentLength="2147483647" />
</requestFiltering>
</security>
</system.webServer>
</location>
</configuration>
B) I have the following in my Startup.cs Configuration section:
//Add support for uploading large files TODO: DO I NEED THIS?????
services.Configure<FormOptions>(x =>
{
x.ValueLengthLimit = int.MaxValue; // Limit on individual form values
x.MultipartBodyLengthLimit = int.MaxValue; // Limit on form body size
x.MultipartHeadersLengthLimit = int.MaxValue; // Limit on form header size
});
services.Configure<IISServerOptions>(options =>
{
options.MaxRequestBodySize = int.MaxValue; //2GB
});
Here's what my form looks like that allows the user to pick the file and submit:
#{
ViewData["Title"] = "Messages";
}
<h1>#ViewData["Title"]</h1>
<p></p>
<form id="uploadForm" action="UploadFile" method="post" enctype="multipart/form-data">
<dl>
<dt>
<label for="file">File</label>
</dt>
<dd>
<input id="file" type="file" name="file" />
</dd>
</dl>
<input class="btn" type="submit" value="Upload" />
<div style="margin-top:15px">
<output form="uploadForm" name="result"></output>
</div>
</form>
Here's what the controller looks like:
[HttpPost]
[RequestSizeLimit(2147483647)] //unit is bytes => 2GB
[RequestFormLimits(MultipartBodyLengthLimit = 2147483647)]
public async void UploadFile()
{
User currentUser = null;
currentUser = await _graphServiceClient.Me.Request().GetAsync();
//nothing have to do with the file has been written yet.
}
When the user clicks on the file button and chooses a large file, I no longer get IIS 413 error messages. Great. The logic hits the right method in my controller.
But I have the following questions for this part of the code:
When the user picks the file ... what is actually happening under the hood? Has the file actually been stuffed into my form and is accessible from my controller?
Is it a stream?
how do i get to the file?
If ultimately, I need to send this file to Sharepoint using this type of an approach (the last example on chunking), it seems that the best approach is to save the file on my server somewhere... and then copy the sample code and try to chunk it out? The sample code seems to be referring to file paths and file sizes, I'm assuming I need to persist it to my web server somewhere first, and then take it from there.
if i do need to save it, can you point me in the right direction - maybe some sample code that shows me how to take the POSTed data in my form and save it?
ultimately, this will need to be refactored os that there is not GUI ... but it's just an API that accepts large files to upload somewhere. But I think i'll try to learn how to do it this way first... and then refactor to change my code to be API only.
Sorry for the noob questions. I have tried to do my research before posting here. But somethings are still a bit fuzzy.
EDIT 1
Per the suggestion in one of the posted answers, i've downloaded sample code that demonstrates how to bypass saving to a local file on the web server. It's based on this article
I have created a web.config file again -to avoid the 413 errors from IIS. I have also edited the list of allowed file extensions to support .pdf and .docx and .mp4.
When I try to run the sample project, and I choose the "Stream a file with AJAX to a controller endpoint" under the "Physical Storage Upload Examples" section, it dies here:
// This check assumes that there's a file
// present without form data. If form data
// is present, this method immediately fails
// and returns the model error.
if (!MultipartRequestHelper
.HasFileContentDisposition(contentDisposition))
if (!MultipartRequestHelper
.HasFileContentDisposition(contentDisposition))
{
ModelState.AddModelError("File",
$"The request couldn't be processed (Error 2).");
// Log error
return BadRequest(ModelState);
}
As is mentioned in the comments above the code, it's checking for form data and then when it finds it... it dies. So i've been playing around with the HTML page which looked like this:
<form id="uploadForm" action="Streaming/UploadPhysical" method="post"
enctype="multipart/form-data" onsubmit="AJAXSubmit(this);return false;">
<dl>
<dt>
<label for="file">File</label>
</dt>
<dd>
<input id="file" type="file" name="file" />asdfasdf
</dd>
</dl>
<input class="btn" type="submit" value="Upload" />
<div style="margin-top:15px">
<output form="uploadForm" name="result"></output>
</div>
</form>
And I've tried to remove the form like this:
<dl>
<dt>
<label for="file">File</label>
</dt>
<dd>
<input id="file" type="file" name="file" />
</dd>
</dl>
<input class="btn" type="button" asp-controller="Streaming" asp-action="UploadPhysical" value="Upload" />
<div style="margin-top:15px">
<output form="uploadForm" name="result"></output>
</div>
But the button doesn't do anything now when I click it.
Also, in case you're wondering / it helps, I manually copied in a file into the c:\files folder on my computer and when the sample app opens, it does list the file - proving it can read the folder.
I added read /write permissions so hopefully the web app can write to it when I get that far.

I've implemented a similar large file controller but using mongoDB GridFS.
In any case, streaming is the way to go for large files because it is fast and lightweight.
And yes, the best option is to save the files on your server storage before you send.
One suggestion is, add some validations to allow specefic extensions and restrict execution permissions.
Back to your questions:
The entire file is read into an IFormFile, which is a C# representation of the file used to process or save the file.
The resources (disk, memory) used by file uploads depend on the number and size of concurrent file uploads. If an app attempts to buffer too many uploads, the site crashes when it runs out of memory or disk space. If the size or frequency of file uploads is exhausting app resources, use streaming.
source 1
The CopyToAsync method enables you to perform resource-intensive I/O operations without blocking the main thread.
source 2
Here you have examples.
Example 1:
using System.IO;
using Microsoft.AspNetCore.Http;
//...
[HttpPost]
[Authorize]
[DisableRequestSizeLimit]
[RequestFormLimits(ValueLengthLimit = int.MaxValue, MultipartBodyLengthLimit = int.MaxValue)]
[Route("upload")]
public async Task<ActionResult> UploadFileAsync(IFormFile file)
{
if (file == null)
return Ok(new { success = false, message = "You have to attach a file" });
var fileName = file.FileName;
// var extension = Path.GetExtension(fileName);
// Add validations here...
var localPath = $"{Path.Combine(System.AppContext.BaseDirectory, "myCustomDir")}\\{fileName}";
// Create dir if not exists
Directory.CreateDirectory(Path.Combine(System.AppContext.BaseDirectory, "myCustomDir"));
using (var stream = new FileStream(localPath, FileMode.Create)){
await file.CopyToAsync(stream);
}
// db.SomeContext.Add(someData);
// await db.SaveChangesAsync();
return Ok(new { success = true, message = "All set", fileName});
}
Example 2 with GridFS:
[HttpPost]
[Authorize]
[DisableRequestSizeLimit]
[RequestFormLimits(ValueLengthLimit = int.MaxValue, MultipartBodyLengthLimit = int.MaxValue)]
[Route("upload")]
public async Task<ActionResult> UploadFileAsync(IFormFile file)
{
if (file == null)
return Ok(new { success = false, message = "You have to attach a file" });
var options = new GridFSUploadOptions
{
Metadata = new BsonDocument("contentType", file.ContentType)
};
using (var reader = new StreamReader(file.OpenReadStream()))
{
var stream = reader.BaseStream;
await mongo.GridFs.UploadFromStreamAsync(file.FileName, stream, options);
}
return Ok(new { success = true, message = "All set"});
}

You are on the right path, but as others have pointed out Microsoft have put up a well written document on file uploading which is a must read in your situation - https://learn.microsoft.com/en-us/aspnet/core/mvc/models/file-uploads?view=aspnetcore-6.0#upload-large-files-with-streaming.
As for your questions
do you need services.Configure<FormOptions>(x =>
No you don't! And you don't need services.Configure<IISServerOptions>(options => either, its read from the maxAllowedContentLength that you have configured in your web.config
When the user picks the file ... what is actually happening under the hood? Has the file actually been stuffed into my form and is accessible from my controller?, Is it a stream?
If you disable the form value model binding and use the MultipartReader the file is streamed and won't be cached into memory or disk, as you drain the stream, more data will be accepted from the client(the browser)
how do i get to the file?
Check the document above, there is a working sample for accessing the stream.
If ultimately, I need to send this file to Sharepoint using this type of an approach (the last example on chunking), it seems that the best approach is to save the file on my server somewhere... and then copy the sample code and try to chunk it out? The sample code seems to be referring to file paths and file sizes, I'm assuming I need to persist it to my web server somewhere first, and then take it from there.
Not necessarily, using the streaming approach you can copy the stream data directly.

How do I use WebClient to make Basic Authentication calls to IIS WebAPI?

Using a PowerShell Cmdlet encapsulating a WebClient call to a WebAPI, I am attempting to implement Basic Authentication. I have tried a couple methods and settled on adding a Filter to the WebAPI as suggested here WebAPI Authentication Filter.
FYI: Since this is to be a file-upload tool I would prefer the data only be sent once, as the files will be up to 10MB text logs, hence I am creating the header for the first contact.
From the PowerShell Cmdlet:
private async Task<string> Run(string postBody, string Username, string Password)
{
using (var client = new WebClient())
{
client.Headers[HttpRequestHeader.ContentType] = Common.DefaultMediaType;
var credentials = Convert.ToBase64String(Encoding.ASCII.GetBytes(Username + ":" + Password));
client.Headers[HttpRequestHeader.Authorization] = $"Basic {credentials}";
string response;
try
{
response = await client.UploadStringTaskAsync($"{_baseAddress}/{_uriTestConPost}", postBody);
}
catch (WebException ex)
{
var webResponse = ex.Response as HttpWebResponse;
if (webResponse == null) throw ex;
WriteWarning("WebException: " + webResponse.StatusCode);
return null;
}
return response;
}
}
Running this code as is produced a 401, and no echoes in the debugger.
Running this code with the credentials and header commented out creates two messages from Fiddler, one without authorization and another with authorization. Both get denied with 401 but the debugger on the WebAPI Controller shows that I am only attempting to authorize the first time, the second attempt is formed correctly with the encoded authorization header but is rejected with code 401 without ever hitting the debugger.
The ApiController is as simple as I could make it to verify:
public class TestReportController : ApiController
{
[HttpPost]
[McpBasicAuthenticationFilter]
public async Task<string> TestConnectionPost()
{
dynamic message = await Request.Content.ReadAsStringAsync();
return "You sent " + message.ToString();
}
}
At this point, I have to think the authorization problem is my web.Config or my IIS setup. For IIS I have Anonymous and Basic Authorization enabled (other parts of the site need anonymous access).
Important area of the WebConfig from what I can tell:
<system.web>
<authentication mode="None" />
<compilation debug="true" targetFramework="4.5.1" />
<httpRuntime targetFramework="4.5.1" executionTimeout="240000" maxRequestLength="1073741824" />
<identity impersonate="false" />
</system.web>
What is blocking the requests that have the credentials?

Take a look at this: https://msdn.microsoft.com/en-us/library/aa292114(v=vs.71).aspx
This is how I understand IIS authentication. I am not an authority by any means but I'll tell how I use it and how it works for me. Basically the authentication on IIS tells the website how to handle the handshake between the client and the server, but they all use windows permissions. (even the anonymous requires the user to be using the IIS default user account or one with permissions to access the site).
There is the ASP.NET authentication which you can use Windows or Forms. When I want to validate a user from a database and not from active directory, I use anonymous authentication in IIS and forms authentication in my web.config. The credentials are passed in either through a logon form or the query string.
I have been using ASP.Net identity lately. It does almost all the plumbing and is very easy to set up. https://blogs.msdn.microsoft.com/webdev/2013/10/20/building-a-simple-todo-application-with-asp-net-identity-and-associating-users-with-todoes/
There of course is a lot of stuff out there on this.

How can one prevent accessing content directly?

Using ASP MVC 4.5, how can one apply security measures in order to prevent users from accessing content directly?
Like for example preventing the access of images or other files stored on the web server just by entering their link.

Place your image in a non-web accessible folder.
Create a server side script (for example, an HttpHandler) that can read the image file and return its contents in the HTTP response.
In that script, perform your user validation to make sure the user has access to that file.
In the HTML, have the src attribute of the img tag point to your script.
The user can still directly type in the URL to your script to see the image. But you can at least require that the user is logged into your site and is authorized to view the image.

Use an authentication system such as ASP .NET Membership and require certain credentials to access the content. Other than that, there really isn't a way. If a user has a direct link and access to that area of your website, by nature of how web servers work, there isn't a way to stop it.
There are certain security measures you can take to help prevent users from getting a direct link though, a simple one would be disabling a right click.

I have produced the following HTTPHandler in order to prevent hotlinking.
It seems to work on my project, however I do not certainly know if this is the best practice.
public void ProcessRequest(HttpContext context)
{
//write your handler implementation here.
//Http
HttpRequest request = context.Request;
HttpResponse response = context.Response;
//Header - Properites
int Index = -1;
string[] Keys = request.Headers.AllKeys;
List<string[]> Values = new List<string[]>();
//Header - Loop to get key values
for (int i = 0; i < Keys.Length; i++)
{
Values.Add(request.Headers.GetValues(i));
//Check if property "Accept" exists
if (Keys[i].CompareTo("Accept") == 0)
Index = i;
}
//Check if URL and URL Referrer are null
if (context.Request.Url != null && context.Request.UrlReferrer != null && Index >= 0)
{
//Check image types
if (!context.Request.UrlReferrer.AbsolutePath.EndsWith(".bmp") ||
!context.Request.UrlReferrer.AbsolutePath.EndsWith(".jpg") ||
!context.Request.UrlReferrer.AbsolutePath.EndsWith(".jpeg") ||
!context.Request.UrlReferrer.AbsolutePath.EndsWith(".png"))
{
//Check header "Accept"
if (Values[Index][0].CompareTo("*/*") == 0)
{
//Get bytes from file
byte[] MyBytes = File.ReadAllBytes(context.Request.PhysicalPath);
//new HttpContext(context.Request, context.Response).Request.MapPath(context.Request.RawUrl).ToString()
context.Response.OutputStream.Write(MyBytes, 0, MyBytes.Length);
context.Response.Flush();
}
else
//Redirect
context.Response.Redirect("/Home");
}
else
//Redirect
context.Response.Redirect("/Home");
}
else
//Redirect
context.Response.Redirect("/Home");
}
Also the Web.config was modified as follows:
<system.webServer>
<handlers>
<!--My-->
<add name="PhotoHandler-BMP" path="*.bmp" verb="GET" type="MVCWebApplication.Handlers.PhotoHandler" resourceType="File" />
<add name="PhotoHandler-JPG" path="*.jpg" verb="GET" type="MVCWebApplication.Handlers.PhotoHandler" resourceType="File" />
<add name="PhotoHandler-JPEG" path="*.jpeg" verb="GET" type="MVCWebApplication.Handlers.PhotoHandler" resourceType="File" />
<add name="PhotoHandler-PNG" path="*.png" verb="GET" type="MVCWebApplication.Handlers.PhotoHandler" resourceType="File" />
</handlers>
</system.webServer>
Feel free to comment on any improvements.

There is little you can do unless you want to bug your users. One possible (and widely used) thing could be checking your referrer (and make it be on your application), but that can easily be spoofed.
If security for this is something critical, the only thing that comes into mind is having everything downloaded through a script which would check for credentials (or any other security measure you might want), but there's not much else you can do.
If the browser has indeed downloaded something to the local machine, there's absolutely no way you can prevent that user to use that data (you can put some barriers, like avoiding right-clicking, etc., but all of them can be avoided in some way or another).

Accessing web.config from Sharepoint web part

I have a VS 2008 web parts project - in this project is a web.config file:
something like this:
<?xml version="1.0"?>
<configuration>
<connectionStrings/>
<system.web>
<appSettings>
<add key="MFOwner" value="Blah" />
</appSettings>
…….
In my web part I am trying to access values in the appSetting section:
I've tried all of the code below and each returns null:
string Owner = ConfigurationManager.AppSettings.Get("MFOwner");
string stuff1 = ConfigurationManager.AppSettings["MFOwner"];
string stuff3 = WebConfigurationManager.AppSettings["MFOwner"];
string stuff4 = WebConfigurationManager.AppSettings.Get("MFOwner");
string stuff2 = ConfigurationManager.AppSettings["MFowner".ToString()];
I've tried this code I found:
NameValueCollection sAll;
sAll = ConfigurationManager.AppSettings;
string a;
string b;
foreach (string s in sAll.AllKeys)
{
a = s;
b = sAll.Get(s);
}
and stepped through it in debug mode - that is getting things like :
FeedCacheTimer
FeedPageURL
FeedXsl1
ReportViewerMessages
which is NOT coming from anything in my web.config file....maybe a config file in sharepoint itself? How do I access a web.config (or any other kind of config file!) local to my web part???
thanks,
Phil J

Those values look like default SharePoint web.config properties. When you create a SharePoint web part, it gets deployed to the IIS virtual directory that hosts the SharePoint site which has it's own (very large) web.config file and NOT your application web.config file.
Depending on your production environment, you can update the SharePoint web.config file with the values you want and your code should work. If you cannot update the web.config file you can look into using a stand alone config file in the IIS directory that you deploy to (custom code then to read it) or possibly the property bag associated with each SP site collection for which you would need to add your values via code.