I have an method which is download images via URL and sometimes URL become like this, which contains very long path (i removed some string,it just didnt beacome long question :) : data:image/jpeg;base64,/9j/4AAQSkZJRgABAQAAAQABAAD/2wCEAAkGBxITEhUSEBAVFhMXGRcXFRgWFRcVGRcYFhcXFhUVGBgYISggGBslGxgaITEhJSkrLi8uGCAzODMsNygtLisBCgoKDg0OFxAQGysfHSUtLS8tLS0tLS0tLSstLS0wLS4tLS0tLS0tLS0tKystNy0uLy0rKy0tLS0rNS0tLS0tLf/AABEIALoBDwMBIgACEQEDEQH/xAAcAAEAAgIDAQAAAAAAAAAAAAAABAUCBgEDBwj/xABLEAABAwMCAgYDCgsHAwUAAAABAAIDBBESBSExQQYTFCJRYXGBkQcjMkJTVJOh0fAVJDNSYpKUscHS4RYXQ1VjcoKEorJEg6TCw//EABkBAQEBAQEBAAAAAAAAAAAAAAABAgMEBf/EACURAQEAAgIABQQDAAAAAAAAAAABAhEDIQQSMWFxQYGh0QUTUf/aAAwDAQACEQMRAD8A9xREQEREBERAREQEREBERAREQEREBERAREQEREBERAREQEREBERAREQEREBERAREQEREBERAREQEREBERAREQEREBERAREQEREBERAREQEREBERAREQEREBERAREQEREBERAREQEREBERAREQEREBERAREQEREBERAREQEREBERAREQERaH0h90yGnnfAyB8hYcXnIMGQ4gbG/p2Qb4i8p1v3S5JYsKZjoJLtOd2v2B7zcS224Vezp9WBtjISfGzRf1Yq6Tb2ZF4vN7o1WxhJJNud2j6sFd9HPdJc98UUrLukIsTxsSG/FAHHyTRt6aiIooiIgIiICIiAiIgIiICIiAiIgIiICLQPdM6eu09zIomgyOAeSd7NJc22J8S3jflwWtRe6lOWA2Nz+ixWQ29kReR/wB5suNrSZ+OMdvYrLT/AHVImsAnhmc/mWiMA/8AcP3Jqm3pSLU+jPT6mrJepY2SN5BLRIG96wuQC0ncAX3W2KAiIgIiIC+Zdbqc6uodfjNKfV1hsvpeeQNaXHgASfULr5WgeS5zjxO59JN1YlV2txh1S+9+DedvihRuzDll+sV36i78Yf8A8f8AxauHnYrUgvNa6MGmfKXOqXxRmMZX6okv3dfLLhwsAd3i9sSDC0mgZO4hjp2hjDI976lrGNawxhzr9X+c4gDxezwJPoPS8Na2okfA17Gy0Tne9NbsDGHMBLiRk1rmk+TmnjYU/Seth1LrYaGJ8kgpy8PdHHE4lk0TnRNYxoLhgBxN+4OO6zbpdKJjbgW1GUE346jHsQDx8r2+tZNikc5rY9Qnc5xxaG1zXnibEhlzb4PAcneVuqHR6kNv2V4cRnbstmhx7mHwdm4d7h8McPjKVQ6HKZYwYJQ0PYS50JjIZEQWuyHxyC644ktbudrZVArqqWJ5Y6rrDaxDm1Jc1zXAOa5p2uC0+A34gWIXXBqErnNa2qri5xa0ATm7nOLQAN+JJcB/x81ues6tRR0r6eeEds6ppwELXWlkizDnTD4OJcBYEEY8CFpPRvTTUztiLywWfI5zW5ODYmOkd1bR8J/dsB4lal63UqZqT6mnf1dRUV8cmIdi6e5s6+JNn7bj2KFJq7x/6yu5/wCMf0rX7/8As/7vJTtboGOpoq6CeeSJ73QWqQOtY6MF1g4Eh7LHa3C9uRVl0a1bTYqGojrIgZiXFvvHWueMW9W1ribMs4OJ4cRv4tzXSR3w9G68sa91ZVR5Nza2SoIfgT3SWh5xuOF7IejFf/mMvK/4y/8ARvxeP0/Y3xNrbpTptdN1U0MdTIHRtPvdzjtvcRkcrAHyB5Kh/A+pgginrfU2Yf8A22F7gLkdTe3GXA6pYdW1+boW2uSQY+HEWHO/eGwFyOuXVIopDBT0TJWxho6yXKR0pc3IOFm37wu4AkbA7AbLlusnb8Tp7DicHgG++RIbwDO/tbu778F6cPBc3JjMscer8JeTGdWq3WqWoe4NfWCYC+5aWAOyc3bEHIWFAiIgIiICIiD/2Q==
And then when i debug or when its come to download image, i get this Exception:
var DL = webClient.DownloadData(base64)
The specified path and / or file name is too long. The fully qualified
name must be less than 260 characters and the folder name must be less
than 248 characters.
i did also research which is suggested to add <httpRuntime maxUrlLength="260" /> in webconfig or used different library , but unfortunately it didnt help to solve the problem.
Can anyone please help me or point me into the right direction :)
Thanks in advance.
Controller:
[HttpPost]
public string DownloadImagesFromLinkViaURL(ImagesViewModel model)
{
var RandomName = Guid.NewGuid().ToString("N").Substring(0,12);
using (WebClient webClient = new WebClient())
{
try
{
string base64 = model.ImageURL.Substring(model.ImageURL.IndexOf(',') + 1);
byte[] data = Convert.FromBase64String(base64);
var DL = webClient.DownloadData(base64);
using (MemoryStream mem = new MemoryStream(DL))
{
using (var content = Image.FromStream(mem))
{
var format = ImageFormat.Png.ToString().ToLower();
var PathIMG = "https://SomeName.com/folder/" + RandomName + "." + format;
content.Save(Path.Combine(Server.MapPath(PathIMG)));
ImageStore img = new ImageStore();
img.ProducentVarenr = model.ImageName;
img.ImageOrginalURL = model.ImageURL;
img.ImageRandomName = RandomName;
img.LinktilBillede = PathIMG;
db.ImageStoreList.Add(img);
db.SaveChanges();
}
}
}
catch (ArgumentException)
{
return "content is not image";
}
}
return "saved";
}
ViewModal:
public class ImagesViewModel
{
public int ImageID { get; set; }
public string ImageURL { get; set; }
public string ImageName { get; set; }
public string ImagePath { get; set; }
public string RandomName { get; set; }
}
Data you see is not url or path. It is image data presented as Base64 string. So no any downloading is needed since you have image data already.
If you paste that very long string i.e this base64-to-image converter tool, you see actual image.
With given base64-string you can save it to file with following style:
File.WriteAllBytes(#"c:\yourfile", Convert.FromBase64String(base64));
Here is (non-tested and non-refactored) fixed version of your original method, as requested in answer comments.
[HttpPost]
public string DownloadImagesFromLinkViaURL(ImagesViewModel model)
{
var RandomName = Guid.NewGuid().ToString("N").Substring(0, 12);
var format = ImageFormat.Png.ToString().ToLower();
var PathIMG = "https://SomeName.com/folder/" + RandomName + "." + format;
if (model.ImageURL.StartsWith("data:image"))
{
string base64 = model.ImageURL.Substring(model.ImageURL.IndexOf(',') + 1);
File.WriteAllBytes($#"c:\temp\{RandomName}.jpeg", Convert.FromBase64String(base64));
ImageStore img = new ImageStore();
img.ProducentVarenr = model.ImageName;
img.ImageOrginalURL = model.ImageURL;
img.ImageRandomName = RandomName;
img.LinktilBillede = PathIMG;
db.ImageStoreList.Add(img);
db.SaveChanges();
return "saved";
}
using (WebClient webClient = new WebClient())
{
try
{
byte[] data = Convert.FromBase64String(base64);
var DL = webClient.DownloadData(base64);
using (MemoryStream mem = new MemoryStream(DL))
{
using (var content = Image.FromStream(mem))
{
content.Save(Path.Combine(Server.MapPath(PathIMG)));
ImageStore img = new ImageStore();
img.ProducentVarenr = model.ImageName;
img.ImageOrginalURL = model.ImageURL;
img.ImageRandomName = RandomName;
img.LinktilBillede = PathIMG;
db.ImageStoreList.Add(img);
db.SaveChanges();
}
}
}
catch (ArgumentException)
{
return "content is not image";
}
}
return "saved";
}
Related
I'm making a simple webserver to serve html, css, js & images (done in c#). I am using HttpListener and I can get the html, javascript and css files to work properly. I am just having trouble with the images. This is what I'm using currently:
if (request.RawUrl.ToLower().Contains(".png") || request.RawUrl.Contains(".ico") || request.RawUrl.ToLower().Contains(".jpg") || request.RawUrl.ToLower().Contains(".jpeg"))
{
string dir = System.IO.Path.GetDirectoryName(System.Reflection.Assembly.GetExecutingAssembly().Location);
string[] img = request.RawUrl.Split('/');
string path = dir + #"\public\imgs\" + img[img.Length - 1];
FileInfo fileInfo = new FileInfo(path);
long numBytes = fileInfo.Length;
FileStream fileStream = new FileStream(path, FileMode.Open, FileAccess.Read);
BinaryReader binaryReader = new BinaryReader(fileStream);
byte[] output = binaryReader.ReadBytes((int)numBytes);
binaryReader.Close();
fileStream.Close();
var temp = System.Text.Encoding.UTF8.GetString(output);
return temp;
}
I am converting the image into a string to return them (it's the way my boss suggested). This is the method where I am handling these requests.
private static string SendResponse(HttpListenerRequest request)
This is my WebServer classes Run() method. The call to SetContentType just goes through the request.RawUrl and determines the content type.
public void Run()
{
ThreadPool.QueueUserWorkItem((o) =>
{
Console.WriteLine("StackLight Web Server is running...");
try
{
while (_listener.IsListening)
{
ThreadPool.QueueUserWorkItem((c) =>
{
var ctx = c as HttpListenerContext;
try
{
// store html content in a byte array
string responderString = _responderMethod(ctx.Request);
// set the content type
ctx.Response.Headers[HttpResponseHeader.ContentType] = SetContentType(ctx.Request.RawUrl);
byte[] buffer = buffer = Encoding.UTF8.GetBytes(responderString);
// this writes the html out from the byte array
ctx.Response.ContentLength64 = buffer.Length;
using(Stream stream = ctx.Response.OutputStream)
{
stream.Write(buffer, 0, buffer.Length);
}
}
catch (Exception ex)
{
ConfigLogger.Instance.LogCritical(LogCategory, ex);
}
}, _listener.GetContext());
}
}
catch (Exception ex)
{
ConfigLogger.Instance.LogCritical(LogCategory, ex);
}
});
}
My html page needs to display an image to the screen, it displays a broken image so far. I know the images directory is correct, I tested that.
This is where I got my code for the webserver: here
I was thinking that maybe I have to change the SendResponse method to not return a string
I figured it out. I created a class to hold the data, content type and the request.RawUrl. Then, where I was passing a string, I changed it to pass the object I created.
So, for my WebServer class, my Run method looks like this:
public void Run()
{
ThreadPool.QueueUserWorkItem((o) =>
{
Console.WriteLine("StackLight Web Server is running...");
try
{
while (_listener.IsListening)
{
ThreadPool.QueueUserWorkItem((c) =>
{
var ctx = c as HttpListenerContext;
try
{
// set the content type
ctx.Response.Headers[HttpResponseHeader.ContentType] = SetContentType(ctx.Request.RawUrl);
WebServerRequestData data = new WebServerRequestData();
// store html content in a byte array
data = _responderMethod(ctx.Request);
string res = "";
if(data.ContentType.Contains("text"))
{
char[] chars = new char[data.Content.Length/sizeof(char)];
System.Buffer.BlockCopy(data.Content, 0, chars, 0, data.Content.Length);
res = new string(chars);
data.Content = Encoding.UTF8.GetBytes(res);
}
// this writes the html out from the byte array
ctx.Response.ContentLength64 = data.Content.Length;
ctx.Response.OutputStream.Write(data.Content, 0, data.Content.Length);
}
catch (Exception ex)
{
ConfigLogger.Instance.LogCritical(LogCategory, ex);
}
finally
{
ctx.Response.OutputStream.Close();
}
}, _listener.GetContext());
}
}
catch (Exception ex)
{
ConfigLogger.Instance.LogCritical(LogCategory, ex);
}
});
}
And my SendResponse method looks like this:
private static WebServerRequestData SendResponse(HttpListenerRequest request)
{
string dir = System.IO.Path.GetDirectoryName(System.Reflection.Assembly.GetExecutingAssembly().Location);
string[] fileUrl = request.RawUrl.Split('/');
// routes
if (request.RawUrl.Contains("/"))
{
// this is the main page ('/'), all other routes can be accessed from here (including css, js, & images)
if (request.RawUrl.ToLower().Contains(".png") || request.RawUrl.ToLower().Contains(".ico") || request.RawUrl.ToLower().Contains(".jpg") || request.RawUrl.ToLower().Contains(".jpeg"))
{
try
{
string path = dir + Properties.Settings.Default.ImagesPath + fileUrl[fileUrl.Length - 1];
FileInfo fileInfo = new FileInfo(path);
path = dir + #"\public\imgs\" + fileInfo.Name;
byte[] output = File.ReadAllBytes(path);
_data = new WebServerRequestData() {Content = output, ContentType = "image/png", RawUrl = request.RawUrl};
//var temp = System.Text.Encoding.UTF8.GetString(output);
//return Convert.ToBase64String(output);
return _data;
}
catch(Exception ex)
{
ConfigLogger.Instance.LogError(LogCategory, "File could not be read.");
ConfigLogger.Instance.LogCritical(LogCategory, ex);
_errorString = string.Format("<html><head><title>Test</title></head><body>There was an error processing your request:<br />{0}</body></html>", ex.Message);
_byteData = new byte[_errorString.Length * sizeof(char)];
System.Buffer.BlockCopy(_errorString.ToCharArray(), 0, _byteData, 0, _byteData.Length);
_data = new WebServerRequestData() { Content = _byteData, ContentType = "text/html", RawUrl = request.RawUrl };
return _data;
}
}
I'm still cleaning up the code a bit but it now serves the images!
Oh... And here is the object I'm using:
public class WebServerRequestData
{
public string RawUrl { get; set; }
public string ContentType { get; set; }
public byte[] Content { get; set; }
public string RawData { get; set; }
}
Some really bad stuff here:
Empty catch. You'll never find out about many bugs.
Stuffing binary data into a string. Why? There's no encoding that is able to roundtrip binary data.
You're not disposing of ctx. I don't see why you need a manual finally block. Use using.
Untrusted callers can inject arbitrary paths into path. I could request your web.config file by navigating to /img/..\..\web.config (something like that).
Consider factoring out some common expressions into variables. You've got a Copy&Paste error with ToLower. Don't do dirty stuff and you'll have less bugs.
I have an existing web site that allows the user (normally me) to upload pictures and then to display the image on the appropriate page. I am now trying to add this functionality to a new web site.
In visual studio I opened the new web site and did an ‘add existing item’ to copy the model/view/controller from the old web site. Made a few changes to remove unneeded functionality from the code and a few other minor things.
In both I store the image in a folder named /data/Images. When I execute the code for the new site (Lat34North) and add an image, the image gets added, and the everything appears to work (no errors).
Problem:
If I look at the folder (new solution) using “File Explorer”, the jpg appears, in this case, on my hard drive and in the correct folder.
If I try and access the folder (~/Data/Images) via Visual Studio, the jpg does not show up.
When I try and display the image from the web site, I just get that funny little icon you get when the image is not found.
Photo Controller
//
[Authorize]
public ActionResult PhotoCreate(string CallingState, string masterMarkerID, string markerTitle)
{
ViewBag.photoCallingState = CallingState;
ViewBag.photoMarkerID = masterMarkerID;
ViewBag.photomarkerTitle = markerTitle;
return View();
}
//
// POST: /Photo/Create
[AcceptVerbs(HttpVerbs.Post)]
[Authorize]
public ActionResult PhotoCreate(Photo photo, HttpPostedFileBase file)
{
if (ModelState.IsValid)
{
photo.PhotoLinkRecID = photo.savePhotoState + photo.saveMarkerID;
if (photo.PhotoSequence == 0)
{
var no_Photo = from s in db.Photo
where s.PhotoLinkRecID == photo.PhotoLinkRecID
select s;
int noPhoto = no_Photo.Count();
int Sequence = (noPhoto * 20);
photo.PhotoSequence = Sequence;
}
photo.PhotoDateCreated = DateTime.Now;
photo.PhotoCreatedBy = #User.Identity.Name;
if (photo.fileUpload != null && photo.fileUpload.ContentLength > 0)
{
// get the file extension
photo.PhotoLinkRecID = photo.savePhotoState + photo.saveMarkerID; // Create key to link photo to the approiate marker -- // unique record identifier
photo.PhotoState = photo.savePhotoState;
photo.PhotoMarkerID = photo.saveMarkerID;
var fileName = Path.GetFileName(photo.fileUpload.FileName);
var FileExtension = fileName.Substring(fileName.LastIndexOf('.') + 1).ToLower();
photo.PhotoFileType = FileExtension;
db.Photo.Add(photo); // add new photo record to the DB
db.SaveChanges();
photo.PhotoRecID = "Photo" + photo.PhotoID.ToString(); // unique record identifier
var saveFile = photo.savePhotoState + photo.PhotoRecID + "." + photo.PhotoFileType; // new file name
var path = Path.Combine(Server.MapPath("~/Data/Images"), saveFile); // save the file
HttpPostedFileBase hpf = photo.fileUpload as HttpPostedFileBase;
hpf.SaveAs(path);
Image image = Image.FromFile(path);
photo.PhotoHeight = image.Height;
photo.PhotoWidth = image.Width;
try
{
//get the date taken from the files metadata
PropertyItem pi = image.GetPropertyItem(0x9003);
string sdate = Encoding.UTF8.GetString(pi.Value).Replace("\0", String.Empty).Trim();
string secondhalf = sdate.Substring(sdate.IndexOf(" "), (sdate.Length - sdate.IndexOf(" ")));
string firsthalf = sdate.Substring(0, 10);
firsthalf = firsthalf.Replace(":", "/");
sdate = firsthalf + secondhalf;
photo.PhotoDateTaken = sdate;
}
catch { }
try
{
double? lat = ImageExtensions.GetLatitude(image);
double? lon = ImageExtensions.GetLongitude(image);
if (lat > 1)
{
photo.PhotoLatDirection = "N";
string latString = lat.ToString();
//photo.PhotoLat = latString.Substring(0, 10);
if (latString.Length < 10)
{
photo.PhotoLat = latString;
}
else
{
photo.PhotoLat = latString.Substring(0, 10);
}
//photo.PhotoLat = lat.ToString();
photo.PhotoLongDirection = "W";
string longString = lon.ToString();
if (longString.Length < 10)
{
photo.PhotoLong = longString;
}
else
{
photo.PhotoLong = longString.Substring(0, 10);
}
//photo.PhotoLong = lon.ToString();
}
}
catch { }
}
db.SaveChanges();
return RedirectToAction("PhotoEdit", new { id=photo.PhotoID });
}
return View(photo);
The old web site uses EntityFramework, Version=5.0.0.0
The new web site uses EntityFramework, Version=6.0.0.0
The other difference between the two are the packages I have installed. Could I be missing one?
I'm fairly new to coding and c#. I'm building an app that accesses google spreadsheets via an API, I turn this data into XML, zip it and write it to a stream. It works fine, but instead of adding every spreadsheet ID manually to my list I want to make another API call that will retrieve every ID in my google sheets account so that I can store them to a list and update it every time I run the app.
I'm banging my head off a wall looking for answers but I think this is not available via a sheets API but maybe through the drive API??, any help would be greatly appreciated, I'm not looking for anyone to write my code, but please point me in the right direction, or am I trying to do something that cannot be done?
public class GoogleSheetsReader
{
private string apiKey;
public Stream OutStream { get; set; }
List<string> SpreadSheetIdList { get; set; }
public GoogleSheetsReader(string apiKey)
{
this.apiKey = apiKey;
}
public Stream Main()
{
SheetsService sheetsService = new SheetsService(new BaseClientService.Initializer
{
HttpClientInitializer = GetCredential(),
ApplicationName = "My Project To XML",
ApiKey = apiKey,
});
using (MemoryStream memory = new MemoryStream())
{
using (ZipArchive zip = new ZipArchive(memory, ZipArchiveMode.Create))
{
SpreadSheetIdList = new List<string>(); // not the best place for the list, just testing funtionality
SpreadSheetIdList.Add("1oZlfj6XZOrPs9Qti9K9iKL6itZChM8dlwwJFSvBNzUc");
SpreadSheetIdList.Add("1oU3sjd7QoOgQ2PvmC7NxciyM1MRXns6-Z9vMayFgOjU");
foreach (var spreadSheetId in SpreadSheetIdList)
{
var ssRequest = sheetsService.Spreadsheets.Get(spreadSheetId);
Data.Spreadsheet service = ssRequest.Execute();
foreach (var sheet in service.Sheets)
{
var sheetName = sheet.Properties.Title;
SpreadsheetsResource.ValuesResource.GetRequest request = sheetsService.Spreadsheets.Values.Get(spreadSheetId, sheetName);
SpreadsheetsResource.GetRequest requestForSpreadSheet = sheetsService.Spreadsheets.Get(spreadSheetId);
requestForSpreadSheet.Ranges = sheetName;
Data.Spreadsheet response1 = requestForSpreadSheet.Execute();
var spreadSheetName = response1.Properties.Title;
Data.ValueRange response = request.Execute();
ZipArchiveEntry entry = zip.CreateEntry($"{spreadSheetName}\\{sheetName}.xml");
using (Stream ZipFile = entry.Open())
{
byte[] data = Encoding.ASCII.GetBytes(SchemaMaker(response)); // SchemaMaker converts my sheet data to required xml format
ZipFile.Write(data, 0, data.Length);
}
}
}
}
OutStream = new MemoryStream(memory.ToArray());
}
return OutStream;
}
public string SchemaMaker(ValueRange _param)
{
var result = "<TABLE>";
var headers = _param.Values[0];
for (int i = 1; i < _param.Values.Count; i++)
{
result = result + "<ROW>";
for (int j = 0; j < _param.Values[i].Count; j++)
{
result = result + $"<{headers[j]}>{_param.Values[i][j]}</{headers[j]}>";
}
result = result + "</ROW>";
}
result = result + "</TABLE>";
var element = XElement.Parse(result);
var settings = new System.Xml.XmlWriterSettings();
settings.OmitXmlDeclaration = true;
settings.Indent = true;
settings.NewLineOnAttributes = true;
var xmlFormat = element.ToString();
return xmlFormat;
}
public static UserCredential GetCredential()
{
string[] Scopes = { SheetsService.Scope.SpreadsheetsReadonly };
return null;
}
}
using .NET SDK v.1.5.21.0
I'm trying to upload a large file (63Mb) and I'm following the example at:
http://docs.aws.amazon.com/AmazonS3/latest/dev/LLuploadFileDotNet.html
But using a helper instead the hole code and using jQuery File Upload
https://github.com/blueimp/jQuery-File-Upload/blob/master/basic-plus.html
what I have is:
string bucket = "mybucket";
long totalSize = long.Parse(context.Request.Headers["X-File-Size"]),
maxChunkSize = long.Parse(context.Request.Headers["X-File-MaxChunkSize"]),
uploadedBytes = long.Parse(context.Request.Headers["X-File-UloadedBytes"]),
partNumber = uploadedBytes / maxChunkSize + 1,
fileSize = partNumber * inputStream.Length;
bool lastPart = inputStream.Length < maxChunkSize;
// http://docs.aws.amazon.com/AmazonS3/latest/dev/LLuploadFileDotNet.html
if (partNumber == 1) // initialize upload
{
iView.Utilities.Amazon_S3.S3MultipartUpload.InitializePartToCloud(fileName, bucket);
}
try
{
// upload part
iView.Utilities.Amazon_S3.S3MultipartUpload.UploadPartToCloud(fs, fileName, bucket, (int)partNumber, uploadedBytes, maxChunkSize);
if (lastPart)
// wrap it up and go home
iView.Utilities.Amazon_S3.S3MultipartUpload.CompletePartToCloud(fileName, bucket);
}
catch (System.Exception ex)
{
// Huston, we have a problem!
//Console.WriteLine("Exception occurred: {0}", exception.Message);
iView.Utilities.Amazon_S3.S3MultipartUpload.AbortPartToCloud(fileName, bucket);
}
and
public static class S3MultipartUpload
{
private static string accessKey = System.Configuration.ConfigurationManager.AppSettings["AWSAccessKey"];
private static string secretAccessKey = System.Configuration.ConfigurationManager.AppSettings["AWSSecretKey"];
private static AmazonS3 client = Amazon.AWSClientFactory.CreateAmazonS3Client(accessKey, secretAccessKey);
public static InitiateMultipartUploadResponse initResponse;
public static List<UploadPartResponse> uploadResponses;
public static void InitializePartToCloud(string destinationFilename, string destinationBucket)
{
// 1. Initialize.
uploadResponses = new List<UploadPartResponse>();
InitiateMultipartUploadRequest initRequest =
new InitiateMultipartUploadRequest()
.WithBucketName(destinationBucket)
.WithKey(destinationFilename.TrimStart('/'));
initResponse = client.InitiateMultipartUpload(initRequest);
}
public static void UploadPartToCloud(Stream fileStream, string destinationFilename, string destinationBucket, int partNumber, long uploadedBytes, long maxChunkedBytes)
{
// 2. Upload Parts.
UploadPartRequest request = new UploadPartRequest()
.WithBucketName(destinationBucket)
.WithKey(destinationFilename.TrimStart('/'))
.WithUploadId(initResponse.UploadId)
.WithPartNumber(partNumber)
.WithPartSize(maxChunkedBytes)
.WithFilePosition(uploadedBytes)
.WithInputStream(fileStream) as UploadPartRequest;
uploadResponses.Add(client.UploadPart(request));
}
public static void CompletePartToCloud(string destinationFilename, string destinationBucket)
{
// Step 3: complete.
CompleteMultipartUploadRequest compRequest =
new CompleteMultipartUploadRequest()
.WithBucketName(destinationBucket)
.WithKey(destinationFilename.TrimStart('/'))
.WithUploadId(initResponse.UploadId)
.WithPartETags(uploadResponses);
CompleteMultipartUploadResponse completeUploadResponse =
client.CompleteMultipartUpload(compRequest);
}
public static void AbortPartToCloud(string destinationFilename, string destinationBucket)
{
// abort.
client.AbortMultipartUpload(new AbortMultipartUploadRequest()
.WithBucketName(destinationBucket)
.WithKey(destinationFilename.TrimStart('/'))
.WithUploadId(initResponse.UploadId));
}
}
my maxChunckedSize is 6Mb (6 * (1024*1024)) as I have read that the minimum is 5Mb...
why am I getting "Your proposed upload is smaller than the minimum allowed size" exception? What am I doing wrong?
The error is:
<Error>
<Code>EntityTooSmall</Code>
<Message>Your proposed upload is smaller than the minimum allowed size</Message>
<ETag>d41d8cd98f00b204e9800998ecf8427e</ETag>
<MinSizeAllowed>5242880</MinSizeAllowed>
<ProposedSize>0</ProposedSize>
<RequestId>C70E7A23C87CE5FC</RequestId>
<HostId>pmhuMXdRBSaCDxsQTHzucV5eUNcDORvKY0L4ZLMRBz7Ch1DeMh7BtQ6mmfBCLPM2</HostId>
<PartNumber>1</PartNumber>
</Error>
How can I get ProposedSize if I'm passing the stream and stream length?
Here is a working solution for the latest Amazon SDK (as today: v.1.5.37.0)
Amazon S3 Multipart Upload works like:
Initialize the request using client.InitiateMultipartUpload(initRequest)
Send chunks of the file (loop until the end) using client.UploadPart(request)
Complete the request using client.CompleteMultipartUpload(compRequest)
If anything goes wrong, remember to dispose the client and request, as well fire the abort command using client.AbortMultipartUpload(abortMultipartUploadRequest)
I keep the client in Session as we need this for each chunk upload as well, keep an hold of the ETags that are now used to complete the process.
You can see an example and simple way of doing this in Amazon Docs itself, I ended up having a class to do everything, plus, I have integrated with the lovely jQuery File Upload plugin (Handler code below as well).
The S3MultipartUpload is as follow
public class S3MultipartUpload : IDisposable
{
string accessKey = System.Configuration.ConfigurationManager.AppSettings.Get("AWSAccessKey");
string secretAccessKey = System.Configuration.ConfigurationManager.AppSettings.Get("AWSSecretKey");
AmazonS3 client;
public string OriginalFilename { get; set; }
public string DestinationFilename { get; set; }
public string DestinationBucket { get; set; }
public InitiateMultipartUploadResponse initResponse;
public List<PartETag> uploadPartETags;
public string UploadId { get; private set; }
public S3MultipartUpload(string destinationFilename, string destinationBucket)
{
if (client == null)
{
System.Net.WebRequest.DefaultWebProxy = null; // disable proxy to make upload quicker
client = Amazon.AWSClientFactory.CreateAmazonS3Client(accessKey, secretAccessKey, new AmazonS3Config()
{
RegionEndpoint = Amazon.RegionEndpoint.EUWest1,
CommunicationProtocol = Protocol.HTTP
});
this.OriginalFilename = destinationFilename.TrimStart('/');
this.DestinationFilename = string.Format("{0:yyyy}{0:MM}{0:dd}{0:HH}{0:mm}{0:ss}{0:fffff}_{1}", DateTime.UtcNow, this.OriginalFilename);
this.DestinationBucket = destinationBucket;
this.InitializePartToCloud();
}
}
private void InitializePartToCloud()
{
// 1. Initialize.
uploadPartETags = new List<PartETag>();
InitiateMultipartUploadRequest initRequest = new InitiateMultipartUploadRequest();
initRequest.BucketName = this.DestinationBucket;
initRequest.Key = this.DestinationFilename;
// make it public
initRequest.AddHeader("x-amz-acl", "public-read");
initResponse = client.InitiateMultipartUpload(initRequest);
}
public void UploadPartToCloud(Stream fileStream, long uploadedBytes, long maxChunkedBytes)
{
int partNumber = uploadPartETags.Count() + 1; // current part
// 2. Upload Parts.
UploadPartRequest request = new UploadPartRequest();
request.BucketName = this.DestinationBucket;
request.Key = this.DestinationFilename;
request.UploadId = initResponse.UploadId;
request.PartNumber = partNumber;
request.PartSize = fileStream.Length;
//request.FilePosition = uploadedBytes // remove this line?
request.InputStream = fileStream; // as UploadPartRequest;
var up = client.UploadPart(request);
uploadPartETags.Add(new PartETag() { ETag = up.ETag, PartNumber = partNumber });
}
public string CompletePartToCloud()
{
// Step 3: complete.
CompleteMultipartUploadRequest compRequest = new CompleteMultipartUploadRequest();
compRequest.BucketName = this.DestinationBucket;
compRequest.Key = this.DestinationFilename;
compRequest.UploadId = initResponse.UploadId;
compRequest.PartETags = uploadPartETags;
string r = "Something went badly wrong";
using (CompleteMultipartUploadResponse completeUploadResponse = client.CompleteMultipartUpload(compRequest))
r = completeUploadResponse.ResponseXml;
return r;
}
public void AbortPartToCloud()
{
// abort.
client.AbortMultipartUpload(new AbortMultipartUploadRequest()
{
BucketName = this.DestinationBucket,
Key = this.DestinationFilename,
UploadId = initResponse.UploadId
});
}
public void Dispose()
{
if (client != null) client.Dispose();
if (initResponse != null) initResponse.Dispose();
}
}
I use DestinationFilename as the destination file so I can avoid the same name, but I keep the OriginalFilename as I needed later.
Using jQuery File Upload Plugin, all works inside a Generic Handler, and the process is something like this:
// Upload partial file
private void UploadPartialFile(string fileName, HttpContext context, List<FilesStatus> statuses)
{
if (context.Request.Files.Count != 1)
throw new HttpRequestValidationException("Attempt to upload chunked file containing more than one fragment per request");
var inputStream = context.Request.Files[0].InputStream;
string contentRange = context.Request.Headers["Content-Range"]; // "bytes 0-6291455/14130271"
int fileSize = int.Parse(contentRange.Split('/')[1]);,
maxChunkSize = int.Parse(context.Request.Headers["X-Max-Chunk-Size"]),
uploadedBytes = int.Parse(contentRange.Replace("bytes ", "").Split('-')[0]);
iView.Utilities.AWS.S3MultipartUpload s3Upload = null;
try
{
// ######################################################################################
// 1. Initialize Amazon S3 Client
if (uploadedBytes == 0)
{
HttpContext.Current.Session["s3-upload"] = new iView.Utilities.AWS.S3MultipartUpload(fileName, awsBucket);
s3Upload = (iView.Utilities.AWS.S3MultipartUpload)HttpContext.Current.Session["s3-upload"];
string msg = System.String.Format("Upload started: {0} ({1:N0}Mb)", s3Upload.DestinationFilename, (fileSize / 1024));
this.Log(msg);
}
// cast current session object
if (s3Upload == null)
s3Upload = (iView.Utilities.AWS.S3MultipartUpload)HttpContext.Current.Session["s3-upload"];
// ######################################################################################
// 2. Send Chunks
s3Upload.UploadPartToCloud(inputStream, uploadedBytes, maxChunkSize);
// ######################################################################################
// 3. Complete Upload
if (uploadedBytes + maxChunkSize > fileSize)
{
string completeRequest = s3Upload.CompletePartToCloud();
this.Log(completeRequest); // log S3 response
s3Upload.Dispose(); // dispose all objects
HttpContext.Current.Session["s3-upload"] = null; // we don't need this anymore
}
}
catch (System.Exception ex)
{
if (ex.InnerException != null)
while (ex.InnerException != null)
ex = ex.InnerException;
this.Log(string.Format("{0}\n\n{1}", ex.Message, ex.StackTrace)); // log error
s3Upload.AbortPartToCloud(); // abort current upload
s3Upload.Dispose(); // dispose all objects
statuses.Add(new FilesStatus(ex.Message));
return;
}
statuses.Add(new FilesStatus(s3Upload.DestinationFilename, fileSize, ""));
}
Keep in mind that to have a Session object inside a Generic Handler, you need to implement IRequiresSessionState so your handler will look like:
public class UploadHandlerSimple : IHttpHandler, IRequiresSessionState
Inside fileupload.js (under _initXHRData) I have added an extra header called X-Max-Chunk-Size so I can pass this to Amazon and calculate if it's the last part of the uploaded file.
Fell free to comment and make smart edits for everyone to use.
I guess you didn't set the content-length of the part inside the UploadPartToCloud() function.
I am writing unit test for this method. I have tried lot of times but still can not write any code for it. Please suggest me how to unit test it. I am using C# , nunit framework and rhino mock.
Thanks in advance.
public FileUploadJsonResult AjaxUploadProfile(int id, string branchName, string filepath, HttpPostedFileBase file)
{
// TODO: Add your business logic here and/or save the file
string statusCode = "1";
string profilePicture = string.Empty;
string fileExtension = System.IO.Path.GetExtension(file.FileName.ToLower());
string fileName = id + "_" + branchName;
string fileNameWithOriginalExtension = fileName + fileExtension;
string fileNameWithJPGExtension = fileName + ".jpg";
string fileServerPath = this.Server.MapPath("~/LO_ProfilePicture/" + fileNameWithJPGExtension);
string statusMessage = string.Empty;
if (string.IsNullOrEmpty(fileExtension) || !Utility.isCorrectExtension(fileExtension))
{
statusMessage = "Profile picture should be of JPG, BMP, PNG, GIF or JPEG format.";
return new FileUploadJsonResult { Data = new { message = string.Format(statusMessage, fileNameWithOriginalExtension), filename = string.Empty, profilepic = profilePicture, statusCode = "0" } };
}
if (file.ContentLength > PageConstants.PROFILE_PICTURE_FILE_SIZE)
{
statusMessage = "Profile picture size should be less than 2MB";
return new FileUploadJsonResult { Data = new { message = string.Format(statusMessage, fileNameWithOriginalExtension), filename = string.Empty, profilepic = profilePicture, statusCode = "0" } };
}
Utility.SaveThumbnailImage(fileServerPath, file.InputStream, PageConstants.BRANCH_PROFILE_PICTURE_FILE_HEIGTH, PageConstants.BRANCH_PROFILE_PICTURE_FILE_WIDTH);
profilePicture = PageConstants.IMAGE_PATH + "LO_ProfilePicture/" + fileNameWithJPGExtension;
// Return JSON
return new FileUploadJsonResult { Data = new { message = string.Format("Profile Picture is successfully uploaded.", fileNameWithOriginalExtension), filename = fileNameWithJPGExtension, profilepic = profilePicture, statusCode } };
}
Make it do just the essential part. Split anything that has nothing to do with the operation you're trying to handle to other classes. Put those behind interfaces, so you can mock these in your unittests. This way you'll notice you don't have to test anything with file i/o in this class. In the class below I split up the function in the essential part, some file i/o and retrieving of settings. Even these settings have nothing to do with the current method you're trying to test. The method just needs verification on, for example, the extension, but it doesn't matter on how it does this.
Tip: try to avoid static utility classes. Give them their own class. Also avoid external components such as network communication or file i/o.
As I don't have a lot of context and it may not compile. But I would go with something like:
class Controller {
public FileUploadJsonResult AjaxUploadProfile(int id, string branchName, string filepath, HttpPostedFileBase file) {
string fileName = id + "_" + branchName;
string fileExtension = _fileIO.GetExtensionForFile(file);
if (!_extensionManager.IsValidExtension(fileExtension)) {
return CreateAjaxUploadProfileError("Profile picture should be of JPG, BMP, PNG, GIF or JPEG format.");
}
if (file.ContentLength > _settingsManager.GetMaximumFileSize()) {
return CreateAjaxUploadProfileError("Profile picture size should be less than 2MB");
}
string fileNameWithJPGExtension = fileName + ".jpg";
string fileServerPath = _fileIO.GetServerProfilePicture(Server, fileNameWithJPGExtension);
string fileClientPath = _fileIO.GetClientProfilePicture(fileNameWithJPGExtension);
var dimensions = _settingsManager.GetThumbnailDimensions();
_fileIO.SaveThumbnailImage(fileServerPath, file, dimensions.Item1, dimensions.Item2);
// Return JSON
var data = new {
message = "Profile Picture is successfully uploaded.",
filename = fileClientPath,
profilepic = profilePicture,
statusCode = "1"
};
return new FileUploadJsonResult { Data = data };
}
private static CreateAjaxUploadProfileError(string message) {
var data = new {
message = message,
filename = string.Empty,
profilepic = string.Empty,
statusCode = "0"
};
return new FileUploadJsonResult { Data = data };
}
}
class FileIO : IFileIO {
public string GetExtensionForFile(HttpPostedFileBase file) {
return System.IO.Path.GetExtension(filePath.FileName.ToLower());
}
public string GetServerProfilePicture(T server, string file) {
return server.MapPath( "~/LO_ProfilePicture/" + file);
}
public void SaveThumbnailImage(string path, HttpPostedFileBase file, int height, int width) {
Utility.SaveThumbnailImage(path, file.InputStream, height, width); // or even inline
}
public string GetClientProfilePicture(string fileName) {
return _settingsManager.GetClientImagePath() + "LO_ProfilePicture/" + fileNameWithJPGExtension;
}
}
class ExtensionManager : IExtensionManager {
public bool IsValidExtension(string extension) {
return Utility.isCorrectExtension(fileExtension); // or even inline
}
}
class SettingsManager : ISettingsManager {
public Tuple<int, int> GetThumbnailDimensions() {
return Tuple.Create<int, int>(PageConstants.BRANCH_PROFILE_PICTURE_FILE_HEIGTH, PageConstants.BRANCH_PROFILE_PICTURE_FILE_WIDTH);
}
public int GetMaximumFileSize() {
return PageConstants.PROFILE_PICTURE_FILE_SIZE;
}
}
You can look at this function as a combination of multiple functions doing specific work. One function is getting target file path, another is validating extension, another is validating size, another creates thumbnail, etc.
The goal is to breakdown the complex code into small testable functions (units) which you can test independently. So when you put them together you have better confidence that your big function works as expected.