We Keep Coding

File transfer between AWS EC2 MVC app and Lambda Serverless Web API app results in corrupted data

I have a MVC application which I am hosting on a Windows Server 2016 AWS EC2 instance. This application is an admin tool. This application uses a Web API application that is hosted as a AWS Lambda Serverless app (https://docs.aws.amazon.com/toolkit-for-visual-studio/latest/user-guide/lambda-build-test-severless-app.html).
One area of my MVC app allows users to upload images using a form file input. This is then posted back to the MVC controller and sent off to an API utility which sends the file to the API. The API then resizes (using Magick.NET) and saves the image to an S3 bucket and the resulting URL to a MySQL database.
This all works perfectly when running locally on my machine. The problem is when I try to upload an image on the live website. The result is that when the image data is loaded into a MagickImage, I get the following error:
ImageMagick.MagickCorruptImageErrorException: Not a JPEG file: starts
with 0xef 0xbf `' # error/jpeg.c/JPEGErrorHandler/332\n
I added in some code to log the first 20 bytes of the data (which is a byte array) both in the MVC app (before the file is posted to the API) and in the API once the file was received. I discovered that the values I received were completely different, as shown below:
MVC: FF-D8-FF-E0-00-10-4A-46-49-46-00-01-01-01-01-2C-01-2C-00-00
API: EF-BF-BD-EF-BF-BD-EF-BF-BD-EF-BF-BD-00-10-4A-46-49-46-00-01
I then did the following when running locally and saw that the values outputted were the same:
MVC: FF-D8-FF-E0-00-10-4A-46-49-46-00-01-01-01-01-2C-01-2C-00-00
API: FF-D8-FF-E0-00-10-4A-46-49-46-00-01-01-01-01-2C-01-2C-00-00
Is there some sort of environment setting that I need to set/change which could be causing this strange behaviour?
Below are the different sections of code that are relevant, in order of occurrence.
MVC controller:
public async Task<IActionResult> AddImage(ImageFormViewModel viewModel)
{
if (!ModelState.IsValid)
{
return RedirectToAction("Details", new { id = viewModel.TourId, errors = string.Join(",", ViewData.ModelState.Values.SelectMany(x => x.Errors.Select(y => y.ErrorMessage))) });
}
var apiResponse = await this.api.PostFile<ApiResponse>($"tours/{viewModel.TourId}/images", viewModel.Image);
if (apiResponse.Success)
{
return RedirectToAction("Details", new { id = viewModel.TourId, message = "Image added successfully!" });
}
{
return RedirectToAction("Details", new { id = viewModel.TourId, errors = string.Join(",", apiResponse.Errors) });
}
}
API Utility (In MVC app):
public async Task<TResponse> PostFile<TResponse>(string uri, IFormFile file) where TResponse : ApiResponse
{
var response = default(TResponse);
if (file != null && file.Length > 0)
{
var url = (!string.IsNullOrWhiteSpace(uri) ? new Uri(new Uri(this.baseUrl), uri) : new Uri(this.baseUrl)).ToString();
using (var http = new HttpClient())
{
byte[] data;
using (var stream = new MemoryStream())
{
await file.CopyToAsync(stream);
data = stream.ToArray();
}
this.logger.Information("First bytes (MVC app): " + BitConverter.ToString(data.Take(20).ToArray()));
var content = new MultipartFormDataContent();
content.Add(new ByteArrayContent(data), "file", file.FileName);
var httpResponse = await http.PostAsync(url, content);
response = JsonConvert.DeserializeObject<TResponse>(await httpResponse.Content.ReadAsStringAsync());
}
}
return response;
}
API controller:
[HttpPost]
public async Task<IActionResult> Post([FromRoute]string tourId)
{
var response = new ApiResponse();
if (Request.HasFormContentType)
{
var form = Request.Form;
foreach (var formFile in form.Files)
{
using (var stream = new MemoryStream())
{
await formFile.CopyToAsync(stream);
var result = await this.tourManagementService.AddImage(HttpUtility.UrlDecode(tourId), Path.GetFileNameWithoutExtension(formFile.FileName), stream.ToArray());
if (!result.Success)
{
...
}
}
}
}
return Ok(response);
}
Service method to save image etc:
public async Task<AddImageResult> AddImage(string tourId, string imageName, byte[] imageData)
{
this.logger.Information("First bytes (API): " + BitConverter.ToString(imageData.Take(20).ToArray()));
...
}
Code where Magick.NET is used and exception is thrown:
private byte[] resizeImage(byte[] imageData, int width, int height)
{
using (var image = new MagickImage(imageData, new MagickReadSettings { Format = MagickFormat.Jpeg }))
{
...
}
}

The problem turned out to be that my AWS API Gateway wasn't accepting binary data. By default, API Gateway treats the message body as JSON as explained here - https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-payload-encodings.html
By default, API Gateway treats the message body as a text payload and
applies any preconfigured mapping template to transform the JSON
string.
I believe the is the source of the corruption. To remedy this, I had to add "image/jpeg" as an accepted binary media type in API Gateway shown below:
I then adjusted my code to just deal with binary data (and scrapped the form content stuff):
MVC side:
public async Task<TResponse> PostFile<TResponse>(string uri, IFormFile file) where TResponse : ApiResponse
{
var response = default(TResponse);
if (file != null && file.Length > 0)
{
var url = (!string.IsNullOrWhiteSpace(uri) ? new Uri(new Uri(this.baseUrl), uri) : new Uri(this.baseUrl)).ToString();
using (var http = new HttpClient())
{
var content = new StreamContent(file.OpenReadStream());
content.Headers.ContentType = new MediaTypeHeaderValue(file.ContentType);
var httpResponse = await http.PostAsync(url, content);
response = JsonConvert.DeserializeObject<TResponse>(await httpResponse.Content.ReadAsStringAsync());
}
}
return response;
}
API side:
[HttpPost]
public async Task<IActionResult> Post([FromRoute]string tourId)
{
var response = new ApiResponse();
if (Request.ContentType.Equals("image/jpeg"))
{
using (var stream = new MemoryStream())
{
await Request.Body.CopyToAsync(stream);
...
}
}
else
{
...
}
return Ok(response);
}

WebAPI and Authorization Basic

I created a WebAPI but now I want to secure it with Basic Authorization.
// POST the data to the API
using (var client = new WebClient())
{
client.Headers.Add("Content-Type", "application/json");
client.Headers.Add(HttpRequestHeader.Authorization, "Basic" + Convert.ToBase64String(Encoding.ASCII.GetBytes(credentials)));
string json = JsonConvert.SerializeObject(ex);
string content = client.UploadString("http://myURL/v1/endpoint", json);
}
Below, how I post the data. Now, I would like to create a function that I can add to my controller or my Application_Start(). It will check:
if the request.Headers.Authorization is != null
if the request.Headers.Authorization.Scheme is != "Basic"
if there are some parameters
get the parameter and decode it to create a pair (SecretId/SecretKey)
call a service to check in the DB if there is a client with this pair
create an identity with IPrincipal
The thing is I don't know the best way is to create a customAttribute or a filter or something else. There is plenty of different way to do this but I would like to understand the difference.

Create the below-mentioned Filter in your project and use it at top of your web API method as :
**[BasicAuth]**
/// <summary>
/// Basic Authentication Filter Class
/// </summary>
public class BasicAuthAttribute : ActionFilterAttribute
{
/// <summary>
/// Called when [action executing].
/// </summary>
/// <param name="filterContext">The filter context.</param>
public override void OnActionExecuting(HttpActionContext filterContext)
{
try
{
if (filterContext.Request.Headers.Authorization == null)
{
// Client authentication failed due to invalid request.
filterContext.Response = new System.Net.Http.HttpResponseMessage()
{
StatusCode = HttpStatusCode.Unauthorized,
Content = new StringContent("{\"error\":\"invalid_client\"}", Encoding.UTF8, "application/json")
};
filterContext.Response.Headers.WwwAuthenticate.Add(new AuthenticationHeaderValue("Basic", "realm=xxxx"));
}
else if (filterContext.Request.Headers.Authorization.Scheme != "Basic" ||
string.IsNullOrEmpty(filterContext.Request.Headers.Authorization.Parameter))
{
// Client authentication failed due to invalid request.
filterContext.Response = new System.Net.Http.HttpResponseMessage()
{
StatusCode = HttpStatusCode.BadRequest,
Content = new StringContent("{\"error\":\"invalid_request\"}", Encoding.UTF8, "application/json")
};
}
else
{
var authToken = filterContext.Request.Headers.Authorization.Parameter;
Encoding encoding = Encoding.GetEncoding("iso-8859-1");
string usernamePassword = encoding.GetString(Convert.FromBase64String(authToken));
int seperatorIndex = usernamePassword.IndexOf(':');
string clientId = usernamePassword.Substring(0, seperatorIndex);
string clientSecret = usernamePassword.Substring(seperatorIndex + 1);
if (!ValidateApiKey(clientId, clientSecret))
{
// Client authentication failed due to invalid credentials
filterContext.Response = new System.Net.Http.HttpResponseMessage()
{
StatusCode = HttpStatusCode.Unauthorized,
Content = new StringContent("{\"error\":\"invalid_client\"}", Encoding.UTF8, "application/json")
};
}
// Successfully finished HTTP basic authentication
}
}
catch (Exception ex)
{
// Client authentication failed due to internal server error
filterContext.Response = new System.Net.Http.HttpResponseMessage()
{
StatusCode = HttpStatusCode.BadRequest,
Content = new StringContent("{\"error\":\"invalid_request\"}", Encoding.UTF8, "application/json")
};
}
}
/// <summary>
/// Validates the API key.
/// </summary>
/// <param name="recievedKey">The recieved key.</param>
/// <returns></returns>
private bool ValidateApiKey(string clientId, string clientSecret)
{
if (your condition satisfies)
{
return true;
}
return false;
}
}

I found few interesting articles about handlers/filter and attribute. I don't want to override [Authorize] so I will probably do an Authentication Filter.
Below some good links:
filter with attribute
filter
handler
#Nkosi: Cheers to confirm. I'm going to change the code a little bit because I don't want to use an Attribute but rather an filter that I put in the WebApiConfig

how to call web api to download document to website directory using webclient

I am struggling with being able to create a file with its data based on the byte array returned from the WebAPI. The following is my code for making the call to the web api
using (var http = new WebClient())
{
string url = string.Format("{0}api/FileUpload/FileServe?FileID=" + fileID, webApiUrl);
http.Headers[HttpRequestHeader.ContentType] = "application/octet-stream";
http.Headers[HttpRequestHeader.Authorization] = "Bearer " + authCookie.Value;
http.DownloadDataCompleted += Http_DownloadDataCompleted;
byte[] json = await http.DownloadDataTaskAsync(url);
}
The api code is
[HttpGet]
[Route("FileServe")]
[Authorize(Roles = "Admin,SuperAdmin,Contractor")]
public async Task<HttpResponseMessage> GetFile(int FileID)
{
using (var repo = new MBHDocRepository())
{
var file = await repo.GetSpecificFile(FileID);
if (file == null)
{
throw new HttpResponseException(HttpStatusCode.BadRequest);
}
var stream = File.Open(file.PathLocator, FileMode.Open);
HttpResponseMessage response = new HttpResponseMessage(HttpStatusCode.OK);
response.Content = new StreamContent(stream);
response.Content.Headers.ContentType = new System.Net.Http.Headers.MediaTypeHeaderValue(file.FileType);
return response;
}
}
I receive a byte array as a response however am unable to create the corresponding file from that byte array. I have no idea how to convert the byte array into the relevant file type (such as jpg, or pdf based on file type in the web api). any help will be appreciated.

Alright so there are a few ways of solving your problem firstly, on the server side of things you can either simply send the content type and leave it at that or you can also send the complete filename which helps you even further.
I have removed the code that is specific to your stuff with basic test code, please just ignore that stuff and use it in terms of your code.
Some design notes here:
[HttpGet]
[Route("FileServe")]
[Authorize(Roles = "Admin,SuperAdmin,Contractor")]
public async Task<HttpResponseMessage> GetFileAsync(int FileID) //<-- If your method returns Task have it be named with Async in it
{
using (var repo = new MBHDocRepository())
{
var file = await repo.GetSpecificFile(FileID);
if (file == null)
{
throw new HttpResponseException(HttpStatusCode.BadRequest);
}
var stream = File.Open(file.PathLocator, FileMode.Open);
HttpResponseMessage response = new HttpResponseMessage(HttpStatusCode.OK);
response.Content = new StreamContent(stream);
response.Content.Headers.ContentType = new System.Net.Http.Headers.MediaTypeHeaderValue(file.FileType);
response.Content.Headers.ContentDisposition = new System.Net.Http.Headers.ContentDispositionHeaderValue("attachment") { FileName=Path.GetFileName(file.PathLocator)};
return response;
}
}
Your client side code has two options here:
static void Main(string[] args)
{
using (var http = new WebClient())
{
string url = string.Format("{0}api/FileUpload/FileServe?FileID={1}",webApiUrl, fileId);
http.Headers[HttpRequestHeader.ContentType] = "application/octet-stream";
http.Headers[HttpRequestHeader.Authorization] = "Bearer " + authCookie.Value;
var response = http.OpenRead(url);
var fs = new FileStream(String.Format(#"C:\Users\Bailey Miller\Downloads\{0}", GetName(http.ResponseHeaders)), FileMode.Create);
response.CopyTo(fs); <-- how to move the stream to the actual file, this is not perfect and there are a lot of better examples
fs.Flush();
fs.Close();
}
}
private static object GetName(WebHeaderCollection responseHeaders)
{
var c_type = responseHeaders.GetValues("Content-Type"); //<-- do a switch on this and return a really weird file name with the correct extension for the mime type.
var cd = responseHeaders.GetValues("Content-Disposition")[0].Replace("\"", ""); <-- this gets the attachment type and filename param, also removes illegal character " from filename if present
return cd.Substring(cd.IndexOf("=")+1); <-- extracts the file name
}

How to sign url in .net for google cloud storage

I want to know that how to generate signurl using google cloud storage classes in .net
I have created string as per the requirement
GET
1388534400
/bucket/objectname
but I now want to sign this url with p12 key and then want to make it url friendly
This library doesn't show specific function for it -> https://developers.google.com/resources/api-libraries/documentation/storage/v1/csharp/latest/annotated.html
So, basically I need .net alternate of Google_Signer_P12 class of php
$signer = new Google_Signer_P12(file_get_contents(__DIR__.'/'."final.p12"), "notasecret");
$signature = $signer->sign($to_sign);

Now there is a UrlSigner in the pre-release package Google.Cloud.Storage.V1 that can be used to to provide read-only access to existing objects:
// Create a signed URL which can be used to get a specific object for one hour.
UrlSigner urlSigner = UrlSigner.FromServiceAccountCredential(credential);
string url = urlSigner.Sign(
bucketName,
objectName,
TimeSpan.FromHours(1),
HttpMethod.Get);
Or write-only access to put specific object content into a bucket:
// Create a signed URL which allows the requester to PUT data with the text/plain content-type.
UrlSigner urlSigner = UrlSigner.FromServiceAccountCredential(credential);
var destination = "places/world.txt";
string url = urlSigner.Sign(
bucketName,
destination,
TimeSpan.FromHours(1),
HttpMethod.Put,
contentHeaders: new Dictionary<string, IEnumerable<string>> {
{ "Content-Type", new[] { "text/plain" } }
});
// Upload the content into the bucket using the signed URL.
string source = "world.txt";
ByteArrayContent content;
using (FileStream stream = File.OpenRead(source))
{
byte[] data = new byte[stream.Length];
stream.Read(data, 0, data.Length);
content = new ByteArrayContent(data)
{
Headers = { ContentType = new MediaTypeHeaderValue("text/plain") }
};
}
HttpResponseMessage response = await httpClient.PutAsync(url, content);

I know the question was for P12, but Google lead me here when I was looking to do this for the newer, preferred JSON method. I pieced this together with other samples and sites I found. Hope this help save some time.
public string GetSignedURL()
{
var myObj = "theObject";
var scopes = new string[] { "https://www.googleapis.com/auth/devstorage.read_write" };
var myBucket = "theBucket";
ServiceAccountCredential cred;
using ( var stream = new FileStream(#"\path to\private-key.json", FileMode.Open, FileAccess.Read) )
{
cred = GoogleCredential.FromStream(stream)
.CreateScoped(scopes)
.UnderlyingCredential as ServiceAccountCredential;
}
var urlSigner = UrlSigner.FromServiceAccountCredential(cred);
return urlSigner.Sign(myBucket, myObj, TimeSpan.FromHours(1), HttpMethod.Get);
}
A list of Scopes can be found here

The .NET client doesn't support signing URLs (it is an XML-only API), so you will need to either make a callout to a tool like gsutil or generate an RSA signature internal to your application (Signing and verifying signatures with RSA C#)

This is my google signer code, One can make it more dynamic as per their needs
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Security.Cryptography;
using System.Web;
using System.Security.Cryptography.X509Certificates;
namespace HHAFSGoogle
{
static class GoogleSigner
{
private static string hashAlgo = "SHA256";
public static string ServiceAccountEmail
{
get
{
return "XXXXXXXXXXXXX-YYYYYYYYYYYYYYYYYYYYYYYY#developer.gserviceaccount.com";
}
}
public static string GoogleSecreat
{
get
{
return "notasecret";
}
}
public static string GoogleBucketDir
{
get
{
return "MyBucketDirectory";
}
}
public static string GoogleBucketName
{
get
{
return "MyBucket";
}
}
public static string CertiFilelocation
{
get
{
return System.Web.HttpContext.Current.Server.MapPath("p12file.p12");
}
}
/// <summary>
/// Get URL signature
/// </summary>
/// <param name="base64EncryptedData"></param>
/// <param name="certiFilelocation"></param>
/// <returns></returns>
public static string GetSignature(string base64EncryptedData, string certiFilelocation)
{
X509Certificate2 certificate = new X509Certificate2(certiFilelocation, GoogleSecreat, X509KeyStorageFlags.Exportable);
RSACryptoServiceProvider csp = (RSACryptoServiceProvider)certificate.PrivateKey;
RSACryptoServiceProvider privateKey1 = new RSACryptoServiceProvider();
privateKey1.ImportParameters(csp.ExportParameters(true));
csp.ImportParameters(privateKey1.ExportParameters(true));
byte[] data = Encoding.UTF8.GetBytes(base64EncryptedData.Replace("\r", ""));
byte[] signature = privateKey1.SignData(data, hashAlgo);
bool isValid = privateKey1.VerifyData(data, hashAlgo, signature);
if (isValid)
{
return Convert.ToBase64String(signature);
}
else
{
return string.Empty;
}
}
/// <summary>
/// Get signed URL by Signature
/// </summary>
/// <param name="fileName"></param>
/// <param name="method"></param>
/// <param name="content_type"></param>
/// <param name="duration"></param>
/// <returns></returns>
public static string GetSignedURL(string fileName, string method = "GET", string content_type = "", int duration = 10)
{
TimeSpan span = (DateTime.UtcNow.AddMinutes(10) - new DateTime(1970, 1, 1, 0, 0, 0, 0, DateTimeKind.Utc));
var expires = Math.Round(span.TotalSeconds, 0);
// Encode filename, so URL characters like %20 for space could be handled properly in signature
fileName = HttpUtility.UrlPathEncode(fileName);
// Generate a string to sign
StringBuilder sbFileParam = new StringBuilder();
sbFileParam.AppendLine(method); //Could be GET, PUT, DELETE, POST
// /* Content-MD5 */ "\n" .
sbFileParam.AppendLine();
sbFileParam.AppendLine(content_type); // Type of content you would upload e.g. image/jpeg
sbFileParam.AppendLine(expires.ToString()); // Time when link should expire and shouldn't work longer
sbFileParam.Append("/" + GoogleBucketName + "/" + fileName);
var signature = System.Web.HttpContext.Current.Server.UrlEncode(GetSignature(sbFileParam.ToString(), CertiFilelocation));
return ("https://storage.googleapis.com/MyBucket/" + fileName +
"?response-content-disposition=attachment;&GoogleAccessId=" + ServiceAccountEmail +
"&Expires=" + expires + "&Signature=" + signature);
}
}
}
and to download file call above class to get signed url
GoogleSigner.GetSignedURL(bucketFileName)
and to upload file call above class to get signed url for upload url
GoogleSigner.GetSignedURL(fileName, "PUT", type);

Lost byte when download file

I have three applications.
First: IIS
Second: Service (ASP.NET MVC)
Third: Client(Winform)
Files are store on IIS. Service public an api to download file as byte array base on URL. Client call api of Service and store file by extension.
After Client call Service, I check on Service, it return 15500 bytes. But I catch on Client, it is 13 bytes.
Below is the code on Service:
[HttpGet]
public byte[] DownloadData(string serverUrlAddress, string path)
{
if (string.IsNullOrWhiteSpace(serverUrlAddress) || string.IsNullOrWhiteSpace(path))
return null;
// Create a new WebClient instance
using (WebClient client = new WebClient())
{
// Concatenate the domain with the Web resource filename.
string url = string.Concat(serverUrlAddress, "/", path);
if (url.StartsWith("http://") == false)
url = "http://" + url;
byte[] data = client.DownloadData(url);
return data;
}
}
Below is the code on Client:
static void Main(string[] args)
{
byte[] data = GetData();
File.WriteAllBytes(#"E:\a.pdf", data);
}
public static byte[] GetData()
{
using (var client = new HttpClient())
{
client.BaseAddress = new Uri("http://localhost:54220/");
client.DefaultRequestHeaders.Accept.Clear();
client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
HttpResponseMessage response = client.GetAsync("API/File/DownloadData?serverUrlAddress=www.x.com&path=Data/Folder/file.pdf").Result;
if (response.IsSuccessStatusCode)
{
var yourcustomobjects = response.Content.ReadAsByteArrayAsync().Result;
return yourcustomobjects;
}
else
{
return null;
}
}
}

When returning CLR types Web API attempts to serialize the object based on either Xml or Json serializers. Neither of these are what you want. You want to return the raw stream of bytes. Try this.
[HttpGet]
public HttpResponseMessage DownloadData(string serverUrlAddress, string path)
{
if (string.IsNullOrWhiteSpace(serverUrlAddress) || string.IsNullOrWhiteSpace(path))
return null;
// Create a new WebClient instance
using (WebClient client = new WebClient())
{
// Concatenate the domain with the Web resource filename.
string url = string.Concat(serverUrlAddress, "/", path);
if (url.StartsWith("http://") == false)
url = "http://" + url;
byte[] data = client.DownloadData(url);
return new HttpResponseMessage() { Content = new StreamContent(data) };
}
}
By returning a HttpResponseMessage you have more control over exactly how Web API returns the response. By default StreamContent will set the Content-Type header to be application/octet-stream. You may want to change that to 'application/pdf' if you are always returning PDF files.