We Keep Coding

Walmart API POST failing with 400 Bad Request (inventory feed) ARCA

I am having problems with a POST request to the Walmart Marketplace API for bulk data exchange and am hoping for some help.
Background:
I have been successful in writing signature authentication routines, and can successfully execute GET commands such as get products etc. This indicates to me that Authentication signatures are properly formatted, and headers (for the most part) are correct.
Problem:
I am receiving a 400 Bad Request response, Request Content is Invalid. response when attempting to submit a test feed to Walmarts API. I have read that this problem is common, but I have yet to find any forum post that clearly explains the actual problem, or how to fix it. Here are my current parameters:
ARCA
ARCA Rest Client For Chrome
URL:
https://marketplace.walmartapis.com/v2/feeds?feedType=inventory
Headers:
Accept: application/xml
WM_SVC.NAME: Walmart Marketplace
WM_CONSUMER.ID: <Consumer ID>
WM_SEC.AUTH_SIGNATURE: <Good Auth Signature>
WM_QOS.CORRELATION_ID: 15649814651
WM_SEC.TIMESTAMP: <Timestamp>
WM_CONSUMER.CHANNEL.TYPE: <Channel Type>
Content-Type: multipart/form-data
File attachment (not raw payload although that has been tried)
<?xml version="1.0" encoding="utf-8"?>
<InventoryFeed xmlns="http://walmart.com/">
<InventoryHeader>
<version>1.4</version>
</InventoryHeader>
<inventory>
<sku>KON04418</sku>
<quantity>
<unit>EACH</unit>
<amount>4</amount>
</quantity>
<fulfillmentLagTime>1</fulfillmentLagTime>
</inventory>
</InventoryFeed>
When I take this exact same XML and test it at Walmart API Explorer
the file is accepted with Response Code 200 (OK).
I have validated with Notepad++ XML Tools plugin that the XML conforms to the XSD provided by Walmart. I have seen numerous posts regarding Boundaries that need to be applied, so I have additionally attempted to change the Content-Type, and add Boundaries but have been unsuccessful in getting the request accepted.
Any help in getting this request to return a response code 200 would be greatly appreciated.
Lastly, once this request validates in ARCA, I will be implementing in C#. I already have all of the code written, but there's a bit of fuzziness about how to add an attachment to an HttpWebRequest vs. just submitting a raw data stream. If any clarity could be provided on the difference I would again, appreciate it.

So this answer isn't clean and elegant, more of a work around than anything. I have spoken with a few individuals inside the Walmart engineering team and have been told that a C# SDK should be forthcoming in the next few months.
After all of my research, it appears there is some tricks to how you submit a multi-part form to Walmart and the system is very inflexible. I've seen posts about adding specifically formatted boundaries into the body of the HTTP request, but had no such luck. I was unable to attach the body as a file, or as a data stream to the request.
The work around is pretty simple, and ugly unfortunately. It takes a bit of setup, but you can create a .jar wrapper around the Walmart Java SDK and call it from your .Net program.
So.. steps in the process:
Grab the appropriate .XSD files and generate C# classes from them.
Build properly formatted XML inventory file. Make sure to include namespaces!! Walmart will fail particular calls if you don't include the appropriate ns2 / ns3 namespaces.
Dynamically generate a batch file to call your Java module. Spawning a command line process directly seemed to make things cranky for some reason so I opted for the batch file instead.
string path = #Directory.GetParent(Environment.CurrentDirectory).ToString();
if (File.Exists(#"../inventory.bat"))
{
File.Delete(#"../inventory.bat");
}
string batchCommand = #"cd " + path + Environment.NewLine + #"java -jar WalmartWrapper.jar SubmitInventoryFeed inventoryFeed.xml";
File.WriteAllText(path + #"\\inventory.bat", batchCommand);
ProcessStartInfo info = new ProcessStartInfo();
info.UseShellExecute = true;
info.FileName = #"inventory.bat";
info.WorkingDirectory = path;
var p = Process.Start(info);
p.WaitForExit();`
From here, the Java module takes over. It took a bit of hacking around to make it work more like an SDK and less like a sample program.. Here's some of the sample code for making things work..
Entry Point
if ("SubmitInventoryFeed".equals(args[0].trim())) {
if (args.length < 2)
{
System.out.println("Need second argument for SubmitInventoryFeed");
return;
}
String filename = args[1];
Feed inventoryFeed = new Feed();
try
{
inventoryFeed.submitInventoryFeed(filename);
} catch (Exception ex) {
System.out.println("submitInventoryFeed failed: " + ex.getMessage());
}
}
SDK Call (This is the bare bones of submitInventoryFeed without error checking)
String path = Paths.get(".").toAbsolutePath().normalize().toString();
File itemFile = FileHandler.getFile(filename.trim());
String filePath = path + "\\" + "MarketplaceClientConfig.properties";
WalmartMarketplace wm = Utils.getClient(filePath);
Response response = wm.submitFeed(MarketplaceFeedType.inventory, itemFile);
You can use ResponseChecker.isResponse200(response, true) to test for successful submissions
Use FeedAcknowledgement ack = response.readEntity(FeedAcknowledgement.class); to grab the actual response to check errors
I will be the first to say I can't wait to replace this work around with the C# SDK pending from Walmart, but for the time being, this is the only way I have been able to submit. I've looked through the walmart code in depth, but unfortunately, there is some Java magic happening under the hood that does the file attachment so there's not really any way to gain access to the exact procedure and reverse engineer for C#. I think that someone who really knew Java inside and out could figure it out, but I have enough Java background to allow me to cobble together a workable, albeit ugly solution.

Trying to Write Web Server Side Implementation of File Download via C# where body of message is the file contents

My question/challenge is very much akin to a couple of different Stack Overflow postings, but I've read them in detail and am still struggling to get even the most basic application to work. Specifically, I believe "Returning binary file from controller in ASP.NET Web API" and "How to return a file (FileContentResult) in ASP.NET WebAPI" are both very close to what I'm trying to achieve.
To give you context, I'm trying to write/stub a simple Firmware OTA (over the air) server. My initial goal is seemingly very simple, but I've spent a lot of time with almost zero progress. I want to provide a URL that can be hit from a browser (and ultimately some firmware running on an IoT device) that will take in some parameters (either via URL parameters or via the header). The call should return a file as the body of the response if a firmware update is available, or a suitable HTTP Reponse code (or just an empty message body) if no updated is currently available.
I'm not very experienced in Visual Studio for Website applications, but plenty comfortable in C# and it seemed like a good environment to get this running quickly. I am open to other implementations, but I thought writing a controller would be the simplest way. I am learning that it seems this Web API is primarily intended for .Net to .Net communication, so if there's a better platform, I'm happy to be pointed in a better direction. Here's what I put together (heavily leveraging links above, so no credit due to me!):
namespace OverTheAirApi.Controllers
{
[Route("api/[controller]")]
public class OTAController : ApiController
{
[HttpGet]
public HttpResponseMessage Download(string name)
{
string fileName = "hello-world.bin";
string filePath = "C:\\Devel\\Code\\...\\hello_world\\build\\" + fileName;
HttpResponseMessage result = new HttpResponseMessage(HttpStatusCode.OK);
var stream = new FileStream(filePath, FileMode.Open);
result.Content = new StreamContent(stream);
result.Content.Headers.ContentDisposition = new ContentDispositionHeaderValue("attachment");
result.Content.Headers.ContentDisposition.FileName = Path.GetFileName(filePath);
result.Content.Headers.ContentType = new MediaTypeHeaderValue("application/octet-stream");
result.Content.Headers.ContentLength = stream.Length;
return result;
}
}
}
When I type in the URL http://localhost:2265/api/ota, I see the following raw output on screen:
{"version":{"major":1,"minor":1,"build":-1,"revision":-1,"majorRevision":-1,"minorRevision":-1},"content":{"headers":[{"key":"Content-Disposition","value":["attachment; filename=hello-world.bin"]},{"key":"Content-Type","value":["application/octet-stream"]},{"key":"Content-Length","value":["407904"]}]},"statusCode":200,"reasonPhrase":"OK","headers":[],"requestMessage":null,"isSuccessStatusCode":true}
The entire response, captured in Fiddler2 looks like this:
HTTP/1.1 200 OK
Transfer-Encoding: chunked
Content-Type: application/json; charset=utf-8
Server: Kestrel
X-SourceFiles: =?UTF-8?B? YzpcdXNlcnNcZG91Z2NcZG9jdW1lbnRzXHZpc3VhbCBzdHVkaW8gMjAxNVxQcm9qZWN0c1xPdmVyVGhl QWlyQXBpXHNyY1xPdmVyVGhlQWlyQXBpXGFwaVxvdGE=?=
X-Powered-By: ASP.NET
Date: Tue, 28 Feb 2017 16:43:01 GMT
192
{"version": {"major":1,"minor":1,"build":-1,"revision":-1,"majorRevision":-1,"minorRevision":-1},"content":{"headers":[{"key":"Content-Disposition","value":["attachment; filename=hello-world.bin"]},{"key":"Content-Type","value":["application/octet-stream"]},{"key":"Content-Length","value":["407904"]}]},"statusCode":200,"reasonPhrase":"OK","headers":[],"requestMessage":null,"isSuccessStatusCode":true}
0
(Sorry if that formatting is a bit ugly, I guess I need some tutoring on using MD for HTTP Responses as well!)
Once again, my goal is to write C# code that will return back an HTTP response where the header indicates that the file is an attachment and where the body of the message is just the file itself.
Thank you again for any assistance/insight you might be able to provide!
Thank You - doug.

Here's my standard method for sending a file to the browser. The method takes two parameters, one the contents of the file and one the default filename that the client will receive. Enjoy.
/// <summary>
/// Sends the fileData to the browser and prompts the visitor to save it with the
/// name specified in fileName.
/// </summary>
/// <param name="defaultFilename">File name to use when browser prompts visitor (spaces will be replaced with dashes)</param>
/// <param name="data">Data to be sent to visitor's browser</param>
/// <param name="errorMsg"></param>
// Mod 08/04/09 Ron C - Reworked code to work right in modern browsers.
public static bool DownloadToBrowser(string data, string defaultFilename, out string errorMsg){
errorMsg = "";
System.Web.HttpResponse response = HttpContext.Current.Response;
try {
defaultFilename = defaultFilename.Replace(' ', '-'); //firefox will cut the name off at the space if there is one, so get rid of 'em
response.Clear();
response.ContentType = "application/octet-stream";
response.AddHeader("Content-Disposition", "attachment; filename=" + defaultFilename);
//8/5/09 Adding a "Content-Length" header was cutting off part of my file. Apparently
// it would need to factor in the length of the headers as well as the conent.
// since I have no easy way to figure out the length of the headers, initially I was gonna
// eliminate "Content-Length" header. Doing so works great in IE 7 and FireFox 3, but not
// in Safari 3 or 4(which doesn't download the file without the "Content-Length" header. So I
// resorted to a cludge of using the header tag and setting it to content length + 1000 for headers.
// I'd love to have a better solution, but his one works and I've already wasted 6+ hours to get to this
// solution. Cross broswer downloading of a file shouldn't have to be so hard. -Ron
int len = data.Length + 1000;
response.AddHeader("Content-Length", len.ToString()); //file size for progress dialog
response.Write(data);
response.Flush();
response.Close(); //Close() needed to prevent html from page being streamed back after the file data we sent.
//Don't use response.End() cause it throws a thread abort exception that can't be caught! Actually you can
//catch it but then it rethrows after the catch block! (What bozo thought that up?). I found lots of threads on this.
} catch (Exception ex) {
errorMsg = "Unable to download file to browser.";
//Add code here to log the error in your environment
return false;
}
return true;
}

End of result from BinaryWrite Flush is missing

I'm having some problems dealing with downloading a file from the server.
The problem is that the end of the downloaded file is missing.
I have found some indications of other people having similar problems, but nothing that helps me in my problem.
When debugging I have learned that the length of fileData is correct, and all data is present in the byte array when calling BinaryWrite.
That leaves the BinaryWrite, Flush or Close calls...
I have read about not using Response.End or Response.Close, for example here:
HttpResponse.End vs HttpResponse.Close vs HttpResponse.SuppressContent and it seems like a probable cause, but what should I use instead (have tried to remove Response.Close completely, but that result in too much data output)?
Someone knows what might cause this behaviour, and how to fix it?
EDIT: Just tried with Response.ContentType = "binary/octet-stream"; and it works like a charm!
What is the difference between text/plain and binary/octet-stream that may cause a behavior like this?
It even works without the Close call...
MORE EDIT: Seems like compression of Responses was activated on the server-side. Apparently there seems to be an issue with plain text streams when compression is active.
The code I have is:
private void DownloadFile(byte[] fileData, string fileName, string fileExtension)
{
Response.Clear();
Response.AddHeader("content-disposition", "attachment; filename=" + fileName + fileExtension);
Response.AddHeader("Content-Length", fileData.Length.ToString(CultureInfo.InvariantCulture));
Response.ContentType = "text/plain";
Response.BinaryWrite(fileData);
Response.Flush();
Response.Close();
}

If compression is active on the server-side (IIS) it apparently causes trouble with text/plain streams.
For you with similar problems, try deactivating, it might help!
It surely did for me!

ASP.NET MVC cannot stream files

I have a ASP.NET MVC 4 site that creates an excel file using OPEN XML SDK. I simply point the hyperlink to the proper controller and it generates the OPEN XML excel document and writes the stream to response header and done. In IE 9 and Chrome this works fine. File gets downloaded with the given file name and proper contents. However, just recently I upgraded my browser to IE 10 and now instead of downloading the file and opening up in excel I get the error that could not open 'URI'. When I click ok it gives another error: Microsoft Excel cannot access the file 'URI'. There are several possible reasons:
I don't understand why this would work in IE 9 and chrome and not in IE 10. I debugged the response headers with fiddlers and it has the proper content type, and content length set:
Content Type: application/vnd.openxmlformats-officedocument.spreadsheetml.sheet
Content Disposition: attachment; filename=result.xlsx
Content length: 1232
Is there something that I am missing?
Code snippet: This all is part of
public override void ExecuteResult(ControllerContext context)
{
...
....
..
extention = "xlsx";
response.ContentType = "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet";
response.AddHeader("Content-Disposition",
String.Format("attachment; filename={0}.{1}", fileName, extention));
response.AddHeader("Content-Length", mem.Length.ToString());
mem.Seek(0, SeekOrigin.Begin); // Go back to the begining.
mem.CopyTo(response.OutputStream);
context.HttpContext.ApplicationInstance.CompleteRequest();
}

As a workaround, Controller.File works for me with ASP.NET MVC 4 and IE 10:
public class DownloadController : Controller
{
public ActionResult GetFile()
{
...
mem.Position = 0;
return File(
mem,
"application/vnd.openxmlformats-officedocument.spreadsheetml.sheet",
"result.xlsx");
}
}
EDIT
So I have been browsing the MVC source code at CodePlex and found a few implementation details that differ from your code snippet:
Content-Length header is never set
Instead of calling Stream.CopyTo, they use a simple Stream.Read, Stream.Write loop with buffer size of 0x1000
When setting Content-Disposition header, they check whether file download name contains UTF-8 chars, and if so, encode them according to RFC 2231. See ContentDispositionUtil.GetHeaderValue
context.HttpContext.ApplicationInstance.CompleteRequest() is never called
Now you could try applying these changes one by one and see which one makes it work with IE 10.

I guess it's an IE10 bug. But maybe you still can work around it.
Based on this post and a few others.
Please check that:
Both full URI and the filename of Content Disposition header do not contains special symbols and that their lengths are less than 105. Just use regular A-Za-z0-9 symbols for instance.
Try to add "X-UA-Compatible" HTTP header with "IE=9" value.

try these MIME types:-
for old version:-
Response.ContentType = "application/vnd.ms-excel";
Response.AppendHeader("content-disposition", "attachment; filename=myfile.xls");
for 2007:-
Response.ContentType = "application/application/vnd.openxmlformats-officedocument.spreadsheetml.sheet";
Response.AppendHeader("content-disposition", "attachment; filename=result.xlsx");
for csv files which you need to open in excel is:-
Response.AddHeader "Content-Disposition", "Attachment;Filename=myfile.csv"
in last you can try:-
Response.ContentType = "application/octet-stream";
Response.AppendHeader("content-disposition", "attachment; filename=result.xlsx");
thanks

Weird characters in email

I have written a mail-processing program, which basically slaps a template on incoming mail and forwards it on. Incoming mail goes to a Gmail account, which I download using POP, then I read the mail (both html and plain text multipart-MIME), make whatever changes I need to the template, then create a new mail with the appropriate plain+html text and send it on to another address.
Trouble is, when the mail gets to the other side, some of the mails have been mangled, with weird characters like à and  magically getting inserted. They weren't in the original mails, they're not in my template, and I can't find any sort of predictable pattern as to when these characters appear. I'm sure it's got something to do with the encoding properties of the mails, but I am making sure to set both the charset and the transfer encoding of the outgoing mail to be the same as the incoming mail. So what else do I need to do?
EDIT: Here's a snipped sample of an incoming mail:
Content-Type: text/plain; charset=iso-8859-1
Content-Transfer-Encoding: quoted-printable
=0A=0ASafari Special:=0A=0A=A0=0A=0ASafari in Thornybush Priv=
ate Game Reserve 9-12=0AJanuary 2012 (3nights)
After processing, this comes out as:
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
=0D=0A=0D=0ASafari Special:=0D=0A=0D=0A=C2=A0=0D=0A=0D=0A=
Safari in Thornybush Private Game Reserve 9-12=0D=0AJanuary=
2012 (3nights)
Notice the insertion of the =0D and =C2 characters (aside from a few =0A's that weren't in the original).
So what does you think is happening here?
ANOTHER CLUE: Here's my code that creates the alternate view:
var htmlView = AlternateView.CreateAlternateViewFromString(htmlBody, null, "text/html");
htmlView.ContentType.CharSet = charSet;
htmlView.TransferEncoding = transferEncoding;
m.AlternateViews.Add(htmlView);
Along the lines of what #mjwills suggested, perhaps the CreateAlternativeViewFromString() method already assumes UTF-8, and changing it later to iso-8859-1 doesn't make a difference?

So every =0A is becoming =0D=0A.
And every =A0 is becoming =C2=A0.
The former looks like it might be related to Carriage Return / Line Feeds.
The latter looks like it might be related to What is "=C2=A0" in MIME encoded, quoted-printable text?.
My guess is that even though you have specified the charset, something alone the line is treating it as UTF8.
You may want to try using this form of CreateAlternateViewFromString, where the ContentType.CharSet is set appropriately.