We Keep Coding

Redirect/overwrite resource request without changing actual file URL in CefSharp

In CefSharp, I need to redirect all resource/file requests from a domain (e.g. fileserver.com) to a different domain (e.g. myfileserver.com which will mirror the same files with sometimes a bit of modifications (like changed graphics/.png files).
I have tried implementing an IRequestHandler with OnBeforeResourceLoad which would check and redirect all resource requests going to fileserver.com to myfileserver.com instead. For example:
public CefReturnValue OnBeforeResourceLoad(IWebBrowser browserControl, IBrowser browser, IFrame frame, IRequest request, IRequestCallback callback)
{
string originalUrl = request.Url;
if (request.Url.contains("http://fileserver.com"))
{
Console.WriteLine("fileserver.com: " + request.Url);
string filename = GetFileName(request.Url);
// e.g. http://fileserver.com/abc.png = abc.png
// We want to use the files from myfileserver.com instead
request.Url = "http://myfileserver.com/" + filename;
}
callback.Dispose();
return CefReturnValue.Continue;
}
Now, the above code should work (works using with a HTML img tag), the problem is that fileserver.com is used for a Flash game and I don't think it handles this sort of redirect correctly as it still expects to get the file from the real URL, and this causes for no files/graphics from fileserver.com or myfileserver.com to be loaded in the Flash game as the original URL got cancelled from the redirect.
So, I think I would have to do this redirect/overwrite the fileserver.com requested resource without doing an actual URL change.
As an alternative, I have been trying to get IResponseFilter with IResponseFilter.Filter to directly overwrite the file [bytes?] instead of redirecting. However, I'm completely lost in how to do it.
Here is some pseudocode on how I imagined it would work (I have myfileserver.com files now on my disk instead):
FilterStatus IResponseFilter.Filter(Stream dataIn, out long dataInRead, Stream dataOut, out long dataOutWritten)
{
// For example, fileserver.com has requested file abc.png
// Read bytes from my local version of the file from fileserver.com
byte[] myLocalFileBytes = File.ReadAllBytes("myfileserver\\abc.png"); // name of requested file, not very sure how to retrieve this here
// Write my file to the requested file from fileserver.com to replace it
memoryStream.Write(myLocalFileBytes, 0, myLocalFileBytes.Length);
// This is obviously completely wrong and probably even the wrong place, sorry!
return FilterStatus.Done;
}
(I got this idea from filter in CefSharp repo, as I couldn't find any other methods to try overwrite file without doing URL redirect, but no clue how to make it read my local file and write it over the requested file)
My other idea is to make myfileserver.com pretend that it's actually fileserver.com but don't think it's possible.
I have no experience with trying to read/write memory like this so I have no idea what to do. I would appreciate any help or feedback, thanks.

e.TotalBytesToReceive returns -1 [duplicate]

This question already has answers here:
Download file from FTP with Progress - TotalBytesToReceive is always -1?
(3 answers)
Closed 4 years ago.
I have a ListBox that contains a list of DirectAdmin user backups. List is populated using WebRequestMethods.Ftp.ListDirectory and it looks like this:
I can download an archive using the button at the bottom right. When I click on the button, another form appears and downloads the archive.
My download code is this:
public static void DownloadFile(string server, string username, ...)
{
Uri URI = new Uri($"ftp://{server}/{targetFilePath}");
using (WebClient client = new WebClient())
{
client.Credentials = new NetworkCredential(username, password);
if (progress != null)
{
client.DownloadProgressChanged += new DownloadProgressChangedEventHandler(progress);
}
if (complete != null)
{
client.DownloadFileCompleted += new AsyncCompletedEventHandler(complete);
}
before?.Invoke();
client.DownloadFileAsync(URI, localFilePath);
}
}
and this is what I pass to the DownloadFile() method for the DownloadProgressChanged event:
delegate (object s2, DownloadProgressChangedEventArgs e2)
{
TransferLabel.Invoke((MethodInvoker)delegate
{
TransferLabel.Text = $"{(e2.BytesReceived / 1024).ToString()} KB / {(e2.TotalBytesToReceive / 1024).ToString()} KB";
});
TransferProgressBar.Invoke((MethodInvoker)delegate
{
TransferProgressBar.Value = (int)(e2.BytesReceived / (float)e2.TotalBytesToReceive * 100);
});
}
I'm using this same approach to upload a file and it works fine, but with download e2.TotalBytesToReceive returns -1 throughout the process:
and only when it's done, I get the correct value:
Why is that?
I've found a workaround to solve the problem. I'll change the ListBox to ListView and also store the filesize of the archives using ListDirectoryDetails. This way I can compare the e.BytesReceived to stored total bytes instead of e.TotalBytesToReceive. This would solve my problem, but I'm still curious about the problem. Why do I get -1? Am I doing something wrong, or is this a server related problem? Also is there anything I can do to fix it (get the correct value)?

With FTP protocol, WebClient in general does not know total download size. So you commonly get -1 with FTP.
See also Download file from FTP with Progress - TotalBytesToReceive is always -1?
Note that the behavior actually contradicts the .NET documentation, which says for FtpWebResponse.ContentLength (where the value of TotalBytesToReceive comes from):
For requests that use the DownloadFile method, the property is greater than zero if the downloaded file contained data and is zero if it was empty.
But you will easily find out many of questions about this (like the one I've linked above), effectively showing that the behavior is not always as documented. The FtpWebResponse.ContentLength has a meaningful value for GetFileSize method only.
The FtpWebRequest/WebClient makes no explicit attempt to find out a size of the file that it is downloading. All it does is that it tries to look for (xxx bytes). string in 125/150 responses to RETR command. No FTP RFC mandates that the server should include such information. ProFTPD (see data_pasv_open in src/data.c) and vsftpd (see handle_retr in postlogin.c) seem to include this information. Other common FTP servers (IIS, FileZilla) do not do this.

Certainly for HTTP downloads it's possible for the server not to supply size information when performing a file download and you're left with no sensible information until the server signals that it's done.
Not sure for FTP (I'd note that there's a separate SIZE command defined in the FTP command set and so including such information during a Retrieve may be considered redundant).
I'm slightly surprised that the documentation for TotalBytesToRetrieve isn't more explicit on the possibility that the information will not be available and what will be returned in such circumstances.

Walmart API POST failing with 400 Bad Request (inventory feed) ARCA

I am having problems with a POST request to the Walmart Marketplace API for bulk data exchange and am hoping for some help.
Background:
I have been successful in writing signature authentication routines, and can successfully execute GET commands such as get products etc. This indicates to me that Authentication signatures are properly formatted, and headers (for the most part) are correct.
Problem:
I am receiving a 400 Bad Request response, Request Content is Invalid. response when attempting to submit a test feed to Walmarts API. I have read that this problem is common, but I have yet to find any forum post that clearly explains the actual problem, or how to fix it. Here are my current parameters:
ARCA
ARCA Rest Client For Chrome
URL:
https://marketplace.walmartapis.com/v2/feeds?feedType=inventory
Headers:
Accept: application/xml
WM_SVC.NAME: Walmart Marketplace
WM_CONSUMER.ID: <Consumer ID>
WM_SEC.AUTH_SIGNATURE: <Good Auth Signature>
WM_QOS.CORRELATION_ID: 15649814651
WM_SEC.TIMESTAMP: <Timestamp>
WM_CONSUMER.CHANNEL.TYPE: <Channel Type>
Content-Type: multipart/form-data
File attachment (not raw payload although that has been tried)
<?xml version="1.0" encoding="utf-8"?>
<InventoryFeed xmlns="http://walmart.com/">
<InventoryHeader>
<version>1.4</version>
</InventoryHeader>
<inventory>
<sku>KON04418</sku>
<quantity>
<unit>EACH</unit>
<amount>4</amount>
</quantity>
<fulfillmentLagTime>1</fulfillmentLagTime>
</inventory>
</InventoryFeed>
When I take this exact same XML and test it at Walmart API Explorer
the file is accepted with Response Code 200 (OK).
I have validated with Notepad++ XML Tools plugin that the XML conforms to the XSD provided by Walmart. I have seen numerous posts regarding Boundaries that need to be applied, so I have additionally attempted to change the Content-Type, and add Boundaries but have been unsuccessful in getting the request accepted.
Any help in getting this request to return a response code 200 would be greatly appreciated.
Lastly, once this request validates in ARCA, I will be implementing in C#. I already have all of the code written, but there's a bit of fuzziness about how to add an attachment to an HttpWebRequest vs. just submitting a raw data stream. If any clarity could be provided on the difference I would again, appreciate it.

So this answer isn't clean and elegant, more of a work around than anything. I have spoken with a few individuals inside the Walmart engineering team and have been told that a C# SDK should be forthcoming in the next few months.
After all of my research, it appears there is some tricks to how you submit a multi-part form to Walmart and the system is very inflexible. I've seen posts about adding specifically formatted boundaries into the body of the HTTP request, but had no such luck. I was unable to attach the body as a file, or as a data stream to the request.
The work around is pretty simple, and ugly unfortunately. It takes a bit of setup, but you can create a .jar wrapper around the Walmart Java SDK and call it from your .Net program.
So.. steps in the process:
Grab the appropriate .XSD files and generate C# classes from them.
Build properly formatted XML inventory file. Make sure to include namespaces!! Walmart will fail particular calls if you don't include the appropriate ns2 / ns3 namespaces.
Dynamically generate a batch file to call your Java module. Spawning a command line process directly seemed to make things cranky for some reason so I opted for the batch file instead.
string path = #Directory.GetParent(Environment.CurrentDirectory).ToString();
if (File.Exists(#"../inventory.bat"))
{
File.Delete(#"../inventory.bat");
}
string batchCommand = #"cd " + path + Environment.NewLine + #"java -jar WalmartWrapper.jar SubmitInventoryFeed inventoryFeed.xml";
File.WriteAllText(path + #"\\inventory.bat", batchCommand);
ProcessStartInfo info = new ProcessStartInfo();
info.UseShellExecute = true;
info.FileName = #"inventory.bat";
info.WorkingDirectory = path;
var p = Process.Start(info);
p.WaitForExit();`
From here, the Java module takes over. It took a bit of hacking around to make it work more like an SDK and less like a sample program.. Here's some of the sample code for making things work..
Entry Point
if ("SubmitInventoryFeed".equals(args[0].trim())) {
if (args.length < 2)
{
System.out.println("Need second argument for SubmitInventoryFeed");
return;
}
String filename = args[1];
Feed inventoryFeed = new Feed();
try
{
inventoryFeed.submitInventoryFeed(filename);
} catch (Exception ex) {
System.out.println("submitInventoryFeed failed: " + ex.getMessage());
}
}
SDK Call (This is the bare bones of submitInventoryFeed without error checking)
String path = Paths.get(".").toAbsolutePath().normalize().toString();
File itemFile = FileHandler.getFile(filename.trim());
String filePath = path + "\\" + "MarketplaceClientConfig.properties";
WalmartMarketplace wm = Utils.getClient(filePath);
Response response = wm.submitFeed(MarketplaceFeedType.inventory, itemFile);
You can use ResponseChecker.isResponse200(response, true) to test for successful submissions
Use FeedAcknowledgement ack = response.readEntity(FeedAcknowledgement.class); to grab the actual response to check errors
I will be the first to say I can't wait to replace this work around with the C# SDK pending from Walmart, but for the time being, this is the only way I have been able to submit. I've looked through the walmart code in depth, but unfortunately, there is some Java magic happening under the hood that does the file attachment so there's not really any way to gain access to the exact procedure and reverse engineer for C#. I think that someone who really knew Java inside and out could figure it out, but I have enough Java background to allow me to cobble together a workable, albeit ugly solution.

Streaming MP3 Chunks on ASP.NET

Currently, I have a feature on an ASP.NET website where the user can play back MP3 Files. The code looks something like this:
Response.Clear();
Response.ContentType = "audio/mpeg";
foreach (DataChunk leChunk in db.Mp3Files.First(mp3 => mp3.Mp3ResourceId.Equals(id)).Data.Chunks.OrderBy(chunk => chunk.ChunkOrder))
{
Response.BinaryWrite(leChunk.Data);
}
Unfortunately, if a larger MP3 file is selected, the audio does not begin to play until the entire file is downloaded, which can cause a noticeable delay. Is there any way to get the MP3 to start playing immediately, even though the entire file may not yet be transferred?

You should be able to do what you want by writing to the outpstream of the response, i.e.:
Response.OutputStream.Write
It is also probably a good idea to check previously if Response.IsClientConnected and give up if not.
I found a demo that allows playback of mp3 files from an asp.net web application:
http://aspsnippets.com/Articles/Save-MP3-Audio-Files-to-database-and-display-in-ASPNet-GridView-with-Play-and-Download-option.aspx

try this:
Response.BufferOutput = false; //sets chunked encoding
Response.ContentType = "audio/mpeg";
using (var bw = new BinaryWriter(Response.OutputStream))
{
foreach (DataChunk leChunk in db.Mp3Files.First(mp3 => mp3.Mp3ResourceId.Equals(id)).Data.Chunks.OrderBy(chunk => chunk.ChunkOrder))
{
if (Response.IsClientConnected) //avoids the host closed the connection exception
{
bw.Write(leChunk.Data);
}
}
}
Also, go yo your web.config file and do this if you still have problems with chunked encoding:
<system.webServer>
<asp enableChunkedEncoding="true" />
</system.webServer>
The error you reported above about the host being closing the connection is happening probably because you are opening the page using the browser and when the browser reads the content type, it opens the media player and closes itself who had the opened connection which was then closed, causing that error, so to avoid this, you need to check periodically whether your client is still connected or not.
Finally, I would use a Generic Handler (.ashx) or a custom handler and set a .mp3 extension for this if you are using a aspx page to avoid the unnecessary overhead of the web page.
I hope this helps.

Try setting Response.BufferOutput = false before streaming the response.

If the location of the MP3 files are publicly available to your user then an alternative approach could be to just return the MP3's URL and use the HTML 5 audio tags in your mark up to stream the music. I am pretty sure that the default behaviour of the audio tag would be to stream the file rather than wait until the whole file has downloaded.

One method to support this would be implementing HTTP byte range requests.
By default I don't believe that ASP.NET does this, and definitely won't if using any of the code in the questions or the answer.
You can implement this manually with a little work though. Another option, which would be much less dev work, would be to let IIS serve a static file. I assume that isn't an option though.
Here's an example implementation:
http://www.codeproject.com/Articles/820146/HTTP-Partial-Content-In-ASP-NET-Web-API-Video

Read only the title and/or META tag of HTML file, without loading complete HTML file

Scenario :
I need to parse millions of HTML files/pages (as fact as I can) & then read only only Title or Meta part of it & Dump it to Database
What I am doing is using System.Net.WebClient Class's DownloadString(url_path) to download & then Saving it to Database by LINQ To SQL
But this DownloadString function gives me complete html source, I just need only Title part & META tag part.
Any ideas, to download only that much content?

I think you can open a stream with this url and use this stream to read the first x bytes, I can't tell the exact number but i think you can set it to reasonable number to get the title and the description.
HttpWebRequest fileToDownload = (HttpWebRequest)HttpWebRequest.Create("YourURL");
using (WebResponse fileDownloadResponse = fileToDownload.GetResponse())
{
using (Stream fileStream = fileDownloadResponse.GetResponseStream())
{
using (StreamReader fileStreamReader = new StreamReader(fileStream))
{
char[] x = new char[Number];
fileStreamReader.Read(x, 0, Number);
string data = "";
foreach (char item in x)
{
data += item.ToString();
}
}
}
}

I suspect that WebClient will try to download the whole page first, in which case you'd probably want a raw client socket. Send the appropriate HTTP request (manually, since you're using raw sockets), start reading the response (which will not be immediately) and kill the connection when you've read enough. However, the rest will have probably already been sent from the server and winging its way to your PC whether you want it or not, so you might not save much - if anything - of the bandwidth.
Depending on what you want it for, many half decent websites have a custom 404 page which is a lot simpler than a known page. Whether that has the information you're after is another matter.

You can use the verb "HEAD" in a HttpWebRequest to return the the response headers (not element. To get the full element with the meta data you'll need to download the page and parse out the meta data you want.
System.Net.WebRequest.Create(uri) { Method = "HEAD" };