I am trying to retrieve an uploaded file from my webserver. As the client sends its files through a webform (random files), I need to parse the request to get the file out and to process it further on.
Basically, the code goes as:
HttpListenerContext context = listener.GetContext();
HttpListenerRequest request = context.Request;
StreamReader r = new StreamReader(request.InputStream, System.Text.Encoding.Default);
// this is the retrieved file from streamreader
string file = null;
while ((line = r.ReadLine()) != null){
// i read the stream till i retrieve the filename
// get the file data out and break the loop
}
// A byststream is created by converting the string,
Byte[] bytes = request.ContentEncoding.GetBytes(file);
MemoryStream mstream = new MemoryStream(bytes);
// do the rest
As a result, i am able to retrieve textfiles, but for all other files, they are corrupted.
Could someone tell me how to parse these HttplistnerRequests properly (or providing a lightweighted alternative)?
I think you are making things harder on yourself than necessary by doing this with an HttpListener rather than using the built in facilities of ASP.Net. But if you must do it this way here is some sample code. Note: 1) I'm assuming you're using enctype="multipart/form-data" on your <form>. 2) This code is designed to be used with a form containing only your <input type="file" /> if you want to post other fields or multiple files you'll have to change the code. 3) This is meant to be a proof of concept/example, it may have bugs, and is not particularly flexible.
static void Main(string[] args)
{
HttpListener listener = new HttpListener();
listener.Prefixes.Add("http://localhost:8080/ListenerTest/");
listener.Start();
HttpListenerContext context = listener.GetContext();
SaveFile(context.Request.ContentEncoding, GetBoundary(context.Request.ContentType), context.Request.InputStream);
context.Response.StatusCode = 200;
context.Response.ContentType = "text/html";
using (StreamWriter writer = new StreamWriter(context.Response.OutputStream, Encoding.UTF8))
writer.WriteLine("File Uploaded");
context.Response.Close();
listener.Stop();
}
private static String GetBoundary(String ctype)
{
return "--" + ctype.Split(';')[1].Split('=')[1];
}
private static void SaveFile(Encoding enc, String boundary, Stream input)
{
Byte[] boundaryBytes = enc.GetBytes(boundary);
Int32 boundaryLen = boundaryBytes.Length;
using (FileStream output = new FileStream("data", FileMode.Create, FileAccess.Write))
{
Byte[] buffer = new Byte[1024];
Int32 len = input.Read(buffer, 0, 1024);
Int32 startPos = -1;
// Find start boundary
while (true)
{
if (len == 0)
{
throw new Exception("Start Boundaray Not Found");
}
startPos = IndexOf(buffer, len, boundaryBytes);
if (startPos >= 0)
{
break;
}
else
{
Array.Copy(buffer, len - boundaryLen, buffer, 0, boundaryLen);
len = input.Read(buffer, boundaryLen, 1024 - boundaryLen);
}
}
// Skip four lines (Boundary, Content-Disposition, Content-Type, and a blank)
for (Int32 i = 0; i < 4; i++)
{
while (true)
{
if (len == 0)
{
throw new Exception("Preamble not Found.");
}
startPos = Array.IndexOf(buffer, enc.GetBytes("\n")[0], startPos);
if (startPos >= 0)
{
startPos++;
break;
}
else
{
len = input.Read(buffer, 0, 1024);
}
}
}
Array.Copy(buffer, startPos, buffer, 0, len - startPos);
len = len - startPos;
while (true)
{
Int32 endPos = IndexOf(buffer, len, boundaryBytes);
if (endPos >= 0)
{
if (endPos > 0) output.Write(buffer, 0, endPos-2);
break;
}
else if (len <= boundaryLen)
{
throw new Exception("End Boundaray Not Found");
}
else
{
output.Write(buffer, 0, len - boundaryLen);
Array.Copy(buffer, len - boundaryLen, buffer, 0, boundaryLen);
len = input.Read(buffer, boundaryLen, 1024 - boundaryLen) + boundaryLen;
}
}
}
}
private static Int32 IndexOf(Byte[] buffer, Int32 len, Byte[] boundaryBytes)
{
for (Int32 i = 0; i <= len - boundaryBytes.Length; i++)
{
Boolean match = true;
for (Int32 j = 0; j < boundaryBytes.Length && match; j++)
{
match = buffer[i + j] == boundaryBytes[j];
}
if (match)
{
return i;
}
}
return -1;
}
To help you better understand what the code above is doing, here is what the body of the HTTP POST looks like:
Content-Type: multipart/form-data; boundary=----WebKitFormBoundary9lcB0OZVXSqZLbmv
------WebKitFormBoundary9lcB0OZVXSqZLbmv
Content-Disposition: form-data; name="my_file"; filename="Test.txt"
Content-Type: text/plain
Test
------WebKitFormBoundary9lcB0OZVXSqZLbmv--
I've left out the irrelevant headers. As you can see, you need to parse the body by scanning through to find the beginning and ending boundary sequences, and drop the sub headers that come before the content of your file. Unfortunately you cannot use StreamReader because of the potential for binary data. Also unfortunate is the fact that there is no per file Content-Length (the Content-Length header for the request specifies the total length of the body including boundaries, sub-headers, and spacing.
You can't use StreamReader because it is meant to read streams in which the bytes are in the UTF8. Instead, you want to read the contents of the stream to a receive buffer, remove all the stuff you don't need, get the file extension of the uploaded file, extract the contents of the uploaded file, then save the file contents to a new file. The code I show in this post assumes your form looks like this:
<form enctype="multipart/form-data" method="POST" action="/uploader">
<input type="file" name="file">
<input type="submit">
</form>
As you can see, the code is only meant to handle a form that only has the file. Since there is no way to extract the contents of a file on the server from a application/x-www-form-urlencoded form, so you have to include the "multipart/form-data".
First, for this method of handling uploaded files, you will first need this little bit of code:
using System;
using System.IO;
using System.Text;
using System.Net;
using System.Collections.Generic;
Second, you need to read the contents of the request.InputStream to a receive buffer, or a byte[]. We do this by making a byte[] buffer with the length of the Content-Length header sent by the browser. Then, we read the contents of the request.InputStream to the buffer. The code would look like this:
int len = int.Parse(request.Headers["Content-Length"]);
byte[] buffer = new byte[len];
request.InputStream.Read(buffer, 0, len);
The stream will look somewhat like this:
------WebKitFormBoundary9lcB0OZVXSqZLbmv
Content-Disposition: form-data; name="file"; filename="example-file.txt"
Content-Type: text/plain
file contents here
------WebKitFormBoundary9lcB0OZVXSqZLbmv--
Next, you need to get the file extension of the uploaded file. We can do this using this code:
string fileExtension = Encoding.UTF8.GetString(bytes).Split("\r\n")[1].Split("filename=\"")[1].Replace("\"", "").Split('.')[^1];
Then, we need to get the contents of the file. We do this by removing the stuff at the beginning (the -----WebKitFormBoundary, the Content-Disposition, the Content-Type, and a blank line), then removing the last line of the request body, plus an extra \r\n at the end. Here is the code that does just that:
// note that the variable buffer is the byte[], and the variable bytes is the List<byte>
string stringBuffer = Encoding.UTF8.GetString(buffer);
List<byte> bytes = new List<byte>(buffer);
string[] splitString = stringBuffer.Split('\n');
int lengthOfFourLines = splitString[0].Length + splitString[1].Length +
splitString[2].Length + splitString[3].Length + 4;
bytes.RemoveRange(0, lengthOfFourLines);
int lengthOfLastLine = splitString[^2].Length+2;
bytes.RemoveRange(bytes.Count - lengthOfLastLine, lengthOfLastLine);
buffer = bytes.ToArray();
Finally, we need to save the contents to a file. The code below generates a random file name with the user-specified file extension, and saves the files contents to it.
string fname = "";
string[] chars = "q w e r t y u i o p a s d f g h j k l z x c v b n m Q W E R T Y U I O P A S D F G H J K L Z X C V B N M 1 2 3 4 5 6 7 8 9 0".Split(" ");
for (int i = 0; i < 10; i++)
{
fname += chars[new Random().Next(chars.Length)];
}
fname += fileExtension;
FileStream file = File.Create(fname);
file.Write(buffer);
file.Close();
Here is the whole code put together:
public static void Main()
{
var listener = new HttpListener();
listener.Prefixes.Add("http://localhost:8080/");
listener.Start();
while(true)
{
HttpListenerContext context = listener.GetContext();
HttpListenerRequest request = context.Request;
HttpListenerResponse response = context.Response;
if(request.HttpMethod=="POST") SaveFile(request);
response.OutputStream.Write(Encoding.UTF8.GetBytes("file successfully uploaded"));
response.OutputStream.Close();
}
}
void SaveFile(HttpListenerRequest request)
{
int len = (int)request.ContentLength64;
Console.WriteLine(len);
byte[] buffer = new byte[len];
request.InputStream.Read(buffer, 0, len);
string stringBuffer = Encoding.UTF8.GetString(buffer);
Console.WriteLine(stringBuffer.Replace("\r\n","\\r\\n\n"));
string fileExtension = stringBuffer.Split("\r\n")[1]
.Split("filename=\"")[1]
.Replace("\"", "")
.Split('.')[^1]
;
List<byte> bytes = new List<byte>(buffer);
string[] splitString = stringBuffer.Split('\n');
int lengthOfFourLines = splitString[0].Length + splitString[1].Length + splitString[2].Length + splitString[3].Length + 4;
bytes.RemoveRange(0, lengthOfFourLines);
int lengthOfLastLine = splitString[^2].Length+2;
bytes.RemoveRange(bytes.Count - lengthOfLastLine, lengthOfLastLine);
buffer = bytes.ToArray();
string fname = "";
string[] chars = "q w e r t y u i o p a s d f g h j k l z x c v b n m Q W E R T Y U I O P A S D F G H J K L Z X C V B N M 1 2 3 4 5 6 7 8 9 0".Split(" ");
for (int i = 0; i < 10; i++)
{
fname += chars[new Random().Next(chars.Length)];
}
fname += "." + fileExtension;
FileStream file = File.Create(fname);
file.Write(buffer);
file.Close();
}
Also, if you want to send an uploaded file to the client, here is a useful function that sends the file to the client.
// Make sure you are using System.IO, and System.Net when making this function.
// Also make sure you set the content type of the response before calling this function.
// fileName is the name of the file you want to send to the client, and output is the response.OutputStream.
public static void SendFile(string fileName, Stream output)
{
FileStream fs = new FileStream(fileName, FileMode.Open, FileAccess.Read);
fs.CopyTo(output);
fs.Close();
output.Close();
}
The problem is you are reading the file as text.
You need to read the file as a bytearray instead and using the BinaryReader is better and easier to use than StreamReader:
Byte[] bytes;
using (System.IO.BinaryReader r = new System.IO.BinaryReader(request.InputStream))
{
// Read the data from the stream into the byte array
bytes = r.ReadBytes(Convert.ToInt32(request.InputStream.Length));
}
MemoryStream mstream = new MemoryStream(bytes);
May have bugs, test thoroughly. This one gets all post, get, and files.
using System.Collections.Generic;
using System.Collections.Specialized;
using System.IO;
using System.Net;
using System.Text;
using System.Web;
namespace DUSTLauncher
{
class HttpNameValueCollection
{
public class File
{
private string _fileName;
public string FileName { get { return _fileName ?? (_fileName = ""); } set { _fileName = value; } }
private string _fileData;
public string FileData { get { return _fileData ?? (_fileName = ""); } set { _fileData = value; } }
private string _contentType;
public string ContentType { get { return _contentType ?? (_contentType = ""); } set { _contentType = value; } }
}
private NameValueCollection _get;
private Dictionary<string, File> _files;
private readonly HttpListenerContext _ctx;
public NameValueCollection Get { get { return _get ?? (_get = new NameValueCollection()); } set { _get = value; } }
public NameValueCollection Post { get { return _ctx.Request.QueryString; } }
public Dictionary<string, File> Files { get { return _files ?? (_files = new Dictionary<string, File>()); } set { _files = value; } }
private void PopulatePostMultiPart(string post_string)
{
var boundary_index = _ctx.Request.ContentType.IndexOf("boundary=") + 9;
var boundary = _ctx.Request.ContentType.Substring(boundary_index, _ctx.Request.ContentType.Length - boundary_index);
var upper_bound = post_string.Length - 4;
if (post_string.Substring(2, boundary.Length) != boundary)
throw (new InvalidDataException());
var raw_post_strings = new List<string>();
var current_string = new StringBuilder();
for (var x = 4 + boundary.Length; x < upper_bound; ++x)
{
if (post_string.Substring(x, boundary.Length) == boundary)
{
x += boundary.Length + 1;
raw_post_strings.Add(current_string.ToString().Remove(current_string.Length - 3, 3));
current_string.Clear();
continue;
}
current_string.Append(post_string[x]);
var post_variable_string = current_string.ToString();
var end_of_header = post_variable_string.IndexOf("\r\n\r\n");
if (end_of_header == -1) throw (new InvalidDataException());
var filename_index = post_variable_string.IndexOf("filename=\"", 0, end_of_header);
var filename_starts = filename_index + 10;
var content_type_starts = post_variable_string.IndexOf("Content-Type: ", 0, end_of_header) + 14;
var name_starts = post_variable_string.IndexOf("name=\"") + 6;
var data_starts = end_of_header + 4;
if (filename_index == -1) continue;
var filename = post_variable_string.Substring(filename_starts, post_variable_string.IndexOf("\"", filename_starts) - filename_starts);
var content_type = post_variable_string.Substring(content_type_starts, post_variable_string.IndexOf("\r\n", content_type_starts) - content_type_starts);
var file_data = post_variable_string.Substring(data_starts, post_variable_string.Length - data_starts);
var name = post_variable_string.Substring(name_starts, post_variable_string.IndexOf("\"", name_starts) - name_starts);
Files.Add(name, new File() { FileName = filename, ContentType = content_type, FileData = file_data });
continue;
}
}
private void PopulatePost()
{
if (_ctx.Request.HttpMethod != "POST" || _ctx.Request.ContentType == null) return;
var post_string = new StreamReader(_ctx.Request.InputStream, _ctx.Request.ContentEncoding).ReadToEnd();
if (_ctx.Request.ContentType.StartsWith("multipart/form-data"))
PopulatePostMultiPart(post_string);
else
Get = HttpUtility.ParseQueryString(post_string);
}
public HttpNameValueCollection(ref HttpListenerContext ctx)
{
_ctx = ctx;
PopulatePost();
}
}
}
I like #paul-wheeler answer. However I needed to modify their code to include some additional data (In this case, the directory structure).
I'm using this code to upload files:
var myDropzone = $("#fileDropZone");
myDropzone.dropzone(
{
url: "http://" + self.location.hostname + "/Path/Files.html,
method: "post",
createImageThumbnails: true,
previewTemplate: document.querySelector('#previewTemplateId').innerHTML,
clickable: false,
init: function () {
this.on('sending', function(file, xhr, formData){
// xhr is XMLHttpRequest
var name = file.fullPath;
if (typeof (file.fullPath) === "undefined") {
name = file.name;
}
formData.append('fileNameWithPath', name);
});
}
});
Here is #paul-wheeler modified code. Thanks #paul-wheeler.
public class FileManager
{
public static void SaveFile(HttpListenerRequest request, string savePath)
{
var tempFileName = Path.Combine(savePath, $"{DateTime.Now.Ticks}.tmp");
if (!Directory.Exists(savePath))
{
Directory.CreateDirectory(savePath);
}
var (res, fileName) = SaveTmpFile(request, tempFileName);
if (res)
{
var filePath = Path.Combine(savePath, fileName);
var fileDir = filePath.Substring(0, filePath.LastIndexOf(Path.DirectorySeparatorChar));
if (!Directory.Exists(fileDir))
{
Directory.CreateDirectory(fileDir);
}
if (File.Exists(filePath))
{
File.Delete(filePath);
}
File.Move(tempFileName, filePath);
}
}
private static (bool, string) SaveTmpFile(HttpListenerRequest request, string tempFileName)
{
var enc = request.ContentEncoding;
var boundary = GetBoundary(request.ContentType);
var input = request.InputStream;
byte[] boundaryBytes = enc.GetBytes(boundary);
var boundaryLen = boundaryBytes.Length;
using (FileStream output = new FileStream(tempFileName, FileMode.Create, FileAccess.Write))
{
var buffer = new byte[1024];
var len = input.Read(buffer, 0, 1024);
var startPos = -1;
// Get file name and relative path
var strBuffer = Encoding.Default.GetString(buffer);
var strStart = strBuffer.IndexOf("fileNameWithPath") + 21;
if (strStart < 21)
{
Logger.LogError("File name not found");
return (false, null);
}
var strEnd = strBuffer.IndexOf(boundary, strStart) - 2;
var fileName = strBuffer.Substring(strStart, strEnd - strStart);
fileName = fileName.Replace('/', Path.DirectorySeparatorChar);
// Find start boundary
while (true)
{
if (len == 0)
{
Logger.LogError("Find start boundary not found");
return (false, null);
}
startPos = IndexOf(buffer, len, boundaryBytes);
if (startPos >= 0)
{
break;
}
else
{
Array.Copy(buffer, len - boundaryLen, buffer, 0, boundaryLen);
len = input.Read(buffer, boundaryLen, 1024 - boundaryLen);
}
}
// Advance to data
var foundData = false;
while (!foundData)
{
while (true)
{
if (len == 0)
{
Logger.LogError("Preamble not Found");
return (false, null);
}
startPos = Array.IndexOf(buffer, enc.GetBytes("\n")[0], startPos);
if (startPos >= 0)
{
startPos++;
break;
}
else
{
// In case read in line is longer than buffer
len = input.Read(buffer, 0, 1024);
}
}
var currStr = Encoding.Default.GetString(buffer).Substring(startPos);
if (currStr.StartsWith("Content-Type:"))
{
// Go past the last carriage-return\line-break. (\r\n)
startPos = Array.IndexOf(buffer, enc.GetBytes("\n")[0], startPos) + 3;
break;
}
}
Array.Copy(buffer, startPos, buffer, 0, len - startPos);
len = len - startPos;
while (true)
{
var endPos = IndexOf(buffer, len, boundaryBytes);
if (endPos >= 0)
{
if (endPos > 0) output.Write(buffer, 0, endPos - 2);
break;
}
else if (len <= boundaryLen)
{
Logger.LogError("End Boundaray Not Found");
return (false, null);
}
else
{
output.Write(buffer, 0, len - boundaryLen);
Array.Copy(buffer, len - boundaryLen, buffer, 0, boundaryLen);
len = input.Read(buffer, boundaryLen, 1024 - boundaryLen) + boundaryLen;
}
}
return (true, fileName);
}
}
private static int IndexOf(byte[] buffer, int len, byte[] boundaryBytes)
{
for (int i = 0; i <= len - boundaryBytes.Length; i++)
{
var match = true;
for (var j = 0; j < boundaryBytes.Length && match; j++)
{
match = buffer[i + j] == boundaryBytes[j];
}
if (match)
{
return i;
}
}
return -1;
}
private static string GetBoundary(string ctype)
{
return "--" + ctype.Split(';')[1].Split('=')[1];
}
}
using Microsoft.AspNetCore.WebUtilities;
using Microsoft.Net.Http.Headers;
var contentType = MediaTypeHeaderValue.Parse(context.Request.ContentType);
var boundary = HeaderUtilities.RemoveQuotes(contentType.Boundary).Value;
var multipartReader = new MultipartReader(boundary, context.Request.InputStream);
var section = (await multipartReader.ReadNextSectionAsync()).AsFileSection();
var fileName = section.FileName;
var fileStream = section.FileStream;
for a given HttpListenerRequest req and a string path the following code saves a file (worked for text file but also png file)
int len = (int)req.ContentLength64;
byte[] buffer = new byte[len];
int totalRead = 0;
while(totalRead < len){
// InputStream.Read does not read always read full stream (so loop until it has)
totalRead += req.InputStream.Read(buffer, totalRead, len - totalRead);
}
string stringBuffer = Encoding.UTF8.GetString(buffer);
string startTag = stringBuffer.Substring(0, stringBuffer.IndexOf("\r\n\r\n") + 4);
string endTag = stringBuffer.Substring(stringBuffer.IndexOf("\r\n------WebKitFormBoundary"));
List<byte> bytes = new List<byte>(buffer);
bytes = bytes.GetRange(startTag.Length, len - (startTag.Length + endTag.Length));
buffer = bytes.ToArray();
File.WriteAllBytes(path, buffer);
(build on top of copee moo solution)
Related
I have the following code, which essentially reads an HTTP Response into a byte array and uses unsafe operations to copy data from the byte buffer into a specified region of a memory mapped file:
private static readonly ArrayPool<byte> Pool = ArrayPool<byte>.Shared;
[Obsolete("Obsolete")]
private static async Task GetStream(string url, long start, long end, MemoryMappedFile mmf, int bufferSize, CancellationToken cancellationToken)
{
var r = (HttpWebRequest)WebRequest.Create(url);
r.Method = "GET";
r.AddRange(start,end);
r.Pipelined = true;
r.KeepAlive = true;
using var response = await r.GetResponseAsync().ConfigureAwait(false);
var buffer = Pool.Rent(bufferSize);
try
{
await using var stream = response.GetResponseStream();
var accessor = mmf.CreateViewAccessor(start, response.ContentLength, MemoryMappedFileAccess.ReadWrite);
var bytesRead = 0;
var brtot = 0;
var total = 0;
int i = -1;
do
{
Array.Clear(buffer);
var offset = 0;
do {
bytesRead = await stream.ReadAsync(buffer, offset, bufferSize - offset, cancellationToken).ConfigureAwait(false);
brtot += bytesRead;
offset += bytesRead;
} while (bytesRead != 0 && offset < bufferSize);
if (offset == 0) {continue;}
i = i + 1;
if (brtot - total > 1)
{
CopyBytes(buffer, total, accessor, i == 0 ? brtot : brtot - total);
Console.WriteLine(String.Format("{0} - {1} ==== {2}", total, brtot - bytesRead, i));
total = brtot + 1;
}
} while (bytesRead != 0);
await stream.FlushAsync(cancellationToken);
}
finally
{
Pool.Return(buffer);
response.Close();
}
}
private static void CopyBytes(byte[] buff, int offset, MemoryMappedViewAccessor s, int length)
{
unsafe
{
var pointer = (byte*)0;
RuntimeHelpers.PrepareConstrainedRegions();
try {
s.SafeMemoryMappedViewHandle.AcquirePointer(ref pointer);
Marshal.Copy(buff, 0, IntPtr.Add(new IntPtr(pointer), offset), length);
}
finally {
if (pointer != null)
s.SafeMemoryMappedViewHandle.ReleasePointer();
}
}
}
Then I have this code that calls it:
public static async Task DownloadFile(string url, String outFile, int parts, int bufferSize)
{
var responseLength = (await WebRequest.Create(url).GetResponseAsync()).ContentLength;
var partSize = (long)Math.Floor(responseLength / (parts + 0.0));
var pieces = new List<ValueTuple<long, long>>();
var uri = new Uri(url);
var filename = outFile ?? Path.GetFileName(uri.LocalPath);
Console.WriteLine(responseLength);
Console.WriteLine(partSize);
ServicePointManager.Expect100Continue = false;
ServicePointManager.DefaultConnectionLimit = 10000;
ServicePointManager.FindServicePoint(new Uri(url)).ConnectionLimit = parts;
#if NET6_0_OR_GREATER
ServicePointManager.ReusePort = true;
#endif
//Loop to add all the events to the queue
for (long i = 0; i < responseLength; i += partSize) {
pieces.Insert(0,(i + partSize < responseLength ? new ValueTuple<long, long>(i, i + partSize) : new ValueTuple<long, long>(i, responseLength)));
}
var cs = new CancellationToken(false);
File.Delete(filename);
var mmf = MemoryMappedFile.CreateFromFile(filename, FileMode.CreateNew,null, responseLength, MemoryMappedFileAccess.ReadWrite);
GCSettings.LatencyMode = GCLatencyMode.SustainedLowLatency;
await Parallel.ForEachAsync(pieces, new ParallelOptions() { MaxDegreeOfParallelism = Environment.ProcessorCount}, async (tuple, token) =>
{
await GetStream(url,tuple.Item1, tuple.Item2, mmf, bufferSize, cs);
});
}
The only issue with the code is that it is corrupting whatever file I download with no errors. I am sure this is due to my parameters specified to CopyBytes(). But unfortunately, I can't get the right calculations for the parameters, can someone help me with this or at least point me on the right direction? I know some of the methods are obsolete, and I will address this, but I am currently more focused on getting the unsafe memory copying to work. Thank you!
I have found the answer for Java: https://stackoverflow.com/a/36357819/202179 and tried to port it to Xamarin.
Here is the code that I've made:
const string COMPRESSED_AUDIO_FILE_MIME_TYPE = "audio/mp4a-latm";
const int COMPRESSED_AUDIO_FILE_BIT_RATE = 64000; // 64kbps
const int SAMPLING_RATE = 48000;
const int BUFFER_SIZE = 48000;
const int CODEC_TIMEOUT_IN_MS = 5000;
void Compress()
{
var inputFile = new Java.IO.File(tempFileWavPath);
var fis = new Java.IO.FileInputStream(inputFile);
var outputFile = new Java.IO.File(fileM4APath);
if (outputFile.Exists())
outputFile.Delete();
var mux = new MediaMuxer(outputFile.AbsolutePath, MuxerOutputType.Mpeg4);
MediaFormat outputFormat = MediaFormat.CreateAudioFormat(COMPRESSED_AUDIO_FILE_MIME_TYPE, SAMPLING_RATE, 1);
outputFormat.SetInteger(MediaFormat.KeyAacProfile, (int)MediaCodecProfileType.Aacobjectlc);
outputFormat.SetInteger(MediaFormat.KeyBitRate, COMPRESSED_AUDIO_FILE_BIT_RATE);
outputFormat.SetInteger(MediaFormat.KeyMaxInputSize, 16384);
MediaCodec codec = MediaCodec.CreateEncoderByType(COMPRESSED_AUDIO_FILE_MIME_TYPE);
codec.Configure(outputFormat, null, null, MediaCodecConfigFlags.Encode);
codec.Start();
MediaCodec.BufferInfo outBuffInfo = new MediaCodec.BufferInfo();
byte[] tempBuffer = new byte[BUFFER_SIZE];
var hasMoreData = true;
double presentationTimeUs = 0;
int audioTrackIdx = 0;
int totalBytesRead = 0;
int percentComplete = 0;
do
{
int inputBufIndex = 0;
while (inputBufIndex != -1 && hasMoreData)
{
inputBufIndex = codec.DequeueInputBuffer(CODEC_TIMEOUT_IN_MS);
if (inputBufIndex >= 0)
{
var dstBuf = codec.GetInputBuffer(inputBufIndex);
dstBuf.Clear();
int bytesRead = fis.Read(tempBuffer, 0, dstBuf.Limit());
if (bytesRead == -1)
{ // -1 implies EOS
hasMoreData = false;
codec.QueueInputBuffer(inputBufIndex, 0, 0, (long)presentationTimeUs, MediaCodecBufferFlags.EndOfStream);
}
else
{
totalBytesRead += bytesRead;
dstBuf.Put(tempBuffer, 0, bytesRead);
codec.QueueInputBuffer(inputBufIndex, 0, bytesRead, (long)presentationTimeUs, 0);
presentationTimeUs = 1000000l * (totalBytesRead / 2) / SAMPLING_RATE;
}
}
}
// Drain audio
int outputBufIndex = 0;
while (outputBufIndex != (int)MediaCodecInfoState.TryAgainLater)
{
outputBufIndex = codec.DequeueOutputBuffer(outBuffInfo, CODEC_TIMEOUT_IN_MS);
if (outputBufIndex >= 0)
{
var encodedData = codec.GetOutputBuffer(outputBufIndex);
encodedData.Position(outBuffInfo.Offset);
encodedData.Limit(outBuffInfo.Offset + outBuffInfo.Size);
if ((outBuffInfo.Flags & MediaCodecBufferFlags.CodecConfig) != 0 && outBuffInfo.Size != 0)
{
codec.ReleaseOutputBuffer(outputBufIndex, false);
}
else
{
mux.WriteSampleData(audioTrackIdx, encodedData, outBuffInfo);
codec.ReleaseOutputBuffer(outputBufIndex, false);
}
}
else if (outputBufIndex == (int)MediaCodecInfoState.OutputFormatChanged)
{
outputFormat = codec.OutputFormat;
audioTrackIdx = mux.AddTrack(outputFormat);
mux.Start();
}
}
percentComplete = (int)Math.Round(((float)totalBytesRead / (float)inputFile.Length()) * 100.0);
} while (outBuffInfo.Flags != MediaCodecBufferFlags.EndOfStream);
fis.Close();
mux.Stop();
mux.Release();
}
This almost works as it converts the file, but the resulting file appears to be encoded too fast - the pitch is too high and speed is too high and the reproduction lasts shorter than expected.
It is likely that just some slight change is needed, but I am not sure what. Can anyone suggest?
I could reproduce the resulting file appears to be encoded too fast when i use the different size of SAMPLING_RATE.
For example, i download a wav file online. The Sampline Rate is 11025. If i use the original rate 48000 in the code, it would play too fast. When i use 11025, it would work.
So we need to know the Sampling Rate of the wav fille and then set it in the code.
const int SAMPLING_RATE = 11025;//44100, 48000
I have been experiencing some issues with imcompatiabilities between the difference implementations of zlib compression.
As a test case I thought to create test data with 10000 doubles ranging from 0 - 10000.
I created some test code to compress and decompress this data that uses the compress and uncompress in zlib.c
unsigned int Test(char* comparisonFile)
{
unsigned long partsSize = 0x80000;
const int arraySize = 10000;
Bytef doubleArray[sizeof(double) * arraySize];
Bytef outBuffer[sizeof(double) * arraySize];
for (int i = 0; i < arraySize; i++)
{
Bytef doubleBytes[sizeof(double)];
*(double*)(doubleBytes) = (double)i;
for (int x = 0; x < 8; x++)
doubleArray[(8 * i) + x] = doubleBytes[x];
}
compress(outBuffer, &partsSize, doubleArray, sizeof(double) * arraySize);
//create file of compressed data
char * filename = "zlibCompressed";
FILE * file = fopen(filename, "w+b");
int compressResult = et_int64(fwrite((char *)outBuffer, 1, size_t(partsSize), file));
fclose(file);
//load file of compressed data either from zlib or other
if (comparisonFile != NULL)
filename = comparisonFile;
FILE * compressedFile = fopen(filename, "r+b");
if (compressedFile == NULL)
return -1;
unsigned long outBufferSize = sizeof(double) * arraySize;
fseek(compressedFile, 0, SEEK_END);
long fsize = ftell(compressedFile);
fseek(compressedFile, 0, SEEK_SET); /* same as rewind(f); */
partsSize = int(fsize);
double * doubleResult = new double [arraySize];
Bytef* inBuffer = (Bytef*)malloc(sizeof(Bytef)*partsSize);
int readresult = et_int64(fread((char *)inBuffer, 1, partsSize , compressedFile));
if (readresult != partsSize)
return -1;
Bytef * uncompressedOutBuffer = static_cast<Bytef*>((void *)doubleResult);
int result = uncompress(uncompressedOutBuffer, &outBufferSize, inBuffer, partsSize);
for (int i = 0; i < arraySize; i++)
{
// uncompressed data does not match expectation
if ((int)doubleResult[i] != i)
return -2;
}
fclose(compressedFile);
return 0;
}
This allows me to test the internal compression and also substitute compression results from C#.
However, when I use ionic or standard deflate in the following manner I can only recover about 8150 of the expected 10000.
The uncompression returns a:
Z_DATA_ERROR
Given that these seem in theory interoperable, i'm not sure why C# compression results can only be partially unpacked with Adler's zlib? Any help out there?
public void ZlibTest()
{
byte[] buffer;
using (var ms = new MemoryStream())
{
for (int i = 0; i < 10000; i++)
ms.Write(BitConverter.GetBytes((double) i), 0, sizeof(double));
buffer = ms.ToArray();
}
var file = "dummy1";
if (File.Exists(file))
File.Delete(file);
using (Stream fs = new FileStream(file, FileMode.OpenOrCreate, FileAccess.ReadWrite))
{
using (var resultStream = new MemoryStream())
{
using (var compressionStream2 = new Ionic.Zlib.ZlibStream(resultStream, Ionic.Zlib.CompressionMode.Compress, CompressionLevel.Default))
{
compressionStream2.Write(buffer, 0, buffer.Length);
var packetLength = (int)resultStream.Length;
fs.Write(resultStream.ToArray(), 0, packetLength);
}
}
}
file = "dummy2";
using (Stream fs = new FileStream(file, FileMode.OpenOrCreate, FileAccess.ReadWrite))
{
using (var resultStream = new MemoryStream())
{
using (var compressionStream2 = new System.IO.Compression.DeflateStream(resultStream, System.IO.Compression.CompressionMode.Compress))
{
compressionStream2.Write(buffer, 0, buffer.Length);
var packetLength = (int)resultStream.Length;
fs.Write(BitConverter.GetBytes((ushort)40056), 0, sizeof(ushort));
fs.Write(resultStream.ToArray(), 0, packetLength);
}
}
}
}
They're both using the same zlib. There must be an error in your C# code. It is possible that you are not reading the file in binary mode in C#, which would cause occasional corruption.
I am try to do some code using BinaryWriter and Then BinaryReader.
When I wanna write I use method Write().
But the problem is that between two lines of Write method there appears a new byte which is in ASCII table in decimal 31 (sometines 24).
You can see it on this image:
You can see that byte at index 4 (5th byte) is of ASCII decimal value 31. I didnt insert it there. As you can see 1st 4 bytes are reserved for a number (Int32), next are other data (some text mostly - this is not important now).
As you can see from the code i write:
- into 1st line a number 10
- into 2nd line text "This is some text..."
How come came that 5th byte (dec 31) in between??
And this is the code I have:
static void Main(string[] args)
{
//
//// SEND - RECEIVE:
//
SendingData();
Console.ReadLine();
}
private static void SendingData()
{
int[] commandNumbers = { 1, 5, 10 }; //10 is for the users (when they send some text)!
for (int i = 0; i < commandNumbers.Length; i++)
{
//convert to byte[]
byte[] allBytes;
using (MemoryStream ms = new MemoryStream())
{
using (BinaryWriter bw = new BinaryWriter(ms))
{
bw.Write(commandNumbers[i]); //allocates 1st 4 bytes - FOR MAIN COMMANDS!
if (commandNumbers[i] == 10)
bw.Write("This is some text at command " + commandNumbers[i]); //HERE ON THIS LINE IS MY QUESTION!!!
}
allBytes = ms.ToArray();
}
//convert back:
int valueA = 0;
StringBuilder sb = new StringBuilder();
foreach (var b in GetData(allBytes).Select((a, b) => new { Value = a, Index = b }))
{
if (b.Index == 0) //1st num
valueA = BitConverter.ToInt32(b.Value, 0);
else //other text
{
foreach (byte _byte in b.Value)
sb.Append(Convert.ToChar(_byte));
}
}
if (sb.ToString().Length == 0)
sb.Append("ONLY COMMAND");
Console.WriteLine("Command = {0} and Text is \"{1}\".", valueA, sb.ToString());
}
}
private static IEnumerable<byte[]> GetData(byte[] data)
{
using (MemoryStream ms = new MemoryStream(data))
{
using (BinaryReader br = new BinaryReader(ms))
{
int j = 0;
byte[] buffer = new byte[4];
for (int i = 0; i < data.Length; i++)
{
buffer[j++] = data[i];
if (i == 3) //SENDING COMMAND DATA
{
yield return buffer;
buffer = new byte[1];
j = 0;
}
else if (i > 3) //SENDING TEXT
{
yield return buffer;
j = 0;
}
}
}
}
}
If you look at the documentation for Write(string), you'll see that it writes a length-prefixed string. So the 31 is the number of characters in your string -- perfectly normal.
You should probably be using Encoding.GetBytes and then write the bytes instead of writing a string
for example
bw.Write(
Encoding.UTF8.GetBytes("This is some text at command " + commandNumbers[i])
);
When a string is written to a binary stream, the first thing it does is write the length of the string. The string "This is some text at command 10" has 31 characters, which is the value you're seeing.
You should check the documentation of methods you use before asking questions about them:
A length-prefixed string represents the string length by prefixing to
the string a single byte or word that contains the length of that
string. This method first writes the length of the string as a UTF-7
encoded unsigned integer, and then writes that many characters to the
stream by using the BinaryWriter instance's current encoding.
;-)
(Though in fact it is an LEB128 and not UTF-7, according to Wikipedia).
The reason this byte is there because you're adding a variable amount of information, so the length is needed. If you were to add two strings, where would you know where the first ended and the second began?
If you really don't want or need that length byte, you can always convert the string to a byte array and use that.
Ok, here is my edited code. I removed BinaryWriter (while BinaryReader is still there!!), and now it works very well - no more extra bytes.
What do you thing? Is there anytihng to do better, to make it run faster?
Expecially Im interesting for that foreach loop, which read from another method that is yield return type!!
New Code:
static void Main(string[] args)
{
//
//// SEND - RECEIVE:
//
SendingData();
Console.ReadLine();
}
private static void SendingData()
{
int[] commands = { 1, 2, 3 };
// 1 - user text
// 2 - new game
// 3 - join game
// ...
for (int i = 0; i < commands.Length; i++)
{
//convert to byte[]
byte[] allBytes;
using (MemoryStream ms = new MemoryStream())
{
// 1.st - write a command:
ms.Write(BitConverter.GetBytes(commands[i]), 0, 4);
// 2nd - write a text:
if (commands[i] == 1)
{
//some example text (like that user sends it):
string myText = "This is some text at command " + commands[i];
byte[] myBytes = Encoding.UTF8.GetBytes(myText);
ms.Write(myBytes, 0, myBytes.Length);
}
allBytes = ms.ToArray();
}
//convert back:
int valueA = 0;
StringBuilder sb = new StringBuilder();
foreach (var b in ReadingData(allBytes).Select((a, b) => new { Value = a, Index = b }))
{
if (b.Index == 0)
{
valueA = BitConverter.ToInt32(b.Value, 0);
}
else
{
sb.Append(Convert.ToChar(b.Value[0]));
}
}
if (sb.ToString().Length == 0)
sb.Append("ONLY COMMAND");
Console.WriteLine("Command = {0} and Text is \"{1}\".", valueA, sb.ToString());
}
}
private static IEnumerable<byte[]> ReadingData(byte[] data)
{
using (MemoryStream ms = new MemoryStream(data))
{
using (BinaryReader br = new BinaryReader(ms))
{
int j = 0;
byte[] buffer = new byte[4];
for (int i = 0; i < data.Length; i++)
{
buffer[j++] = data[i];
if (i == 3) //SENDING COMMAND DATA
{
yield return buffer;
buffer = new byte[1];
j = 0;
}
else if (i > 3) //SENDING TEXT
{
yield return buffer;
j = 0;
}
}
}
}
}
if I want to partially download a file and define a single range in the request Header, I get
the byte-stream of the requested file in the response body.
But if i specify multiple ranges as below, I always get for each defined range an additional
response header (wich describes the requested range) within the response body that corrupts
the downloaded file.
static void Main(string[] args)
{
Console.Write("Please enter target File: ");
string Target = Console.ReadLine();
string Source = #"http://mozilla-mirror.3347.voxcdn.com/pub/mozilla.org/firefox/releases/3.6/win32/de/Firefox%20Setup%203.6.exe";
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(Source);
request.Credentials = CredentialCache.DefaultCredentials;
// define multiple Ranges
request.AddRange( 0, 1000000);
request.AddRange(1000000, 2000000);
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
using (Stream source = response.GetResponseStream())
{
using (FileStream target = File.Open(Target, FileMode.OpenOrCreate, FileAccess.Write, FileShare.ReadWrite))
{
byte[] buffer = new byte[4096];
int BytesRead = 0;
int TotalBytesRead = 0;
while((BytesRead = source.Read(buffer, 0, buffer.Length)) > 0)
{
target.Write(buffer, 0, BytesRead);
TotalBytesRead += BytesRead;
Console.WriteLine("{0}", TotalBytesRead);
}
}
}
Console.WriteLine("Downloading Finished!");
Console.ReadLine();
}
Request as shown in Wireshark:
http://img197.imageshack.us/img197/8199/requesty.png
Response Body should only contain the Byte-Stream of the file, but additionally contains the unwanted Response-Header at the beginning of each defined Range:
Is it possible to avoid the additional response header in the body without requesting each Range separately?
or
Is there a build-in way to filter the additional response header, whose size could vary depending on the HTTP-Server?
No, that's how multiple ranges in HTTP/1.1 work. See RFC 2616, Section 19.2.
thanks for your help, as described in the link above it is the supposed
way http responds to a request with multiple ranges.
so....
Is it possible to avoid the additional response header in the body without
requesting each Range separately?
=> No.
Is there a build-in way to filter the additional response header, whose size
could vary depending on the HTTP-Server?
=> I don't know but ...
maybe some of you could have a critical look at the following chunk of code wich
filters the headers from the file data:
public void DoDownload(Range[] Ranges)
{
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(m_Source);
request.Credentials = CredentialCache.DefaultCredentials;
foreach (Range r in Ranges)
{
request.AddRange(r.From, r.To);
}
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
string boundary = "";
Match m = Regex.Match(response.ContentType, #"^.*boundary=(?<boundary>.*)$");
if (m.Success)
{
boundary = m.Groups["boundary"].Value;
}
else
{
throw new InvalidDataException("invalid packet data: no boundary specification found.");
}
using (Stream source = response.GetResponseStream())
{
using (FileStream target = File.Open(m_TargetFile, FileMode.OpenOrCreate, FileAccess.Write, FileShare.ReadWrite))
{
// buffer for payload
byte[] buffer = new byte[4096];
// buffer for current range header
byte[] header = new byte[200];
// next header after x bytes
int NextHeader = 0;
// current position in header[]
int HeaderPosition = 0;
// current position in buffer[]
int BufferPosition = 0;
// left data to proceed
int BytesToProceed = 0;
// total data written without range-headers
long TotalBytesWritten = 0;
// count of last data written to target file
int BytesWritten = 0;
// size of processed header data
int HeaderSize = 0;
// count of last data read from ResponseStream
int BytesRead = 0;
while ((BytesRead = source.Read(buffer, 0, buffer.Length)) > 0)
{
BufferPosition = 0;
BytesToProceed = BytesRead;
HeaderSize = 0;
while (BytesToProceed > 0)
{
if (NextHeader == 0)
{
for (;HeaderPosition < header.Length; HeaderPosition++, BufferPosition++, HeaderSize++)
{
if (BytesToProceed > HeaderPosition && BufferPosition < BytesRead)
{
header[HeaderPosition] = buffer[BufferPosition];
if (HeaderPosition >= 4 &&
header[HeaderPosition - 3] == 0x0d &&
header[HeaderPosition - 2] == 0x0a &&
header[HeaderPosition - 1] == 0x0d &&
header[HeaderPosition] == 0x0a)
{
string RangeHeader = Encoding.ASCII.GetString(header, 0, HeaderPosition + 1);
Match mm = Regex.Match(RangeHeader,
#"^\r\n(--)?" + boundary + #".*?(?<from>\d+)\s*-\s*(?<to>\d+)/.*\r\n\r\n", RegexOptions.Singleline);
if (mm.Success)
{
int RangeStart = Convert.ToInt32(mm.Groups["from"].Value);
int RangeEnd = Convert.ToInt32(mm.Groups["to"].Value);
NextHeader = (RangeEnd - RangeStart) + 1;
target.Seek(RangeStart, SeekOrigin.Begin);
BufferPosition++;
BytesToProceed -= HeaderSize + 1;
HeaderPosition = 0;
HeaderSize = 0;
break;
}
else { throw new InvalidDataException("invalid header: missing range specification.");}
}
}
else { goto READ_NEW; }
}
if (NextHeader == 0)
throw new InvalidDataException("invalid packet data: no range-header found.");
}
BytesWritten = (NextHeader > BytesToProceed) ? BytesToProceed : NextHeader;
target.Write(buffer, BufferPosition, BytesWritten);
BytesToProceed -= BytesWritten;
NextHeader -= BytesWritten;
BufferPosition += BytesWritten;
TotalBytesWritten += BytesWritten;
}
READ_NEW:;
}
}
}
}
and could give me some hints if there is a another/better way to do that.
best regards
cap_Chap