C# gRPC file streaming, original file smaller than the streamed one - c#

I am having some problems with setting up a request-stream type gRPC architecture. The code below is just for testing purposes and it is missing various validation checks, but the main issue is that the original file is always smaller than the received one.
Could the cause here be encoding? It doesn't matter what the file type is, the end result is always that the file sizes are different.
Protobuf inteface:
syntax = "proto3";
package FileTransfer;
option csharp_namespace = "FileTransferProto";
service FileTransferService {
rpc DownloadFile(FileRequest) returns (stream ChunkMsg);
}
message ChunkMsg {
string FileName = 1;
int64 FileSize = 2;
bytes Chunk = 3;
}
message FileRequest {
string FilePath = 1;
}
Server side (sending):
public override async Task DownloadFile(FileRequest request, IServerStreamWriter<ChunkMsg> responseStream, ServerCallContext context)
{
string filePath = request.FilePath;
if (!File.Exists(filePath)) { return; }
FileInfo fileInfo = new FileInfo(filePath);
ChunkMsg chunk = new ChunkMsg();
chunk.FileName = Path.GetFileName(filePath);
chunk.FileSize = fileInfo.Length;
int fileChunkSize = 64 * 1024;
byte[] fileByteArray = File.ReadAllBytes(filePath);
byte[] fileChunk = new byte[fileChunkSize];
int fileOffset = 0;
while (fileOffset < fileByteArray.Length && !context.CancellationToken.IsCancellationRequested)
{
int length = Math.Min(fileChunkSize, fileByteArray.Length - fileOffset);
Buffer.BlockCopy(fileByteArray, fileOffset, fileChunk, 0, length);
fileOffset += length;
ByteString byteString = ByteString.CopyFrom(fileChunk);
chunk.Chunk = byteString;
await responseStream.WriteAsync(chunk).ConfigureAwait(false);
}
}
Client side (receiving):
public static async Task GetFile(string filePath)
{
var channel = Grpc.Net.Client.GrpcChannel.ForAddress("https://localhost:5001/", new GrpcChannelOptions
{
MaxReceiveMessageSize = 5 * 1024 * 1024, // 5 MB
MaxSendMessageSize = 5 * 1024 * 1024, // 5 MB
});
var client = new FileTransferProto.FileTransferService.FileTransferServiceClient(channel);
var request = new FileRequest { FilePath = filePath };
string tempFileName = $"temp_{DateTime.UtcNow.ToString("yyyyMMdd_HHmmss")}.tmp";
string finalFileName = tempFileName;
using (var call = client.DownloadFile(request))
{
await using (Stream fs = File.OpenWrite(tempFileName))
{
await foreach (ChunkMsg chunkMsg in call.ResponseStream.ReadAllAsync().ConfigureAwait(false))
{
Int64 totalSize = chunkMsg.FileSize;
string tempFinalFilePath = chunkMsg.FileName;
if (!string.IsNullOrEmpty(tempFinalFilePath))
{
finalFileName = chunkMsg.FileName;
}
fs.Write(chunkMsg.Chunk.ToByteArray());
}
}
}
if (finalFileName != tempFileName)
{
File.Move(tempFileName, finalFileName);
}
}

To add to Marc's answer, I feel like you can simplify your code a little bit.
using var fs = File.Open(filePath, System.IO.FileMode.Open);
int bytesRead;
var buffer = new byte[fileChunkSize];
while ((bytesRead = await fs.ReadAsync(buffer)) > 0)
{
await call.RequestStream.WriteAsync(new ChunkMsg
{
// Here the correct number of bytes must be sent which is starting from
// index 0 up to the number of read bytes from the file stream.
// If you solely pass 'buffer' here, the same bug would be present.
Chunk = ByteString.CopyFrom(buffer[0..bytesRead]),
});
}
I've used the array range operator from C# 8.0 which makes this cleaner or you can also use the overload of ByteString.CopyFrom which takes in an offset and count of how many bytes to include.

In your write loop, the chunk you actually send is for the oversized buffer, not accounting for length. This means that the last segment includes some garbage and is oversized. The received payload will be oversized by this same amount. So: make sure you account for length when constructing the chunk to send.

I tested the code and modified it to transfer the correct size.
The complete code is available at the following URL: https://github.com/lisa3907/grpc.fileTransfer
server-side-code
while (_offset < _file_bytes.Length)
{
if (context.CancellationToken.IsCancellationRequested)
break;
var _length = Math.Min(_chunk_size, _file_bytes.Length - _offset);
Buffer.BlockCopy(_file_bytes, _offset, _file_chunk, 0, _length);
_offset += _length;
_chunk.ChunkSize = _length;
_chunk.Chunk = ByteString.CopyFrom(_file_chunk);
await responseStream.WriteAsync(_chunk).ConfigureAwait(false);
}
client-side-code
await foreach (var _chunk in _call.ResponseStream.ReadAllAsync().ConfigureAwait(false))
{
var _total_size = _chunk.FileSize;
if (!String.IsNullOrEmpty(_chunk.FileName))
{
_final_file = _chunk.FileName;
}
if (_chunk.Chunk.Length == _chunk.ChunkSize)
_fs.Write(_chunk.Chunk.ToByteArray());
else
{
_fs.Write(_chunk.Chunk.ToByteArray(), 0, _chunk.ChunkSize);
Console.WriteLine($"final chunk size: {_chunk.ChunkSize}");
}
}

Related

OneDrive REST API - Requested Range Error While Uploading Chunk

I've been trying to upload files to my OneDrive via HTTP Requests following this document (https://learn.microsoft.com/en-us/onedrive/developer/rest-api/api/driveitem_createuploadsession?view=odsp-graph-online) without success. I have the following steps rounded up (Authentication, folder creation for the file, create an upload session) but when I try the last step, byte upload to the created session, I get this error in the second PUT request:
Requested Range Not Satisfiable {"error":{"code":"invalidRange","message":"Optimistic concurrency failure during fragmented upload"}}
This is my code:
//Get File Data
byte[] FileByteArray = File.ReadAllBytes(FilePath);
//Create Upload Session
OutlookEndpoint = $"{AppSettings.DriveSettings.OneDriveSettings.Endpoint}/me/drive/items/{FolderId}:/{Name}:/createuploadsession";
OutlookResponseMessage = await OutlookClient.PostAsync(OutlookEndpoint, new StringContent("{}", Encoding.UTF8, "application/json"));
OutlookResponseContent = await OutlookResponseMessage.Content.ReadAsStringAsync();
if (OutlookResponseMessage.IsSuccessStatusCode)
{
OutlookUpload OutlookUpload = JsonConvert.DeserializeObject<OutlookUpload>(OutlookResponseContent);
//Check the Created URL
if (!string.IsNullOrEmpty(OutlookUpload.UploadUrl))
{
//Chunk Calculation
int TotalSize = FileByteArray.Length;
int AcumulativeSize = 0;
int ChunkSize = 327680;
int ChunkBuffer = ChunkSize;
int ChunkNumber = TotalSize / ChunkSize;
int ChunkLeftover = TotalSize - ChunkSize * ChunkNumber;
int ChunkCounter = 0;
while (true)
{
if (ChunkNumber == ChunkCounter)
{
ChunkSize = ChunkLeftover;
}
byte[] ChunkData = FileByteArray.Skip(ChunkBuffer * ChunkCounter).Take(ChunkSize).ToArray();
AcumulativeSize += ChunkData.Length;
//PUT Upload of Chunk
string UploadEndpoint = OutlookUpload.UploadUrl;
string BytesHeader = $"bytes {AcumulativeSize - ChunkSize}-{AcumulativeSize - 1}/{TotalSize}";
OutlookClient.DefaultRequestHeaders.Clear();
OutlookClient.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Bearer", AccessToken);
OutlookClient.DefaultRequestHeaders.TryAddWithoutValidation("Content-Length", ChunkSize.ToString());
OutlookClient.DefaultRequestHeaders.TryAddWithoutValidation("Content-Range", BytesHeader);
OutlookResponseMessage = await OutlookClient.PutAsync(UploadEndpoint, new ByteArrayContent(ChunkData));
OutlookResponseContent = await OutlookResponseMessage.Content.ReadAsStringAsync();
if (OutlookResponseMessage.IsSuccessStatusCode)
{
Console.WriteLine("SUCCESS");
}
else
{
Console.WriteLine(OutlookResponseMessage.ReasonPhrase);
}
if (ChunkNumber == ChunkCounter)
{
break;
}
ChunkCounter++;
}
}
}
Perhaps I'm missing something. I only get a SUCCESS message in the first PUT request, the others always give me the error described above. Here's an image of the error with the headers I send. Image
I'd appreciate any help, thanks for reading this far.
EDIT:
Got it working after modifying the the header configuration for the request and changing the way chunks are created.
//Get File Data
byte[] FileByteArray = File.ReadAllBytes(FilePath);
//Create Upload Session
OutlookEndpoint = $"{AppSettings.DriveSettings.OneDriveSettings.Endpoint}/me/drive/items/{FolderId}:/{Name}:/createuploadsession";
OutlookResponseMessage = await OutlookClient.PostAsync(OutlookEndpoint, new StringContent("{}", Encoding.UTF8, "application/json"));
OutlookResponseContent = await OutlookResponseMessage.Content.ReadAsStringAsync();
if (OutlookResponseMessage.IsSuccessStatusCode)
{
OutlookUpload OutlookUpload = JsonConvert.DeserializeObject<OutlookUpload>(OutlookResponseContent);
//Check the Created URL
if (!string.IsNullOrEmpty(OutlookUpload.UploadUrl))
{
using MemoryStream FileStream = new MemoryStream(FileByteArray);
//Chunk Calculation
int ChunkSize = 320 * 1024;
int ChunkRemaining = 0;
byte[] ByteBuffer = new byte[ChunkSize];
int BytesRead = 0;
while ((BytesRead = FileStream.Read(ByteBuffer, 0, ByteBuffer.Length)) > 0)
{
if (BytesRead < ChunkSize)
{
byte[] LastBuffer = new byte[BytesRead];
Buffer.BlockCopy(ByteBuffer, 0, LastBuffer, 0, BytesRead);
ByteBuffer = new byte[BytesRead];
ByteBuffer = LastBuffer;
}
try
{
OutlookClient.DefaultRequestHeaders.Clear();
string UploadEndpoint = OutlookUpload.UploadUrl;
string BytesHeader = $"bytes {ChunkRemaining}-{ChunkRemaining + ByteBuffer.Length - 1}/{FileByteArray.Length}";
HttpRequestMessage MicrosoftResponseMessage = new HttpRequestMessage()
{
Content = new ByteArrayContent(ByteBuffer),
RequestUri = new Uri(UploadEndpoint),
Method = HttpMethod.Put,
};
MicrosoftResponseMessage.Content.Headers.Add("Content-Length", ByteBuffer.Length.ToString());
MicrosoftResponseMessage.Content.Headers.Add("Content-Range", BytesHeader);
OutlookResponseMessage = await OutlookClient.SendAsync(MicrosoftResponseMessage);
OutlookResponseContent = await OutlookResponseMessage.Content.ReadAsStringAsync();
if (OutlookResponseMessage.IsSuccessStatusCode)
{
Console.WriteLine("SUCCESS");
ChunkRemaining += ByteBuffer.Length;
if (ChunkRemaining == FileByteArray.Length)
{
Console.WriteLine("COMPLETED");
}
}
else
{
Console.WriteLine(OutlookResponseMessage.ReasonPhrase);
}
}
catch (Exception Exception)
{
Console.WriteLine(Exception.Message);
break;
}
}
}
}
Please note that on failures when the client sent a fragment the server had already received, the server will respond with HTTP 416 Requested Range Not Satisfiable. You can request upload status to get a more detailed list of missing ranges. Apparently the content-range and content-length were the problem. You changed the header configuration from the HttpClient to a HttpRequestMessage and it worked perfectly now.

MVC WebAPI return multiple images

I have 3 images in a directory but my code always returns one of them. I'd like to return 3 images image1.jpg, image2.jpg, image3.jpg and get them in my Xamarin app.
I think returning the result like an array might solve the problem but I don't understand what I need.
var result = new HttpResponseMessage(HttpStatusCode.OK);
MemoryStream ms = new MemoryStream();
for (int i = 0; i < 3; i++)
{
String filePath = HostingEnvironment.MapPath("~/Fotos/Empresas/Comer/" + id + (i + 1) + ".jpg");
FileStream fileStream = new FileStream(filePath, FileMode.OpenOrCreate);
Image image = Image.FromStream(fileStream);
image.Save(ms, ImageFormat.Jpeg);
fileStream.Close();
byte[] bytes = File.ReadAllBytes(filePath);
byte[] length = BitConverter.GetBytes(bytes.Length);
// Write length followed by file bytes to stream
ms.Write(length, 0, 3);
ms.Write(bytes, 0, bytes.Length);
}
result.Content = new StreamContent(ms);
return result;
Now i getting bytes, i edit a little bit the code now
byte[] imageAsBytes = client.GetByteArrayAsync(url).Result;
MemoryStream stream1 = new MemoryStream(imageAsBytes);
img.Source = ImageSource.FromStream(() => { return stream1; });
this is my xamarin code to get images, but i still getting nothing =/
If you just return a memorystream is not easy to differentiate one image from the other in the stream, instead of this, you can return a List of byte arrays, then you can access each position in the array and convert from byte array to image...
Here is a fully functional dotnet core webapi controller :
public class GetImagesController : Controller
{
private readonly IWebHostEnvironment _host;
public GetImagesController(IWebHostEnvironment host)
{
_host = host;
}
[HttpGet("{images}")]
public async Task<List<byte[]>> Get([FromQuery]string images)
{
List<byte[]> imageBytes = new List<byte[]>();
String[] strArray = images.Split(',');
for (int i = 0; i < strArray.Length; i++)
{
String filePath = Path.Combine(_host.ContentRootPath, "images", strArray[i]+".jpg");
byte[] bytes = System.IO.File.ReadAllBytes(filePath);
imageBytes.Add(bytes);
}
return imageBytes;
}
}
This controller can be called like this :
https://localhost:44386/getImages?images=P1,P2,P3
Given that you have a folder called images with files P1.jpg, P2.jpg and P3.jpg under your ContentRooPath.
https://learn.microsoft.com/en-us/aspnet/core/fundamentals/host/web-host?view=aspnetcore-3.0
You'll need something in the response to delimit where each image starts and finishes. As a basic solution, you could write the image length as an Int32 and follow it with the image data. On the other end, you'll need to read the 4-byte length followed by that x number of bytes:
[HttpGet]
public HttpResponseMessage Get(string id)
{
var result = new HttpResponseMessage(HttpStatusCode.OK);
String[] strArray = id.Split(',');
var ms = new MemoryStream();
for (int i = 0; i < strArray.Length; i++)
{
String filePath = HostingEnvironment.MapPath("~/Fotos/Empresas/Comer/" + strArray[i] + (i + 1) + ".jpg");
byte[] bytes = File.ReadAllBytes(filePath);
byte[] length = BitConverter.GetBytes(bytes.Length);
// Write length followed by file bytes to stream
ms.Write(length, 0, 4);
ms.Write(bytes, 0, bytes.Length);
}
result.Content = new StreamContent(ms);
return result;
}

Fixed read-only folders

I am trying to copy or move folders and files in my C# application, but the folders are Read-Only, and they cannot be disabled, since when I try, it's enabled again. I tried many solutions, but not worked... Yes, I am administrator with all rights. I tried disable Read-only in script too, but don't work.
const int CopyBufferSize = 64 * 1024;
public void CopyFile(string source, string destination)
{
//File.Copy(source, destination);
//Stopwatch swTotal = Stopwatch.StartNew();
using (var outputFile = File.Create(destination))
{
using (var inputFile = File.OpenRead(source))
{
// we need two buffers so we can ping-pong
var buffer1 = new byte[CopyBufferSize];
var buffer2 = new byte[CopyBufferSize];
var inputBuffer = buffer1;
int bytesRead;
IAsyncResult writeResult = null;
while ((bytesRead = inputFile.Read(inputBuffer, 0, CopyBufferSize)) != 0)
{
// Wait for pending write
if (writeResult != null)
{
writeResult.AsyncWaitHandle.WaitOne();
outputFile.EndWrite(writeResult);
writeResult = null;
}
// Assign the output buffer
var outputBuffer = inputBuffer;
// and swap input buffers
inputBuffer = (inputBuffer == buffer1) ? buffer2 : buffer1;
// begin asynchronous write
writeResult = outputFile.BeginWrite(outputBuffer, 0, bytesRead, null, null);
}
if (writeResult != null)
{
writeResult.AsyncWaitHandle.WaitOne();
outputFile.EndWrite(writeResult);
}
}
}
//swTotal.Stop();
//Console.WriteLine("Total time: {0:N4} seconds.", swTotal.Elapsed.TotalSeconds);
}
I tried with
File.Copy(source, destination);
too.
Thank you.
more details..
var fileName = "sourceFile.txt";
var source = Path.Combine(Environment.CurrentDirectory, fileName);
var destination = Path.Combine(destinationFolder, fileName);
File.Copy(source, destination);
OR
File.Copy(#"someDirectory\someFile.txt", #"otherDirectory\someFile.txt");

File got corrupted when I Read the file's all bytes on list type of object then write a file again Using c#

I've lot of tried to write file from collection of bytes. but file always get corrupted. not sure why its happening. If somebody knows about it would be helpful me more.
Note: Its always working good when I uncomment under while loop this line //AppendAllBytes(pathSource, bytes);
but I need bytes from object. later on I will use this concept on p2p.
namespace Sender
{
static class Program
{
static void Main(string[] args)
{
string pathSource = "../../Ok&SkipButtonForWelcomeToJakayaWindow.jpg";
using (FileStream fsSource = new FileStream(pathSource,
FileMode.Open, FileAccess.Read))
{
// Read the source file into a byte array.
const int numBytesToRead = 100000; // Your amount to read at a time
byte[] bytes = new byte[numBytesToRead];
int numBytesRead = 0;
if (File.Exists(pathSource))
{
Console.WriteLine("File of this name already exist, you want to continue?");
System.IO.FileInfo obj = new System.IO.FileInfo(pathSource);
pathSource = "../../Files/" + Guid.NewGuid() + obj.Extension;
}
int i = 0;
byte[] objBytes = new byte[numBytesRead];
List<FileInfo> objFileInfo = new List<FileInfo>();
Guid fileID = Guid.NewGuid();
FileInfo fileInfo = null;
while (numBytesToRead > 0)
{
// Read may return anything from 0 to numBytesToRead.
int n = fsSource.Read(bytes, numBytesRead, numBytesToRead);
i++;
//AppendAllBytes(pathSource, bytes);
fileInfo = new FileInfo { FileID = fileID, FileBytes = bytes, FileByteID = i };
objFileInfo.Add(fileInfo);
// Break when the end of the file is reached.
if (n == 0)
{
break;
}
// Do here what you want to do with the bytes read (convert to string using Encoding.YourEncoding.GetString())
}
//foreach (var b in objFileInfo.OrderBy(m => m.FileByteID))
//{
// AppendAllBytes(pathSource, b.FileBytes);
//}
foreach (var item in objFileInfo)
{
AppendAllBytes(pathSource, item.FileBytes);
}
fileInfo = null;
}
}
static void AppendAllBytes(string path, byte[] bytes)
{
using (var stream = new FileStream(path, FileMode.Append))
{
stream.Write(bytes, 0, bytes.Length);
}
}
}
class FileInfo
{
public Guid FileID { get; set; }
public int FileByteID { get; set; }
public byte[] FileBytes { get; set; }
}
}
You don't increase numBytesRead and don't decrease numBytesToRead.
objFileInfo contains a List of FileInfo which contains a reference type byte[].
You copy the reference to the bytes when you create a new FileInfo and then repeatedly overwrite those bytes until you reach the end of the file.
byte[] bytes = new byte[numBytesToRead];
//...
List<FileInfo> objFileInfo = new List<FileInfo>();
//...
//...
while (numBytesToRead > 0)
{
int n = fsSource.Read(bytes, numBytesRead, numBytesToRead);
//First time here bytes[0] == the first byte of the file
//Second time here bytes[0] == 10000th byte of file
//...
//The following line should copy the bytes into file info instead of the reference to the existing byte array
fileInfo = new FileInfo { ..., FileBytes = bytes, ... };
objFileInfo.Add(fileInfo);
//First time here objFileInfo[0].FileBytes[0] == first byte of file
//Second time here objFileInfo[0].FileBytes[0] == 10000th byte of file because objFileInfo[All].FileBytes == bytes
//...
}
You can test this by looking in the FileBytes variable for multiple FileInfo. I'd bet the contents look similar
There is two problem in your code :
The block of data is all of size 100000, which cannot work most of time unless the file size is exactly a multiple of it. So, the last block of data will contains 0s.
FileInfo.FileBytes will change, if you change the buffer to something new causing the every single block of data being the identical to the last block read.
using System;
using System.Collections.Generic;
using System.IO;
static class Program
{
static void Main(string[] args)
{
string pathSource = "test.jpg";
using (FileStream fsSource = new FileStream(pathSource, FileMode.Open, FileAccess.Read))
{
// Read the source file into a byte array.
const int BufferSize = 100000; // Your amount to read at a time
byte[] buffer = new byte[BufferSize];
if (File.Exists(pathSource))
{
Console.WriteLine("File of this name already exist, you want to continue?");
System.IO.FileInfo obj = new System.IO.FileInfo(pathSource);
pathSource = "Files/" + Guid.NewGuid() + obj.Extension;
}
int i = 0, offset = 0, bytesRead;
List<FileInfo> objFileInfo = new List<FileInfo>();
Guid fileID = Guid.NewGuid();
while (0 != (bytesRead = fsSource.Read(buffer, offset, BufferSize)))
{
var data = new byte[bytesRead];
Array.Copy(buffer, data, bytesRead);
objFileInfo.Add(new FileInfo { FileID = fileID, FileBytes = data, FileByteID = ++i });
}
foreach (var item in objFileInfo)
{
AppendAllBytes(pathSource, item.FileBytes);
}
}
}
static void AppendAllBytes(string path, byte[] bytes)
{
using (var stream = new FileStream(path, FileMode.Append))
{
stream.Write(bytes, 0, bytes.Length);
}
}
}
class FileInfo
{
public Guid FileID { get; set; }
public int FileByteID { get; set; }
public byte[] FileBytes { get; set; }
}

Making self-extracting executable with C#

I'm creating simple self-extracting archive using magic number to mark the beginning of the content.
For now it is a textfile:
MAGICNUMBER .... content of the text file
Next, textfile copied to the end of the executable:
copy programm.exe/b+textfile.txt/b sfx.exe
I'm trying to find the second occurrence of the magic number (the first one would be a hardcoded constant obviously) using the following code:
string my_filename = System.Diagnostics.Process.GetCurrentProcess().MainModule.FileName;
StreamReader file = new StreamReader(my_filename);
const int block_size = 1024;
const string magic = "MAGICNUMBER";
char[] buffer = new Char[block_size];
Int64 count = 0;
Int64 glob_pos = 0;
bool flag = false;
while (file.ReadBlock(buffer, 0, block_size) > 0)
{
var rel_pos = buffer.ToString().IndexOf(magic);
if ((rel_pos > -1) & (!flag))
{
flag = true;
continue;
}
if ((rel_pos > -1) & (flag == true))
{
glob_pos = block_size * count + rel_pos;
break;
}
count++;
}
using (FileStream fs = new FileStream(my_filename, FileMode.Open, FileAccess.Read))
{
byte[] b = new byte[fs.Length - glob_pos];
fs.Seek(glob_pos, SeekOrigin.Begin);
fs.Read(b, 0, (int)(fs.Length - glob_pos));
File.WriteAllBytes("c:/output.txt", b);
but for some reason I'm copying almost entire file, not the last few kilobytes. Is it because of the compiler optimization, inlining magic constant in while loop of something similar?
How should I do self-extraction archive properly?
Guessed I should read file backwards to avoid problems of compiler inlining magic constant multiply times.
So I've modified my code in the following way:
string my_filename = System.Diagnostics.Process.GetCurrentProcess().MainModule.FileName;
StreamReader file = new StreamReader(my_filename);
const int block_size = 1024;
const string magic = "MAGIC";
char[] buffer = new Char[block_size];
Int64 count = 0;
Int64 glob_pos = 0;
while (file.ReadBlock(buffer, 0, block_size) > 0)
{
var rel_pos = buffer.ToString().IndexOf(magic);
if (rel_pos > -1)
{
glob_pos = block_size * count + rel_pos;
}
count++;
}
using (FileStream fs = new FileStream(my_filename, FileMode.Open, FileAccess.Read))
{
byte[] b = new byte[fs.Length - glob_pos];
fs.Seek(glob_pos, SeekOrigin.Begin);
fs.Read(b, 0, (int)(fs.Length - glob_pos));
File.WriteAllBytes("c:/output.txt", b);
}
So I've scanned the all file once, found that I though would be the last occurrence of the magic number and copied from here to the end of it. While the file created by this procedure seems smaller than in previous attempt it in no way the same file I've attached to my "self-extracting" archive. Why?
My guess is that position calculation of the beginning of the attached file is wrong due to used conversion from binary to string. If so how should I modify my position calculation to make it correct?
Also how should I choose magic number then working with real files, pdfs for example? I wont be able to modify pdfs easily to include predefined magic number in it.
Try this out. Some C# Stream IO 101:
public static void Main()
{
String path = #"c:\here is your path";
// Method A: Read all information into a Byte Stream
Byte[] data = System.IO.File.ReadAllBytes(path);
String[] lines = System.IO.File.ReadAllLines(path);
// Method B: Use a stream to do essentially the same thing. (More powerful)
// Using block essentially means 'close when we're done'. See 'using block' or 'IDisposable'.
using (FileStream stream = File.OpenRead(path))
using (StreamReader reader = new StreamReader(stream))
{
// This will read all the data as a single string
String allData = reader.ReadToEnd();
}
String outputPath = #"C:\where I'm writing to";
// Copy from one file-stream to another
using (FileStream inputStream = File.OpenRead(path))
using (FileStream outputStream = File.Create(outputPath))
{
inputStream.CopyTo(outputStream);
// Again, this will close both streams when done.
}
// Copy to an in-memory stream
using (FileStream inputStream = File.OpenRead(path))
using (MemoryStream outputStream = new MemoryStream())
{
inputStream.CopyTo(outputStream);
// Again, this will close both streams when done.
// If you want to hold the data in memory, just don't wrap your
// memory stream in a using block.
}
// Use serialization to store data.
var serializer = new System.Runtime.Serialization.Formatters.Binary.BinaryFormatter();
// We'll serialize a person to the memory stream.
MemoryStream memoryStream = new MemoryStream();
serializer.Serialize(memoryStream, new Person() { Name = "Sam", Age = 20 });
// Now the person is stored in the memory stream (just as easy to write to disk using a
// file stream as well.
// Now lets reset the stream to the beginning:
memoryStream.Seek(0, SeekOrigin.Begin);
// And deserialize the person
Person deserializedPerson = (Person)serializer.Deserialize(memoryStream);
Console.WriteLine(deserializedPerson.Name); // Should print Sam
}
// Mark Serializable stuff as serializable.
// This means that C# will automatically format this to be put in a stream
[Serializable]
class Person
{
public String Name { get; set; }
public Int32 Age { get; set; }
}
The easiest solution is to replace
const string magic = "MAGICNUMBER";
with
static string magic = "magicnumber".ToUpper();
But there are more problems with the whole magic string approach. What is the file contains the magic string? I think that the best solution is to put the file size after the file. The extraction is much easier that way: Read the length from the last bytes and read the required amount of bytes from the end of the file.
Update: This should work unless your files are very big. (You'd need to use a revolving pair of buffers in that case (to read the file in small blocks)):
string inputFilename = System.Diagnostics.Process.GetCurrentProcess().MainModule.FileName;
string outputFilename = inputFilename + ".secret";
string magic = "magic".ToUpper();
byte[] data = File.ReadAllBytes(inputFilename);
byte[] magicData = Encoding.ASCII.GetBytes(magic);
for (int idx = magicData.Length - 1; idx < data.Length; idx++) {
bool found = true;
for (int magicIdx = 0; magicIdx < magicData.Length; magicIdx++) {
if (data[idx - magicData.Length + 1 + magicIdx] != magicData[magicIdx]) {
found = false;
break;
}
}
if (found) {
using (FileStream output = new FileStream(outputFilename, FileMode.Create)) {
output.Write(data, idx + 1, data.Length - idx - 1);
}
}
}
Update2: This should be much faster, use little memory and work on files of all size, but the program your must be proper executable (with size being a multiple of 512 bytes):
string inputFilename = System.Diagnostics.Process.GetCurrentProcess().MainModule.FileName;
string outputFilename = inputFilename + ".secret";
string marker = "magic".ToUpper();
byte[] data = File.ReadAllBytes(inputFilename);
byte[] markerData = Encoding.ASCII.GetBytes(marker);
int markerLength = markerData.Length;
const int blockSize = 512; //important!
using(FileStream input = File.OpenRead(inputFilename)) {
long lastPosition = 0;
byte[] buffer = new byte[blockSize];
while (input.Read(buffer, 0, blockSize) >= markerLength) {
bool found = true;
for (int idx = 0; idx < markerLength; idx++) {
if (buffer[idx] != markerData[idx]) {
found = false;
break;
}
}
if (found) {
input.Position = lastPosition + markerLength;
using (FileStream output = File.OpenWrite(outputFilename)) {
input.CopyTo(output);
}
}
lastPosition = input.Position;
}
}
Read about some approaches here: http://www.strchr.com/creating_self-extracting_executables
You can add the compressed file as resource to the project itself:
Project > Properties
Set the property of this resource to Binary.
You can then retrieve the resource with
byte[] resource = Properties.Resources.NameOfYourResource;
Search backwards rather than forwards (assuming your file won't contain said magic number).
Or append your (text) file and then lastly its length (or the length of the original exe), so you only need read the last DWORD / few bytes to see how long the file is - then no magic number is required.
More robustly, store the file as an additional data section within the executable file. This is more fiddly without external tools as it requires knowledge of the PE file format used for NT executables, q.v. http://msdn.microsoft.com/en-us/library/ms809762.aspx

Categories