We Keep Coding

Difference between loading a file from a path and from a stream C#

This is a follow up question to this question:
Difference between file path and file stream?
I didn't fully understand everything answered in the linked question.
I am using the Microsoft.SqlServer.Dac.BacPackage which contains a Load method with 2 overloads - one that receives a string path and one that receives a Stream.
This is the documentation of the Load method:
https://learn.microsoft.com/en-us/dotnet/api/microsoft.sqlserver.dac.bacpackage.load?view=sql-dacfx-150
What exactly is the difference between the two? Am I correct in assuming that the overloading of the string path saves all the file in the memory first, while the stream isn't? Are there other differences?

No, the file will not usually be fully loaded all at once.
A string path parameter normally means it will just open the file as a FileStream and pass it to the other version of the function. There is no reason why the stream should fully load the file into memory unless requested.
A Stream parameter means you open the file and pass the resulting Stream. You could also pass any other type of Stream, such as a network stream, a zip or decryption stream, a memory-backed stream, anything really.

Short answer:
The fact that you have two methods, one that accepts a filename and one that accepts a stream is just for convenience. Internally, the one with the filename will open the file as a stream and call the other method.
Longer answer
You can consider a stream as a sequence of bytes. The reason to use a stream instead of a byte[] or List<byte>, is, that if the sequence is really, really large, and you don't need to have access to all bytes at once, it would be a waste to put all bytes in memory before processing them.
For instance, if you want to calculate the checksum for all bytes in a file: you don't need to put all data in memory before you can start calculating the sum. In fact, anything that efficiently can deliver you the bytes one by one would suffice.
That is the reason why people would want to read a file as a stream.
The reason why people want a stream as input for their data, is that they want to give the caller the opportunity to specify the source of their data: callers can provide a stream that reads from a file, but also a stream with data from the internet, or from a database, or from a textBox, the procedure does not care, as long as it can read the bytes one by one or sometimes per chunk of bytes:
using (Stream fileStream = File.Open(fileName)
{
ProcessInputData(fileStream);
}
Or:
byte[] bytesToProcess = ...
using (Stream memoryStream = new MemoryStream(bytesToProcess))
{
ProcessInputData(memoryStream);
}
Or:
string operatorInput = this.textBox1.Text;
using (Stream memoryStream = new MemoryStream(operatorInput))
{
ProcessInputData(memoryStream);
}
Conclusioin
Methods use streams in their interface to indicate that they don't need all data in memory at once. One-by-one, or per chunk is enough. The caller is free to decide where the data comes from.

Change encoding of text file using Stream class object

I am using UploadCsv method of BigQuery Client class. This class accepts Stream class object. Can I change encoding of my file and pass this file to stream object without converting my text file to ByteArray? I'm using a large file and in case of ByteArray it gives out of memory exception.
UploadCsvAsync(string datasetId, string tableId, TableSchema schema, Stream input, UploadCsvOptions options = null, CancellationToken cancellationToken = null);

It seems that it is not possible to change the encoding of your input. This is because the UploadCSVAsync requires the input to be a System.IO.Stream class.
This class is the abstract base class of all streams in C#. A stream, on the other hand is a sequence of bytes, such as a file, input/output devices, an inter-process communication pipe or a TCP/IP socket. If you changed the type or encoding of the input, it might not match with the System.IO.Stream remarks.
This covers the main question "If it possible to change the encoding?", the answer is No.
Now, the next you may ask should be: "If it is not possible, then what do we need to do?", and that's a great question!
For that particular matter, the Out of memory Exception Error message can be solved by changing the way we are reading the data. Having a single loop statement to read a single file might not be the best option due to the allocated memory.
Instead, use a buffer to read data as they did on this other question or use smaller arrays or jagged arrays as done on this question.
Hope this is helpful! :)

How to I read any file in binary using C#? [duplicate]

This question already has answers here:
C# - How do I read and write a binary file?
(4 answers)
Closed 9 years ago.
The application I'm attempting to create would read the binary code of any file and create a file with the exact same binary code, creating a copy.
While writing a program that reads a file and writes it somewhere else, I was running into encoding issues, so I hypothesize that reading as straight binary will overcome this.
The file being read into the application is important, as after I get this to work I will add additional functionality to search within or manipulate the file's data as it is read.
Update:
I'd like to thank everyone that took the time to answer, I now have a working solution. Wolfwyrd's answer was exactly what I needed.

BinaryReader will handle reading the file into a byte buffer. BinaryWriter will handle dumping those bytes back out to another file. Your code will be something like:
using (var binReader = new System.IO.BinaryReader(System.IO.File.OpenRead("PATHIN")))
using (var binWriter = new System.IO.BinaryWriter(System.IO.File.OpenWrite("PATHOUT")))
{
byte[] buffer = new byte[512];
while (binReader.Read(buffer, 0, 512) != 0)
{
binWriter.Write(buffer);
}
}
Here we cycle a buffer of 512 bytes and immediately write it out to the other file. You would need to choose sensible sizes for your own buffer (there's nothing stopping you reading the entire file if it's reasonably sized). As you mentioned doing pattern matching you will need to consider the case where a pattern overlaps a buffered read if you do not load the whole file into a single byte array.
This SO Question has more details on best practices on reading large files.

Look at MemoryStream and BinaryReader/BinaryWriter:
http://www.dotnetperls.com/memorystream
http://msdn.microsoft.com/en-us/library/system.io.binaryreader.aspx
http://msdn.microsoft.com/en-us/library/system.io.binarywriter.aspx

Have a look at using BinaryReader Class
Reads primitive data types as binary values in a specific encoding.
and maybe BinaryReader.ReadBytes Method
Reads the specified number of bytes from the current stream into a
byte array and advances the current position by that number of bytes.
also BinaryWriter Class
Writes primitive types in binary to a stream and supports writing
strings in a specific encoding.
Another good example C# - Copying Binary Files

for instance, one char at a time.
using (BinaryReader writer = new BinaryWrite(File.OpenWrite("target"))
{
using (BinaryReader reader = new BinaryReader(File.OpenRead("source"))
{
var nextChar = reader.Read();
while (nextChar != -1)
{
writer.Write(Convert.ToChar(nextChar));
nextChar = reader.Read();
}
}
}

The application I'm attempting to create would read the binary code of any file and create a file with the exact same binary code, creating a copy.
Is this for academic purposes? Or do you actually just want to copy a file?
If the latter, you'll want to just use the System.IO.File.Copy method.

c# reading/writing bytes to/from console

I have a byte[] array and want to write it to stdout: Console.Out.Write(arr2str(arr)). How to convert byte[] to string, so that app.exe > arr.txt does the expected thing? I just want to save the array to a file using a pipe, but encodings mess things up.
I'd later want to read that byte array from stdin: app.exe < arr.txt and get the same thing.
How can I do these two things: write and read byte arrays to/from stdin/stdout?
EDIT:
I'm reading with string s = Console.In.ReadToEnd(), and then System.Text.Encoding.Default.GetBytes(s). I'm converting from array to string with System.Text.Encoding.Default.GetString(bytes), but this doesn't work when used with <,>. By "doesn't work" I mean that writing and reading over a pipe does not return the same thing.

To work with binary files you want Console.OpenStandardInput() to retrieve a Stream that you can read from. This has been covered in other threads here at SO, this one for example: Read binary data from Console.In

If you are writing to Console.WriteLine you need to encode the text in to a printable format. If you want to output to a file as a binary you can't use Console.WriteLine
If you still need to output to the console you either need to open the raw stream with Console.OpenStandardOutput() or call Convert.ToBase64String to turn the byte array to a string. There is also Convert.FromBase64String to come back from base64 to a byte array.

How to write data at a particular position in c#?

I am making an application in c#. In that application I have one byte array and I want to write that byte array data to particular position.
Here i used the following logic.
using(StreamWriter writer=new StreamWriter(#"D:\"+ FileName + ".txt",true))
{
writer.WriteLine(Encoding.ASCII.GetString(Data),IndexInFile,Data.Length);
}
But whenever i am writing data in file, it starts writing from starting.
My condition is that suppose at the initial I have empty file and I want to start writing in file from position 10000. Please help me .Thanks in advance.

Never try to write binary data as strings, like you do. It won't work correctly. Write binary data as binary data. You can use Stream for that instead of StreamWriter.
using (Stream stream = new FileStream(fileName, FileMode.OpenOrCreate))
{
stream.Seek(1000, SeekOrigin.Begin);
stream.Write(Data, 0, Data.Length);
}

You can set the position inside the stream like this:
writer.BaseStream.Seek( 1000, SeekOrigin.Begin);
Add this before your WriteLine code.
Note that I have not included any code to check that there is at least 1000 chars inside the file to start with.

The WriteLine overload you are using is this one:
TextWriter.WriteLine Method (Char[], Int32, Int32)
In particular the IndexInFile argument supplied is actually the index in the buffer from which to begin reading not the index in the file at which to begin writing - this explains why you are writing at the start of the file not at IndexInFile as you expect.
You should obtain access to the underlying Stream and Seek to the desired position in the file first and then write to the file.

Maybe a bit hacky, but you can access the BaseStream property of the StreamWriter and use Seek(long offset, SeekOrigin origin) on it. (But be warned, as not every stream can use Seek.)