Why am I missing bytes when reading from a WebClient stream as follows?
const int chuckDim = 80;
System.Net.WebClient client = new System.Net.WebClient();
Stream stream = client.OpenRead("http://media-cdn.tripadvisor.com/media/photo-s/01/70/3e/a9/needed-backup-lol.jpg");
//Stream stream = client.OpenRead("file:///C:/Users/Tanganello/Downloads/needed-backup-lol.jpg");
//searching file length
WebHeaderCollection whc = client.ResponseHeaders;
int totalLength = (Int32.Parse(whc["Content-Length"]));
byte[] buffer = new byte[totalLength];
//reading and writing
FileStream filestream = new FileStream("C:\\Users\\Tanganello\\Downloads\\clone1.jpg", FileMode.Create, FileAccess.ReadWrite);
int accumulator = 0;
while (accumulator + chuckDim < totalLength) {
stream.Read(buffer, accumulator, chuckDim);
filestream.Write(buffer, accumulator, chuckDim);
accumulator += chuckDim;
}
stream.Read(buffer, accumulator, totalLength - accumulator);
filestream.Write(buffer, accumulator, totalLength - accumulator);
stream.Close();
filestream.Flush();
filestream.Close();
this is what I get with the first stream:
http://img839.imageshack.us/img839/830/clone1h.jpg
The problem is that you are ignoring the return value of the Stream.Read Method:
count
The maximum number of bytes to be read from the current stream.
Return Value
The total number of bytes read into the buffer. This can be less than the number of bytes requested
You can avoid the whole business of reading and writing streams by simply using the WebClient.DownloadFile Method:
using (var client = new WebClient())
{
client.DownloadFile(
"http://media-cdn.tripadvisor.com/media/photo-s/01/70/3e/a9/needed-backup-lol.jpg",
"C:\\Users\\Tanganello\\Downloads\\clone1.jpg");
}
Alternatively, if you really want to use streams, you can simply use the Stream.CopyTo Method:
using (var client = new WebClient())
using (var stream = client.OpenRead("http://..."))
using (var file = File.OpenWrite("C:\\..."))
{
stream.CopyTo(file);
}
If you insist on really copying the bytes yourself, the correct way to do this would be as follows:
using (var client = new WebClient())
using (var stream = client.OpenRead("http://..."))
using (var file = File.OpenWrite("C:\\..."))
{
var buffer = new byte[512];
int bytesReceived;
while ((bytesReceived = stream.Read(buffer, 0, buffer.Length)) != 0)
{
file.Write(buffer, 0, bytesReceived);
}
}
Related
request = MakeConnection(uri, WebRequestMethods.Ftp.DownloadFile, username, password);
response = (FtpWebResponse)request.GetResponse();
Stream responseStream = response.GetResponseStream();
//This part of the code is used to write the read content from the server
using (StreamReader responseReader = new StreamReader(responseStream))
{
using (var destinationStream = new FileStream(toFilenameToWrite, FileMode.Create))
{
byte[] fileContents = Encoding.UTF8.GetBytes(responseReader.ReadToEnd());
destinationStream.Write(fileContents, 0, fileContents.Length);
}
}
//This part of the code is used to write the read content from the server
using (var destinationStream = new FileStream(toFilenameToWrite, FileMode.Create))
{
long length = response.ContentLength;
int bufferSize = 2048;
int readCount;
byte[] buffer = new byte[2048];
readCount = responseStream.Read(buffer, 0, bufferSize);
while (readCount > 0)
{
destinationStream.Write(buffer, 0, readCount);
readCount = responseStream.Read(buffer, 0, bufferSize);
}
}
The former ones writes the content to the file but when I try to open the file it says it is corrupted. But the later one does the job perfectly when downloading zip files. Is there any specific reason why the former code doesn't work for zip files as it works perfectly for text files?
byte[] fileContents = Encoding.UTF8.GetBytes(responseReader.ReadToEnd());
You try to interpret a binary PDF file as an UTF-8 text. That just cannot work.
For a correct code, see Upload and download a binary file to/from FTP server in C#/.NET.
Use BinaryWriter and pass it FileStream.
//This part of the code is used to write the read content from the server
using (var destinationStream = new BinaryWriter(new FileStream(toFilenameToWrite, FileMode.Create)))
{
long length = response.ContentLength;
int bufferSize = 2048;
int readCount;
byte[] buffer = new byte[2048];
readCount = responseStream.Read(buffer, 0, bufferSize);
while (readCount > 0)
{
destinationStream.Write(buffer, 0, readCount);
readCount = responseStream.Read(buffer, 0, bufferSize);
}
}
here is my solution that worked for me
C#
public IActionResult GetZip([FromBody] List<DocumentAndSourceDto> documents)
{
List<Document> listOfDocuments = new List<Document>();
foreach (DocumentAndSourceDto doc in documents)
listOfDocuments.Add(_documentService.GetDocumentWithServerPath(doc.Id));
using (var ms = new MemoryStream())
{
using (var zipArchive = new ZipArchive(ms, ZipArchiveMode.Create, true))
{
foreach (var attachment in listOfDocuments)
{
var entry = zipArchive.CreateEntry(attachment.FileName);
using (var fileStream = new FileStream(attachment.FilePath, FileMode.Open))
using (var entryStream = entry.Open())
{
fileStream.CopyTo(entryStream);
}
}
}
ms.Position = 0;
return File(ms.ToArray(), "application/zip");
}
throw new ErrorException("Can't zip files");
}
don't miss the ms.Position = 0; here
Problem still there while i tried below three methods.
Using Window API "URLDownloadToFile"
WebClient Method
webclient.DownloadFile(url,dest) ''With/Without credientials
HTTP WebRequest Method:
public static void Download(String strURLFileandPath, String strFileSaveFileandPath)
{
HttpWebRequest wr = (HttpWebRequest)WebRequest.Create(strURLFileandPath);
HttpWebResponse ws = (HttpWebResponse)wr.GetResponse();
Stream str = ws.GetResponseStream();
byte[] inBuf = new byte[100000];
int bytesToRead = (int) inBuf.Length;
int bytesRead = 0;
while (bytesToRead > 0)
{
int n = str.Read(inBuf, bytesRead,bytesToRead);
if (n==0)
break;
bytesRead += n;
bytesToRead -= n;
}
FileStream fstr = new FileStream(strFileSaveFileandPath, FileMode.OpenOrCreate, FileAccess.Write);
fstr.Write(inBuf, 0, bytesRead);
str.Close();
fstr.Close();
}
Still i m facing the problem, file i am able to download at my local system, but when i open that it show Corrupt pdf.
!!!!I just want to download the pdf from URL and thats my query in VB.net/C# not using response method of ASP.net.
Please help if someone face this real problem.
Thanks in Advance!!!
Your code only writes 100000 bytes of the downloaded PDF and hence every PDF that is bigger than 100000 bytes gets corrupted.
To read more bytes you have to write the contents of every buffer to the FileStream.
The following should do it:
HttpWebRequest wr = (HttpWebRequest)WebRequest.Create(strURLFileandPath);
using (HttpWebResponse ws = (HttpWebResponse)wr.GetResponse())
using (Stream str = ws.GetResponseStream())
using (FileStream fstr = new FileStream(strFileSaveFileandPath, FileMode.OpenOrCreate, FileAccess.Write))
{
byte[] inBuf = new byte[100000];
int bytesRead = 0;
while ((bytesRead = str.Read(inBuf, 0, inBuf.Length)) > 0)
fstr.Write(inBuf, 0, bytesRead);
}
(It's good coding practice to use a using on every IDisposable instead of manually closing the streams.)
I am calling a WebAPI that returns a PDF in a ConnectStream. I write that ConnectStream to a file and then read it back in to a memorystream and I can attach it to an email. How can I accomplish the same thing without the IO overhead?
System.Net.HttpWebResponse webResponse = (System.Net.HttpWebResponse)webRequest.GetResponse();
System.IO.Stream stream = webResponse.GetResponseStream();
System.IO.StreamReader reader = new System.IO.StreamReader(stream, Encoding.Default);
string finalPath = System.IO.Path.Combine(outputPath, $"{startDate:yyyy-MM-dd}_{endDate:yyyy-MM-dd}.pdf");
System.IO.Stream fileStream = System.IO.File.OpenWrite(finalPath);
using (System.IO.StreamWriter sw = new System.IO.StreamWriter(fileStream, Encoding.Default))
{
sw.Write(reader.ReadToEnd());
sw.Flush();
sw.Close();
}
using (MemoryStream ms = new MemoryStream(File.ReadAllBytes(finalPath)))
{
using (MailMessage mailMessage = new MailMessage())
{
mailMessage.From = new MailAddress("noreply#somedomain.com");
mailMessage.To.Add("someone#somedomain.com");
mailMessage.Attachments.Add(new Attachment(ms, new System.Net.Mime.ContentType(System.Net.Mime.MediaTypeNames.Application.Pdf)));
SmtpClient smtpClient = new SmtpClient() { DeliveryMethod = System.Net.Mail.SmtpDeliveryMethod.SpecifiedPickupDirectory, PickupDirectoryLocation = outputPath };
smtpClient.Send(mailMessage);
}
}
First, the request stream is a one-way read only stream and cannot be passed to most methods that allow a stream, so you need to read it into something you can manipulate:
public byte[] ReadStreamBinary(Stream stream, int bufferSize)
{
using (var ms = new MemoryStream())
{
var buffer = CreateBuffer(bufferSize);
var finished = false;
while (!finished)
{
buffer.Initialize();
var bytes = stream.Read(buffer, 0, buffer.Length);
if (bytes > 0)
{
ms.Write(buffer, 0, bytes);
}
else
{
finished = true;
}
}
return ms.ToArray();
}
}
Then you can just create your MemoryStream from this array of bytes.
Since the default internal buffer for a stream is 4k, I almost always use that as my buffer size (4096). In your case, it may be easier to just modify the method to return the MemoryStream directly.
If you decide to return the stream, you'll need to remove the using (so the stream won't get closed/disposed) and return the stream pointer to the beginning.
public MemoryStream ReadStreamBinary(Stream stream, int bufferSize)
{
var ms = new MemoryStream();
var buffer = CreateBuffer(bufferSize);
var finished = false;
while (!finished)
{
buffer.Initialize();
var bytes = stream.Read(buffer, 0, buffer.Length);
if (bytes > 0)
{
ms.Write(buffer, 0, bytes);
}
else
{
finished = true;
}
}
ms.Seek(0, SeekOrigin.Begin);
return ms;
}
Just remember to close/dispose of the stream in the calling method.
Oops. almost forgot to include the CreateBuffer code, just put this anywhere in your class:
public static Func<int, byte[]> CreateBuffer = x => (byte[])Array.CreateInstance(typeof(byte), x);
I have a method for FTP download file, but I do not save file locally rather I parse the file in memory through ftp response. My question is, is returning stream reader after getting ftp response stream a good practice? Because do not want to do parsing and other stuff in the same method.
var uri = new Uri(string.Format("ftp://{0}/{1}/{2}", "somevalue", remotefolderpath, remotefilename));
var request = (FtpWebRequest)FtpWebRequest.Create(uri);
request.Credentials = new NetworkCredential(userName, password);
request.Method = WebRequestMethods.Ftp.DownloadFile;
var ftpResponse = (FtpWebResponse)request.GetResponse();
/* Get the FTP Server's Response Stream */
ftpStream = ftpResponse.GetResponseStream();
return responseStream = new StreamReader(ftpStream);
For me there are 2 disadvantages of using the stream directly, if you can live with them, you shouldn't waste memory or disk space.
In this stream you can not seek to a specific position, you can only read the contents as it comes in;
Your internet connection could suddenly drop and you will get an exception while parsing and processing your file, either split the parsing and processing or make sure your processing routine can handle the case that a file is processed for a second time (after a failure halfway through the first attempt).
To work around these issues, you could copy the stream to a MemoryStream:
using (var ftpStream = ftpResponse.GetResponseStream())
{
var memoryStream = new MemoryStream()
while ((bytesRead = ftpStream.Read(buffer, 0, buffer.Length)) > 0)
{
memoryStream.Write(buffer, 0, bytesRead);
}
memoryStream.Flush();
memoryStream.Position = 0;
return memoryStream;
}
If you are working with larger files I prefer writing it to a file, this way you minimize the memory footprint of your application:
using (var ftpStream = ftpResponse.GetResponseStream())
{
var fileStream = new FileStream(Path.GetTempFileName(), FileMode.CreateNew)
while ((bytesRead = ftpStream.Read(buffer, 0, buffer.Length)) > 0)
{
fileStream.Write(buffer, 0, bytesRead);
}
fileStream.Flush();
fileStream.Position = 0;
return fileStream;
}
I see more practical returning a responseStream when you are performing an HttpWebRequest. If you are using FtpWebRequest it means you are working with files. I would read the responseStream to byte[] and return the byte file content of the downloaded file, so you can easily work with the System.IO.Fileclasses to handle the file.
Thanks Carlos it was really helpful . I just return the byte[]
byte[] buffer = new byte[16 * 1024];
using (MemoryStream ms = new MemoryStream())
{
int read;
while ((read = ftpStream.Read(buffer, 0, buffer.Length)) > 0)
{
ms.Write(buffer, 0, read);
}
memoryStream=ms;
}
return memoryStream.ToArray();
and used byte[] in the method like this
public async Task ParseReport(byte[] bytesRead)
{
Stream stream = new MemoryStream(bytesRead);
using (StreamReader reader = new StreamReader(stream))
{
string line = null;
while (null != (line = reader.ReadLine()))
{
string[] values = line.Split(';');
}
}
stream.Close();
}
Right now I am using this code to download files (with a Range header). Most of the files are large, and it is running 99% of CPU currently as the file downloads. Is there any way that the file can be written periodically so that it does not remain in RAM constantly?
private byte[] GetWebPageContent(string url, long start, long finish)
{
byte[] result = new byte[finish];
HttpWebRequest request;
request = WebRequest.Create(url) as HttpWebRequest;
//request.Headers.Add("Range", "bytes=" + start + "-" + finish);
request.AddRange((int)start, (int)finish);
using (WebResponse response = request.GetResponse())
{
return ReadFully(response.GetResponseStream());
}
}
public static byte[] ReadFully(Stream stream)
{
byte[] buffer = new byte[32768];
using (MemoryStream ms = new MemoryStream())
{
while (true)
{
int read = stream.Read(buffer, 0, buffer.Length);
if (read <= 0)
return ms.ToArray();
ms.Write(buffer, 0, read);
}
}
}
Instead of writing the data to a MemoryStream (which stores the data in memory), write the data to a FileStream (which stores the data in a file on disk).
byte[] buffer = new byte[32768];
using (FileStream fileStream = File.Create(path))
{
while (true)
{
int read = stream.Read(buffer, 0, buffer.Length);
if (read <= 0)
break;
fileStream.Write(buffer, 0, read);
}
}
Using .NET 4.0:
using (FileStream fileStream = File.Create(path))
{
stream.CopyTo(fileStream);
}