Hello everybody.
I would like to know how to download and save a file to my hard drive, specifically a zip file from a HTTP server using the System.Net.Socket.Sockets class.
I know there are allot easier ways to download a file with .Net, but i would like to know how to do it with Sockets, if possible of course although I'm pretty sure it is.
I've tried a few things, nothing worked once i don't have any background experience with sockets.
Your help satisfying my curiosity is appreciated.
Any question just ask. Thank you.
Note:
The file is a standard zip file, however i would like a way that would work with any file type.
The file size is different every day.
The file is downloaded every minute, caching of such file must be disabled to get a accurate and update file version from the server.
File url sample: www.somewhere.com/files/feed/list.zip
You could do this directly with a .NET socket, but it would require parsing and understanding the HTTP request.
The standard way to do this would just be to use the higher level System.Net classes. For example, this can be done in two lines of code via WebClient.DownloadFile - why make life more difficult for yourself?
If you really must do this from raw sockets, it will just take a lot of work. At it's core, you can connect to port 80 (assuming http) via a TCP connection, write the correct strings to the socket, and start receiving data.
That being said, getting everything correct, and handling all of the issues required is far beyond a standard StackOverflow answer's scope. If you want to go down this road, take a look at the HTTP Protocol specifications - you'll need to implement the proper aspects of this specification.
For this you can simply use the "HttpWebRequest" and "HttpWebResponse" classes in .net.
Below is a sample console app I wrote to demonstrate how easy this is.
using System;
using System.Collections.Generic;
using System.Text;
using System.Net;
using System.IO;
namespace Test
{
class Program
{
static void Main(string[] args)
{
string url = "www.somewhere.com/files/feed/list.zip";
string fileName = #"C:\list.zip";
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
request.Timeout = 5000;
try
{
using (WebResponse response = (HttpWebResponse)request.GetResponse())
{
using (FileStream stream = new FileStream(fileName, FileMode.Create, FileAccess.Write))
{
byte[] bytes = ReadFully(response.GetResponseStream());
stream.Write(bytes, 0, bytes.Length);
}
}
}
catch (WebException)
{
Console.WriteLine("Error Occured");
}
}
public static byte[] ReadFully(Stream input)
{
byte[] buffer = new byte[16 * 1024];
using (MemoryStream ms = new MemoryStream())
{
int read;
while ((read = input.Read(buffer, 0, buffer.Length)) > 0)
{
ms.Write(buffer, 0, read);
}
return ms.ToArray();
}
}
}
}
Enjoy!
Related
I have a question regarding the CopyTo() method of the Stream class:
https://learn.microsoft.com/en-us/dotnet/api/system.io.stream.copyto
This approach works for small file circa 15kb as I tried it, but anything higher (I tested with 2mbs, 4 mbs and so on) and it just hangs on the CopyTo() method. Can't really figure out why.
Code sample:
Server's handle client :
public void HandleClient(object c)
{
string path = "some path";
using (TcpClient client = (TcpClient)c)
{
using (NetworkStream netStream = client.GetStream())
{
using (FileStream fileStream = new FileStream(path, FileMode.Create))
{
netStream.CopyTo(fileStream);
}
}
}
}
Client send :
public void Send()
{
IPEndPoint remoteEndPoint = new IPEndPoint(IPAddress.Parse("some address"), 12345);
string path = "some path";
using (TcpClient client = new TcpClient(remoteEndPoint))
{
using (NetworkStream netStream = client.GetStream())
{
using (FileStream fileStream = new FileStream(path, FileMode.Open))
{
fileStream.CopyTo(netStream);
}
}
}
}
P.s. As I research my way into Network Programming, I often find people advising other people to switch to WCF for this kind of tasks since, apparently, WCF makes everything a lot easier. What do you guys suggest and could you provide some links for a WCF noob that would be useful in modeling a LAN file sharing application since that's what my goal is?
I managed to solve the CopyTo() issue. The problem was that I was sending the file on the main thread so the whole application just chocked for larger files that took more than an instant to transfer.
Put the sending operation in a separate thread and tested sending up to 3 GB, works as it should. Now, is there any way I could track the progress of the CopyTo() operation? Seems to me that I can't and that I should do manual transfer if I want to track the progress.
Thanks to everyone involved :)
I'm not sure you can do a CopyTo with a file larger than the buffer size. Maybe, you can try to write it by splitting your file every buffer size.
I would like to load an excel file directly from an ftp site into a memory stream. Then I want to open the file in the FarPoint Spread control using the OpenExcel(Stream) method. My issue is I'm not sure if it's possible to download a file directly into memory. Anyone know if this is possible?
Yes, you can download a file from FTP to memory.
I think you can even pass the Stream from the FTP server to be processed by FarPoint.
WebRequest request = FtpWebRequest.Create("ftp://asd.com/file");
using (WebResponse response = request.GetResponse())
{
Stream responseStream = response.GetResponseStream();
OpenExcel(responseStream);
}
Using WebClient you can do nearly the same. Generally using WebClient is easier but gives you less configuration options and control (eg.: No timeout setting).
WebClient wc = new WebClient();
using (MemoryStream stream = new MemoryStream(wc.DownloadData("ftp://asd.com/file")))
{
OpenExcel(stream);
}
Take a look at WebClient.DownloadData. You should be able to download the file directory to memory and not write it to a file first.
This is untested, but something like:
var spreadSheetStream
= new MemoryStream(new WebClient().DownloadData(yourFilePath));
I'm not familiar with FarPoint though, to say whether or not the stream can be used directly with the OpenExcel method. Online examples show the method being used with a FileStream, but I'd assume any kind of Stream would be accepted.
Download file from URL to memory.
My answer does not exactly show, how to download a file for use in Excel, but shows how to create a generic-purpose in-memory byte array.
private static byte[] DownloadFile(string url)
{
byte[] result = null;
using (WebClient webClient = new WebClient())
{
result = webClient.DownloadData(url);
}
return result;
}
I had using the BinaryFormatter to Serialize an object through NetworkStream
The code like this
//OpenConnection ...
TCPClient client = server.AcceptTCPConnection();
Message message = new Message("bla bla"); // This is the serializable class
NetworkStream stream = client.GetStream(); // Get Stream
BinaryFormatter bf = new BinaryFormatter();
bf.Serialize(stream, message);
stream.Flush();
stream.Close(); //Close Connection
And in client Code, we just need to Read from stream
bf.Deserialize(stream) as Message
to get the object we just sent from Server.
But there is a problem here, if I delete the line stream.Close(); the client cannot read this Object. Or I can change to stream.Dispose();
However, I want to use this stream again to send another Message, how I can do? Please help, it make me feel so headache ##
UPDATE:
I found the reason of this issue. Because I used one machine to run both client and server. It definitely worked well in two different machines. Someone can tell me why? Get big problem with this for a couple day ago.
Sending multiple separate messages involves "framing" - splitting the single channel into separate chunks that don't ever require the client to "read to end". Oddly, though, I was under the impression that BinaryFormatter already implemented basic framing - but: I could be wrong. In the general case, when working with a binary protocol, the most common approach is to prefix each message with the length of the payload, i.e.
using(var ms = new MemoryStream()) {
while(...)
{
// not shown: serialize to ms
var len BitConverter.GetBytes((int)ms.Length);
output.Write(len, 0, 4);
output.Write(ms.GetBuffer(), 0, (int) ms.Length);
ms.SetLength(0); // ready for next cycle
}
}
the caller has to:
read exactly 4 bytes (at least, for the above), or detect EOF
determine the length
read exactly that many bytes
deserialize
repeat
If that sounds like a lot of work, maybe just use a serializer that does all this for you; for example, with protobuf-net, this would be:
while(...) { // each item
Serializer.SerializeWithLengthPrefix(output, PrefixStyle.Base128, 1);
}
and the reader would be:
foreach(var msg in Serializer.DeserializeItems<Message>(
input, PrefixStyle.Base128, 1))
{
// ...
}
(note: this does not use the same format / rules as BinaryFormatter)
I have a WCF Service, which uploads the document using Stream class.
Now after this, i want to get the Size of the document(Length of Stream), to update the fileAttribute for FileSize.
But doing this, the WCF throws an exception saying
Document Upload Exception: System.NotSupportedException: Specified method is not supported.
at System.ServiceModel.Dispatcher.StreamFormatter.MessageBodyStream.get_Length()
at eDMRMService.DocumentHandling.UploadDocument(UploadDocumentRequest request)
Can anyone help me in solving this.
Now after this, i want to get the Size of the document(Length of Stream), to update the fileAttribute for FileSize.
No, don't do that. If you are writing a file, then just write the file. At the simplest:
using(var file = File.Create(path)) {
source.CopyTo(file);
}
or before 4.0:
using(var file = File.Create(path)) {
byte[] buffer = new byte[8192];
int read;
while((read = source.Read(buffer, 0, buffer.Length)) > 0) {
file.Write(buffer, 0, read);
}
}
(which does not need to know the length in advance)
Note that some WCF options (full message security etc) require the entire message to be validated before processing, so can never truly stream, so: if the size is huge, I suggest you instead use an API where the client splits it and sends it in pieces (which you then reassemble at the server).
If the stream doesn't support seeking you cannot find its length using Stream.Length
The alternative is to copy the stream to a byte array and find its cumulative length. This involves processing the whole stream first , if you don't want this, you should add a stream length parameter to your WCF service interface
I have a C# application that communicates with a PHP-based SOAP web service for updates and licensing.
I am now working on a feedback system for users to submit errors and tracelogs automatically through the software. Based on a previous question I posted, I felt that a web service would be the best way to do it (most likely to work properly with least configuration).
My current thought is to use .NET built-in gzip compression to compress the text file, convert to base64, send to the web-service, and have the PHP script convert to binary and uncompress the data.
Can PHP decompress data compressed with GZipStream, and if so, how?
I actually tried this. GZipStream doesn't work. On the other hand, compressing with DeflateStream on .NET side and decompressing with gzinflate on PHP side do work. Your mileage may vary...
If the http-level libraries implements it (Both client and server), http has support for gzip-compression, in which case there would be no reason to manually compress anything. You should check if this is already happening before you venture any further.
Since the server is accepting web requests you really should be checking the HTTP headers to determine if any client accepts GZIP encoding rather than just guessing and gzipping each and every time.
If the PHP client can do gzip itll set the header and your code will then react according and do the right thing. Assuming or guessing is a poor choice when the facility is provided for your code to learn the capabilities of the client.
I wrote an article I recently posted that shows how to compress/decompress in C#. I used it for almost the same scenario. I wanted to transfer log files from the client to the server and they were often quite large. However in my case my webservice was running in .NET so I could use the decompress method. But looks like PHP does support a method called gzdecode that would work.
http://coding.infoconex.com/post/2009/05/Compress-and-Decompress-using-net-framework-and-built-in-GZipStream.aspx
Yes, PHP can decompress GZIP compressed strings, with or without headers.
gzdecode for GZIP file format (ie, compatible with gzip)
gzinflate for "raw" DEFLATE format
gzuncompress for ZLIB format (GZIP format without some header info)
I don't know for sure which one you'd want as I'm unfamiliar with .NET GZipStream. It sounds a little like gzuncompress, as the ZLIB format is kind of a "streaming" format, but try all three.
I was able to demo this with Gzip on C# and PHP.
Gzip Compressing in C#:
using System;
using System.IO;
using System.IO.Compression;
using System.Text;
public class Program {
public static void Main() {
string s = "Hi!!";
byte[] byteArray = Encoding.UTF8.GetBytes(s);
byte[] b2 = Compress(byteArray);
Console.WriteLine(System.Convert.ToBase64String(b2));
}
public static byte[] Compress(byte[] bytes) {
using (var memoryStream = new MemoryStream()) {
using (var gzipStream = new GZipStream(memoryStream, CompressionLevel.Optimal)) {
gzipStream.Write(bytes, 0, bytes.Length);
}
return memoryStream.ToArray();
}
}
public static byte[] Decompress(byte[] bytes) {
using (var memoryStream = new MemoryStream(bytes)) {
using (var outputStream = new MemoryStream()) {
using (var decompressStream = new GZipStream(memoryStream, CompressionMode.Decompress)) {
decompressStream.CopyTo(outputStream);
}
return outputStream.ToArray();
}
}
}
}
the code above prints the base64 encoded compressed string which is H4sIAAAAAAAEAPPIVFQEANxaFPgEAAAA for the Hi!! input.
Here's the code to decompress in PHP:
echo gzdecode(base64_decode('H4sIAAAAAAAEAPPIVFQEANxaFPgEAAAA'));