In C#, I'm using a FileStream to open files across a network that are being hosted on another Windows box, and I was curious what impact this would have on the computer hosting the file. Does the accessing computer simply grab it chunk by chunk from the HDD directly? Does the host computer put the file into memory? I guess this is sort of outside of the actual programming area as this may be something that is more at the OS level, but I figured I would ask here.
My main concerns are if the host computer actually has to open the files into memory to send them across, I may use up its memory if accessing a lot of files simultaneously.
Windows uses SMB protocol for remote file read/write. Reference information can be found at msdn.
It will not load everything on the host server's memory, transfer will be streamed. Even if you call File.ReadAllBytes() on the client, host server will stream it to the client through SMB protocol.
Memory utilization on host will depend on the number of clients connected at a time. More clients simultaneously transferring file would naturally mean more memory utilization on the host. Other than that, individual file transfer itself should have very little impact on additional memory allocation in the host.
Related
I am looking for a way to transfer a lot of files over a long period of time from a client to a server.
The connection between the client and the server is not reliable and slow.
I thought about using the FTP protocol. I saw the netftp client.
I now need a ftp server in .net also.
The most important feature that I need is reliable connection resuming. Something that I can rely on to just start and end over a period of time reliably.
I didn't find many ftp servers that were written in c#.
Thank you.
you can use Background Intelligent Transfer Service in windows.
http://msdn.microsoft.com/en-us/library/bb968799(v=vs.85).aspx
Use BITS for applications that need to:
Asynchronously transfer files in the foreground or background.
Preserve the responsiveness of other network applications.
Automatically resume file transfers after network disconnects and computer restarts.
a .net wrapper is available for BITS - http://sharpbits.codeplex.com/
I have a windows service which monitors a directory and whenever it detects a new file it will send that file to my web service for processing. I've now noticed that it's become a bit of a bottle neck sending the file using the web service request, so I was trying to work out what the alternatives are?
I've thought of having the Windows Service doing the processing directly (which I'd ideally like), but this isn't an option. Would it be better to be using WCF? In 90% of deployments the Web Service is on the same server as the Windows Service, but there is that 10% where it's on different servers. Just not sure what the best approach would be here...
EDIT: I'm sending the file as a byte[] to the Web Service, this is what I am wanting to somehow speed up. So the question I have is, would using another approach help speed this up, such as using WCF and a different protocol? I understand there is always an overhead, but trying to minimize this.
WCF & Bindings: Changing to WCF offers several bindings you can use that are more efficient to transmit data in a LAN, e.g. NetTcpBinding or Named Pipes (local only). So using WCF is a good step if you don't want to introduce bigger changes into your application.
Hybrid approach:
However, the best way to speed up at least for the 90% of deployments that host both components on the same machine is to remove the process boundary in these cases. As you mention, you've already thought about that. So just in case that the reason for putting the idea aside are the 10% of deployments that would require a distributed installation: By using an interface with an implementation for local procession and one for remote transmission, you could implement a configurable approach that supports both scenarios (90% very efficient, 10% at least not slower as before).
Scaling down data sizes:
Another - obvious - way to speed things up is to filter or compress the file contents before transmitting them to the service.
File path instead of contents:
As I understand your environment, the machines that host the services are at least close to each other (LAN, no firewall issues, ...). So it might also be a viable option not to transmit the file contents to the service, but to notify the service of the file path and have the web service access the file directly. Though not a very beautiful way with certain downsides (e.g. account of web service must be able to access the file, path must also be accessible from the web service) you'd at least get rid of the inefficient transmission of the files and substitute that with a protocol that is built for file access. Also in the 90% of installations where both services run on the same machine, the web service would do a local file access that should be very fast.
I have a C# service application which receives files via UDP and saves them into a local drive. The application runs fine on local OS, but when it's running from a VM I get these exceptions from the Socket lib and the IO lib.
System.Net.Sockets.SocketException (0x80004005): An operation on a socket could not be performed because the system lacked sufficient buffer space or because a queue was full
System.IO.IOException: Insufficient quota to complete the requested service.
I've tested the disk space, but It's 80% free. Also the Ram on the VM is set to 2GB.
I've google but couldn't find much on this topic.
# David Stratton:
Info About the VM system:
Using ESXi 4.1
Host OS windows 2003
# Seth Noble
Your right. The first exception happens when it's sending a packet to the source.
And the Second IO exception when it receives the packets to write to disk.
If this is only happening in thw VM it is most likely a issue with the implementation of some of the features of the actual VM software and it is safe to assume that it is not due to a programming mistake on your part.
When you have a network path open in Windows explorer, and you drag it to a local folder does it open a socket? Also, when you use c# FileStream fin = new FileStream(#"//networkpath/file); does that open a socked? my question is this, would it be just as fast to stream a file over a socket manually as it would be to read it over a network using c#'s filestream?
The Windows file service works over TCP/IP by default (although not necessarily), so typically, there's a socket involved. Yes, there's some overhead from the SMB protocol that Windows uses. However, for files where transfer time matters, the overhead is small compared to the data.
In addition, coming up with your own file sharing protocol without a very good reason is a bad idea. It's a lot of development and debugging work, you have to install the server part somehow, you have to think of security implications (user authentication, etc), firewalls will break it... Just not worth it.
To gauge the amount of work involved, read the description of the FTP protocol.
I want to set up a system with a Client-Server architecture.
Problem: Sending daily log data from the client application to server. Both the client and server are on an internal network with different IP addresses.
Possible Methods...
1) Using Socket Programming
2) Sending over Active Directory (if possible)
3) Sending file from one IP address to another.
Please suggest some methods (apart from http) and how to implement the same?
NOTE: Information on client side is stored in SQLite database.
Thanks in advance.
How about something as simple as a shared network drive, and a regular file copy? If the servers are on the same network?
Yes, it's not sexy, but it does the job pretty easily.
DNS is not required to use HTTP or any other network protocol. You can use the computers IP Address in place of the domain name entry. As Jakob Jenkov said you could just use standard File.Copy to copy the files from the client to a central server. If you want to use some sort of custom network connection for such a transfer you should look at WCF. WCF supports many different network protocols and is very reusable.
More difficult options would using Sockets, TcpListner/TcpClient, and Remoting.
It should probably be noted that Active Directory is a directory listing service, not a network transfer protocol.
Since you consider "sending a file from one ip address to another" an option, why not setup a network share (\serverShare\mylogshare) and write a simple program that runs on the client that will copy the client log data file to the network share. Run the program as a scheduled task in whatever time frame you desire.