I have a .NET client that needs to connect to a remote SQL Server over the WAN, is it possible to compress SQL traffic between the client and the server?
I am using .NET 3.5 and SQL Server 2005 and greater.
Looking at the connectionstrings.com here for SQL Server 2008, the database providers do not have some kind of compression scheme...You may need to write a wrapper on a different port, that compresses the data, by using the front end, send the data across that port, from there, compress it, send it across to the remote endpoint, decompress it, and forward it on to the real tcp/ip port where the server is sitting on.
Usually SQL Server sits on port 1433...
Since a picture is worth a thousand words....
+--------+ +--------+
| CLIENT | | SERVER |
+--------+ +--------+
Port 1234 Port 1433 <+--+
| |
| |
+={Module}= <=> TX/RX <=> ={Module}= -+-----------+
The module will sit there on both ends compressing/decompressing...
To be quite honest, it sounds like there will be work involved as the Firewall's holes would have to be punctured to allow the compressed data in and out...throw in NAT/SNAT could make things complicated...
Have a look at this article that I wrote on Codeproject, that code acts as a traffic redirector and could easily be modified to use the compression/decompression scheme..
As others have said there is no compression built in to the SQL Server TDS Protocol. It's also worth saying that by default there is no encryption either. To enable encryption you must use certificates and specify it in the connection strings.
The easiest solution to solve both issues is to open up a VPN tunnel with encryption and compression enabled. Simple Microsoft PPTP solves both issues and is easy to setup.
I don't think there is compression implemented in SQL server connection - if you are in need of compressing data, you should use web service and HTTP compression when communication with the service.
I know this question is over a year old but I found myself looking for this so I thought I would share what I found. There is this (quite expensive) software that compresses SQL server traffic. I am testing it at the moment for one of my clients, it works very well, achieving 60% compression ratios on average.
http://www.nitrosphere.net/store/nitroaccelerator
It is also compatible with clients that don't have this service installed.
if you want to create a tunnel with compression and encryption (can be disabled to save process) without having to create a vpn and also is cross plataform for your delight, here you have one that functions as a client server and using listening ports all life also functions as a firewall as a tunnel to have a single port as a channel to manage remote connections and ports), this tools exists 10 years ago: http://www.winton.org.uk/zebedee/
I'm doing experiments to compress (at level 3) a connection unencrypted SQL Server and I 'm getting good ratios tuneando the level of compression, let the intention that children spend large queries possible data for the limited channel...
updated in: https://sourceforge.net/projects/zebedee/
In this case I suggest to use web services or WCF to send the data instead of using connection to the database.
Check this out : http://www.toonel.net/tcpany.htm
Btw, I also think that SQL Server itself cannot compress trafic, but, with a network tier within application - you can do the compression there.
We are currently also testing the NitroSphere software over our WAN network, and we have a 73% compression rate, and a big speed improvement.
My opinion is that the software is actually cheap compared to SQL Server licensing, Riverbed devices and MPLS WAN connections. So for sure have a look if you have bandwidth troubles. It also support encryption but we do not plan on using this since everything will stay on internal MPLS network.
Related
My system has a server and multiple clients. the server has a service for the clients and each client has a service too, for talking to other clients.
I forward the server's service port manually on the router but the future client can not do it by him self after the installation.
Is there a way to automatically forward ports by code from the client side through the installation?
My main question is - Does this approach is wise? Should the system needs to be build deferentially?
Project details:
C# - WCF, Communication - NetTcpBinding.
The server is on my computer (Home network). Server's service port : 8080.
The clients can be installed everywhere. Client's service port: 8081.
*I'm not known with the IIS technology, can it help in this scenario?
The model you're describing sounds like a mesh network, generally you do not want clients to forward ports, be it automatically or not.
If it's absolutely necessary you could implement UPnP, there is an elaborate article here describing how to do so in .NET with a library. Note that you will have to select a different port.
I would strongly recommend to go for a different option though, having the server manage connections between clients is more managable and safer. There are very few valid arguments in favor of a model where a server is present and clients omit it at times:
Bandwidth, the server might not be able to handle all the data with reasonable throughput (i.e. torrent)
Security, the server might be only there for client updates (i.e. P2P chatclient with updater)
From the sound of it, your project does not apply to either.
EDIT: Because you have indicated the project is basically a torrent client, I would recommend reading up on the UPnP article.
I have an application in C# that is a TCP server listening to a port. GPS devices connect to this port. The application is accepting the TCP client and creating a new thread for each client. The client ID in maintained in a hash table that is updated when a client is connected. this was all working fine until around 400 units. Once the number of units increased, the server was unable to handle all connections. The connections are being continuously dropped and once in awhile leads eating up the server CPU and memory and brings it down. Work around was to open another instance of the TCP server listening to a different port and diverted some units to that port. Currently some 1800 units are somehow running in 8 different ports. The server is extremely unstable and units are still unable to stay connected. Facing too many issues on a daily basis. Also using remoting to send settings via the remoting port - this is working only sometimes.
Please help by giving a solution for TCP socket/threading/thread pooling etc. that is both scalable and robust and can in a single port.
This TCP server is running in Windows server 2008 R2 Enterprise with IIS7 and SQL server 2008.
Processor: Intel Xenon CPU E3-1270 V2 #3.50GHz
RAM: 32GB
System: 64-bit operating system
Thanks
Jonathan
Basically, don't use a thread per socket; use one of the async APIs (BeginReceive / ReceiveAsync), or some kind of socket polling (Socket.Select for example, although note that this is implemented in a very awkward way; when I use this, I actually use P/Invoke to get to the raw underlying API). Right at this moment, I have > 30k sockets per process talking to our web-sockets server (which is implemented via via Socket). Note that for OS reasons we do split that over a few different ports - mainly due to limitations of our load-balancer:
One thread per connection is not a really good idea specially when you have to handle 100s of client concurrently.Asynchronous is the way to go with some buffer pooling/managing. If you are looking for something to start with asynchronous sockets have a look at this basic implementation if you are looking for something complete Take a look at this(explanation: Here)
If you are willing check this out too.
In C# you can go with classical BeginXXX/EndXXX methods. Microsoft also have a High Performance Socket API which can be leveraged using XXXAsync methods. A few articles which explain the High Performance Socket API Here and Here
We have applications that need to send out email notifications from time to time. We have been using the .net mail classes but recently we are running into the problem of some virus checkers blocking port 25. To me the correct solution is to set up the correct exclusion rules to allow our app to keep working. However, some are afraid that this may become a huge hassle in the future, especially as new organizations come on line, and would prefer to use SQL dbmail instead. I don't particularly like this choice, I would much rather have the application handle this aspect but I'm not sure if it's worth the fight. Are there any real advantages of using .net mail over the SQL mail?
You can set up dedicated mail server, isolate it from network except for certain IPs and only for inbound connections on port 25. I think it is safe enough. Even if you move to SQL server, you will open port 25 and become vulnerable. You (your coulegues) can manage risks instead of hiding them.
You should ask yourself the following question
Do I really want to use the database server to send out email?
It all depends on how many emails you'll be sending and how busy the SQL server is.
If you'll not be sending many emails and you've got a pretty quiet SQL server; go right ahead.
If (like most of us) you're not that lucky, then look elsewhere.
They are both SMTP solutions using port 25.
You'll have the same issue most likely, it depends on where the scanner is checking/blocking
My thoughts would be to redesign this aspect of the system so that email requests are queued in a database table, and a single machine processes them.
Benefits:
you don't have to tie up resources on your SQL Server processing volumes of email
you don't have to go through the paperwork / exclusion sets for all client machines, just the one that's actually going to do the work.
Downsides:
more work for your developers!
I am about to develop a Network measurement tool. The objective is to make a tool, which can measure the responsetime in between a client and a server machine (from the client side). It is s side-application to a main application - If the main applicaiton experiences that the responsetime from the server is above a certain threshold, the tool will be kicked alive, and performs network connectivity tests, to determine of the client server connection is stable (it might be unstable, due to the network being wireless etc.)
The tests I need to perform are not just ping operations, but also transmitting packages of different size.
I have however very little experience in communications technology.
Is ICMP protocol the way to go? and if yes, is it possible to send packages of differnet sizes (to measure if the network is able to transfer eg. 2 MB of data in a reasonable time)?
I have a second concern. What should I look out for in regards to firewalls? It would be a shame to develop an application which works fine on my local network, but as soon as it is used out in the real life, it fails misserably because the tests are blocked by a firewall.
I hope my questions aren't too noobish, but know that any help is much appreciated.
All the best
/Sagi
To keep clear of firewalls, you should do a test using the same protocol and port you use, and create inside of your application a new type of message that should be responded as soon as it is read by the server: You should program your ping measures.
Then the client would measure the times spent in travel traveled and compute your ping and relay it back to your server. This also gives a better reading when in case of some ISPs that give a ICMP protocol packets a huge advantage over other packages on their QoS server, artificially creating(faking) lower latency. And also, you would not have to worry about the firewall not allowing your ICMP packets, because you would have to be allowed to conect on the standart port you use.
Also, most games work this way (Half-Life, Age Of Empires etc.) , and not by sending standard Ping packets.
I am using http remoting in C# to talk between two applications.
Earlier when I was running both applications on a single machine/two machines, it was very slow.
We have a proxy server to connect to the internet. We set "Bypass proxy server for local addresses" in IE and now when I run both applications on one machine the communication is happening quite fast. (by a factor of 10 nearly)
But if I run each application on a different machine then it is still slow, like it was the case before setting the "Bypass proxy server..."
Any insight will be helpful.
What sort of data are you sending, and at what rate? And are you using lots of calls to remote (MarshalByRefObject) objects?
In reality, it is hard to fully control the data with remoting. Personally, I would recommend something message-based, for example WCF. This makes the interfaces between systems much more explicit and predictable.
If the proxy is being a pinch-point, you can still bypass this for remote addresses via proxycfg.exe
There aren't a lot of remote calls, as feared.
A single remote call takes 10 secs on different machines while it takes only 350ms when on a single machine.
when I run proxycfg.exe, it shows "Direct Access (no proxy server)"