如何提高FtpWebRequest的性能?

I have an application written in .NET 3.5 that uses FTP to upload/download files from a server. The app works fine but there are performance issues:

我有一个用。net 3.5编写的应用程序，它使用FTP从服务器上传/下载文件。该应用程序运行良好，但存在性能问题:

1. It takes a lot of time to make connection to the FTP server. The FTP server is on a different network and has Windows 2003 Server (IIS FTP). When multiple files are queued for upload, the change from one file to another creates a new connection using FTPWebRequest and it takes a lot of time (around 8-10 seconds).

   连接到FTP服务器需要很多时间。FTP服务器位于另一个网络上，并具有Windows 2003服务器(IIS FTP)。当多个文件排队上载时，从一个文件到另一个文件的更改将使用FTPWebRequest创建一个新的连接，这将花费大量时间(大约8-10秒)。

2. Is is possible to re-use the connection? I am not very sure about the KeepAlive property. Which connections are kept alive and reused.

   是否可以重新使用连接?我不太确定是否有保存财产。哪些连接保持活动并被重用。

3. The IIS-FTP on Windows Server 2003 does not support SSL so anyone can easily see the username/password through a packet sniffer such as WireShark. I found that windows Server 2008 supports SSL over FTP in its new version if IIS 7.0.

   Windows Server 2003上的is - ftp不支持SSL，因此任何人都可以通过包嗅探器(如WireShark)轻松查看用户名/密码。我发现windows Server 2008在其新版本iis7.0中支持SSL / FTP。

I basically want to improve the upload/download performance of my application. Any ideas will be appreciated.

我主要想改进我的应用程序的上传/下载性能。任何想法都会受到欢迎。

** Please note that 3 is not an issue but I would like people to have comments on it

**请注意，3不是问题，但我希望人们对它有意见

18 个解决方案

#1

I have done some experimentation (uploading about 20 files on various sizes) on FtpWebRequest with the following factors

我在FtpWebRequest上做了一些实验(上传了20个不同大小的文件)，有以下几个因素

KeepAlive = true/false

KeepAlive = true / false

ftpRequest.KeepAlive = isKeepAlive;

Connnection Group Name = UserDefined or null

Connnection Group Name = UserDefined或null

ftpRequest.ConnectionGroupName = "MyGroupName";

Connection Limit = 2 (default) or 4 or 8

连接限制= 2(默认)或4或8

ftpRequest.ServicePoint.ConnectionLimit = ConnectionLimit;

Mode = Synchronous or Async

模式=同步或异步

see this example

看这个例子

My results:

我的结果:

1. Use ConnectionGroupName,KeepAlive=true took (21188.62 msec)

   使用ConnectionGroupName,KeepAlive=true take (21188.62 msec)

2. Use ConnectionGroupName,KeepAlive=false took (53449.00 msec)

   使用ConnectionGroupName,KeepAlive=false take (53449.00 msec)

3. No ConnectionGroupName,KeepAlive=false took (40335.17 msec)

   没有ConnectionGroupName,KeepAlive=false (40335.17 msec)

4. Use ConnectionGroupName,KeepAlive=true;async=true,connections=2 took (11576.84 msec)

   使用ConnectionGroupName,KeepAlive=true;async=true,connections=2取得(11576.84 msec)

5. Use ConnectionGroupName,KeepAlive=true;async=true,connections=4 took (10572.56 msec)

   使用ConnectionGroupName,KeepAlive=true;async=true，连接=4 (10572.56 msec)

6. Use ConnectionGroupName,KeepAlive=true;async=true,connections=8 took (10598.76 msec)

   使用ConnectionGroupName,KeepAlive=true;async=true，连接=8 (10598.76 msec)

Conclusions

结论

1. FtpWebRequest has been designed to support an internal connection pool. To ensure, the connection pool is used, we must make sure the ConnectionGroupName is being set.

   FtpWebRequest是用来支持内部连接池的。为了确保使用连接池，我们必须确保设置了ConnectionGroupName。

2. Setting up a connection is expensive. If we are connecting to the same ftp server using the same credentials, having the keep alive flag set to true will minimize the number of connections setup.

   建立连接是昂贵的。如果我们使用相同的凭据连接到同一个ftp服务器，将keep alive标志设置为true将最小化连接设置的数量。

3. Asynchronous is the recommended way if you have a lot of files to ftp.

   如果有很多文件要ftp，建议使用异步方式。

4. The default number of connections is 2. In my environment, a connection limit of 4 will give to me the most overall performance gain. Increasing the number of connections may or may not improve performance. I would recommend that you have the connection limit as a configuration parameter so that you can tune this parameter in your environment.

   默认连接数是2。在我的环境中，4的连接限制将使我获得最大的总体性能收益。增加连接的数量可能会提高性能，也可能不会。我建议您将连接限制作为配置参数，以便在您的环境中对该参数进行调优。

Hope you would find this useful.

希望你能发现这个有用。

#2

It doesn't matter if the individual connections take long to connect as long as you can launch many in parallel. If you have many items to transfer (say hundreds) then it makes sense to launch tens and even hundreds of WebRequests in parallel, using the asynchronous methods like BeginGetRequestStream and BeginGetResponse. I worked on projects that faced similar problems (long connect/authenticate times) but by issuing many calls in parallel the overall throughput was actually very good.

如果单个连接要花很长时间才能连接，这并不重要，只要可以同时启动多个连接。如果您有许多项要传输(比如数百个)，那么使用诸如BeginGetRequestStream和BeginGetResponse之类的异步方法来并行地启动数十个甚至数百个webrequest是很有意义的。我曾在一些项目中遇到过类似的问题(长时间的连接/验证时间)，但通过同时发出许多调用，总体吞吐量实际上非常好。

Also it makes a huge difference if you use the async methods or the synchronous one, as soon as you have many (tens, hundreds) of requests. This applies not only to your WebRequests methods, but also to your Stream read/write methods you'll use after obtaining the upload/download stream. The Improving .Net Performance and Scalability book is a bit outdated, but much of its advice still stands, and is free to read online.

如果您使用异步方法或同步方法，只要您有许多(数十个、数百个)请求，它就会产生巨大的差异。这不仅适用于WebRequests方法，还适用于获取上传/下载流后使用的流读/写方法。提高。net性能和可扩展性的书籍有点过时了，但是它的许多建议仍然有效，可以在网上免费阅读。

One thing to consider is that the ServicePointManager class sits there lurking in the Framwework with one sole purpose: to ruin your performance. Make sure you obtain the ServicePoint of your URL and change the ConnectionLimit to a reasonable value (at least as high as how many concurrent requests you intend).

需要考虑的一件事是ServicePointManager类位于Framwework中，其唯一目的是:破坏您的性能。确保您获得了URL的ServicePoint，并将ConnectionLimit更改为一个合理的值(至少与您想要的并发请求数量相同)。

#3

Debug Network

调试网络

A few tricks for simple network debugging:

一些简单的网络调试技巧:

1. Check the response times when you ping the FTP server from the application server.
2. 当您从应用服务器上ping FTP服务器时，检查响应时间。
3. Check the response times for a trace route (tracert from a DOS shell).
4. 检查跟踪路径的响应时间(来自DOS shell的tracert)。
5. Transfer a file from the command-line using the ftp command.
6. 使用ftp命令从命令行传输文件。
7. Connect to the FTP server via Telnet: telnet server 21.
8. 通过Telnet: Telnet server 21连接到FTP服务器。

The results will provide clues to solving the problem.

这些结果将为解决这个问题提供线索。

Network Hardware

网络硬件

For a slow trace route:

对于缓慢的跟踪路径:

• Determine why the two computers are having network issues.
• 确定为什么这两台计算机有网络问题。
• Upgrade the network hardware between the slowest link.
• 在最慢的链路之间升级网络硬件。

Network Configuration

网络配置

For a slow ping:

平的缓慢:

• Check the network configuration on each machine.
• 检查每台机器上的网络配置。
• Ensure the settings are optimal.
• 确保设置是最优的。

Validate API

验证API

A slow command-line FTP session will tell you that the problem is not isolated to the FTP API you are using. It does not eliminate the API as a potential problem, but certainly makes it less likely.

一个缓慢的命令行FTP会话将告诉您，这个问题并不孤立于您正在使用的FTP API。它并没有将API作为一个潜在的问题加以消除，但肯定会降低这种可能性。

Network Errors

网络错误

If packets are being dropped between the source and destination, ping will tell you. You might have to increase the packet size to 1500 bytes to see any errors.

如果信息包在源和目标之间被丢弃，ping将告诉您。您可能需要将包大小增加到1500字节才能看到任何错误。

FTP Queue Server

FTP服务器队列

If you have no control over the destination FTP server, have an intermediary server receive uploaded files. The intermediary then sends the files to the remote server at whatever speed it can. This gives the illusion that the files are being sent quickly. However, if the files must exist on the remote server as soon as they are uploaded, then this solution might not be viable.

如果您无法控制目标FTP服务器，请让一个中间服务器接收上传的文件。然后，中介以任何可能的速度将文件发送到远程服务器。这给人一种文件被快速发送的错觉。但是，如果文件必须在上传之后就存在远程服务器上，那么这个解决方案可能是不可行的。

FTP Server Software

FTP服务器软件

Use a different FTP daemon on the FTP server, such as ProFTPd as a Windows service. (ProFTPd has plug-ins for various databases that allow authentication using SQL queries.)

在FTP服务器上使用不同的FTP守护程序，比如ProFTPd作为Windows服务。(ProFTPd为各种数据库提供插件，允许使用SQL查询进行身份验证。)

FTP Server Operating System

FTP服务器操作系统

A Unix-based operating system might be a better option than a Microsoft-based one.

基于unix的操作系统可能比基于microsoft的更好。

FTP Client Software

FTP客户端软件

There are a number of different APIs for sending and receiving files via FTP. It might take some work to make your application modular enough that you can simply plug in a new file transfer service. A few different APIs are listed as answers here.

通过FTP发送和接收文件有许多不同的api。要使应用程序模块化到只需插入一个新的文件传输服务，可能需要做一些工作。这里列出了一些不同的api作为答案。

Alternate Protocol

替代协议

If FTP is not an absolute requirement, try:

如果FTP不是绝对要求，请尝试:

• a Windows network drive
• 一个Windows网络驱动器
• HTTPS
• HTTPS
• scp, rsync, or similar programs (Cygwin might be required)
• scp、rsync或类似的程序(可能需要Cygwin)

#4

This link describes ConnectionGroupName and KeepAlive affects: WebRequest ConnectionGroupName

这个链接描述ConnectionGroupName和KeepAlive影响:WebRequest ConnectionGroupName。

#5

You should definitely check out BITS which is a big improvement over FTP. The clear-text passwords aren't the only weakness in FTP. There's also the issue of predicting the port it will open for a passive upload or download and just overall difficulty when clients are using NAT or firewalls.

你一定要检查位，这是一个很大的改进比FTP。纯文本密码并不是FTP的唯一缺点。还有一个问题是，当客户端使用NAT或防火墙时，它会为被动的上传或下载打开一个端口，并且会有整体的困难。

BITS works over HTTP/HTTPS using IIS extensions and supports queued uploads and downloads that can be scheduled at low priority. It's overall just a lot more flexible than FTP if you are using Windows on the client and server.

比特使用IIS扩展来处理HTTP/HTTPS，支持排队上传和下载，这些都可以排在低优先级。如果你在客户端和服务器上使用Windows，它比FTP灵活得多。

BITS for PowerShell

PowerShell的位

BITS for .NET

位为。net

#6

Personally I have migrated all of our apps away from using FTP for file upload/download, and instead rolled a solution based on XML Web Services in ASP.NET.

就我个人而言，我已经将所有的应用程序从使用FTP迁移到文件上传/下载，转而使用ASP.NET中的XML Web服务来进行解决。

Performance is much improved, security is as much or as little as you want to code (and you can use the stuff built in to .NET) and it can all go over SSL with no issues.

性能得到了很大的改善，安全性和您想要的代码一样多或一样少(您可以使用内建到。net中的东西)，并且它可以在SSL上运行，没有问题。

Our success rate getting our clients' connections out through their own firewalls is FAR better than running FTP.

我们通过客户的防火墙建立连接的成功率比运行FTP要好得多。

#7

Look at this page - http://www.ietf.org/rfc/rfc959.txt

看看这个页面——http://www.ietf.org/rfc/rfc959.txt

It says of using different port when connecting to be able to reuse the connection.
Does that work?

它表示在连接时使用不同的端口，以便能够重用连接。这工作吗?

#8

I strongly suggest Starksoft FTP/FTPS Component for .NET and Mono. It has a connection object that you can cache and reuse.

我强烈建议使用。net和Mono的Starksoft FTP/FTPS组件。它有一个可以缓存和重用的连接对象。

#9

I'd recommend switching to rsync.
Pros :
Optimised for reducing transfer time.
Supports SSH for secure transfer
Uses TCP so makes your IT dept/firewall guys happier

我建议切换到rsync。优点:优化减少传输时间。支持SSH的安全传输使用TCP，使您的IT部门/防火墙人员更高兴

Cons:
No native .NET support
Geared towards linux server installations - though there are decent windows ports like DeltaCopy

缺点:没有针对linux服务器安装的本机。net支持——尽管有不错的windows端口，比如DeltaCopy

Overall though it's a much better choice than FTP

总的来说，它比FTP更好

#10

I have had good results with EDT's ftp library:

我对EDT的ftp库有很好的结果:

http://www.enterprisedt.com/products/edtftpnet/overview.html

http://www.enterprisedt.com/products/edtftpnet/overview.html

#11

To resolve the problem about performance you simply need to set:

要解决您只需要设置的性能问题:

ftpRequest.ConnectionGroupName = "MyGroupName"; ftpRequest.KeepAlive = false; ftpRequest.ServicePoint.CloseConnectionGroup("MyGroupName");

ftpRequest。ConnectionGroupName = " MyGroupName”;ftpRequest。KeepAlive = false;ftpRequest.ServicePoint.CloseConnectionGroup(“MyGroupName”);

#12

I know it's an old thread, but I recently had to go through a similar situation.

我知道这是一条古老的线索，但最近我不得不经历类似的情况。

We needed to download 70+ XML files from an ftp server in under 25 minutes without opening more than 5 connections during that time-frame.

我们需要在25分钟内从ftp服务器上下载70+ XML文件，在此期间不打开超过5个连接。

These were all the alternatives we tried:

这些都是我们尝试过的方法:

• wget - It was fast, but every GET meant a new connection. We had to stop due to the amount of connections created. We were having some issues with GETM that's well-documented in the web so that wasn't an option.
• wget——速度很快，但每次GET都意味着新的连接。由于创建的连接数量太多，我们不得不停止。我们在web上有很多关于GETM的文档，所以这不是一个选项。
• FtpWebRequest - Every Create call would log a new connection, even though we used the KeepAlive and ConnectionGroupName properties. Plus it was very slow.
• FtpWebRequest——尽管我们使用了KeepAlive和ConnectionGroupName属性，但每个Create调用都会记录一个新的连接。而且速度很慢。
• Webclient - We didn't check if it created a new connection for every file (although I assume it does), but it copied 1 file/minute. So it wasn't an option.
• Webclient——我们没有检查它是否为每个文件创建了一个新的连接(尽管我假设它为)，但是它复制了一个文件/分钟。所以这不是一个选择。

We ended up using old-fashioned ftp batch script. It's fast and I only use one connection to download all the files. It isn't flexible, but it's much faster than everything else I've tried (75 files in under 20 minutes).

我们最终使用了老式的ftp批处理脚本。它很快，我只用一个连接下载所有的文件。它不是灵活的，但是它比我尝试过的任何东西都快得多(在20分钟之内有75个文件)。

#13

KeepAlive is working. FtpWebRequest caches connections inside, so they can be reused after some time. For details and explanation of this mechanism you can look to ServicePoint.

KeepAlive是有效的。FtpWebRequest缓存内部的连接，所以一段时间后可以重用它们。有关该机制的详细信息和说明，您可以查看ServicePoint。

Another good source of information is to look into FtpWebRequest source (you can do it on VS2008).

另一个好的信息来源是查看FtpWebRequest源(您可以在VS2008上完成)。

#14

AFAIK, each FtpWebRequest has to set up a new connection - including logon to the server. If you want to speed up the FTP transfers, I would recommend that you use an alternate FTP client instead. Some of these alternate clients can login and then perform multiple actions using the same command connection.

每个FtpWebRequest都必须建立一个新的连接——包括登录到服务器。如果您想加快FTP传输速度，我建议您使用另一个FTP客户端。这些备用客户机中的一些可以登录，然后使用相同的命令连接执行多个操作。

Examples of such clients incldue: http://www.codeproject.com/KB/IP/FtpClient.aspx which also includes a good explanation as to why these libraries can operate faster than the standard FtpWebRequest and http://www.codeproject.com/KB/macros/ftp_class_library.aspx which looks like a simple enough implementation also.

此类客户机的示例包括:http://www.codeproject.com/KB/IP/FtpClient.aspx，其中还包含了一个很好的解释，说明为什么这些库可以比标准的FtpWebRequest和http://www.codeproject.com/KB/macros/ftp_class_library.aspx运行得更快，这看起来也是一个很简单的实现。

Personally, I rolled my own implementation of FTP back in the .NET 1.1 days before the FtpWebRequest was introduced and this still works well for .NET 2.0 onwards.

就我个人而言，在FtpWebRequest发布之前的1.1天，我就在。net中推出了自己的FTP实现。

#15

Single Point of advice:

单点建议:

LOWER BUFFER/CHUNK-SIZES SIGNIFICANTLY REDUCE PERFORMANCE

较低的缓冲区/块大小会显著降低性能

Reason: Many more disk i/o, memory i/o, ftp stream init and many many more factors

原因:更多的磁盘i/o、内存i/o、ftp流init和更多的因素

#16

I did some benchmarks on FtpWebRequest, similar to @Sid 's response above. Setting KeepAlive to true does improve, but not the asynchronous calls in my test. The test consists of

我在FtpWebRequest上做了一些基准测试，类似于上面@Sid的响应。将KeepAlive设置为true确实有所改进，但我的测试中没有异步调用。测试包括

1) FtpWebRequest for check of file existence 2) FtpWebRequest for upload 3) FtpWebRequest for rename file on server

1) FtpWebRequest检查文件是否存在2)FtpWebRequest for upload 3) FtpWebRequest for rename file on server

Test FTP client 30 files of size 5 Kbytes took ... 14.897 seconds Test upload (alive true, connection name) 30 files of size 5 Kbytes took ... 34.229 seconds Test async(alive true, connection name) 30 files of size 5 Kbytes took ... 40.659 seconds Test send thread (alive true, connection name) 30 files of size 5 Kbytes took ... 38.926 seconds, 30 files

测试FTP客户端30个大小为5 Kbytes的文件…14.897秒测试上传(活动真实，连接名)30个文件大小为5 Kbytes。34.229秒异步测试(live true, connection name) 30个大小为5 Kbytes的文件…40.659秒测试发送线程(活动真实，连接名)30个文件大小为5 Kbytes。38.926秒,30个文件

what did improve was an implementation of an FTP client made using the Socket class

使用Socket类实现FTP客户端有什么改进吗

the benchmark is here

这里的基准

https://github.com/pedro-vicente/ftp_t

https://github.com/pedro-vicente/ftp_t

#17

i was working a few days with that... and speed was really low, nothing to compare with FileZilla... finally i solved with multithreads. 10 threads making connections for download gives me a good rate, maybe even could be improved more.. with a standar ftprequest configuration

我在那工作了几天…而且速度真的很低，根本无法与FileZilla相比……最后我用多线程解决了。10个线程连接下载给了我一个很好的速率，也许甚至可以改进更多。使用标准的ftprequest配置。

PeticionFTP.ConnectionGroupName = "MyGroupName"
PeticionFTP.ServicePoint.ConnectionLimit = 4
PeticionFTP.ServicePoint.CloseConnectionGroup("MyGroupName")

PeticionFTP.KeepAlive = False 
PeticionFTP.UsePassive = False

PeticionFTP.UseBinary = True

PeticionFTP.Credentials = New NetworkCredential(lHost.User, lHost.Password)

#18

Try this below code, you will get better performence:

试试下面的代码，你会有更好的表现:

private void Upload144_Click(object sender, EventArgs e)
{
    OpenFileDialog fileobj = new OpenFileDialog();
    fileobj.InitialDirectory = "C:\\";
    //fileobj.Filter = "Video files (*.mp4)";
    //fileobj.ShowDialog();

    if (fileobj.ShowDialog() == DialogResult.OK)
    {
        if (fileobj.CheckFileExists)
        {
            string test = Properties.Settings.Default.Connection;
            SqlConnection con = new SqlConnection(test);
            con.Open();
            string correctfilename = System.IO.Path.GetFileName(fileobj.FileName);
            SqlCommand cmd = new SqlCommand("Insert into Path(ID,Path5) VALUES   ((select isnull(MAX(id),0) + 1 from Path),'\\Videos\\" + correctfilename + "')", con);

            cmd.ExecuteNonQuery();

            string path = Application.StartupPath.Substring(0, Application.StartupPath.Length - 10);
            con.Close();

            //For Progressbar
            DataTable dt = new DataTable();
       //   SqlDataAdapter da = new SqlDataAdapter(cmd);
       //   da.Fill(dt);

            timer5.Enabled = true;

            // FOR FtpServer File Upload::
            string uploadfile = fileobj.FileName;
            string uploadFileName = new FileInfo(uploadfile).Name;

            string uploadUrl = "ftp://ftp.infotech.com/";
            FileStream fs = new FileStream(uploadfile, FileMode.Open, FileAccess.Read);
            try
            {
                long FileSize = new FileInfo(uploadfile).Length; // File size of file being uploaded.
                Byte[] buffer = new Byte[FileSize];

                fs.Read(buffer, 0, buffer.Length);

                fs.Close();
                fs = null;
                string ftpUrl = string.Format("{0}/{1}", uploadUrl, uploadFileName);
                FtpWebRequest requestObj = FtpWebRequest.Create(ftpUrl) as FtpWebRequest;
                requestObj.Method = WebRequestMethods.Ftp.UploadFile;
                requestObj.Credentials = new NetworkCredential("[e-mail]", "test@123");
                Stream requestStream = requestObj.GetRequestStream();
                requestStream.Write(buffer, 0, buffer.Length);

                requestStream.Flush();
                requestObj = null;
            }
            catch (Exception ex)
            {
                //MessageBox.Show("File upload/transfer Failed.\r\nError Message:\r\n" + ex.Message, "Succeeded", MessageBoxButtons.OK, MessageBoxIcon.Information);
            }
        }
    }
}