查找[主机]:在Go中没有这样的主机错误

I have this test program which will fetch url parallel, but when I increase the parallel number to about 1040, I start to get lookup www.httpbin.org: no such host error.

我有这个将获取url并行的测试程序，但是当我将并行数增加到大约1040时，我开始查找www.httpbin.org:没有这样的主机错误。

After some Google, I found others say that not close the response will cause the problem, but I do close that with res.Body.Close().

经过一些谷歌之后，我发现其他人说不关闭响应将导致问题，但是我用res.Body.Close()关闭它。

What's the problem here? thanks very much.

这里的问题是什么?非常感谢。

package main

import (
    "fmt"
    "net/http"
    "io/ioutil"
)

func get(url string) ([]byte, error) {

    client := &http.Client{}
    req, _ := http.NewRequest("GET", url, nil)

    res, err := client.Do(req)

    if err != nil {
        fmt.Println(err)
        return nil, err
    } 

    bytes, read_err := ioutil.ReadAll(res.Body)
    res.Body.Close()

    fmt.Println(bytes)

    return bytes, read_err
}

func main() {
    for i := 0; i < 1040; i++ {
        go get(fmt.Sprintf("http://www.httpbin.org/get?a=%d", i))
    }
}

2 个解决方案

#1

That's because you may have up to 1040 concurrent calls in your code so you may very well be in a state with 1040 body opened and none yet closed.

这是因为您的代码中可能有多达1040个并发调用，所以您很可能处于打开了1040个主体而没有关闭的状态。

You need to limit the number of goroutines used.

你需要限制使用goroutines的数量。

Here's one possible solution with a limit to 100 concurrent calls max :

这里有一个可能的解决方案，限制100个并发调用的最大值:

func getThemAll() {
    nbConcurrentGet := 100
    urls :=  make(chan string, nbConcurrentGet)
    for i := 0; i < nbConcurrentGet; i++ {
        go func (){
            for url := range urls {
                get(url)
            }
        }()
    }
    for i:=0; i<1040; i++ {
        urls <- fmt.Sprintf("http://www.httpbin.org/get?a=%d", i)
    }
}

If you call this in the main function of your program, it may stop before all tasks are finished. You can use a sync.WaitGroup to prevent it :

如果在程序的主函数中调用这个函数，它可能在所有任务完成之前停止。你可以使用同步。WaitGroup防止:

func main() {
    nbConcurrentGet := 100
    urls :=  make(chan string, nbConcurrentGet)
    var wg sync.WaitGroup
    for i := 0; i < nbConcurrentGet; i++ {
        go func (){
            for url := range urls {
                get(url)
                wg.Done()
            }
        }()
    }
    for i:=0; i<1040; i++ {
        wg.Add(1)
        urls <- fmt.Sprintf("http://www.httpbin.org/get?a=%d", i)
    }
    wg.Wait()
    fmt.Println("Finished")
}

#2

well technically your process is limited (by the Kernel) to about 1000 open file descriptors. Depending on the context you might need to increase this number.

从技术上讲，您的过程是有限的(由内核)到大约1000个打开的文件描述符。根据上下文，您可能需要增加这个数字。

In your shell run (note the last line):

在shell中运行(注意最后一行):

$ ulimit -a
-t: cpu time (seconds)         unlimited
-f: file size (blocks)         unlimited
-d: data seg size (kbytes)     unlimited
-s: stack size (kbytes)        8192
-c: core file size (blocks)    0
-v: address space (kb)         unlimited
-l: locked-in-memory size (kb) unlimited
-u: processes                  709
-n: file descriptors           2560

To increase (temporarly):

增加(瞬时):

$ ulimit -n 5000
(no output)

Then verify the fd limit:

然后验证fd极限:

$ ulimit -n
5000

#1