grep多个模式单个文件参数列表太长

I am currently searching for multiple patterns in a file. The file is of 90GB in size, I am searching on a particular field(from position 6-17 in each line). I am trying to get all the lines that contain any of a particular list of numbers. The current syntax I am using is:

我目前正在文件中搜索多个模式。该文件大小为90GB,我在特定字段(每行6-17位)搜索。我试图获得包含任何特定数字列表的所有行。我使用的当前语法是:

grep '^.\{6\}0000000012345\|^.\{6\}0000000012543' somelargeFile.txt > outputFile.txt

For small number of patterns this works. For a large number of patterns I get the "Argument list too long" error.

对于少量模式,这是有效的。对于大量模式,我得到“参数列表太长”错误。

One alternative I have tried is to search for each patters separately (using a for loop over the patterns), but this will require multiple passes over the large data file(57102722 lines) which is not efficient.

我尝试过的另一种方法是分别搜索每个模式(在模式上使用for循环),但这需要多次传递大数据文件(57102722行),这是无效的。

From what I understand about the "Argument list too long" error, it is related to bash cmds in general and not specific to grep. Is there any setting that can be used to get around this error? Or alternatively, any ideas as to how to do this using awk or sed or another tool?

根据我对“参数列表太长”错误的理解,它通常与bash cmds有关,而不是特定于grep。有没有可用于解决此错误的设置?或者,有关如何使用awk或sed或其他工具执行此操作的任何想法?

Thank you!

2 个解决方案

#1

You can avoid the problem by putting the patterns in a file, and using the -f command line option to grep.

您可以通过将模式放在文件中,并使用-f命令行选项grep来避免此问题。

The most convenient is to put each alternative in a separate line of the file:

最方便的是将每个备选项放在文件的单独行中:

patterns.txt

^.\{6\}0000000012345
^.\{6\}0000000012543

invocation

grep -f patterns.txt somelargeFile.txt > outputFile.txt

#2

Try using alternation operator.

尝试使用交替运算符。

grep '^.\{6\}0000000012\(345\|543\)'