使用numpy.genfromtxt在Python 3中加载UTF-8文件

I have a CSV file that I downloaded from WHO site (http://apps.who.int/gho/data/view.main.52160 , Downloads, "multipurpose table in CSV format"). I try to load the file into a numpy array. Here's my code:

我有一个CSV文件，我从世卫组织网站下载（http://apps.who.int/gho/data/view.main.52160，下载，“CSV格式的多用途表格”）。我尝试将文件加载到numpy数组中。这是我的代码：

import numpy
#U75 - unicode string of max. length 75
world_alcohol = numpy.genfromtxt("xmart.csv", dtype="U75", skip_header=2, delimiter=",")
print(world_alcohol)

And I get

我明白了

UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 2: ordinal not in range(128).

UnicodeDecodeError：'ascii'编解码器无法解码位置2中的字节0xc3：序数不在范围内（128）。

I guess that numpy has a problem reading the string "Côte d'Ivoire". The file is properly encoded UTF-8 (according to my text editor). I am using Python 3.4.3 and numpy 1.9.2.

我猜numpy在阅读字符串“Côted'Ivoire”时遇到了问题。该文件正确编码为UTF-8（根据我的文本编辑器）。我正在使用Python 3.4.3和numpy 1.9.2。

What am I doing wrong? How can I read the file into numpy?

我究竟做错了什么？如何将文件读入numpy？

1 个解决方案

#1

In Python3 I can do:

在Python3中我可以做到：

In [224]: txt = "Côte d'Ivoire"
In [225]: x = np.zeros((2,),dtype='U20')
In [226]: x[0] = txt
In [227]: x
Out[227]: 
array(["Côte d'Ivoire", ''],   dtype='<U20')

Which means I probably could open a 'UTF-8' file (regular, not byte mode), and readlines, and assign them to elements of an array like x.

这意味着我可能打开一个'UTF-8'文件（常规，非字节模式）和readlines，并将它们分配给像x这样的数组元素。

But genfromtxt insists on operating with byte strings (ascii) which can't handle the larger UTF-8 set (7 bytes v 8). So I need to apply decode at some point to get an U array.

但genfromtxt坚持使用字节字符串（ascii）进行操作，这些字符串无法处理较大的UTF-8集（7字节v 8）。所以我需要在某些时候应用解码来获得一个U数组。

I can load it into a 'S' array with genfromtxt:

我可以使用genfromtxt将其加载到'S'数组中：

In [258]: txt="Côte d'Ivoire"
In [259]: a=np.genfromtxt([txt.encode()],delimiter=',',dtype='S20')
In [260]: a
Out[260]: 
array(b"C\xc3\xb4te d'Ivoire",  dtype='|S20')

and apply decode to individual elements:

并将解码应用于单个元素：

In [261]: print(a.item().decode())
Côte d'Ivoire

In [325]: print _
Côte d'Ivoire

Or use np.char.decode to apply it to each element of an array:

或者使用np.char.decode将其应用于数组的每个元素：

In [263]: np.char.decode(a)
Out[263]: 
array("Côte d'Ivoire", dtype='<U13')
In [264]: print(_)
Côte d'Ivoire

genfromtxt lets me specify converters:

genfromtxt让我指定转换器：

In [297]: np.genfromtxt([txt.encode()],delimiter=',',dtype='U20',
    converters={0:lambda x: x.decode()})
Out[297]: 
array("Côte d'Ivoire", dtype='<U20')

If the csv has a mix of strings and numbers, this converters approach will be easier to use than the np.char.decode. Just specify the converter for each string column.

如果csv混合了字符串和数字，那么这种转换器方法将比np.char.decode更容易使用。只需为每个字符串列指定转换器。

(See my earlier edits for Python2 tries).

（请参阅我之前对Python2尝试的编辑）。

#1