python 搜索引擎Whoosh中文文档和代码 以及jieba的使用

时间:2023-03-09 22:52:25
python 搜索引擎Whoosh中文文档和代码 以及jieba的使用

注意, 数据库的表最好别有下划线

中文文档链接:

      https://mr-zhao.gitbooks.io/whoosh/content/%E5%A6%82%E4%BD%95%E7%B4%A2%E5%BC%95%E6%96%87%E6%A1%A3.html

      https://mr-zhao.gitbooks.io/whoosh/content/如何索引文档.html?q=

代码:

 https://github.com/renfanzi/myWhoosh

jieba的使用案例 =====》版本3.5

from jieba.analyse import ChineseAnalyzer
import jieba analyzer = ChineseAnalyzer()
a = analyzer("我的好朋友是李明;我爱北京*;IBM和Microsoft; I have a dream. this is intetesting and interested me a lot") # 这样的是只过滤词,更加简洁,更方便
print([i.text for i in a]) seg_list = jieba.cut("我的好朋友是李明;我爱北京*;IBM和Microsoft; I have a dream. this is intetesting and interested me a lot",cut_all=False) # cut_all = False 就是连标点等都不过滤,完全切割cut
seg_list1 = jieba.cut("IBM和Microsoft; I have a dream. this is intetesting and interested me a lot",cut_all=True) # cut_all = False 就是过滤标点等
print([i for i in seg_list if i])
print([i for i in seg_list1 if i])
# 注意,其实也可以用join来拼接

结果:

Building prefix dict from the default dictionary ...
Loading model from cache /tmp/jieba.cache
Loading model cost 0.865 seconds.
Prefix dict has been built succesfully.
['我', '好', '朋友', '是', '李明', '我', '爱', '北京', '天安', '*', 'ibm', 'microsoft', 'dream', 'intetest', 'interest', 'me', 'lot']
['我', '的', '好', '朋友', '是', '李明', ';', '我', '爱', '北京', '*', ';', 'IBM', '和', 'Microsoft', ';', ' ', 'I', ' ', 'have', ' ', 'a', ' ', 'dream', '.', ' ', 'this', ' ', 'is', ' ', 'intetesting', ' ', 'and', ' ', 'interested', ' ', 'me', ' ', 'a', ' ', 'lot']
['IBM', '和', 'Microsoft', 'I', 'have', 'a', 'dream', 'this', 'is', 'intetesting', 'and', 'interested', 'me', 'a', 'lot']