Python NLTK | nltk.tokenize.mwe()
借助NLTK nltk.tokenize.mwe()
方法,我们可以将音频流标记为 multi_word 表达式标记,这有助于使用nltk.tokenize.mwe()
方法将标记与下划线绑定。请记住,它区分大小写。
Syntax : MWETokenizer.tokenize()
Return : Return bind tokens as one if declared before.
示例 #1:
在本例中,我们使用MWETokenizer.tokenize()
方法,该方法用于绑定之前定义的令牌。我们还可以使用tokenizer.add_mwe()
方法添加预定义的标记。
# import MWETokenizer() method from nltk
from nltk.tokenize import MWETokenizer
# Create a reference variable for Class MWETokenizer
tk = MWETokenizer([('g', 'f', 'g'), ('geeks', 'for', 'geeks')])
# Create a string input
gfg = "geeks for geeks g f g"
# Use tokenize method
geek = tk.tokenize(gfg.split())
print(geek)
输出 :
[‘geeks_for_geeks’, ‘g_f_g’]
示例 #2:
# import MWETokenizer() method from nltk
from nltk.tokenize import MWETokenizer
# Create a reference variable for Class MWETokenizer
tk = MWETokenizer([('g', 'f', 'g'), ('geeks', 'for', 'geeks')])
tk.add_mwe(('who', 'are', 'you'))
# Create a string input
gfg = "who are you at geeks for geeks"
# Use tokenize method
geek = tk.tokenize(gfg.split())
print(geek)
输出 :
[‘who_are_you’, ‘at’, ‘geeks_for_geeks’]