Problem Descriptions:
When I implement a Chinese character-based Bert tokenizer, I need first to extend the BertTokenizer model, then initiate the parameters, and finally create a character-based tokenizer method.
The following is the parameter initalization code.

class BERTChineseCharacterTokenizer(BertTokenizer):
    def __init__(self, vocab_file, do_lower_case=True):
        super(BERTChineseCharacterTokenizer, self).__init__(vocab_file, do_lower_case)
    def char_tokenize(self, text, unk_token='[UNK]'):

Errors:

TypeError: __init__() got an unexpected keyword argument 'max_len'

Solution: initialize the max_len value.

class BERTChineseCharacterTokenizer(BertTokenizer):
    def __init__(self, vocab_file, do_lower_case=True, max_len=512):
        super(BERTChineseCharacterTokenizer, self).__init__(vocab_file, do_lower_case, max_len=512)
    def char_tokenize(self, text, unk_token='[UNK]'):

更多推荐