AttributeError: Can't get attribute 'xxx' on <module '__main__' from 'xxx'

Posted by RyanJones on Sat, 25 Sep 2021 11:51:34 +0200

AttributeError: Can't get attribute 'xxx' on < module 'main' from 'xxx' another possible solution

Problem Restatement: when learning pytorch, I learned to serialize the Word2Seq class using the pickle module and save the serialization in the ws.pkl file. Then I created the lib.py file again. This problem occurs when I deserialize ws.pkl with the following code.

	ws = pickle.load(open('ws.pkl', 'rb')) 

First, explain the first solution (which is not always effective):

I wrote the following code in the lib.py file I created

	import pickle
	from utils.word2seq import Word2Seq  # This Word2Seq is the class serialized for you
	ws = pickle.load(open('ws.pkl', 'rb'))

In this way, run the lib.py file directly, and ws can be loaded correctly without error
But there will be new problems
Problem Description: I created another build_dataset.py file, and expect in build_ Import the ws object in the lib.py file from the dataset.py file. But the problem arises, that is: AttributeError: Can't get attribute 'xxx' on < module 'main' from 'xxx'. The above method will not work at all.
Therefore, this solution is not the most fundamental solution

The second solution (complete solution):

The reason why there is a problem in the problem is that when you construct the class to be serialized by pickle, you are wrong to write it. The py file of the class to be serialized by pickle must be * * "clean" * *, that is, the py file can only write this class. The following example:
If I want to serialize the Word2Seq class in the word2seq.py file, don't write any more content in the PY file, such as the def function, except the content in class Word2Seq()

# word2seq.py
'''
    Construct a dictionary and realize the method of transforming sentences into digital sequences and their inversion
'''
class Word2Seq():
    UNK_TAG = 'UNK'  # UNK represents special characters. Words not seen are replaced by UNK, which corresponds to the number 0
    PAD_TAG = 'PAD'  # Fill in the short sentence with PAD, and the corresponding number of PAD is 1

    UNK = 0
    PAD = 1

    def __init__(self):
        # Match words with numbers
        self.dict = {
            self.UNK_TAG: self.UNK,
            self.PAD_TAG: self.PAD
        }
        self.count = {}  # Statistical word frequency

    def fit(self, text):
        '''
        Save individual sentences to dict in, And count the word frequency of each word
        :param text: [word1, word2, word3, ...]
        :return:
        '''
        for word in text:
            '''Tips:  Programming skills
                self.count.get(word, 0) + 1
                If in the current dictionary'word'Return if present key The corresponding value and+1,If'word'Returns 0 if it does not exist+1
            '''
            self.count[word] = self.count.get(word, 0) + 1

    def build_vocab(self, min=5, max=None, max_features=None):
        '''
        Generate dictionary, Eliminate words that do not meet the quantity requirements
        :param min: Minimum occurrences of words
        :param max: Maximum occurrences
        :param max_features: How many words are reserved
        :return:
        '''
        # Delete words with word frequency less than min in count
        if min is not None:
            '''
                PS: When traversing a dictionary, you are actually traversing key
            '''
            self.count = {word: value for word, value in self.count.items() if value >= min}
        # Delete words with word frequency greater than max in count
        if max is not None:
            self.count = {word: value for word, value in self.count.items() if value <= max}
        # Limit the number of reserved words
        if max_features is not None:
            '''
                sorted The tuple will become a list
                self.count.items() Is an iteratable object, Each of these values is a(key,value)yes
                key=lambda x:x[-1] Make in the dictionary key according to items Medium value Sort, x[-1]Means to take the last value, that is value
                reverse=True From large to small, in descending order
                [:max_features] Put the sorted first max_features Take out the number(because sorted Already dict_items Become list,Therefore, it can be taken this way)
            '''
            temp = sorted(self.count.items(), key=lambda x: x[-1], reverse=True)[
                   :max_features]  # The result is a list in which each element is a binary tuple
            self.count = dict(temp)  # Convert [(key, value), (key, value)] to {key:value, key:value}

        # Number each word {word:num}
        for word in self.count:
            '''
                Because the original self.dict Already in self.UNK_TAG: self.UNK and self.PAD_TAG: self.PAD Two sets of key value pairs
                Therefore, the numbering of new words starts from 2 and will not repeat the previous ones
            '''
            self.dict[word] = len(self.dict)

        # Get a flipped dict dictionary {num:word}
        self.inverse_dict = dict(zip(self.dict.values(), self.dict.keys()))

    def transform(self, text, max_len=None):
        '''
        Convert sentences into sequences
        :param text: [word1, word2, ...]
        :param max_len: int, Fill in or cut out sentences
        :return: [1, 2, 4, ...]
        '''
        '''
            stay self.dict Find the number corresponding to each word in the sentence list return
        '''
        if max_len is not None:
            if max_len > len(text):
                text = text + [self.PAD_TAG] * (max_len - len(text))  # If the sentence length is less than max_len fills the sentence with max_len len (text) 'PAD'
            else:
                text = text[:max_len]  # If the sentence length is greater than max_len, then cut the sentence and take the first max_len

        return [self.dict.get(word, self.UNK) for word in text]

    def inverse_transform(self, indices):
        '''
        Convert sequences into sentences
        :param indices: [1, 2, 4, 5, 3, ...]
        :return: [word1, word2, word4, word3, ...]
        '''
        return [self.inverse_dict.get(index) for index in indices]

    def __len__(self):
        # Number of words
        return len(self.dict)


if __name__ == '__main__':
    pass

At this point, I write the function in the dataset.py file to generate the ws.pkl file

	# Save word to number mapping
def fit_save_word_seq(max_features=10000):
    import os
    import pickle
    from utils.word2seq import Word2Seq

    ws = Word2Seq()
    path = '../data/aclImdb'
    temp_data_path = [os.path.join(path, 'train/pos'),
                      os.path.join(path, 'train/neg'),]
                      # os.path.join(path, 'test/pos'),
                      # os.path.join(path, 'test/neg')]
    for data_path in temp_data_path:
        file_list = os.listdir(data_path)  # Get all file names in the directory
        file_path_list = [os.path.join(data_path, file_name) for file_name in file_list if file_name.endswith('.txt')]  # Get file full path
        for file_path in tqdm(file_path_list):
            text = tokenlize(open(file_path, encoding='utf-8').read())  # Segment each sentence
            ws.fit(text)  # Map each word to a sequence

    if max_features is not None: # Limit the maximum number of words
        ws.build_vocab(min=10, max_features=max_features)
    else:
        ws.build_vocab(min=10)  # Number each word
    pickle.dump(ws, open('../model_data/ws.pkl', 'wb'), protocol=4)
    print(ws.dict, len(ws))


if __name__ == '__main__':
    fit_save_word_seq()

Then load the ws.pkl file in the lib.py file

import pickle
# Note that the biggest difference here is that you don't need to import the Word2Seq class like method 1
ws = pickle.load(open('model_data/ws.pkl', 'rb'))
print(ws)

Finally, I'm building_ Directly import the ws objects deserialized from the lib.py file into the dataset.py file, so that no error will be reported. The ws objects in the lib.py file will be imported in the file where you want to import them. There will be no more problems in the problem!

from utils.lib import ws

PS: don't ask me why I want to move around like this, because if the amount of code is large, you must write a configuration file to store adjustable parameters, so using method 2 is the best solution! Make sure that the py file of the class to be serialized by pickle does not contain anything other than class Word2Seq()

Topics: Python Deep Learning NLP