Problem: pickle} cannot save namedtuple
Specific description:
-
Error message
Traceback (most recent call last): File "/home/liyd/anaconda3/envs/py3/lib/python3.6/site-packages/numpy/lib/npyio.py", line 529, in save pickle_kwargs=dict(fix_imports=fix_imports)) File "/home/liyd/anaconda3/envs/py3/lib/python3.6/site-packages/numpy/lib/format.py", line 664, in write_array pickle.dump(array, fp, protocol=3, **pickle_kwargs) _pickle.PicklingError: Can't pickle <class '__main__.cam'>: attribute lookup cam on __main__ failed python-BaseException
Check the above error message and locate the error in a field called__ main__. In the class of cam, pickle is in__ main__ In the middle of the call, so try it in main__ It's not surprising to find cam in. Take a look at the location of pickle call:
-
find by hard and thorough search
for sequence in seqList:
audioDataDic= {}
for cam_number in range(1, 4):
DATA, CFGforGCF = InitGCF(datasetPath_au, sequence, cam_number)
audioData = {f'{sequence}_cam{cam_number}': DATA}
audioDataDic.update(audioData)
# save the imgDataList as {sequence}_sampleList.npz
folderPath = f'{datasetPath_save}/audio/{sequence}'
if not os.path.exists(folderPath):
os.makedirs(folderPath)
np.save(f'{folderPath}/{sequence}_audio.npz', audioDataDic)# audioDic=audioDataDic
print(f'save audio.npz for {sequence}')The problem is not pickle, but cam. Why can't cam be found in main? Because the cam class is defined in the function, the definition of cam disappears when the function class comes out. Cam has become an orphan object. Of course, pickle can't find the definition of cam. Take a look at the definition of cam:
-
The orphan object cannot be saved
class DataMain:
def __init__(self,cfgGCF):
......
def loadCamAlign(self,datasetPath):
......
cam = {
'Pmat': np.concatenate(([data[0][0]], [data[1][0]], [data[2][0]]), axis=0),
'K': np.concatenate(([data[0][1]], [data[1][1]], [data[2][1]]), axis=0),
'alpha_c': np.concatenate(([data[0][2]], [data[1][2]], [data[2][2]]), axis=0),
'kc': np.concatenate(([data[0][3]], [data[1][3]], [data[2][3]]), axis=0),
}
self.cam = namedtuple('cam',cam.keys())(**cam)
dataPath = f'{datasetPath}/rigid010203.mat'
data = scio.loadmat(dataPath)['rigid'][0]
self.align_mat = data[0][1]Cam is actually defined in datamain loadCamAlign. There is no reference name in cam, and pickle only knows to go when looking for it__ main__. I can't find it in cam. So how can I find it? The answer is to put the class definition outside.
solve:
-
Define cam class
Analyze and define self Cam = namedtuple ('cam ', cam. Keys()) (* * CAM):
namedtuple('cam',cam.keys())
First, namedtuple is a type factory. You can output a new class by entering the name and parameters of the type. This new class currently has no reference to it. If you want pickle to find it, you should give it a name and put it outside the function to avoid destroying the class definition when exiting the function. This principle is explained in reference 1.
(**cam)
Secondly, a new object is created by using this "anonymous class", and its parameters are determined by the dictionary cam. Finally, return the singleton object to self cam.
-
Redefine cam class
camCls = namedtuple('cam',['Pmat','K','alpha_c','kc'])camCls stores the index of this class at this time. If pickle obtains the information of this class from camCls, cam can be saved. However, the actual operation found that pickle could not find camCls. Why?
When pickle is saved, it only gets the class object. The class name cam is stored in the object. Pickle takes cam to find the definition. If the class content is stored in camCls, pickle does not know. It is generally believed that all names in Python are unimportant. Class names and object names can be changed at will. As long as the entity pointed to is correct, the program can work normally. However, in the save operation, the name is important and you need to rely on the name to find the correct entity. A similar example is given in reference 2 to illustrate the problem of changing names.
-
Make sure the cam class name is correct
The correct naming method is as follows:
camCls = namedtuple('camCls',['Pmat','K','alpha_c','kc'])In retrospect, why doesn't this problem occur when defining classes? Because most class definitions occur in__ main__ The name of the class will not be changed to an abbreviation at will. Even if it is changed, it will not just need to be saved. However, the singleton object created by namedtuple can meet these requirements at the same time.
Summary:
The improved code becomes:
camClsAttr = ['Pmat','K','alpha_c','kc']
camCls = namedtuple('camCls', camClsAttr)
class DataMain:
def __init__(self,cfgGCF):
......
def loadCamAlign(self,datasetPath):
......
cam = {
'Pmat': np.concatenate(([data[0][0]], [data[1][0]], [data[2][0]]), axis=0),
'K': np.concatenate(([data[0][1]], [data[1][1]], [data[2][1]]), axis=0),
'alpha_c': np.concatenate(([data[0][2]], [data[1][2]], [data[2][2]]), axis=0),
'kc': np.concatenate(([data[0][3]], [data[1][3]], [data[2][3]]), axis=0),
}
self.cam = camCls(**cam)
dataPath = f'{datasetPath}/rigid010203.mat'
data = scio.loadmat(dataPath)['rigid'][0]
self.align_mat = data[0][1]The definition class is placed outside the function to avoid the object becoming an orphan object, so that the definition of the class can be found when pickle is saved. At this time, cam is no longer an anonymous class, but degenerates into an ordinary class. Also note that in the process of unpickling, you need to make pickle find the same type in the same way.
reference resources: