Issue on the dimension of weight when training highlevel planner

[KiriyamaGK]

Hi,what a brilliant work of mimicplay!
In fact,when implementing training lowlevel controller with my real-world playdata,aftering training my highlevel human playdata, error occured like this:

Traceback (most recent call last):
File "/home/kiriyamagk/桌面/MimicPlay/mimicplay/scripts/train.py", line 379, in main
train(config, device=device)
File "/home/kiriyamagk/桌面/MimicPlay/mimicplay/scripts/train.py", line 197, in train
step_log = TrainUtils.run_epoch(model=model, data_loader=train_loader, epoch=epoch, num_steps=train_num_steps)
File "/home/kiriyamagk/桌面/robomimic/robomimic/utils/train_utils.py", line 559, in run_epoch
info = model.train_on_batch(input_batch, epoch, validate=validate)
File "/home/kiriyamagk/桌面/robomimic/robomimic/algo/bc.py", line 137, in train_on_batch
predictions = self._forward_training(batch)
File "/home/kiriyamagk/桌面/MimicPlay/mimicplay/algo/mimicplay.py", line 328, in _forward_training
_, mlp_feature = self.human_nets.policy._get_latent_plan(batch['obs'], batch["goal_obs"])
File "/home/kiriyamagk/桌面/MimicPlay/mimicplay/algo/mimicplay.py", line 137, in _get_latent_plan
dists, enc_out, mlp_out = self.nets["policy"].forward_train( #调用策略网络
File "/home/kiriyamagk/桌面/MimicPlay/mimicplay/models/policy_nets.py", line 226, in forward_train
out, enc_out, mlp_out = MIMO_MLP.forward(self, return_latent=return_latent, obs=obs_dict, goal=goal_dict)
File "/home/kiriyamagk/桌面/MimicPlay/mimicplay/models/obs_nets.py", line 594, in forward
mlp_out = self.nets"mlp"
File "/home/kiriyamagk/anaconda3/envs/mimicplay/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/home/kiriyamagk/anaconda3/envs/mimicplay/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
return forward_call(*args, **kwargs)
File "/home/kiriyamagk/桌面/robomimic/robomimic/models/base_nets.py", line 286, in forward
return self._model(inputs)
File "/home/kiriyamagk/anaconda3/envs/mimicplay/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/home/kiriyamagk/anaconda3/envs/mimicplay/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
return forward_call(*args, **kwargs)
File "/home/kiriyamagk/anaconda3/envs/mimicplay/lib/python3.8/site-packages/torch/nn/modules/container.py", line 217, in forward
input = module(input)
File "/home/kiriyamagk/anaconda3/envs/mimicplay/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/home/kiriyamagk/anaconda3/envs/mimicplay/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
return forward_call(*args, **kwargs)
File "/home/kiriyamagk/anaconda3/envs/mimicplay/lib/python3.8/site-packages/torch/nn/modules/linear.py", line 118, in forward
return F.linear(input, self.weight, self.bias)

However,error disappeared after I converted the script in demo_mp4.py like this:
all_hand_loc = np.concatenate((hand_loc_1[:, :, -2:], hand_loc[:, :, -2:]), axis=2)
,setting ac_dim=30 in the highlevel_human.json and finally retraining highlevel planner and lowlevel controller.Also,it seems no use setting ac_dim=40 in the lowlevel.json to train lowlevel controller successfully without changing all_hand_loc in demo_mp4.py.

[KiriyamaGK]

Sorry,I have to rectify my statement.Error disappeared after I converted the script in demo_mp4.py like this:
all_hand_loc = np.concatenate((hand_loc_1[:, :, -1:], hand_loc[:, :, -2:]), axis=2)

[jinpeng1018]

Sorry,I have to rectify my statement.Error disappeared after I converted the script in demo_mp4.py like this: all_hand_loc = np.concatenate((hand_loc_1[:, :, -1:], hand_loc[:, :, -2:]), axis=2)
I'm glad to see this issue being addressed here. Recently, I've also been working on this topic and am at the stage of applying it to real robotic arms. I was wondering if you have successfully implemented this and could share any experience or code related to applying the model to real robotic arms?

[KiriyamaGK]

Sorry,I have to rectify my statement.Error disappeared after I converted the script in demo_mp4.py like this: all_hand_loc = np.concatenate((hand_loc_1[:, :, -1:], hand_loc[:, :, -2:]), axis=2)
I'm glad to see this issue being addressed here. Recently, I've also been working on this topic and am at the stage of applying it to real robotic arms. I was wondering if you have successfully implemented this and could share any experience or code related to applying the model to real robotic arms?

Hi,the problem is due to the action dim is set 30(10xyz) in simulation, but 40(10xy*2(dual imgs)) in real world code.What I have done then is to project the x-y coordinates of each camara into x-y-z coordinates in robot base frame,and after which error disappeared. Also,don't forget to add gradient clipping in case of gradient explosion,which worses the training process most.Finally, if you want to use human playdata(I haven't tried it though) ,there should be a KL-loss set in latent planner,which is mentioned in the paper but not implemented in simulation code.