POEM: Human POse EMbedding

We target learning representations/embeddings for 3D human poses from their 2D projections, including:

• Pr-VIPE: Learning a view-invariant probabilistic pose embedding space. [ECCV'20 paper][IJCV'21 paper][website]
• CV-MIM: Learning disentangled view-invariant pose representations and view representations. [CVPR'21 paper]

Please refer to our papers for details and consider citing them if you find the code useful:

@inproceedings{sun2020view,
  title={View-Invariant Probabilistic Embedding for Human Pose},
  author={Sun, Jennifer J and Zhao, Jiaping and Chen, Liang-Chieh and Schroff, Florian and Adam, Hartwig and Liu, Ting},
  booktitle={ECCV},
  year={2020}
}

@article{liu2021view,
  title={View-Invariant, Occlusion-Robust Probabilistic Embedding for Human Pose},
  author={Liu, Ting and Sun, Jennifer J and Zhao, Long and Zhao, Jiaping and Yuan, Liangzhe and Wang, Yuxiao and Chen, Liang-Chieh and Schroff, Florian and Adam, Hartwig},
  journal={International Journal of Computer Vision},
  year={2021},
  publisher={Springer}
}

@inproceedings{zhao2021learning,
  title={Learning View-Disentangled Human Pose Representation by Contrastive Cross-View Mutual Information Maximization},
  author={Zhao, Long and Wang, Yuxiao and Zhao, Jiaping and Yuan, Liangzhe and Sun, Jennifer J and Schroff, Florian and Adam, Hartwig and Peng, Xi and Metaxas, Dimitris and Liu, Ting},
  booktitle={CVPR},
  year={2021}
}

Getting Started

The installation requires Python3, virtualenv, and pip pre-installed. Please refer to requirements.txt and run.sh for how to install the required packages.

Note: The run.sh script only provides a sample command for how to launch a training job on a small dummy data table. Running it as is does NOT launch training job on any real datasets or generate any meaningful model checkpoints.

This repository is organized with the following folders:

• pr_vipe: Pr-VIPE project code.
• cv_mim: CV-MIM project code.
• core: Common utility libraries.
• tools: Common utility tools.
• testdata: Data for code testing (not for real training/evaluation).
• doc: Documentation files.

Getting Help

Please report issues related to this repository to the tracker and make sure to include the "[POEM]" prefix in the issue title.

Contact

• Ting Liu
• Jennifer J. Sun
• Long Zhao
• Liangzhe Yuan

Updates

• 12/03/2021: Added Temporal Pr-VIPE training support. Please refer to our IJCV'21 paper for details.
• 11/23/2021: Added individual random keypoint dropout support. Please refer to our IJCV'21 paper for details.
• 06/08/2021: Added CV-MIM code for the CVPR'21 paper.
• 03/25/2021: Moved Pr-VIPE code into a subfolder. Added CV-MIM subfolder and updated documentations.
• 03/17/2021: Fixed an issue in camera augmentation.
• 03/04/2021: Added a program for running model inference.
• 10/21/2020: Added cross-view pose retrieval evaluation frame keys.
• 10/15/2020: Added training TFRecords generation program.
• 07/02/2020: First release. Included core TensorFlow code for model training.