mingyi_s paper

content/publication/camdm-2024/index.md

@@ -0,0 +1,53 @@
+---
+# Documentation: https://wowchemy.com/docs/managing-content/
+
+title: 'Taming Diffusion Probabilistic Models for Character Control'
+subtitle: 'Equal contributions: Rui and Mingyi'
+summary: ''
+authors:
+- Rui Chen
+- Mingyi Shi
+- Shaoli Huang
+- Ping Tan
+- Taku Komura
+- Xuelin Chen
+
+tags:
+- '3D human generation'
+- 'character controller'
+- 'diffusion model'
+
+categories: []
+date: '2024-07-01'
+lastmod: 2021-01-15T21:34:50Z
+featured: false
+draft: false
+
+# Featured image
+# To use, add an image named `featured.jpg/png` to your page's folder.
+# Focal points: Smart, Center, TopLeft, Top, TopRight, Left, Right, BottomLeft, Bottom, BottomRight.
+image:
+  caption: ''
+  focal_point: ''
+  preview_only: false
+
+# Projects (optional).
+#   Associate this post with one or more of your projects.
+#   Simply enter your project's folder or file name without extension.
+#   E.g. `projects = ["internal-project"]` references `content/project/deep-learning/index.md`.
+#   Otherwise, set `projects = []`.
+projects: []
+publishDate: '2024-07-01T21:34:50.388741Z'
+publication_types:
+# 1 Conference paper
+# 2 Journal article
+# 3 Preprint
+# 4 Report
+# 5 Book
+# 6 Book section
+# 7 Thesis
+# 8 Patent
+- '1'
+abstract: We present a novel character control framework that effectively utilizes motion diffusion probabilistic models to generate high-quality and diverse character animations, responding in real-time to a variety of dynamic user-supplied control signals. At the heart of our method lies a transformer-based Conditional Autoregressive Motion Diffusion Model (CAMDM), which takes as input the character's historical motion and can generate a range of diverse potential future motions conditioned on high-level, coarse user control. To meet the demands for diversity, controllability, and computational efficiency required by a real-time controller, we incorporate several key algorithmic designs. These include separate condition tokenization, classifier-free guidance on past motion, and heuristic future trajectory extension, all designed to address the challenges associated with taming motion diffusion probabilistic models for character control. As a result, our work represents the first model that enables real-time generation of high-quality, diverse character animations based on user interactive control, supporting animating the character in multiple styles with a single unified model. We evaluate our method on a diverse set of locomotion skills, demonstrating the merits of our method over existing character controllers.
+publication: 'SIGGRAPH 2024'
+---

content/publication/motionet-2020/index.md

@@ -0,0 +1,52 @@
+---
+# Documentation: https://wowchemy.com/docs/managing-content/
+
+title: 'MotioNet: 3D Human Motion Reconstruction from Monocular Video with Skeleton Consistency'
+subtitle: ''
+summary: ''
+authors:
+- Mingyi Shi
+- Kfir Aberman
+- Andreas Aristidou
+- Taku Komura
+- Dani Lischinski
+- Daniel Cohen-Or
+- Baoquan Chen
+
+tags:
+- '3D motion reconstruction'
+
+categories: []
+date: '2020-07-01'
+lastmod: 2020-01-15T21:34:50Z
+featured: false
+draft: false
+
+# Featured image
+# To use, add an image named `featured.jpg/png` to your page's folder.
+# Focal points: Smart, Center, TopLeft, Top, TopRight, Left, Right, BottomLeft, Bottom, BottomRight.
+image:
+  caption: ''
+  focal_point: ''
+  preview_only: false
+
+# Projects (optional).
+#   Associate this post with one or more of your projects.
+#   Simply enter your project's folder or file name without extension.
+#   E.g. `projects = ["internal-project"]` references `content/project/deep-learning/index.md`.
+#   Otherwise, set `projects = []`.
+projects: []
+publishDate: '2020-07-01T21:34:50.388741Z'
+publication_types:
+# 1 Conference paper
+# 2 Journal article
+# 3 Preprint
+# 4 Report
+# 5 Book
+# 6 Book section
+# 7 Thesis
+# 8 Patent
+- '1'
+abstract: We introduce MotioNet, a deep neural network that directly reconstructs the motion of a 3D human skeleton from monocular video. While previous methods rely on either rigging or inverse kinematics (IK) to associate a consistent skeleton with temporally coherent joint rotations, our method is the first data-driven approach that directly outputs a kinematic skeleton, which is a complete, commonly used, motion representation. At the crux of our approach lies a deep neural network with embedded kinematic priors, which decomposes sequences of 2D joint positions into two separate attributes - a single, symmetric, skeleton, encoded by bone lengths, and a sequence of 3D joint rotations associated with global root positions and foot contact labels.
+publication: 'ToG 2020'
+---

content/publication/normpose-2021/index.md

@@ -0,0 +1,50 @@
+---
+# Documentation: https://wowchemy.com/docs/managing-content/
+
+title: 'Normalized Human Pose Features for Human Action Video Alignment'
+subtitle: ''
+summary: ''
+authors:
+- Jingyuan Liu
+- Mingyi Shi
+- Qifeng Chen
+- Hongbo Fu
+- Chiew-Lan Tai
+
+tags:
+- 'Human Action Recognization'
+
+categories: []
+date: '2021-07-01'
+lastmod: 2021-07-01T21:34:50Z
+featured: false
+draft: false
+
+# Featured image
+# To use, add an image named `featured.jpg/png` to your page's folder.
+# Focal points: Smart, Center, TopLeft, Top, TopRight, Left, Right, BottomLeft, Bottom, BottomRight.
+image:
+  caption: ''
+  focal_point: ''
+  preview_only: false
+
+# Projects (optional).
+#   Associate this post with one or more of your projects.
+#   Simply enter your project's folder or file name without extension.
+#   E.g. `projects = ["internal-project"]` references `content/project/deep-learning/index.md`.
+#   Otherwise, set `projects = []`.
+projects: []
+publishDate: '2021-07-01T21:34:50.388741Z'
+publication_types:
+# 1 Conference paper
+# 2 Journal article
+# 3 Preprint
+# 4 Report
+# 5 Book
+# 6 Book section
+# 7 Thesis
+# 8 Patent
+- '2'
+abstract: We present a novel approach for extracting human pose features from human action videos. The goal is to let the pose features capture only the poses of the action while being invariant to other factors, including video backgrounds, the video subjects’ anthropometric characteristics and viewpoints. Such human pose features facilitate the comparison of pose similarity and can be used for downstream tasks, such as human action video alignment and pose retrieval. The key to our approach is to first normalize the poses in the video frames by mapping the poses onto a pre-defined 3D skeleton to not only disentangle subject physical features, such as bone lengths and ratios, but also to unify global orientations of the poses. Then the normalized poses are mapped to a pose embedding space of highlevel features, learned via unsupervised metric learning. We evaluate the effectiveness of our normalized features both qualitatively by visualizations, and quantitatively by a video alignment task on the Human3.6M dataset and an action recognition task on the Penn Action dataset.
+publication: 'ICCV 2021 Oral'
+---

content/publication/phasemp-2023/index.md

@@ -0,0 +1,50 @@
+---
+# Documentation: https://wowchemy.com/docs/managing-content/
+
+title: 'PhaseMP: Robust 3D Pose Estimation via Phase-conditioned Human Motion Prior'
+subtitle: ''
+summary: ''
+authors:
+- Mingyi Shi
+- Sebastian Starke
+- Yuting Ye
+- Taku Komura
+- Jungdam Won
+
+tags:
+- '3D human reconstruction'
+
+categories: []
+date: '2023-07-01'
+lastmod: 2021-01-15T21:34:50Z
+featured: false
+draft: false
+
+# Featured image
+# To use, add an image named `featured.jpg/png` to your page's folder.
+# Focal points: Smart, Center, TopLeft, Top, TopRight, Left, Right, BottomLeft, Bottom, BottomRight.
+image:
+  caption: ''
+  focal_point: ''
+  preview_only: false
+
+# Projects (optional).
+#   Associate this post with one or more of your projects.
+#   Simply enter your project's folder or file name without extension.
+#   E.g. `projects = ["internal-project"]` references `content/project/deep-learning/index.md`.
+#   Otherwise, set `projects = []`.
+projects: []
+publishDate: '2021-01-15T21:34:50.388741Z'
+publication_types:
+# 1 Conference paper
+# 2 Journal article
+# 3 Preprint
+# 4 Report
+# 5 Book
+# 6 Book section
+# 7 Thesis
+# 8 Patent
+- '1'
+abstract: We present a novel motion prior, called PhaseMP , modeling a probability distribution on pose transitions conditioned by a frequency domain feature extracted from a periodic autoencoder. The phase feature further enforces the pose transitions to be unidirectional (i.e. no backward movement in time), from which more stable and natural motions can be generated. Specifically, our motion prior can be useful for accurately estimating 3D human motions in the presence of challenging input data, including long periods of spatial and temporal occlusion, as well as noisy sensor measurements. Through a comprehensive evaluation, we demonstrate the efficacy of our novel motion prior, showcasing its superiority over existing state-of-the-art methods by a significant margin across various applications, including video-to-motion and motion estimation from sparse sensor data, and etc.
+publication: 'ICCV 2023'
+---