makehuman.js

A Makehuman variant with different features.

currently writing a tool to create morph targets for face animation from the blend shapes in ARKit / ICT FaceKit

Currently Working On

Animation!

• Face animation with MediaPipe/Live Link Face App

  • data is provided as Apple ARKit Blendshape parameters
  • MH's default face rig is not expressive enough to mimic the ARKit blendshapes
  • current approach is to create morph targets by mapping ARKit and ICT FaceKit meshes to the MH mesh (see screenshot above)
  • if that isn't good enough, i'll try to use MH's rigigy face rig

• Body animation with MediaPipe/FreeMoCap

  • data is provided as BlazePose landmarks
  • there's no user interface yet for this
  • demo videos
    • input: FreeMoCap with 3 cameras
    • result: makehuman.js' animation

(also worth looking into: OpenPose, Apple VisionFramework, ...)

This needs the makehuman.js backend to be running locally.

Once that is working, the next step will be to record animations and export them to Blender.

Notes on MediaPipe:

• better go through something like freemocap
• on live preview
  • older versions of MediaPipe had filters to reduce jitter, now we need to do it on our own
    • Kalman seems to be a favourite
    • Simple Moving Average (SMA)
    • Savitzky-Golay filter needs lookahead
    • EMA
  • z-axis is not good when using world coordinates
    • use pose_world_landmarks with translating with the root of pose_landmarks?
  • the holistic task lacks good neck, hand and foot orientation

Current Status

• Have a look at build 2023-10-31
• Edit, load and save morph
• Pose and load pose
• Select one of 32 pre-defined facial expressions and/org edit facial pose units
• Render various proxy meshes instead of the basemesh
• Export the mesh with rig and texture coordinates as Collada for Blender
• Nothing else... 😅

How to build

• npm install
• npm run dev:prepare
• npm run dev:build
• npm run dev:serve (in another terminal)

Why?

• I've been using MakeHuman for more than a decade but often struggled with the UI and the source code.
• I'm up to something with Blender and Chordata and in need for full artistic control of the toolchain. 😎

Animation

Capturing

• MediaPipe

  Contains image recognition for facial landmarks, fingers and body.

• Chordata

  Body pose capturing using 15 or 17 MARGS (accelerometer + gyroscope + magnetometer) attached to the body.

• The FreeMoCap Project

  Not used in makehuman.js (yet): Python project which bundles and extends various free motion capture tools.

Face

• https://en.wikipedia.org/wiki/Facial_Action_Coding_System

  For psychology. Emotions only. Not speech. Doesn't detail muscle movements.

• Facial Representations: Modeling, Rigging, Retargeting; Iain Mattherys; 2012-03-11; Disney Research, Pittsburgh PDF

• Siggraph 2021 3D Morphable Face Models - Past, Present and Future

  • https://github.com/3d-morphable-models/curated-list-of-awesome-3D-Morphable-Model-software-and-data

• MakeHuman uses a facial rig, which can be animated by combining 60 predefined poses (aka. pose units)

• Google Mediapipe / Apple ARKit / ...

  52 Blendshapes/Morph Targets

  • https://hinzka.hatenablog.com/entry/2021/12/21/222635#Blendshapes-LIST
  • https://arkit-face-blendshapes.com
  • https://github.com/ICT-VGL/ICT-FaceKit Blendshapes from scans. Here's how the data can look:
    • https://news.productioncrate.com/free-3d-facial-expression-blendshapes-pack/
    • Video: https://youtu.be/QAa2j-B2fa4?si=4d_xlBEOfO2FbOAg

Body

TBD

How does Makehuman work?

Morph

Data

• data/3dobjs/base.obj contains a 3d model of a human body, called the base mesh.

  It is completely made of quads, which well give good results when applying a Catmul and Clark subdivision to it.

  Further reading: Mesh Topology.

• data/target/ contains 1258 morph targets, which can deform the base mesh's shape, gender, age and ethnicity.

  The morph targets are handmade by editing the basemesh in a 3d editor and extracting the changes with MakeTarget.

• data/modifiers/ bundles those morph targets into 249 more user friendly modifiers

Code

// render's the morphed base meshfunctionrender(canvas: HTMLCanvasElement,scene: HumanMesh)// the morphed base meshclassHumanMesh{// input
obj: Mesh// the base mesh from the Wavefront Object
human: Human// the morph targets// processingupdate()// calculate vertex from obj and human// output
vertex: number[]// the morphed obj.vertex}// aggregates all the modifiers and creates a list of morph targetsclassHuman{// inputmodifiers: Map<string,Modifier>modifierGroups: Map<string,Modifier[]>// output targetsDetailStack: Map<string,number>// morph targets// for posing and skinning (see below)meshData!: WavefrontObj__skeleton!: Skeleton}// creates a list of ui elements (sliders, text fields) to edit the modifier valuesfunctionloadSliders(filename: string)

Pose

The skeleton aggregates bones and weights. Bones can be rotated.

Posing the skeleton directly, especially those in the face, can be a bit tedious. Hence there are pre-defined pose units, e.g. "LipsKiss" or "HandBendOutLeft", which can control multiple bones at once while also restricting bone movement.

Data

• Skeleton

  • data/rigs/default.mhskel the bones making up the skeleton

    For the actual bone positions little cubes within the mesh are referenced, so when the mesh is morphed, the skeleton is morphed along with it.

    Further reading: Base Mesh and Rig.

  • data/rigs/default_weights.mhw the weights each bone has on each vertex

• Pose Units

  • data/poseunits/body-poseunits.json defines 63 pose units for the body.

  • data/poseunits/face-poseunits.bvh and face-poseunits.json defines 60 pose units for the face.

  • data/expressions/*.mhpose defines 32 face expressions based upon pose units, e.g. like "laugh01" or "fear02".

Code

// aggregates the bone tree and weight listclassSkeleton{}// a single boneclassBone{parent?: Bonechildren: Bone[]=[]name: stringyvector4?: vec4// direction vector of this bone (along y-axis)matRestGlobal?: mat4// bone relative to world
...
// user defined rotationmatPose: mat4}// weightsclassVertexBoneWeights{// bone name -> [[vertex numbers, ...], [weight for vertex, ...]]_data: Map<string,Array<Array<number>>>}

Proxy

Proxies provide additional meshes, e.g. teeth, tounge, eyes, alternative body meshes and cloth.

The proxy files contain data which is used to transform the morphed/posed basemesh into a proxy mesh. These files are created with MakeClothes.

classProxy{// return proxy mesh vertices, adjusted to basemesh morph/posegetCoords(baseMeshVertices: number[]): number[]}

Development

# install packages
npm install
# populate data directory
npm run dev:prepare
# run (in separate terminals)
npm run dev:build
npm run dev:serve

Run single test

 npm run dev:test --file=build/test/skeleton/Skeleton.spec.js