[head posture] Wrong 3d world data when face is located on side

[remmel]

My 3d world face is desfigured when I'm located in the side of the camera (top; bottom; left; right).
FYI, the projected face texture is still correct (the one produced by mediapipe matches my camera projection on the mesh). Thus it's look like this is only the depth of the landmark which is incorrect (distance from the pinhole to that landmark).

Note that my focalLengh is set https://github.com/remmel/mediapipeDemos/blob/main/head_posture_static.py

Also, this is good to note that when I'm located in the center; the 3d world face is impressively good (less than ~1cm depth error! I'm using a Kinect v2 and compare them to its depth).

Gif of the correct face (center) and deformed face (left) - the left face should have the same shape than the center face :

https://sketchfab.com/3d-models/wrong-3d-world-coordinates-tech-issue-94c0d4f8c20a45089353dd9f3b6f8940 (I added the center face matching RGBD to also show you how good is the depth precision!).

Let me know what other information I could provide (create and share open3d snippet to display face; sketchab output with more faces...)

(For the record I compare https://techtee.medium.com/real-time-face-mesh-point-cloud-with-three-js-tensorflow-js-and-typescript-1f37ae844e1f which seems to output a face with good local coordinate but not https://remmel.github.io/FaceMeshFaceGeometry/examples/video/index.html - note that the 2nd webpage use a cropped image)

[remmel]

Note that when displaying the face in my webviewer using face_landmarks.landmark[idx].{x,y,z} the center face and side face have same shape (not deformed). Thus the problem seems to comes from the landmarks -> metric_landmarks transformation in get_metric_landmarks

[remmel]

FYI, this issue talks about the 3d world coordinates google-ai-edge/mediapipe#3362 (comment) especially https://github.com/andrechen/mediapipe/blob/ca7ddc37c77fa237576161b054c9f795d794a77f/visualstudio/face_effect/face_effect_renderer.cc#L110

[Rassibassi]

Hi,

I am not sure if I can help, this all was some time ago. When I build the facemesh algorithm in python and the landmarks -> metric_landmarks stuff, I actually copied all the cpp code from the original mediapipe into a file and tested my python version against it. It could be that the mediapipe cpp code has changes by now, though.

See this repository: https://github.com/Rassibassi/mediapipeFacegeometryPython

There is a main.cpp file and also a test_face_geometry.ipynb file which tests the cpp implementation against my python implementation.

Best,
Rasmus

[remmel]

Thanks Ramus for the answser. I'll have a look. The original inspired file seems to be the (older) geometry_pipeline.cc

[Rassibassi]

Yes, you're right

[binodil]

I have got same results when using facial__transformation_matrixes from mediapipe with solvePnP. Please look to the figures.

print("starting")
import numpy as np
import cv2
import mediapipe as mp
from mediapipe.tasks.python import vision
from mediapipe.tasks import python

img_path = "p_z_60_y_0_x__30.jpg"
frame = cv2.imread(img_path)
frame_rgb = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
base_options = python.BaseOptions("../WebEyeTrack/webeyetrack/python/webeyetrack/model_weights/face_landmarker_v2_with_blendshapes.task")
landmarker = vision.FaceLandmarker.create_from_options(vision.FaceLandmarkerOptions(base_options, output_face_blendshapes=True, output_facial_transformation_matrixes=True, num_faces=1))
mp_image = mp.Image(image_format=mp.ImageFormat.SRGB, data=frame_rgb)

detection_result = landmarker.detect(mp_image)
face_landmarks = np.array([[lm.x, lm.y, lm.z, lm.visibility, lm.presence] for lm in detection_result.face_landmarks[0]])
face_rt = detection_result.facial_transformation_matrixes[0]
LANDMARKS = np.load("../yhyst_face_model_cm.npy")
face  = LANDMARKS     # (478, 3)
hR = face_rt[:3, :3].T  # (3, 3)
ht = face_rt[:3, 3:].T  * [1, -1, -1] # (1, 3)
Fc = face @ hR + ht

vect = np.array([0., 0, -1.]) @ hR #+ ht
np.save(f"{img_path[:-4]}_mp.npy", Fc)
np.save(f"{img_path[:-4]}_mp_vect.npy", vect)
print(vect)
#LANDMARKS = LANDMARKS @ np.array([[-1, 0, 0], [0, -1, 0,], [0, 0, -1]])
face_model = LANDMARKS.reshape(468, 1, 3)

####### USING cv2.solvePnP ########
reshaped_landmarks = face_landmarks[:, :2] * np.array([frame.shape[1], frame.shape[0]])
reshaped_landmarks = reshaped_landmarks.reshape(478, 1, 2)
from camera_calib import camera_matrix, dist
camera = camera_matrix  # my camera fx, fy, cx, cy
distortion = dist  # my camera distortion
ret, rvec, tvec = cv2.solvePnP(face_model, reshaped_landmarks[:468, :], camera, distortion, flags=cv2.SOLVEPNP_EPNP)
ret, rvec, tvec = cv2.solvePnP(face_model, reshaped_landmarks[:468, :], camera, distortion, rvec, tvec, True)
hR = cv2.Rodrigues(rvec)[0]
ht = tvec.T  # 3, 1
Fc2 = face@hR + ht
vect = np.array([0., 0, -1.]) @ hR
np.save(f"{img_path[:-4]}_solvePnP.npy", Fc2)
np.save(f"{img_path[:-4]}_solvePnP_vect.npy", vect)
#### it is the same i think use from the camera information
# there are 100 rvecs and tvecs given in the camera.mat file from MPIIFaceGaze