Mesh Audio Editor

Real-time mesh viewer and editor supporting conversion of meshes to rigid body audio models using Linear Modal Analysis/Synthesis.

Features

General features:

• Create/delete meshes and mesh instances
  • Editable mesh primitives (Rect, Circle, Cube, IcoSphere, UVSphere, Torus, Cylinder, Cone)
  • Load .obj and .ply mesh files (via tinyobjloader and tinyply)
• Select meshes or mesh elements (vertices, edges, or faces)
• Flat/smooth/wireframe mesh rendering
• Bounding Volume Hierarchy (BVH) ray intersection acceleration structure
  • Used for mouse interactions - this is not a ray tracing renderer :)
• Translate / rotate / (nonuniformly) scale meshes and instances with numeric inputs or a custom transform gizmo designed to look and behave just like Blender's
• Simple camera + scene lighting model, roughly matching Blender visually
• Edit camera with mouse wheel, numeric inputs, or with a custom orientation gizmo designed to look and behave just like Blender's
• Edit lighting parameters
• Render face/vertex normals as lines for debugging
• Render bounding box / BVH wireframes for debugging
• Edge-detection-based silhouette outline of active mesh/instance, embedded into the scene with accurate per-pixel depth
• Fast infinite grid with horizon fade

Audio-specific features:

• Enable/disable audio output and change device and native format/sample rate
• Volume / Mute
• Generate an efficient physical audio model for any mesh. (See Physical audio modeling.)
  • Click on an audio mesh to excite the nearest vertex, or trigger a selected vertex in the audio menu
  • Edit model DSP params in real-time (via an embedded ImGui Faust parameter editor)
  • View/navigate the generated audio graph (via an embedded Faust SVG diagram navigator)
• Load RealImpact datasets, including the object mesh and instanced cylinders for each microphone position.

Noteworthy dev bits:

• Terse and direct usage of Vulkan-Hpp
• Shader hot reloading: Edit shader code and recompile/reload GLSL->SPIRV at runtime in the UI
• Resource reflection: Use SPIRV-Cross to automatically create descriptor set layout bindings for all shader pipelines
• Instanced rendering of shared geometry with variable transforms

Physical audio modeling

This project supports generating an efficient physical audio model for any mesh using Linear Modal Analysis/Synthesis

The physical audio modeling components were implemented as a final project for PHYS-6260 - Computational Physics at Georgia Tech during my Master's, and this is the final report. Here is the final report paper.

And here is a 36X48 poster:

Impact audio experiments

Below are audio examples synthesized by "striking" modal audio models (by injecting a short wideband pulse at the selected vertex) for various meshes, with comparisons to impact recordings of their real-world counterparts being struck at the same position. The audio recordings and 3D-scanned meshes come from the RealImpact dataset. See the blog post for embedded audio players.

The cylinders shown in the images represent recorded microphone positions, but all recordings come from a single microphone centered near the impacted object, and the modal audio model does not implement any audio wave radiation modeling. All modal audio samples are generated by extracting estimated surface vibrations, as if recorded from a contact microphone.

Object Name	Mesh	Real Impact Audio	Modal Impact Audio
Ceramic Koi Bowl		Impact	Modal
Ceramic Pitcher		Impact	Modal
Glass Cup		Impact	Modal
Iron Mortar		Impact	Modal
Iron Skillet		Impact	Modal
Plastic Scoop		Impact	Modal
Small Swan Ceramic		Impact	Modal

Build & run

Install dependencies

• Download and install the latest SDK from https://vulkan.lunarg.com/sdk/home
• Set the VULKAN_SDK environment variable. For example, add the following to your .zshrc file:

  export VULKAN_SDK="$HOME/VulkanSDK/{version}/macOS"

Mac

$ git clone --recursive [email protected]:khiner/MeshEditor.git
$ brew install cmake pkgconfig llvm sdl3 fftw eigen
$ brew link llvm --force

Linux

(Only tested on Ubuntu, and it's been awhile, so I'd honestly be suprised if it works. If you want a Linux build, I'd be happy to get this working!)

$ sudo apt install llvm sld3 libeigen3-dev
$ export PATH="$(llvm-config --bindir):$PATH"

Install GTK (for native file dialogs):

$ sudo apt install build-essential libgtk-3-dev

Clone, clean, build, and run

$ git clone --recurse-submodules [email protected]:khiner/MeshEditor.git
$ cd MeshEditor
$ ./script/Clean # optionally clean first
$ ./script/Build [--release]
$ cd build && ./MeshEditor

Stack

• Vulkan + ImGui + SDL3: Graphics + immediate-mode UI/UX
• glm: Small numeric vector/matrix types + math
• entt: Entity Component System (ECS) for an efficient and scalable mixin-style architectural pattern
• Faust: Functional audio programming language, used to render modal audio models to audio graphs
• miniaudio: Audio stream I/O
• tetgen: Fast conversion of triangular 3D surface meshes into tetrahedral volume meshes
• Spectra Estimate eigenvalues/vectors for modal analysis
• VulkanMemoryAllocator: Efficient Vulkan memory allocation
• nativefiledialog-extended: Native file dialogs (TODO SDL3 now has SDL_Dialog)
• ImPlot: Plotting
• fftw for computing spectrograms (visualized with ImPlot)
• tinyobjloader and tinyply: Load .obj and .ply mesh files
• lunasvg: Render Faust SVGs to bitmaps, parse SVG link bounding boxes, render SVG icons

Development

Update submodules

All submodules are in the lib directory. Currently, no submodules are forked. Here is my process for updating to the tip of all the submodule branches:

$ git submodule update --remote lib/{submodule}
$ git add .
$ git cm -m "Update {submodule} ..."