Add a VST3 host filter to obs-studio

[pkviet]

Description

This PR implements a VST3 host for obs-studio, limited to audio effects (MIDI & instruments are excluded; they would require that obs supports MIDI natively which is not (yet?) the case).
Sidechain is supported.
Only one main input bus and one main output bus are supported.

This follows the announcement by Steinberg that VST3 SDK is from v3.8.0 licensed under a dual license (MIT or proprietary).
Clarification was also done by Steinberg on request of obs-studio project about logo usage which used to be mandatory and was in contradiction to GPL v3.
Ref: https://www.steinberg.net/developers/vstsdk/

VST3 has been written for windows, macOS and linux X11 platforms.
Note that 3.8.0 adds also a preview Wayland support (contributed by Presonus).
Ref: https://steinbergmedia.github.io/vst3_dev_portal/pages/Versions/Version+3.8.0.html
Wayland support will be added later.

Screenshots:

VST3 with sidechain

Motivation and Context

VST3 are the current standard for audio filters so they are a must have for any tool doing audio processing.
It's been a feature often requested by users.

How Has This Been Tested?

The following tests have been done on windows 11 23H2, macOS 15, linux Ubuntu 24.04 (X11 with proprietary nvidia drivers).

✅ Filter Lifecycle and Scene Integration

Test Case	Description	Pass/Fail	Notes
Swap filters	Swap filters seamlessly (no crashes)	Pass
Add/Remove During Streaming	Add/remove VST3 filter during stream/recording, no crash	Pass
Enable/Disable Rapidly	Toggle filter on/off quickly, verify plugin state consistency	Pass
Undo/Redo Filter Action	Undo/redo adding/removing filter, no leaks or hangs	Pass
Scene Activation Behavior	Filter activates only when scene becomes active	Pass
No leaks on exit	Exit OBS	Pass
No crashes on exit	Exit OBS	Pass

---

✅ Plugin Compatibility Tests

Test Case	Description	Plugin tested	Pass/Fail	Notes
EQ Plugin	Load EQ plugin and check real-time parameter update	TDR Nova	Pass
Compressor with Sidechain	Check gain reduction based on sidechain input	RoughRider 3	Pass
Reverb	Plugin with long tail, test render stability	BC Chorus, ValhallaSuperMassive	Pass
Delay Plugin	Check delay feedback and sync accuracy	ValhallaSuperMassive	Pass
Distortion Plugin	Ensure clipping/distortion plugin does not crash	IVGI	Pass
GUI-only Plugin	Ensure plugin loads and filters audio even without processing	BC FreqANALYST, TDR Prism,	Pass
GUI-only Plugin	Ensure plugin loads and filters audio even without processing	Voxengo Span	Pass
Linear Phase EQ	Check that plugin initializes with proper latency reporting	Voxengo Marvel GEQ	Pass
Linux Studio Plugins	Check that plugin bundle enumerates all plugins and works	Compressor etc	Pass

---

✅ Sidechain and Source Enumeration

Test Case	Description	Pass/Fail	Notes
Source List Correctness	Only audio sources are listed as sidechain candidates	Pass
Source List Updates	Add/remove audio sources dynamically, check list refresh	Pass
Source Selection Persistence	Restart OBS, sidechain selection should persist	Pass
Sidechain Audio Matching	Inject tone into sidechain, confirm plugin envelope triggers	Pass
Sidechain Removal	Delete active sidechain source, verify graceful degradation	Pass
Mono Sidechain	Test with a Mono Sidechain like RoughRider 3	Pass
Stereo Sidechain	Test with a Stereo Sidechain like Melda MCompressor	Pass

---

✅ Stress and Performance Tests

Test Case	Description	Pass/Fail	Notes
Heavy Plugin + High CPU Load	Load demanding plugin under system stress	Pass
Multiple Filter Instances	Add same plugin to multiple sources simultaneously	Pass
Add several VST3s to same source	Add several VST3s to one source, change order, Enable/Disable	Pass
Test with Steinberg hostchecker	Load VST3 host checker	Pass

---

✅ Error Handling & Logging

Test Case	Description	Pass/Fail	Notes
Plugin Load Failure	Attempt to load a corrupt/nonexistent plugin, check error log	Pass

---

✅ Multiplatform support

Test Case	Description	Pass/Fail	Notes
Windows 11	Make all tests on Windows 11	Pass
macOS	Make all tests on macOS	Pass
Ubuntu	Make all tests on Ubuntu linux	Pass	Tested LSP vst3

Types of changes

• New feature (non-breaking change which adds functionality)

Checklist:

• My code has been run through clang-format.
• I have read the contributing document.
• My code is not on the master branch.
• The code has been tested.
• All commit messages are properly formatted and commits squashed where appropriate.
• I have included updates to all appropriate documentation.

[PatTheMav]

GitHub's new PR view doesn't seem to like the immensity of this PR.. 😄

But as a first action, the VST3 SDK should not be vendored-in, it should be added to obs-deps instead (where it's then pre-built as a library) and a CMake find module added.

Also as this is a "fresh" plugin, we should do things the right way:

• Use platform-specific implementation files and platform-independent header files.
  • This allows the API(s) to be generic and called in a generic fashion and then the implementation of that generic function does the platform specific thing
  • E.g. there should be an implementation of getDefaultSearchPaths in a file called VST3Scanner_Windows.cpp, VST3Scanner_macOS.cpp, VST3Scanner_Linux.cpp, etc. implementing the general definition of the same function.
• Each C++ class has to be put into its own pair of header and implementation file. This avoids unnecessary recompilations of every class just because a single detail of one class is changed in VST3Plugin.h
  • This also makes reviews and maintenance easier, as everything is properly encapsulated and separation of concerns is replicated by separation of implementation.
• Split up implementation of the libobs source API and then libobs properties API into separate files (see mac-avcapture as an example). In general opt for shorter source files that implement a single aspect of the API instead of combining everything into a single "obs-whatever.c" file. In general I find it better to have a plugin be split up into these parts:
  • Implementation of the module API
  • Implementation(s) of a specific "source" API, one for each type (source, filter, output, etc.)
  • Implementation(s) of the property API for each source type
  • Implementation of the actual self-contained object that manages resources (for anything but a pure C plugin that would mean the C++/ObjC/Swift instance of an object)
  • Again, see mac-avcapture or this Swift-based example
• Every time a source file has a long "line separating comment", it means the file is too long and contains too much - split it up per the comment above.
• Do not use _prefix underscores for "private" variables in C++, those names are reserved by the standard library. Always use postfix_ underscores.
• Use our modern header include order (see the updated files in frontend for examples`:
  1. #include "working_directory_file.h" (file in the same directory as the one doing the inclusion)
  2. #include <first_party_file.h> (file provided by some other part of the project or a first-party shared library)
  3. #include <third_party/file.h> (file provided by a third party library)
  4. #include <standard_library.h> (file provided by the C or C++ standard library, with C++ includes coming first)
• Also use #import by default for ObjC/ObjC++

Those are just the things I saw at first glance, will look into the implementation in-depth after those things are addressed.

[pkviet]

The Steinberg sdk is not conceived as a precompiled library but as a set of interfaces and optional helper classes which each plugin or host implements selectively.
If you look at open source vst3 hosts (juce, ardour) or plugins (linux studio plugins), they all include the sdk for that reason.
In our case I included only relevant parts of the sdk , not the whole of it, useful for a host (and it is already a mouthful indeed).
So although it could be made into a static library, the SDK doesn't do so itself, it was not thought to provide one.
Note that it is not header only in my case because some of the base sdk headers and some helpers headers have accompanying cpp implementations. So it is not a case where a single header needs inclusion from obs-deps. Maybe there is some magic from cmake which would allow it to work ?

[PatTheMav]

In that case we'd just omit the compilation step, but it should still be provided as a "header-only" library via obs-deps (just like SIMDe and other third party dependencies).

Despite the name, as long as the target provided by the find module provides header files and source files as its INTERFACE_SOURCES property, they will be added to a target linking it just as if they were part of the source tree (but without the baggage of actually having to manage them in the same tree).

[pkviet]

In that case we'd just omit the compilation step, but it should still be provided as a "header-only" library via obs-deps (just like SIMDe and other third party dependencies).

Despite the name, as long as the target provided by the find module provides header files and source files as its INTERFACE_SOURCES property, they will be added to a target linking it just as if they were part of the source tree (but without the baggage of actually having to manage them in the same tree).

Ah, that was the info i was missing. If it can be done through cmake, that's nice.

[PatTheMav]

In that case we'd just omit the compilation step, but it should still be provided as a "header-only" library via obs-deps (just like SIMDe and other third party dependencies).
Despite the name, as long as the target provided by the find module provides header files and source files as its INTERFACE_SOURCES property, they will be added to a target linking it just as if they were part of the source tree (but without the baggage of actually having to manage them in the same tree).

Ah, that was the info i was missing. If it can be done through cmake, that's nice.

Note that this is only necessary if you want the SDK source files to also appear with your plugin sources in the IDE. Usually you only provide the include path and just include the SDK's files where necessary.

But you can of course add them as sources and then in your target you can set the "FOLDER" property for all these files to put them visually in a subdirectory. Here's an example from the frontend project that looks at all the SOURCES on the project and then filters out headers, sources, and tells CMake to create directories based on their location relative to the main source directory:

obs-studio/frontend/CMakeLists.txt

Lines 105 to 117 in c025f21

	get_target_property(target_sources obs-studio SOURCES)
	set(target_cpp_sources ${target_sources})
	set(target_hpp_sources ${target_sources})
	set(target_qt_sources ${target_sources})
	list(FILTER target_cpp_sources INCLUDE REGEX ".+\\.(cpp|mm|c|m)")
	list(SORT target_cpp_sources COMPARE NATURAL CASE SENSITIVE ORDER ASCENDING)
	list(FILTER target_hpp_sources INCLUDE REGEX ".+\\.(hpp|h)")
	list(SORT target_hpp_sources COMPARE NATURAL CASE SENSITIVE ORDER ASCENDING)
	list(FILTER target_qt_sources INCLUDE REGEX ".+\\.(ui|qrc)")
	list(SORT target_qt_sources COMPARE NATURAL CASE SENSITIVE ORDER ASCENDING)
	source_group(TREE "${CMAKE_CURRENT_SOURCE_DIR}" PREFIX "Source Files" FILES ${target_cpp_sources})
	source_group(TREE "${CMAKE_CURRENT_SOURCE_DIR}" PREFIX "Header Files" FILES ${target_hpp_sources})
	source_group(TREE "${CMAKE_CURRENT_SOURCE_DIR}" PREFIX "Qt Files" FILES ${target_qt_sources})

Haven't checked whether SOURCES will implicitly include the INTERFACE_SOURCES of linked libraries, but that's how I'd approach it, if this is desired.

[pkviet]

The devil's in the details.
Your suggestion to move the sdk to obs-deps forgets about linux and that a patch is required for compatibility with Linux Studio Plugins (a similar patch is done by JUCE and Ardour). obs-deps doesn't cover linux while the plugin works on all our platforms (except flatpak because our flatpak doesn't allow /usr nor $HOME locations while they are the ones used by vst3s).
Since the vst3 sdk is on github, we could pull it as a submodule, from a fork under the obsproject umbrella or directly from origin.
We only use a subset though of the vst3 sdk which I patched, one of the patches being to allow compatibility with Linux Studio Plugins.
I found on launchpad a vst3sdk package but it is out of date (version 3.7.10 from January 24).
It is difficult to work around a workflow which has been thought of as having the sdk in-tree.

[PatTheMav]

The devil's in the details. Your suggestion to move the sdk to obs-deps forgets about linux and that a patch is required for compatibility with Linux Studio Plugins (a similar patch is done by JUCE and Ardour).

What does the patch do? If it's necessary for Linux compatibility to begin with it should be upstreamed to the SDK repo and implemented by Steinberg then.

We are in no hurry to implement VST3 support in the project, and it's not OBS Studio's responsibility to contort itself to somehow "make it work". We need to aim for zero band-aids, in-tree patches, or other workarounds, as those require the most maintenance work and have the potential to break at even the smallest upstream change.

[pkviet]

One of the patches could be upstreamed (zeroing of the buffers on creation).
The other is not upstreamable. It ensures compatibility with some plugins which use messages but not on main ui thread. The sdk requires ui thread but for instance JUCE and Ardour don't enforce that prescription. Maybe this could ne negotiated with Steinberg to not be compulsory as current usage shows it is not enforced.
Anyway, with or without the patches, the linux case is not solved by moving the vst3 sdk to obs-deps.
So I don't see any alternative to in-tree sdk.

[PatTheMav]

So what this reads to me is that the VST3 SDK is not really an SDK in the common sense, but more a "VST template": You take their source code and then write/change your own stuff around it, throw away the pieces you don't need and keep the ones you do?

[pkviet]

So what this reads to me is that the VST3 SDK is not really an SDK in the common sense, but more a "VST template": You take their source code and then write/change your own stuff around it, throw away the pieces you don't need and keep the ones you do?

It is both.
In order to implement either a host or a plugin, it would in principle be enough to use the classes defined in :

• base,
• interfaces.
  The public.sdk subfolder has however implementations of these interfaces. Sometimes it is not useful to reinvent the wheel and some other times it is. Hosts and plugins are free to use or not these implementations. Only what is in 'base' and 'interfaces' are compulsory.
  For the patches, since we have contacts with Steinberg, I'll discuss directly with them to see the potential for upstreaming.

Going forward i see these options:

• we drop support for linux , which allows us to pull the sdk from obs-deps. (Anyway support for flatpak is not possible.)
• we keep support for linux. But then either we leave the responsibility to users to pull the sdk and compile themselves vst3 plugin support or we find a solution which is not obs-deps. Like pulling a submodule.

From talks with @Fenrirthviti it was conveyed to me that linux support was desired but given that sdk difficulty, it seems a new arbitration is required.

[PatTheMav]

I think the main issue with Linux seems to me that architecturally there is no need to provide a system package? Because it sounds like every VST is effectively compiled "on top" of the SDK code, so the SDK becomes the VST.

And because of that, no VST would ever need to dynamically link to a system-wide VST library.

I'll need to marinate on that.

@tytan652 what's your take? If we need to carry a dependency around in-tree (despite all our efforts to stop doing that) because Linux needs it and thus cannot be cordoned off to obs-deps, where should we put it?

Using obs-deps for Linux would probably only work for header-only libraries (as those would be distro-independent), but then we could also potentially make the CEF setup work the same way it does on Windows and macOS?

[pkviet]

And because of that, no VST would ever need to dynamically link to a system-wide VST library.
Correct. All vst3s and hosts are compiled with the sdk. There is no lib in system to which there is a link.
This is true for all platforms supported by the vst3 sdk.

[tytan652]

what's your take? If we need to carry a dependency around in-tree (despite all our efforts to stop doing that) because Linux needs it and thus cannot be cordoned off to obs-deps, where should we put it?

VST3 assumes two system path:

• /usr/lib/vst3
• /usr/local/lib/vst3

According to those paths:

• Does VST3 support Ubuntu ?
  • No, binaries in the lib folder needs to be in the corresponding triplets subfolder so /usr/lib/x86_64-linux-gnu/vst3.
• Does VST3 support Flatpak ?
  • No since we are not able to point the correct extension path through an environment variable like we did for VST2. And we can't use the APPFOLDER environment variable with a custom path unless the vst is loaded in a isolated process since user should not be able to mess with the variable.
• Which distro VST3 actually support ?
  • A distro where /usr/lib is used directly to store binaries matching the installation architecture, so Arch Linux and Gentoo… (I wish I was joking)
• But $HOME/.vst3 ?
  • Only non-sandboxed packages allows it since there is no risk of escaping the sandbox and does not mean that the distro is properly supported.

None of our official packages are correctly supported.
When it comes to adding features, I heavily prefer if the Flatpak has support since it has the biggest coverage (most distros including immutable).

Throwing the VST3 SDK in obs-deps-buildstream is not inconceivable, same to the PPA (it has already its own build of libajantv2).
But in-tree should definitively be a no-go in my opinion.

PS: For the sake of sanity and simplicity I do not take in account unofficial packaging, it will be on the concerned packager⋅s.

[pkviet]

When it comes to adding features, I heavily prefer if the Flatpak has support since it has the biggest coverage (most distros including immutable).

Throwing the VST3 SDK in obs-deps-buildstream is not inconceivable, same to the PPA (it has already its own build of libajantv2).
But in-tree should definitively be a no-go in my opinion.

This bars complete linux support (which I really don't mind ;) ).
Because even if you throw the vst3 sdk in buildstream, this doesn't solve the fact that currently the vst3 ecosystem wants install in locations which are not accessible to flatpak, as you summarized.
Imo flatpak is a flat no in terms of support of vst3 , at least currently. To change that, it would require Steinberg to allow alternate locations than /usr or $HOME , and then plugin authors to allow these new locations. So in the short term, it is illusory to hope to get flatpak working with vst3.

This leaves only linux support away from flatpak, say ubuntu. This means either one modifies obs-deps to accommodate vst3 sdk for linux or we pull it in deps folder as a submodule or in-tree in the obs-vst3 folder.

Maybe the simplest is to plainly remove support for linux. I have no preference tbh.

[pkviet]

VST3 assumes two system path:

• /usr/lib/vst3
• /usr/local/lib/vst3

According to those paths:

• Does VST3 support Ubuntu ?
  • No, binaries in the lib folder needs to be in the corresponding triplets subfolder so /usr/lib/x86_64-linux-gnu/vst3.

No this is incorrect.
You are assuming the host scans the system lib folders.
The host scans the above mentioned folders in addition to $HOME/.vst3 where vst3 plugins get installed.
So ubuntu can definitely be supported.
And actually i compiled the plugin and tested it on ubuntu 24.04 on several linux vst3s (LSP, RoughRider 3 ...) and also windows vst3s through wine and yabridge.