Removing a sounding object from picture and sound — after filtering out speech

Twelve results from an agentic LangGraph pipeline run over source clips that a speech filter had already cleared of dialogue. The pipeline captions each clip, decides what to remove, segments it, erases it from the video, verifies the erasure by re-segmenting, then separates its sound out of the mix. Play with sound on — the point of each pair is what you stop hearing while everything else stays.

101,503 clips screened for speech
31,845 no-speech clips kept
846 clips processed · 12 shown
200 of 239 passes verified two-sided

Why the speech filter mattered

An earlier run of the same pipeline on unfiltered sources kept choosing people as the thing to remove — 25 of 31 targets were a man, a woman or a person — because both source corpora are human-centric and a talking person is the loudest thing in frame. Removing a talking person is speech removal, not object-sound removal.

Screening the sources first with FireRedASR2S (VAD, then ASR confirmation on every positive) and an AudioSet tagger cut 101,503 clips to 31,845 with no speech but a real sounding object. Run on those, the pipeline's own autonomous choices changed completely:

target chosen autonomouslybefore filteringafter filtering
a person (man / woman / person)25 of 31 — 81%144 of 200 — 72%
an animal or object6 of 31 — 19%56 of 200 — 28%

At 846 clips the picture is steadier than it looked at 36: people are still the plurality, but the tail of real sounding objects is now substantial and varied — typewriter keys, a food processor, a military helicopter, an engine, a car door, a volcano, a lioness, a bear cub. None of those appeared in the unfiltered run. Nothing about the pipeline changed; only its input did.

typewriter keys insave_add_and_remove_09458
1280×704 · 23.98 fps · 3.42s

SOURCE

source frames
source spectrogram
source audio

RESULT · typewriter keys removed from picture and sound

result frames
result spectrogram
after removal — the rest of the mix survives
waveforms
source vs object removed
16%mask area
1.000visual removal score
71%of the mix pulled into the removed object
63%of the mix left behind
visual · 10/10winning prompt · candidates passing both tests
military helicopteraudio re-selected insave_general_editing_43315
1280×704 · 23.98 fps · 5.09s

SOURCE

source frames
source spectrogram
source audio

RESULT · military helicopter removed from picture and sound

result frames
result spectrogram
after removal — the rest of the mix survives
waveforms
source vs object removed
16%mask area
1.000visual removal score
86%of the mix pulled into the removed object
49%of the mix left behind
text · 4/10winning prompt · candidates passing both tests
food processor insave_add_and_remove_08918
1280×704 · 23.98 fps · 6.71s

SOURCE

source frames
source spectrogram
source audio

RESULT · food processor removed from picture and sound

result frames
result spectrogram
after removal — the rest of the mix survives
waveforms
source vs object removed
26%mask area
1.000visual removal score
81%of the mix pulled into the removed object
46%of the mix left behind
visual · 3/10winning prompt · candidates passing both tests
German Shepherd dog insave_add_and_remove_06295
1280×704 · 23.98 fps · 6.71s

SOURCE

source frames
source spectrogram
source audio

RESULT · German Shepherd dog removed from picture and sound

result frames
result spectrogram
after removal — the rest of the mix survives
waveforms
source vs object removed
19%mask area
1.000visual removal score
84%of the mix pulled into the removed object
47%of the mix left behind
text · 7/10winning prompt · candidates passing both tests
small white kitten insave_add_and_remove_05262
1280×704 · 23.98 fps · 3.42s

SOURCE

source frames
source spectrogram
source audio

RESULT · small white kitten removed from picture and sound

result frames
result spectrogram
after removal — the rest of the mix survives
waveforms
source vs object removed
20%mask area
1.000visual removal score
84%of the mix pulled into the removed object
55%of the mix left behind
visual · 5/10winning prompt · candidates passing both tests
car dooraudio re-selected insave_general_editing_43833
1280×704 · 23.98 fps · 6.71s

SOURCE

source frames
source spectrogram
source audio

RESULT · car door removed from picture and sound

result frames
result spectrogram
after removal — the rest of the mix survives
waveforms
source vs object removed
32%mask area
0.987visual removal score
83%of the mix pulled into the removed object
53%of the mix left behind
visual · 1/10winning prompt · candidates passing both tests
lioness insave_add_and_remove_09088
1280×704 · 24 fps · 5.04s

SOURCE

source frames
source spectrogram
source audio

RESULT · lioness removed from picture and sound

result frames
result spectrogram
after removal — the rest of the mix survives
waveforms
source vs object removed
35%mask area
1.000visual removal score
68%of the mix pulled into the removed object
73%of the mix left behind
visual · 8/10winning prompt · candidates passing both tests
horse insave_add_and_remove_05116
1280×704 · 23.98 fps · 3.42s

SOURCE

source frames
source spectrogram
source audio

RESULT · horse removed from picture and sound

result frames
result spectrogram
after removal — the rest of the mix survives
waveforms
source vs object removed
24%mask area
1.000visual removal score
77%of the mix pulled into the removed object
62%of the mix left behind
text · 3/10winning prompt · candidates passing both tests
volcano insave_add_and_remove_05305
1280×704 · 23.98 fps · 3.42s

SOURCE

source frames
source spectrogram
source audio

RESULT · volcano removed from picture and sound

result frames
result spectrogram
after removal — the rest of the mix survives
waveforms
source vs object removed
40%mask area
1.000visual removal score
74%of the mix pulled into the removed object
65%of the mix left behind
text · 6/10winning prompt · candidates passing both tests
engine insave_add_and_remove_06189
1280×704 · 23.98 fps · 5.09s

SOURCE

source frames
source spectrogram
source audio

RESULT · engine removed from picture and sound

result frames
result spectrogram
after removal — the rest of the mix survives
waveforms
source vs object removed
50%mask area
1.000visual removal score
74%of the mix pulled into the removed object
44%of the mix left behind
text · 9/10winning prompt · candidates passing both tests
bear cub insave_general_editing_47433
1280×704 · 23.98 fps · 5.09s

SOURCE

source frames
source spectrogram
source audio

RESULT · bear cub removed from picture and sound

result frames
result spectrogram
after removal — the rest of the mix survives
waveforms
source vs object removed
26%mask area
1.000visual removal score
78%of the mix pulled into the removed object
68%of the mix left behind
visual · 2/10winning prompt · candidates passing both tests
cow insave_add_and_remove_05373
1280×704 · 23.98 fps · 5.09s

SOURCE

source frames
source spectrogram
source audio

RESULT · cow removed from picture and sound

result frames
result spectrogram
after removal — the rest of the mix survives
waveforms
source vs object removed
59%mask area
0.997visual removal score
82%of the mix pulled into the removed object
48%of the mix left behind
text · 1/10winning prompt · candidates passing both tests

How “removed” is verified — and a metric I had backwards

SAM-Audio splits the soundtrack into a target (the object's sound) and a residual (everything else). The residual is what gets muxed into the result video. The pipeline picks among ten candidates by ImageBind similarity between the prompt and the target — a score that is maximised by dumping the whole mix into the target, leaving a silent residual. About one selection in four collapsed that way.

My first correction only guarded that end: it required the residual to survive. That let through the opposite failure, and an earlier version of this page led with three of them. A separation that extracts nothing leaves the residual at 98% of the source — which a residual-only metric scores as the best possible result, when in fact the object was never removed. The barred owl pulled 3% of the mix into the target and left 98% behind; the goose, 0%.

A usable separation has to fail in neither direction, so the test is now three-sided: the residual keeps at least 30% of the source (the rest of the scene survives), the target holds at least 20% (something was actually taken out), and the target matches the prompt better than the residual does (ib_ta > ib_ta_res — otherwise the object's sound stayed in the leftovers). Both numbers are shown on every card above.

Against that test, 200 of the 239 clips that passed the pipeline's own gates hold up (84%), and 43 do not. The twelve here are drawn from the 56 whose target is a genuine non-human sounding object, chosen for variety and for extraction strength.