Languages

Guiding Audio Source Separation by Video Object Information

Companion web page for the WASPAA 2017 paper

Sanjeel Parekh, Slim Essid, Alexey Ozerov, Ngoc Q. K. Duong, Patrick Pérez, Gaël Richard

Some separation results computed using proposed methods:

  • LS: NMF + Non-negative least squares (NNLS)
  • spLS: NMF + Sparse NNLS
  • JLS Rand: Joint NMF and sparse NNLS with random initialization
  • JLS NMF: Joint NMF and sparse NNLS with W and H initialized using audio mixture NMF output
To play the audio click on the link between square brackets [].
More examples will be uploaded in the coming days, stay tuned!
Please see the paper for more details.

The particularly challenging and interesting case of same instrument mixtures has been highlighted in blue below. Motion information plays a crucial role for such cases.

EEP MoCap Dataset 

Examples Separation Method Estimated Sources
P1: violin + cello [Mixture] LS [violin] [cello]
spLS [violin] [cello]
JLS Rand [violin] [cello]
JLS NMF [violin] [cello]
P1: violin + viola [Mixture] LS [violin] [viola]
spLS [violin] [viola]
JLS Rand [violin] [viola]
JLS NMF [violin] [viola]
violin (P1) + violin (P2) [Mixture] LS [violin P1] [violin P2]
spLS [violin P1] [violin P2]
JLS Rand [violin P1] [violin P2]
JLS NMF [violin P1] [violin P2]
P1: violin + viola +cello [Mixture] LS [violin] [viola] [cello]
spLS [violin] [viola] [cello]
JLS Rand [violin] [viola] [cello]
JLS NMF [violin] [viola] [cello]

 

URMP Video Set 

Examples Separation Method Estimated Sources
violin 1 + violin 2 [Mixture] LS [violin 1] [violin 2]
spLS [violin 1] [violin 2]
JLS Rand [violin 1] [violin 2]
JLS NMF [violin 1] [violin 2]
violin 2 + viola [Mixture] LS [violin 2] [viola]
spLS [violin 2] [viola]
JLS Rand [violin 2] [viola]
JLS NMF [violin 2] [viola]
violin 1 + viola + cello [Mixture] LS [violin 1] [viola] [cello]
spLS [violin 1] [viola] [cello]
JLS Rand [violin 1] [viola] [cello]
JLS NMF [violin 1] [viola] [cello]
violin 1 + violin 2 + viola + cello [Mixture] LS [violin 1] [violin 2] [viola] [cello]
spLS [violin 1] [violin 2] [viola] [cello]
JLS Rand [violin 1] [violin 2] [viola] [cello]
JLS NMF [violin 1] [violin 2] [viola] [cello]

Comments are closed.