To Encourage or to Restrict: the Label Dependency in Multi-Label Learning

Overview

Abstract

Multi-label learning addresses the problem that one instance can be associated with multiple labels simultaneously. More or less, these labels are usually dependent on each other in different ways. Understanding and exploiting the Label Dependency (LD) is well-accepted as the key to build high-performance multi-label classifiers, i.e., classifiers having abilities including but not limited to generalizing well on clean data and being robust under evasion attack. 

From the perspective of generalization on clean data, previous works have proved the advantage of exploiting LD in multi-label classification. To further verify the positive role of LD in multi-label classification and address previous limitations, we originally propose an approach named Prototypical Networks for Multi- Label Learning (PNML). Specially, PNML addresses multi-label classification from the angle of estimating the positive and negative class distribution of each label in a shared nonlinear embedding space. PNML achieves the State-Of-The-Art (SOTA) classification performance on clean data.

From the perspective of robustness under evasion attack, as a pioneer, we firstly define the attackability of an multi-label classifier as the expected maximum number of flipped decision outputs by injecting budgeted perturbations to the data’s feature distribution. Denote the attackability of a multi-label classifier as C∗, and the empirical evaluation of C∗ is an NP-hard problem. We thus develop a method named Greedy Attack Space Exploration (GASE) to estimate C∗ efficiently. More interestingly, we derive an information-theoretic upper bound for the adversarial risk faced by multi-label classifiers. The bound unveils the key factors determining the attackability of multi-label classifiers and points out the negative role of LD in multi-label classifiers’ adversarial robustness, i.e. LD helps the transfer of attack across labels, which makes multi-label classifiers more attackable. One step forward, inspired by the derived bound, we propose a Soft Attackability Estimator (SAE) and further develop Adversarial Robust Multi-label learning with regularized SAE (ARM-SAE) to improve the adversarial robustness of multi-label classifiers. 

This work gives a more comprehensive understanding of LD in multi-label learn- ing. The exploiting of LD should be encouraged since its positive role in models’ generalizition on clean data, but be restricted because of its negative role in models’ adversarial robustness. These insights inspire people to build high-performance multi- label classifiers with balanced ability of generalization on clean data and adversarial robustness. 

Brief Biography

Zhuo Yang received his Bachelor and Master degrees from Nankai University.  After that, he came to KAUST for his PhD majored in Computer Science. His research interests include adversarial machine learning, multi-label learning, etc.