Soft-Margin SVM

for a sample $\displaystyle S$ of $\displaystyle m$ instances $\displaystyle (x_i,y_i)$

\displaystyle argmin_{\theta,b}\:\: \frac{1}{2} \left\| \theta \right\|_2^2 + c \sum_i \xi_i

$\displaystyle s.t. \:\:\: y_i \left( \theta^T \mu(x_i) + b \right) \geq 1- \xi_i \:,\:$

$\displaystyle \xi_i \geq 0$

$\displaystyle \theta$ , $\displaystyle b$ the parameters of the linear separator
$\displaystyle \mu$ a mapping function, so that $\displaystyle \mu(x_i)^T\mu(x_j) = K(x_i,x_j)$

Supervised Learning

In Brief

Learning a classifier from a fully labeled set

Issues

Label assignment is difficult and expensive:
- difficult: unique and reliable labels
- expensive: great amount of data, need for experts
Datasets are generally noisy

How to handle the confidence in the labels?

Weak-Label Learning

labels are incorrect, missing or not unique

Sub-problems

Semi-Supervised Learning
Unsupervised Learning
Label Proportions Learning
Multi-Instance Learning
Multi-Expert Learning
Noisy-Tolerant Learning

Empirical Surrogate $\displaystyle \beta$ -Risk

For any margin-based loss function $\displaystyle F_{\phi}$

\displaystyle R_{\phi}^{\beta}(X,h) = \frac{b_{\phi}}{m} \sum_{i=1}^{m} \sum_{\sigma \in -1,1} \beta_i^{\sigma} F_{\phi}(\sigma h(x_i))

\displaystyle \beta

degree of confidence / probability of labels

$\displaystyle \beta_i^{\text{-}1} \in [0,1]$ , $\displaystyle \beta_i^{\text{+}1} \in [0,1]$

$\displaystyle \beta_i^{\text{-}1} + \beta_i^{\text{+}1} = 1$

Margin-based loss functions

Soft-Margin $\displaystyle \beta$ -SVM

primal problem

\displaystyle argmin_{\theta,b}\:\: \frac{1}{2} \left\| \theta \right\|_2^2 + c \sum_i \left(\beta_i^{\text{-}1}\xi_i^{\text{-}1}+\beta_i^{\text{+}1}\xi_i^{\text{+}1} \right)

\displaystyle s.t. \:\:\: \sigma (\theta^T \mu(x_i) + b) \geq 1- \xi_i^{\sigma} \:,\:

\displaystyle \xi_i^{\sigma} \geq 0

Lagrangian dual problem

\displaystyle \max_{\alpha} \:\: -\frac{1}{2} \sum_{i,j} \sum_{\sigma,{\sigma}'} \alpha_i^{\sigma} \sigma \alpha_j^{{\sigma}'} {\sigma}' K(x_i,x_j) + \sum_i \sum_{\sigma} \alpha_i^\sigma

\displaystyle s.t. \:\: 0 \leq \alpha_i^\sigma \leq c \beta_i^\sigma \:,\:

\displaystyle \sum_{i=1}^m \sum_{\sigma} \alpha_i^\sigma \sigma = 0

How the margin is affected

Relation with the Classical Risk

Let's rewrite the classical risk

\displaystyle R_{\phi}(X,Y,h)

=

\displaystyle R_{\phi}^{\beta}(X,h)

-

\displaystyle \frac{1}{m} \sum_i \beta_i^{-y_i} y_i h(x_i)

$\displaystyle R_{\phi}^{\beta}(X,h)$ is the $\displaystyle \beta$ -risk
$\displaystyle \frac{1}{m} \sum_i \beta_i^{-y_i} y_i h(x_i)$ is a penality term on the missclassified instances

Iterative Algorithm

Learn $\displaystyle h$ $\displaystyle \:\:argmin_h \:\: N(h) + c R_{\phi}^{\beta}(X,h)$
Estimate $\displaystyle y$ $\displaystyle \forall i=1..m,\:\:\:y_i = sign \left(\beta_i^{\text{+}1}-\frac{1}{2} \right)$
Learn $\displaystyle \beta$ $\displaystyle \:\:argmin_{\beta} R_{\phi}^{\beta}(X,h)$
$\displaystyle s.t.\: \sum_{i=1}^{m}\beta_i^{\text{-}y_i} y_i h(x_i) = 0$
$\displaystyle \beta_i^{\text{-}1} + \beta_i^{\text{+}1} = 1$

Semi-Supervised Learning

with $\displaystyle m_l$ labeled instances and $\displaystyle m_u$ unlabeled instances

Initialization of $\displaystyle \beta$
- $\displaystyle \forall i=1..m_l \:\: \beta_i^{\sigma} = 1 \:\text{if}\: \sigma = y_i, 0 \:\text{otherwise}$
- $\displaystyle \forall i=m_l+1..m_u \:\: \beta_i^{\sigma} = 0.5$
Iterative Algorithm
- Learning $\displaystyle \beta$ of unlabeled set

Results

WellSVM:

Li Yu-Feng, Tsang Ivor W, Kwok James T, Zhou Zhi-Hua. Convex and scalable weakly labeled SVMs The Journal of Machine Learning Research, 2013.

Perspectives: Differential Privacy

How to accuratly learn while preserving the user privacy?

Learn on bags of instances:

the labels of each single instance are unknown
we have access to the proportions of the classes per bag

a New Surrogate Risk for Learning from Weakly Labeled Data
comment